An artificial neural network is a chain of simple calculations: each unit multiplies its inputs by adjustable weights, adds them up, and passes the sum through an activation function, and layers of such units feed one another from input to output. Learning means comparing the output with a known target, measuring the gap with a loss, and nudging every weight in the direction that shrinks it, a procedure called back-propagation that Rumelhart, Hinton, and Williams described in Nature in October 1986. The walk-through below uses a six-weight network small enough to check by hand.
Key takeaways
- A unit computes a weighted sum of its inputs plus a bias, then applies an activation function such as ReLU, defined as max(0, x).
- A forward pass runs the inputs through the layers to a prediction; a backward pass sends the error back to find how much each weight contributed.
- In the worked example below, one update lowers the loss from 0.16 to about 0.019.
- A ReLU unit whose input is negative passes no gradient, so its incoming weights do not change on that example.
- Real networks follow the same steps with millions of weights, many examples at once, and many repeated updates.
Reading the diagram: inputs, hidden layer, output
A standard diagram draws circles in columns. The first column holds the inputs, the last holds the output, and the columns between them are hidden layers. A line between two circles stands for one weight. In a fully connected layer, every unit connects to every unit in the neighboring layer, and no units connect within the same layer. Stanford's CS231n course notes count a network with three inputs, one hidden layer of four units, and two outputs as 20 weights plus six biases, or 26 learnable parameters. Each unit also has a bias, a number added before the activation, which the diagram usually omits.
The same notes describe the unit itself as a dot product of inputs and weights plus a bias, followed by an activation function. Without the activation, stacking layers would add nothing, because a chain of weighted sums collapses into one weighted sum. The activation is what lets extra layers represent more than a straight-line rule.
One forward pass with real numbers
The example is an illustrative example invented for this page, not taken from a paper. The network has two inputs, two hidden units with ReLU activation, and one output with no activation. Biases are set to zero to keep the arithmetic short. The input is x1 = 1 and x2 = 2, the target is 1.0, and the starting weights are shown in the table.
| Step | Calculation | Result |
|---|---|---|
| Hidden unit A (weights 0.5 and 0.5) | max(0, 0.5 x 1 + 0.5 x 2) | 1.5 |
| Hidden unit B (weights 1 and -1) | max(0, 1 x 1 + (-1) x 2) = max(0, -1) | 0 |
| Output (weights 0.4 and 0.6) | 0.4 x 1.5 + 0.6 x 0 | 0.6 |
| Loss (squared error) | (0.6 - 1.0) squared | 0.16 |
The prediction is 0.6 against a target of 1.0. The loss is a single number that summarizes how wrong the network is on this example, and training tries to push it toward zero.
The backward pass: who is responsible for the error
Back-propagation applies the chain rule of calculus. The CS231n notes on backpropagation describe it as a recursive application of that rule, working from the output back toward the inputs, where each step needs only its own local gradient and the gradient arriving from downstream. A multiply step multiplies the incoming gradient by the other input's value, and a max step sends the gradient only to the larger input, which is how ReLU behaves.
| Quantity | Calculation | Gradient |
|---|---|---|
| Loss with respect to output | 2 x (0.6 - 1.0) | -0.8 |
| Output weight from A | -0.8 x 1.5 | -1.2 |
| Output weight from B | -0.8 x 0 | 0 |
| Unit A's activation | -0.8 x 0.4 | -0.32 |
| Weights into A | -0.32 x each input (1 and 2) | -0.32 and -0.64 |
| Weights into B | blocked, because B's input was negative | 0 and 0 |
Every gradient that is negative says that raising the weight would lower the loss. Unit B shows the blocked case: its incoming weights get no signal on this example, because ReLU output nothing for it. The CS231n notes add that ReLU units can stay inactive for good if the learning rate is set too high, and report that as many as 40 percent of a network's units can end up dead in that situation.
One update and its effect
Gradient descent moves each weight a small step against its gradient. With a learning rate of 0.1, the output weight from A moves from 0.4 to 0.52, and the weights into A move from 0.5 to 0.532 and from 0.5 to 0.564. All other weights stay as they were. Running the forward pass again gives hidden unit A = 0.532 x 1 + 0.564 x 2 = 1.66 and an output of 0.52 x 1.66 = 0.8632. The loss falls from 0.16 to about 0.019, as the prediction moves from 0.6 toward the target of 1.0.
One step rarely lands exactly on the target, and a step that is too large can overshoot it. Real training repeats the cycle thousands or millions of times, usually on small batches of examples, and averages the gradients over each batch.
What a single diagram leaves out
- Scale: a research network has millions or billions of weights, not four.
- Data: one example can be fit perfectly and still teach nothing, so training uses large sets and holds some data back to test generalization.
- Overfitting: a large network can memorize its training set, which the CS231n notes address with regularization such as weight decay and dropout.
- Interpretation: the weights are numbers, not rules. Reading what a hidden unit "means" takes separate analysis.
- Architecture: images, text, and sequences use specialized layouts, not only fully connected layers.
FAQ
Is a neural network a copy of the brain?
No. The CS231n notes call the unit a coarse model of a biological neuron, using a firing-rate interpretation. The mathematics is a weighted sum and a function, and it is not a simulation of a brain.
What is the difference between a weight and a bias?
A weight scales one input. A bias is a number added to the weighted sum, and it lets a unit respond even when all of its inputs are zero.
Does back-propagation mean the network works backward in time?
No. It means the error signal is calculated from the output layer back toward the input layer, using stored values from the forward pass.
Who introduced back-propagation?
The 1986 Nature paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams describes it as a new learning procedure that repeatedly adjusts weights to reduce the difference between actual and desired output. Its abstract also notes that hidden units come to represent important features of the task.
Related reading
- What artificial neural networks are used for
- A longer guide to neural networks with examples
- Alternatives to neural networks in machine learning
- How AI programs sort large data sets
More in AI Basics.
Last verified: October 2026
Sources
- Rumelhart DE, Hinton GE, Williams RJ, Learning representations by back-propagating errors, Nature 323, 533-536, October 1986
- Neural Networks Part 1: Setting up the Architecture, CS231n course notes, Stanford University
- Backpropagation, Intuitions, CS231n course notes, Stanford University