Deep learning concept

How neural networks learn

Reference

Backpropagation works by tracing how a prediction error depends on a network’s weights and biases, starting at the output and moving backward through its layers. The network uses those calculations to adjust its parameters, usually with gradient descent, so its loss can decrease on training examples. Repeating the process is how a neural network learns from examples.

Reviewed 2026-10-09

The basic idea

A neural network starts with adjustable numbers called weights and biases. It uses them to turn an input into a prediction. A loss function measures how different that prediction is from the target. Training aims to adjust the parameters to reduce that loss.

Backpropagation calculates how the loss depends on the parameters by working backward from the output through the network. An optimizer such as gradient descent uses those calculations to choose parameter changes. Backpropagation supplies the gradients; the optimizer uses them to update the weights and biases.

The steps, in order

First, the network makes a prediction in a forward pass: each layer transforms the values it receives using its current weights, biases and activation function. Next, the loss function compares the prediction with the target. This gives training a measure of how far that prediction is from the example’s target.

Then backpropagation works backward, calculating how much each parameter contributed to the loss. The optimizer updates the parameters using those gradients. The process repeats over training examples; a complete pass over the training data is called an epoch. Backpropagation calculates changes, but it does not by itself decide whether the model will perform well on new examples.

What can go wrong

A gradient indicates how the loss changes when a parameter changes. If gradients reaching early layers become very small, those layers may train slowly or not at all; this is called the vanishing-gradient problem. If gradients become too large, updates can be disruptive; this is the exploding-gradient problem.

A network can also fit its training examples without doing as well on new data. Training loss measures fit on the examples used for training, so it cannot alone tell you how well the network generalizes. Compare performance on separate validation or test data. Backpropagation is a way to calculate parameter gradients, not a guarantee that every update improves performance on unseen examples.

Check your understanding

Try explaining the training loop without using a formula: the network predicts, the loss scores the prediction against a target, backpropagation traces the loss backward to calculate gradients, and an optimizer uses those gradients to update parameters. Then explain why the loop repeats: one update changes the parameters, which changes later predictions and losses.

A useful check is to distinguish the two backward-and-update roles. Backpropagation calculates how the loss changes with the weights and biases. Gradient descent is one method for using that information to move the parameters toward lower loss. If you can say what each step contributes, you have the core idea.

Questions people ask

How are neural networks trained?

They make predictions on training examples, measure prediction error with a loss function, calculate parameter gradients by backpropagation, and use an optimizer to update weights and biases. The cycle repeats.

Does backpropagation change the weights itself?

Backpropagation calculates how the loss changes with the weights and biases. An optimizer, such as gradient descent, uses those gradients to make parameter updates.

Does a lower training loss mean the network learned to generalize?

Not by itself. Training loss describes fit on training examples; checking separate validation or test data helps assess performance on examples that were not used for those updates.

Learn more

Principles of Data Science, 7.2: Backpropagation
A step-by-step explanation of forward passes, loss, backpropagation and parameter updates.
Neural networks: Interactive exercises
Interactive practice connecting parameter changes, predictions, training and loss.

What changed

Sources

Part of Deep learning: How a neural network learns What training results do and do not show