How Machine Learning Models Actually Learn: Backpropagation Explained

F(X) = Y describes what a model does once it’s trained. But how does F gets to that state in the first place? F starts as a set of weights, initialised close to randomly, and training is the process of adjusting those weights until F(X) reliably matches Y. Backpropagation is the adjustment.

The forward pass produces a guess

Push an input X through the model and each layer performs the same basic operation: multiply the input by the layer’s weights, producing an activation, and pass that activation to the next layer, and so on. The output of the final layer is the model’s prediction, F(X).

The error is measured against the ground truth

This is where Y, the labelled, correct answer, is applied. A loss function compares F(X) to Y and produces a single number: how wrong the prediction was. For a model that’s still close to its random initialisation, this number starts large. The entire purpose of training is to make it smaller, across the whole dataset, not just for one example.

The error propagates backward through the weights

A large loss doesn’t say which weight caused it, and a network can have millions of them. Backpropagation answers that question by working backward from the output, using the chain rule to compute how much each weight, at every layer, contributed to the final error. The name describes exactly this: the error is propagated backward, layer by layer, from output to input, and each layer’s weights are updated by a small amount in the direction that would have reduced the error.

Here is why training and inference place such different demands on memory. Inference only needs the forward pass, input to output, and can discard each layer’s activation once the next layer has consumed it. Training needs those activations kept until the backward pass reaches them, because the gradient calculation at each layer depends on what that layer actually produced during the forward pass. A model that runs comfortably at inference can still be far too large to train on the same hardware, for exactly this reason.

Repeat, and the model converges

One pass of forward, measure, backward, adjust moves every weight a small step in a better direction. Repeated over the dataset, many times, those small steps accumulate into a model whose predictions track the training data closely. This is where machine learning becomes an engineering discipline rather than a research gamble: given a model architecture with enough capacity for the problem, and data that specifies the problem correctly, this process reliably converges on a good model. Not necessarily the best possible model, and not without its own failure modes, but a good one, every time.

For a hands-on version of the same process, Andrew Trask’s neural network in eleven lines of Python remains one of the clearest minimal implementations available, small enough to read start to finish in one sitting, with the forward pass, the error calculation, and the backward pass each doing exactly what this article describes, just in code rather than prose.

That reliability is also why the four types of machine learning aren’t the same for all problems. Supervised learning fits this description exactly, because Y is fixed and known for every example before training starts. Reinforcement learning uses the same underlying mechanism but generates its own Y through interaction with an environment over time, which is part of why it’s a harder and less predictable process to get right.

(The cost implication of all this is direct: the cost of machine learning breaks down why Y, the labelled data this whole process depends on, is usually the most underestimated line item in the budget.)