Backpropagation Algorithm: Math, Numbers and Python, Step by Step

Every neural network guide eventually gets to the point where it says "the network learns from its errors" and just moves on. Backpropagation is the part where that sentence actually gets explained.

In plain terms: backpropagation is the algorithm that trains a neural network by measuring how wrong a prediction was, then working backward through the network to figure out exactly how much each weight contributed to that wrongness, so it can adjust each one accordingly. It's not a separate technique from gradient descent, it's the piece that makes gradient descent possible for a network with many layers. Without it, you'd be guessing at weight updates instead of calculating them.

This piece walks through it properly. The intuition first, then the math (gently, with real numbers), a full worked example on a tiny network you can check by hand, working Python code, and the two things every ML interview loves asking about: backprop vs gradient descent, and why deep sigmoid networks used to stall out completely.

What Is Backpropagation in Machine Learning?

Backpropagation, short for "backward propagation of errors," is the algorithm neural networks use to learn. It runs a prediction forward, compares it to the correct answer, then propagates that error backward through every layer, computing exactly how much each individual weight should change to make the next prediction a little less wrong.

The method was popularised in a 1986 paper by Rumelhart, Hinton, and Williams, and it's genuinely hard to overstate how much modern deep learning rests on it. Every CNN, every transformer, every LLM you've used gets trained this way. Nobody's found a fundamentally better replacement in almost four decades, which, in a field that reinvents itself every eighteen months, is worth sitting with for a second.

How the Backpropagation Algorithm Works: 4 Steps

Strip away the math for a second and backpropagation is four steps, repeated over and over:

1. Forward pass, run the input through the network and get a prediction.

2. Compute the loss, how far off that prediction was from the actual answer. This is where a loss function like mean squared error comes in.

3. Backward pass, work backward through the network computing how much each weight contributed to the loss, using the chain rule.

4. Update the weights, nudge every weight slightly in the direction that reduces the loss, using gradient descent.

Then you do it again. One full pass through all four steps, across your entire training set, is called an epoch. Most networks need dozens or hundreds of epochs before the loss actually flattens out.

Here's the analogy that tends to stick: you're hiking down a mountain in thick fog. You can't see the valley, but you can feel which way the ground slopes under your feet right now. So you take a small step downhill, feel the slope again, take another step. Backpropagation is how the network feels the slope. Gradient descent is the act of actually stepping.

The Math Behind Backpropagation (Chain Rule, Gently)

Here's the one-sentence version: the chain rule lets you figure out how much a weight buried several layers deep affected the final error, without recalculating the whole network from scratch for every single weight.

Say your loss is E, and a particular weight is w5, connecting a hidden neuron's output to the final output neuron. You want dE/dw5, how much E changes if w5 nudges slightly. The chain rule breaks that into three smaller, much easier questions:

dE/dw5 = (dE/dout_o1) × (dout_o1/dnet_o1) × (dnet_o1/dw5)

In words: how much the loss changes when the output changes, times how much the output changes when the neuron's raw input changes, times how much that raw input changes when w5 changes. Multiply the three together and you've got one clean number telling you exactly how responsible w5 was.

One thing worth memorising if you're using sigmoid: its derivative simplifies to sigmoid(x) × (1 − sigmoid(x)). Hang on to this fact, we'll reuse it later when we get to vanishing gradients, because that same formula is also the reason deep sigmoid networks stall out.

For hidden-layer weights it's the same three-part chain, just one link longer, since the error has to travel back through an extra layer first. The pattern doesn't change. It just gets one hop further from the output each time.

If the derivative rules themselves feel shaky, Scaler's free Maths for Machine Learning course is worth a detour before continuing, and our activation functions page covers sigmoid, ReLU, and the rest in more depth than we need here.

Hiring Partners:

GoogleAmazonMicrosoftFlipkartAdobe1200+ more

Backpropagation Step-by-Step Worked Example

Time for actual numbers. Here's a tiny 2-2-1 network: two inputs, one hidden layer with two neurons, one output neuron. Every neuron uses sigmoid activation. Small enough to check by hand with a calculator, which is the whole point.

Forward pass first, run the inputs through and see what the network predicts before it's learned anything:

That's a pretty bad first guess. Target was 0.01, the network predicted 0.7513, which tracks since the weights were just picked arbitrarily to start. That gap is exactly what backpropagation is about to go fix.

Now the backward pass, computing how much each weight contributed to that 0.2748 loss:

With a learning rate of 0.5, every weight moves a small step opposite its gradient: new weight = old weight − (learning rate × gradient). Run that arithmetic across all eight weights and biases:

Run the forward pass again with these updated weights and the new prediction comes out to roughly 0.7281, loss drops to about 0.2578. Still far from perfect after one update, that's expected, real training loops through this thousands of times, but the loss did drop, which is the only thing that matters here. That's backpropagation working exactly as intended, on numbers you can rerun yourself. For the fuller picture of how layers like this stack together into bigger networks, our multilayer perceptron page is the natural next read.

Backpropagation in Python (From Scratch)

Same network, same numbers, now in code. About 30 lines of NumPy, no framework magic hiding what's actually happening.

That's the whole algorithm, no shortcuts. Every framework you've actually used for real work, PyTorch, TensorFlow, JAX, does this exact same chain-rule bookkeeping automatically. It's called autograd. You call loss.backward() and the framework walks the computational graph backward, applying the chain rule at every node, for every one of potentially billions of parameters, without anyone writing a single derivative by hand. What we just did manually for eight weights, autograd does for entire transformer models. Worth appreciating once, then never doing by hand again unless an interviewer asks.

If you'd rather build full networks hands-on instead of just reading about them, Scaler's free Deep Learning course walks through exactly this, plus everything that comes after a single weight update.

Backpropagation vs Gradient Descent

This is the question that trips up almost everyone at some point, usually in an interview, at the worst possible moment.

Short version: backpropagation computes the gradients. Gradient descent uses those gradients to actually update the weights. They're not competing techniques, they're two steps in the same loop, and one is useless without the other.

In practice, you rarely run plain gradient descent on the whole dataset at once, it's slow and memory-hungry. Most training uses stochastic gradient descent (SGD) or mini-batch gradient descent instead, updating weights on small chunks of data at a time, sometimes with momentum-based variants that smooth out the descent path. Most modern models go further still and use adaptive optimizers like Adam, which adjust the learning rate per parameter based on how gradients have been behaving recently. Our optimizers in deep learning page covers Adam, RMSprop, and the rest, but the core idea underneath every one of them is still backpropagation computing the gradient first.

Common Problems: Vanishing & Exploding Gradients

Here's where that sigmoid derivative fact from earlier comes back to bite you. Sigmoid's maximum possible derivative is 0.25, right at its midpoint, and it only gets smaller from there.

Now stack ten or twenty layers of sigmoid neurons. Backpropagation multiplies gradients together as it travels backward through each layer. Multiply a bunch of numbers that are all 0.25 or smaller, over and over, and the gradient shrinks toward zero exponentially fast. By the time the error signal reaches the early layers, there's barely anything left to learn from. Those layers essentially stop training while the later ones keep chugging along. This is the vanishing gradient problem, and it's a big part of why deep networks were genuinely hard to train before roughly the mid-2010s.

The opposite failure mode also exists, exploding gradients, where the multiplied values grow instead of shrink and weight updates become wildly unstable. Less common, equally annoying when it happens.

The fixes that actually stuck, one line each:

• ReLU activation, its derivative is either 0 or 1, no shrinking multiplication chain

• Better weight initialisation (Xavier, He), starting the network somewhere the gradients don't immediately implode

• Batch normalisation, keeps activations in a stable range as they pass through layers

• Gradient clipping, caps gradients at a maximum value so they can't explode

• Residual connections, the trick behind ResNets and most modern architectures, giving gradients a shortcut path backward that skips some of the multiplication entirely

Goodfellow, Bengio, and Courville's Deep Learning Book covers this in far more depth if you want the full derivation.

Advantages, Limitations & Where Backpropagation Is Used

What it gets right: it's efficient. The chain rule means you compute each gradient once and reuse intermediate results, instead of recalculating the whole network from scratch for every single weight. That efficiency is the entire reason training a network with millions or billions of parameters is even remotely feasible.

What it needs to work: every operation in the network has to be differentiable, and you need labelled data to compare predictions against. That second part is why unsupervised and self-supervised setups get creative, they construct a pseudo-label from the data itself so backprop still has something to compare against. Our supervised vs unsupervised learning page gets into that distinction if it's new to you.

Where it's actually used: everywhere, honestly. CNNs recognising images, RNNs and transformers processing language, the LLM you might be using to help draft this exact kind of article. Different architectures, wildly different jobs, same backward pass underneath all of them.

Want to build and train production neural networks instead of just reading the math behind them? Scaler's AI & ML Program covers this from first principles through to deployment.