← Theory · Neural Networks

Backpropagation

The chain rule, run in reverse over a computation graph — how a network turns one loss number into a gradient for every weight.

Mode

Key idea

One forward pass to compute the loss, one backward pass to blame every weight for it. A network is just a long chain of simple operations — multiply, add, apply a nonlinearity — stacked into a computation graph. To train it you need to know how much each of its millions of weights contributed to the error. Backpropagation gets all of those numbers in a single sweep: run the input forward and remember every intermediate value, then walk the graph backward applying the chain rule, multiplying the gradient coming from above by each operation's local derivative. The cost is roughly the same as one forward pass — which is the only reason training deep networks is feasible at all.

Task A tiny network drawn as a computation graph. Press Forward → to compute each value and cache it, then keep stepping to watch the gradient flow backward — each step multiplying the gradient from above by a local derivative that reuses a cached value. Drag the sliders to change the inputs and see every number update at once.
step 0 / 10 · ready

Why not just perturb each weight?

The obvious way to estimate how weight w affects the loss is to nudge it by a tiny ε, re-run the whole network, and see how much the loss moved: (L(w+ε) − L(w)) / ε. That works, but it costs one full forward pass per weight. A network with 100 million parameters would need 100 million forward passes for a single training step — completely hopeless.

Backpropagation gets every partial derivative in one backward pass, at roughly the cost of a single forward pass — regardless of how many weights there are. That is the whole reason deep learning is practical.

The trick: reuse work with the chain rule

The gradient of the loss with respect to an early weight is a long product of local derivatives, one for each operation between that weight and the loss. Backprop computes that product once, from the output end, and reuses the running total for every weight that shares a prefix. Instead of recomputing the whole chain for each weight, it carries a single "gradient so far" backward through the graph and multiplies in each operation's local slope as it passes.

Forward remembers, backward reuses

Notice in the figure that the backward pass never re-runs tanh or re-multiplies the inputs. Each local derivative is written in terms of values the forward pass already computed — the output of tanh, the input x, the activation h. So the forward pass caches those intermediate values, and the backward pass reads them straight back. This is the memory–compute trade at the heart of every deep-learning framework: you keep the forward activations around precisely because backprop will need them.

What it handles

  • Any model you can write as a graph of differentiable operations — MLPs, CNNs, transformers, and most of what people call "deep learning"
  • Getting exact gradients (to floating-point precision), not noisy estimates
  • Millions to billions of parameters, at the cost of ~one extra forward pass
  • Custom losses and layers — if you can differentiate the pieces, autodiff assembles the rest

Where it struggles

  • Non-differentiable steps — hard sampling, argmax, discrete choices (need surrogates, straight-through estimators, or REINFORCE)
  • Very deep or recurrent graphs where gradients vanish or explode on the way back
  • Memory: caching every forward activation is often the real bottleneck on large models
  • It gives you the gradient, not a good step — you still need an optimizer and a learning rate

import torch

# The same tiny graph as the figure, in an autodiff engine.
x  = torch.tensor(1.1)
w1 = torch.tensor(0.9, requires_grad=True)
w2 = torch.tensor(1.3, requires_grad=True)
y  = torch.tensor(0.5)

h_pre = w1 * x
h     = torch.tanh(h_pre)
o     = w2 * h
loss  = (o - y) ** 2

loss.backward()          # one reverse sweep fills in every .grad
print(w1.grad, w2.grad)  # ∂L/∂w1, ∂L/∂w2 — ready for the optimizer
Want the chain-rule derivation and a worked example?

The backward recursion

$$ \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(\ell)}} \;=\; \left(\frac{\partial \mathbf{z}^{(\ell+1)}}{\partial \mathbf{z}^{(\ell)}}\right)^{\!\top} \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(\ell+1)}} $$

  • z(ℓ)the pre-activation (or any intermediate value) at layer ℓ
  • ∂L/∂z(ℓ+1)the gradient already computed one step downstream — the upstream gradient
  • (∂z(ℓ+1)/∂z(ℓ))⊤the layer's local Jacobian, transposed
  • Start with ∂L/∂L = 1 at the output and apply this rule right-to-left

$$ \text{gradient at this node} \;=\; (\text{local derivative})\;\times\;(\text{gradient at the next node}) $$

In words. Backprop is one rule applied over and over. Seed the output with a gradient of 1 (the loss with respect to itself). Then step backward one operation at a time: the gradient at a node equals the gradient that arrived from the node in front of it, multiplied by that operation's local derivative — how much this node's output changes when its input wiggles. For layers of vectors the "multiply" is a matrix product with the layer's Jacobian (the ⊤ just means transposed, so shapes line up), but the idea is identical to the scalar case. Keep multiplying your way back to the inputs and you've collected ∂L/∂w for every weight along the way.

A worked example — the graph in the figure. Take x = 1.1, w₁ = 0.9, w₂ = 1.3, target y = 0.5.

Forward (and cache each value):

  • h_pre = w₁·x = 0.99
  • h = tanh(h_pre) = 0.757
  • o = w₂·h = 0.984
  • L = (o − y)² = 0.234

Backward (chain rule, right to left):

  • ∂L/∂o = 2(o − y) = 0.968
  • ∂L/∂w₂ = ∂L/∂o · h = 0.968 × 0.757 = 0.733  ← reuses cached h
  • ∂L/∂h = ∂L/∂o · w₂ = 0.968 × 1.3 = 1.259
  • ∂L/∂h_pre = ∂L/∂h · (1 − h²) = 1.259 × (1 − 0.757²) = 0.538  ← tanh′ reuses cached h
  • ∂L/∂w₁ = ∂L/∂h_pre · x = 0.538 × 1.1 = 0.592  ← reuses cached x

Every local derivative — h, w₂, 1 − h², x — is either an input or a value the forward pass already produced. Nothing is recomputed.

The four-equation summary (dense layers). For a network of layers z = Wa + b, a = σ(z), backprop is exactly four equations repeated per layer, where δ = ∂L/∂z is the "error signal":

  • Output error: δ_out = ∇_o L ⊙ σ′(z_out)
  • Propagate back: δ_ℓ = (W_{ℓ+1}ᵀ δ_{ℓ+1}) ⊙ σ′(z_ℓ)
  • Weight gradient: ∂L/∂W_ℓ = δ_ℓ a_{ℓ−1}ᵀ
  • Bias gradient: ∂L/∂b_ℓ = δ_ℓ

The ⊙ is elementwise multiply; σ′ is the activation's derivative. This is what a framework runs under the hood for every nn.Linear.

Forward-mode vs reverse-mode. There are two ways to apply the chain rule automatically. Forward-mode propagates derivatives input→output and is cheap when there are few inputs and many outputs. Reverse-mode — what "backprop" means — propagates output→input and is cheap when there are many inputs (weights) and one output (the loss). Deep learning has millions of inputs and a single scalar loss, so reverse-mode is the obvious fit. The price is that you must first go forward and store the intermediate values, because the backward pass consumes them.

Reach for the derivation when

  • You're implementing a custom layer and need to write its backward by hand
  • A gradient is NaN or zero and you need to reason about which local derivative killed it
  • You want to understand why residual connections and normalization help gradients flow
  • You're deciding what to checkpoint vs recompute to fit a model in memory

Let the framework handle it when

  • You're using standard layers — PyTorch / JAX autodiff is exact and faster than hand-rolled gradients
  • The graph is large; manual bookkeeping is error-prone and buys you nothing
  • You just need the numbers — call .backward() and move on
  • You're prototyping — get correctness first, optimize the backward pass later if it matters

import numpy as np

# One dense layer's backward pass, by hand: z = W @ a + b, out = tanh(z)
def layer_backward(grad_out, cache):
    a, W, z = cache["a"], cache["W"], cache["z"]  # cached from forward
    grad_z = grad_out * (1 - np.tanh(z) ** 2)     # local deriv of tanh
    grad_W = grad_z[:, None] @ a[None, :]         # δ aᵀ
    grad_b = grad_z                               # δ
    grad_a = W.T @ grad_z                         # upstream for prev layer
    return grad_a, grad_W, grad_b
Want the graph view, checkpointing, and how frameworks store the tape?

Reverse-mode autodiff

$$ \bar{v} \;\equiv\; \frac{\partial \mathcal{L}}{\partial v}, \qquad \bar{u} \;\mathrel{+}=\; \frac{\partial v}{\partial u}\,\bar{v} \quad \text{for each edge } u \to v $$

  • v̄the adjoint of node v — the loss's sensitivity to v
  • u → van edge of the graph: u is an input to the operation producing v
  • +=accumulate — a node feeding several children sums the gradients from all of them
  • Process nodes in reverse topological order so every v̄ is final before it's used

$$ \text{grad of each input} \;\mathrel{+}=\; \text{local derivative} \times \text{grad of the output} $$

In words. A framework builds a graph of the operations you ran (the "tape"), then visits them in reverse. Each node holds an adjoint v̄ = how much the loss depends on that node. For every input u of an operation, it adds (local derivative) × v̄ into u's adjoint. The += matters: if a value is used in two places (say an activation fed into two later layers), gradients arrive from both paths and must be summed — this is the multivariable chain rule. Visiting nodes in reverse topological order guarantees a node's adjoint is complete before it passes gradient to its parents.

How frameworks actually do it — the tape. In define-by-run engines (PyTorch, JAX with tracing) every operation on a tensor that requires_grad records a node: the op, references to its input tensors, and a closure that knows the local backward. This linked structure is the autograd graph (the "tape"). Calling .backward() walks it in reverse topological order, running each closure and accumulating into .grad. The graph is rebuilt every forward pass, which is why Python control flow "just works" — the tape simply records whatever branch ran.

What gets cached, and why it's the real cost. The backward closures need forward values: tanh's backward needs its output, a matmul's backward needs both operands, max-pooling needs the argmax indices. So the forward pass keeps those tensors alive — they're pinned in memory from the moment they're produced until the backward pass consumes them. For a deep model the stored activations, not the parameters, dominate memory: training memory scales with depth × batch × activation size. This is why you can infer with a model that won't train on the same GPU — inference frees each activation immediately; training cannot.

Gradient checkpointing — trade compute for memory. If storing every activation is too expensive, checkpoint only a subset (say, one tensor per transformer block) and recompute the rest during the backward pass by re-running that block's forward. Memory drops from O(n) to roughly O(√n) in the number of layers, at the cost of ~one extra forward pass (~33% more compute). This single technique is what lets large models fit at all, and it's a one-line wrapper (torch.utils.checkpoint) in practice.

Where the chain of multiplications breaks. Because the backward pass is a long product of Jacobians, its magnitude is a product of factors. If those factors are consistently < 1, the gradient vanishes before it reaches early layers; if > 1, it explodes. The standard fixes all target this product directly: ReLU (derivative exactly 1 on the positive side), residual connections (y = x + F(x) adds an identity path whose local derivative is 1, a "gradient highway"), normalization (keeps activations — and thus σ′ — in a sane range), and gradient clipping (caps the norm to stop explosions in RNNs/transformers). See Neural Networks for the failure modes and Gradient Descent for what happens to the gradient once backprop hands it off.

Backprop through time (BPTT). Unrolling a recurrent network over T steps turns it into a deep feedforward graph sharing one weight matrix. Backprop then sums that weight's gradient across all T copies — but the same repeated-multiplication problem makes long-range gradients vanish, which is exactly what LSTMs and, later, attention were designed to sidestep. See Recurrent Neural Networks.

Higher-order and implicit gradients. Because the backward pass is itself a differentiable computation graph, you can backprop through it to get second derivatives (create_graph=True) — used in meta-learning (MAML), Hessian-vector products, and some physics-informed models. When a forward "layer" is the solution of an optimization or a fixed-point equation (deep equilibrium models, differentiable optimization), you skip unrolling entirely and get the gradient from the implicit function theorem instead — constant memory regardless of how many inner iterations ran.

A note on biological plausibility. Backprop needs a symmetric backward pass that reuses the forward weights and a global error signal — neither of which the brain obviously has. This "weight transport problem" motivates alternatives like feedback alignment, target propagation, and predictive coding. None matches backprop's efficiency on real tasks yet, but they're an active research thread on how learning might work without it.

Reach for it when

  • Training memory is the bottleneck — checkpoint activations to fit bigger models or batches
  • Debugging vanishing/exploding gradients — reason about the product of Jacobians layer by layer
  • You need second-order information — Hessian-vector products via double backward
  • A layer is an implicit solve (optimization, fixed point) — use implicit differentiation, not unrolling

Skip / rethink it when

  • The graph has non-differentiable ops — reach for straight-through estimators, Gumbel-softmax, or REINFORCE
  • Gradients are pathological through a very long recurrence — restructure (attention, gating) rather than fight BPTT
  • The forward pass has huge fan-in with few outputs — forward-mode autodiff may be cheaper
  • You only ever call .backward() on standard layers — you don't need any of this; the framework has it covered

import torch
from torch.utils.checkpoint import checkpoint

# Gradient checkpointing: don't store this block's internal activations —
# recompute them during backward. Trades ~one extra forward for big memory savings.
def block(x):
    return torch.tanh(layer2(torch.relu(layer1(x))))

# Standard forward stores all intermediates; checkpointed forward stores only x.
out = checkpoint(block, x, use_reentrant=False)
loss = (out - target).pow(2).mean()
loss.backward()          # block's forward is re-run here to get the gradients
Too dense?