backprop-efficient-recursive-gradient-factorization

IN derived (depth 1)

Created 2026-06-21T10:01:28+00:00 · Reviewed 2026-06-21T15:37:01+00:00

Backpropagation's computational efficiency stems from a recursive factored structure: the error signal delta propagates backward layer-by-layer via matrix-vector products, each weight gradient factors into a compact o_i * delta_j form, and the full weight gradient is a simple outer product of local quantities — avoiding redundant computation entirely.

Justifications

SL — Three complementary gradient computation beliefs reveal that backprop's efficiency is not incidental but arises from a recursive factorization that makes each gradient local

Antecedents (all must be IN):

IN backprop-gradient-factors-to-oi-delta-j — The gradient ∂E/∂w_ij factors into the compact form o_i · δ_j, where o_i is the activation of the sending neuron and δ_j is the error signal at the receiving neuron.
IN backprop-delta-recursive-computation — The key intermediate quantity delta^l (gradient of the weighted input at layer l) is computed recursively from delta^(l+1) via δ^(l-1) = (f^(l-1))' ∘ (W^l)^T · δ^l, and is sufficient to compute all weight gradients at that layer.
IN backprop-weight-gradient-outer-product — The weight gradient at layer l is computed as an outer product: ∇_{W^l} C = δ^l · (a^(l-1))^T, which is why activations from the forward pass must be cached.

Dependents

These beliefs depend on this one:

IN backprop-three-mathematical-perspectives-converge — Neural network training is illuminated by three independent mathematical perspectives that converge on the same computation — backprop's recursive factorization provides computational efficiency, reverse-mode autodiff provides theoretical abstraction, and gradient flow ODEs provide dynamical systems interpretation — a rare case of multi-framework convergence in ML.