residual-connections-enable-arbitrary-depth

IN derived (depth 1)

Created 2026-06-21T10:06:02+00:00 · Reviewed 2026-06-21T15:37:01+00:00

Residual connections (F(x) + x) are the key architectural innovation enabling arbitrarily deep networks — ResNet solved the degradation problem for CNNs, and the same mechanism was adopted by transformers to enable deep stacking without vanishing gradients.

Justifications

SL — Three independent solutions to vanishing gradients share the principle of identity-preserving shortcuts

Antecedents (all must be IN):

IN resnet-2015-skip-connections-degradation — ResNet (2015) by He et al. solved the degradation problem of very deep networks (20-30+ layers) using skip/residual connections
IN residual-connection-formula — Residual connections in transformers use the formula x -> F(x) + x, originating from ResNet, preventing vanishing gradients in deep stacking.
IN lstm-designed-to-solve-vanishing-gradient — LSTM was designed specifically to solve the vanishing gradient problem using forget gates as the key mechanism.

Dependents

These beliefs depend on this one:

IN residual-connections-and-lstm-gating-are-convergent-solutions — Residual connections (ResNet) and LSTM gating are convergent solutions to the same underlying problem — both enable gradient flow through deep computation graphs, with LSTM's forget gate solving vanishing gradients in recurrent (temporal) depth and skip connections solving the degradation problem in feedforward (spatial) depth — suggesting that identity-preserving shortcuts are a universal architectural principle for deep networks regardless of topology.
IN trainability-not-capacity-was-the-depth-bottleneck — The theory-practice gap in neural network depth — where universal approximation says shallow networks suffice but practice demands deep ones — reveals that the real bottleneck was never representational capacity but trainability, and residual connections resolved exactly this optimization barrier, enabling the hierarchical representations that shallow networks could theoretically but not practically learn.