residual-connections-and-lstm-gating-are-convergent-solutions
IN derived (depth 2)
Created 2026-06-21T11:27:22+00:00 · Reviewed 2026-06-21T15:37:01+00:00
Residual connections (ResNet) and LSTM gating are convergent solutions to the same underlying problem — both enable gradient flow through deep computation graphs, with LSTM's forget gate solving vanishing gradients in recurrent (temporal) depth and skip connections solving the degradation problem in feedforward (spatial) depth — suggesting that identity-preserving shortcuts are a universal architectural principle for deep networks regardless of topology.
Justifications
SL — Two independently invented architectures (LSTM 1997, ResNet 2015) solved analogous gradient flow problems in different topologies using the same principle: additive identity shortcuts
Antecedents (all must be IN):
- IN residual-connections-enable-arbitrary-depth — Residual connections (F(x) + x) are the key architectural innovation enabling arbitrarily deep networks — ResNet solved the degradation problem for CNNs, and the same mechanism was adopted by transformers to enable deep stacking without vanishing gradients.
- IN lstm-designed-to-solve-vanishing-gradient — LSTM was designed specifically to solve the vanishing gradient problem using forget gates as the key mechanism.
- IN lstm-solves-vanishing-gradient — LSTM (Long Short-Term Memory), invented by Hochreiter and Schmidhuber (1995/1997), uses gating mechanisms to preserve long-term dependencies and was specifically designed to address the vanishing gradient problem in vanilla RNNs.
Dependents
These beliefs depend on this one:
- IN convergent-discovery-reveals-mathematical-necessity — Three of deep learning's foundational mechanisms — gradient computation (backprop independently discovered across fields), gradient flow solutions (residual connections and LSTM gating converging independently), and weight sharing (appearing independently across architectures) — were all independently discovered or converged upon, suggesting these are mathematical necessities of the problem structure rather than contingent design choices.
- IN gradient-flow-triply-convergent-across-architectures — The gradient flow bottleneck has been addressed through at least three architecturally distinct mechanisms — LSTM gating (1997), GRU simplified gating without output gate (2014, achieving comparable performance with fewer parameters), and residual connections (2015) — suggesting that gradient flow is among the most convergently addressed problems in deep learning and that the bottleneck admits multiple valid solutions of varying complexity.
- IN gradient-flow-universal-deep-learning-bottleneck — Gradient flow is the universal bottleneck of deep learning — the independent convergent discovery of residual connections (for CNNs, 2015) and gating mechanisms (for RNNs, 1997) as solutions to gradient propagation, combined with the finding that trainability rather than capacity was the depth barrier, establishes that enabling gradient flow, not increasing model expressiveness, was the key to unlocking deep architectures.