trainability-not-capacity-was-the-depth-bottleneck
IN derived (depth 2)
Created 2026-06-21T10:09:45+00:00 · Reviewed 2026-06-21T15:37:01+00:00
The theory-practice gap in neural network depth — where universal approximation says shallow networks suffice but practice demands deep ones — reveals that the real bottleneck was never representational capacity but trainability, and residual connections resolved exactly this optimization barrier, enabling the hierarchical representations that shallow networks could theoretically but not practically learn.
Justifications
SL — Skip connections proved that depth's limitation was optimization, not representation
Antecedents (all must be IN):
- IN residual-connections-enable-arbitrary-depth — Residual connections (F(x) + x) are the key architectural innovation enabling arbitrarily deep networks — ResNet solved the degradation problem for CNNs, and the same mechanism was adopted by transformers to enable deep stacking without vanishing gradients.
- IN universal-approximation-vs-practical-depth — Universal approximation theory (Cybenko, 1989) establishes that shallow networks with sigmoidal activations can approximate any continuous function on compact subsets of R^n, but this existence guarantee has practical caveats: deep networks leverage hierarchical feature learning across multiple layers, and ReLU networks require width strictly greater than input dimension to maintain universal approximation — suggesting that theoretical approximation capacity alone does not determine architectural effectiveness.
Dependents
These beliefs depend on this one:
- IN data-geometry-not-capacity-determines-architecture — Architecture design is governed by data geometry rather than model capacity — the manifold hypothesis explains why inductive biases (local connectivity, weight sharing) succeed, while the depth bottleneck reveals that theoretical capacity (universal approximation) matters less than trainability (residual connections enabling gradient flow through the data manifold's structure).
- IN gradient-flow-universal-deep-learning-bottleneck — Gradient flow is the universal bottleneck of deep learning — the independent convergent discovery of residual connections (for CNNs, 2015) and gating mechanisms (for RNNs, 1997) as solutions to gradient propagation, combined with the finding that trainability rather than capacity was the depth barrier, establishes that enabling gradient flow, not increasing model expressiveness, was the key to unlocking deep architectures.