universal-approximation-vs-practical-depth

IN derived (depth 1)

Created 2026-06-21T10:06:02+00:00 · Reviewed 2026-06-21T15:37:01+00:00

Universal approximation theory (Cybenko, 1989) establishes that shallow networks with sigmoidal activations can approximate any continuous function on compact subsets of R^n, but this existence guarantee has practical caveats: deep networks leverage hierarchical feature learning across multiple layers, and ReLU networks require width strictly greater than input dimension to maintain universal approximation — suggesting that theoretical approximation capacity alone does not determine architectural effectiveness.

Justifications

SL — Theoretical sufficiency of shallow networks vs. practical necessity of depth

Antecedents (all must be IN):

IN cybenko-universal-approximation-theorem-1989 — Cybenko's Universal Approximation Theorem (1989) proves that a single hidden layer with sigmoidal activation functions can approximate any continuous function on compact subsets of R^n.
IN ml-deep-learning-hierarchical-features — Deep learning discovers hierarchical features where higher-level abstract features are built from lower-level ones across multiple layers
IN relu-width-universal-approximation-constraint — Lu et al. proved that for ReLU networks, the width must be strictly larger than the input dimension to be a universal approximator; width ≤ input dimension fails

Dependents

These beliefs depend on this one:

IN trainability-not-capacity-was-the-depth-bottleneck — The theory-practice gap in neural network depth — where universal approximation says shallow networks suffice but practice demands deep ones — reveals that the real bottleneck was never representational capacity but trainability, and residual connections resolved exactly this optimization barrier, enabling the hierarchical representations that shallow networks could theoretically but not practically learn.