relu-width-universal-approximation-constraint

IN premise — entries/2026/06/21/wiki-Deep_learning-chunk-1.md

Created 2026-06-21T09:55:49+00:00

Lu et al. proved that for ReLU networks, the width must be strictly larger than the input dimension to be a universal approximator; width ≤ input dimension fails

Dependents

These beliefs depend on this one:

• IN universal-approximation-vs-practical-depth — Universal approximation theory (Cybenko, 1989) establishes that shallow networks with sigmoidal activations can approximate any continuous function on compact subsets of R^n, but this existence guarantee has practical caveats: deep networks leverage hierarchical feature learning across multiple layers, and ReLU networks require width strictly greater than input dimension to maintain universal approximation — suggesting that theoretical approximation capacity alone does not determine architectural effectiveness.