theorem-1-capacity-exponential-bounds
IN premise — summaries/2026/08/24/engels-2024-not-all-features-linear-sR-references-chunk-1.md
Created 2026-08-25T02:58:02+00:00
Theorem 1 establishes that the number of pairwise δ-orthogonal projection matrices in a d-dimensional residual stream has a lower bound of e^{C₁(d/d′²)δ²} and an upper bound of e^{C₂(d−d′)δ² log(1/δ)}, showing exponential capacity with a large gap between bounds.
Summary
The residual stream in a d-dimensional architecture can hold an exponentially large number of nearly independent directional channels, meaning the system's capacity for distinct information paths grows far faster than any polynomial in the dimension. The practical upshot is that the architecture has vastly more "wiring" available than a simple dimension count would suggest, though the exact scaling is still bracketed between two exponential rates that differ significantly, leaving room for tighter analysis.