data-geometry-not-capacity-determines-architecture
IN derived (depth 3)
Created 2026-06-21T10:13:05+00:00 · Reviewed 2026-06-21T15:37:01+00:00
Architecture design is governed by data geometry rather than model capacity — the manifold hypothesis explains why inductive biases (local connectivity, weight sharing) succeed, while the depth bottleneck reveals that theoretical capacity (universal approximation) matters less than trainability (residual connections enabling gradient flow through the data manifold's structure).
Justifications
SL — depth-3 — both depth-2 conclusions independently point to the same principle: what matters is matching architecture to data structure, not maximizing representational power
Antecedents (all must be IN):
- IN manifold-structure-explains-inductive-bias-effectiveness — The manifold hypothesis offers a theoretical lens for understanding why certain architectural inductive biases are effective — if high-dimensional data lies on low-dimensional manifolds (and prediction is formally equivalent to compression of that structure), then architectures exploiting local connectivity and weight sharing can be seen as responses to data geometry rather than purely ad hoc engineering choices.
- IN trainability-not-capacity-was-the-depth-bottleneck — The theory-practice gap in neural network depth — where universal approximation says shallow networks suffice but practice demands deep ones — reveals that the real bottleneck was never representational capacity but trainability, and residual connections resolved exactly this optimization barrier, enabling the hierarchical representations that shallow networks could theoretically but not practically learn.
Dependents
These beliefs depend on this one:
- IN architecture-design-dual-constraint-geometry-hardware — ML architecture design is dually constrained by data geometry and hardware capability — the manifold hypothesis dictates which inductive biases should work in principle, while hardware evolution determines which architectures can scale in practice, and only architectures satisfying both constraints survive.
- IN effective-architectures-are-geometry-matched-compression — Since prediction and compression are formally equivalent, and data geometry (the manifold hypothesis) offers a geometric explanation for why certain architectural inductive biases succeed, effective ML architectures can be understood as implicit compression algorithms whose success depends on alignment with the data's intrinsic geometry. CNNs exploit spatial locality and transformers exploit relational structure, and this framework suggests they succeed when the data's geometric properties match their compression strategy — though the formal connection between manifold geometry and the prediction-compression equivalence remains conceptual rather than proven.