architecture-design-dual-constraint-geometry-hardware
IN derived (depth 4)
Created 2026-06-21T10:16:38+00:00 · Reviewed 2026-06-21T15:37:01+00:00
ML architecture design is dually constrained by data geometry and hardware capability — the manifold hypothesis dictates which inductive biases should work in principle, while hardware evolution determines which architectures can scale in practice, and only architectures satisfying both constraints survive.
Justifications
SL — One depth-3 chain establishes geometry as the theoretical constraint, the other establishes hardware as the practical constraint — architectures must satisfy both to dominate
Antecedents (all must be IN):
- IN data-geometry-not-capacity-determines-architecture — Architecture design is governed by data geometry rather than model capacity — the manifold hypothesis explains why inductive biases (local connectivity, weight sharing) succeed, while the depth bottleneck reveals that theoretical capacity (universal approximation) matters less than trainability (residual connections enabling gradient flow through the data manifold's structure).
- IN sequence-modeling-arc-recapitulates-hardware-evolution — The evolutionary arc of sequence modeling (RNN→LSTM→Transformer) was shaped in part by hardware evolution: transformers' parallel self-attention architecture achieved dominance partly because of its synergy with GPU parallelism, making hardware-architecture alignment a primary factor — though attention as a bridging mechanism and fundamental tradeoffs between memory range and parallelism were equally essential to the transition.
Dependents
These beliefs depend on this one:
- IN architecture-selection-two-stage-filter — ML architecture selection operates as a two-stage filter: data geometry determines which inductive biases are effective (first filter), and hardware scalability determines which effective architectures survive (second filter) — scalability can veto geometric fit but not vice versa, explaining why theoretically superior architectures are routinely displaced.