effective-architectures-are-geometry-matched-compression

IN derived (depth 4)

Created 2026-06-21T10:27:01+00:00 · Reviewed 2026-06-21T15:37:01+00:00

Since prediction and compression are formally equivalent, and data geometry (the manifold hypothesis) offers a geometric explanation for why certain architectural inductive biases succeed, effective ML architectures can be understood as implicit compression algorithms whose success depends on alignment with the data's intrinsic geometry. CNNs exploit spatial locality and transformers exploit relational structure, and this framework suggests they succeed when the data's geometric properties match their compression strategy — though the formal connection between manifold geometry and the prediction-compression equivalence remains conceptual rather than proven.

Justifications

SL — Learning-compression equivalence (d1) + geometry-driven architecture (d3) reveals architectures as manifold-matched compressors

Antecedents (all must be IN):

IN prediction-compression-manifold-unified-view — Prediction and compression are formally equivalent (Delétang et al., 2023), and the manifold hypothesis — that high-dimensional data lies on low-dimensional manifolds — offers a geometric explanation for why compression is effective in practice. Together, these ideas suggest a connection between learning, compression, and geometry, though the formal link between the manifold hypothesis and the prediction-compression equivalence is conceptual rather than proven.
IN data-geometry-not-capacity-determines-architecture — Architecture design is governed by data geometry rather than model capacity — the manifold hypothesis explains why inductive biases (local connectivity, weight sharing) succeed, while the depth bottleneck reveals that theoretical capacity (universal approximation) matters less than trainability (residual connections enabling gradient flow through the data manifold's structure).

Dependents

These beliefs depend on this one:

IN architecture-design-has-geometry-but-lacks-reliability — ML architecture design possesses a principled theoretical foundation (manifold-matched compression from data geometry) but this foundation addresses only which architectures work, not whether they work safely — the manifold hypothesis explains inductive bias effectiveness without addressing adversarial robustness or deployment reliability.
IN architecture-taxonomy-recapitulates-data-structure-hierarchy — The neural network architecture taxonomy (MLP → CNN → RNN/LSTM → Transformer → Mamba) mirrors a hierarchy of data structure assumptions — MLPs assume no structure, CNNs assume local spatial structure, RNNs assume sequential structure, Transformers assume global relational structure, and each architecture's effectiveness is explained by matching its inductive bias to the manifold geometry of its target data type.
OUT geometry-matched-compression-principled-design-methodology — Manifold-matched compression would provide a principled, non-biological methodology for architecture design — replacing neuroscience analogy with information geometry to predict which inductive biases will succeed for a given data domain — but only if the quadratic scaling limitation of the current best geometry-exploiting architecture (Transformers) can be overcome.
OUT manifold-geometry-rescues-architecture-design-from-crisis — The manifold hypothesis would rescue ML architecture design from its theoretical crisis — providing principled geometry-matched compression as a design basis while no other reliable foundation exists — but only if classical generalization theory's collapse doesn't undermine the manifold framework's own ability to guarantee generalization.
OUT manifold-geometry-sufficient-for-architecture-design — The manifold hypothesis would provide a sufficient principled foundation for architecture design — manifold geometry explains the full CNN-to-Transformer inductive bias spectrum and prediction-compression equivalence grounds architecture selection in information theory, enabling principled rather than empirical design choices.
IN weight-sharing-implements-geometry-matched-compression — Weight sharing can be understood as a primary mechanism through which neural architectures implement geometry-matched compression — CNN shared kernels exploit spatial structure via parameter reuse, transformer weight tying exploits distributional symmetry across embedding spaces, and both achieve efficiency by matching parameter reuse patterns to the data's intrinsic geometry.