prediction-compression-manifold-unified-view

IN derived (depth 1)

Created 2026-06-21T10:06:02+00:00 · Reviewed 2026-06-21T15:37:01+00:00

Prediction and compression are formally equivalent (Delétang et al., 2023), and the manifold hypothesis — that high-dimensional data lies on low-dimensional manifolds — offers a geometric explanation for why compression is effective in practice. Together, these ideas suggest a connection between learning, compression, and geometry, though the formal link between the manifold hypothesis and the prediction-compression equivalence is conceptual rather than proven.

Justifications

SL — Three independently established results that converge on the same insight about data structure

Antecedents (all must be IN):

IN ml-compression-learning-equivalence — Prediction and compression are formally equivalent; language models can exceed PNG/FLAC in lossless compression (Delétang et al., 2023: 'language modeling is compression')
IN compression-prediction-equivalence — Optimal prediction and optimal data compression are formally equivalent: an optimal predictor of sequence probabilities can be used for optimal compression via arithmetic coding, and vice versa
IN ml-manifold-hypothesis — The manifold hypothesis proposes that high-dimensional data lies along low-dimensional manifolds, and is a foundational assumption for many dimensionality reduction techniques

Dependents

These beliefs depend on this one:

IN effective-architectures-are-geometry-matched-compression — Since prediction and compression are formally equivalent, and data geometry (the manifold hypothesis) offers a geometric explanation for why certain architectural inductive biases succeed, effective ML architectures can be understood as implicit compression algorithms whose success depends on alignment with the data's intrinsic geometry. CNNs exploit spatial locality and transformers exploit relational structure, and this framework suggests they succeed when the data's geometric properties match their compression strategy — though the formal connection between manifold geometry and the prediction-compression equivalence remains conceptual rather than proven.
IN manifold-structure-explains-inductive-bias-effectiveness — The manifold hypothesis offers a theoretical lens for understanding why certain architectural inductive biases are effective — if high-dimensional data lies on low-dimensional manifolds (and prediction is formally equivalent to compression of that structure), then architectures exploiting local connectivity and weight sharing can be seen as responses to data geometry rather than purely ad hoc engineering choices.