superposition-unifies-data-and-representation-redundancy

OUT derived (depth 2)

Created 2026-08-25T04:19:08+00:00 · Reviewed 2026-08-25T04:28:09+00:00

Over-complete superposition is the single structural cause of both the data-side redundancy (5× correlated corpora yield only marginal gains, Spearman 0.87–0.97 inter-correlation) and the representation-side redundancy (SAE dead features at 2–48%, reconstruction shrinkage): in both cases the available "capacity" exceeds the unique information it must encode.

Justifications

This belief has 3 justifications — it is IN if any one holds.

SL — Each antecedent independently demonstrates the same over-completeness from a different vantage (training data, feature capacity, reconstruction fidelity); the conclusion survives if any one holds.

Antecedents (all must be IN):

  • IN corpus-redundancy-overcounts-unique-information — The high inter-correlation of pre-training corpora (Spearman 0.87–0.97) combined with marginal accuracy gains from 5× data indicates that "relevant document count" systematically overcounts unique information, making the long-tail scaling estimate an upper bound on true information scarcity rather than a lower bound on required capacity.

Unless (any of these IN defeats this justification):

  • IN superposition-unifies-data-and-representation-redundancy-v2 — Over-complete superposition is a primary structural factor consistent with both the data-side redundancy (5× correlated corpora at Spearman 0.87–0.97 yielding only marginal accuracy gains, suggesting 'relevant document count' overcounts unique information) and the representation-side redundancy (SAE dead features ranging from ~2% at 1M to ~65% at 34M, alongside systematic under-reconstruction): in both cases the observed patterns are consistent with available capacity exceeding the unique information to be encoded, though the antecedents establish these as descriptive correlations rather than isolating over-completeness as the sole causal mechanism.
SL — Each antecedent independently demonstrates the same over-completeness from a different vantage (training data, feature capacity, reconstruction fidelity); the conclusion survives if any one holds.

Antecedents (all must be IN):

  • IN sae-dead-feature-proportions-by-size — Dead feature proportions (zero activation across 10⁷ tokens) are approximately 2% for the 1M SAE, 35% for the 4M SAE, and 65% for the 34M SAE.

Unless (any of these IN defeats this justification):

  • IN superposition-unifies-data-and-representation-redundancy-v2 — Over-complete superposition is a primary structural factor consistent with both the data-side redundancy (5× correlated corpora at Spearman 0.87–0.97 yielding only marginal accuracy gains, suggesting 'relevant document count' overcounts unique information) and the representation-side redundancy (SAE dead features ranging from ~2% at 1M to ~65% at 34M, alongside systematic under-reconstruction): in both cases the observed patterns are consistent with available capacity exceeding the unique information to be encoded, though the antecedents establish these as descriptive correlations rather than isolating over-completeness as the sole causal mechanism.
SL — Each antecedent independently demonstrates the same over-completeness from a different vantage (training data, feature capacity, reconstruction fidelity); the conclusion survives if any one holds.

Antecedents (all must be IN):

  • IN sae-shrinkage-problem — The SAE shrinkage problem refers to sparse autoencoders under-reconstructing the original activations, with mitigations including finetuning approaches (Wright & Sharkey) and gating activation functions (Rajamanoharan et al.).

Unless (any of these IN defeats this justification):

  • IN superposition-unifies-data-and-representation-redundancy-v2 — Over-complete superposition is a primary structural factor consistent with both the data-side redundancy (5× correlated corpora at Spearman 0.87–0.97 yielding only marginal accuracy gains, suggesting 'relevant document count' overcounts unique information) and the representation-side redundancy (SAE dead features ranging from ~2% at 1M to ~65% at 34M, alongside systematic under-reconstruction): in both cases the observed patterns are consistent with available capacity exceeding the unique information to be encoded, though the antecedents establish these as descriptive correlations rather than isolating over-completeness as the sole causal mechanism.