sae-granularity-as-superposition-resolution

OUT derived (depth 2)

Created 2026-08-25T03:15:17+00:00 · Reviewed 2026-08-25T04:02:18+00:00

SAE's empirical granularity scaling (broad categorical features at small run sizes → specific entity features at large run sizes) is the operational resolution of superposition: the 10–200× expansion ratio creates discrete "zoom levels" at which the same underlying covariance geometry manifests as features of different semantic specificity.

Justifications

SL — Connects SAE's empirical granularity observation (base) to the superposition framework (derived), showing that the "zoom" is not an SAE artifact but the quantitative signature of over-complete encoding resolving at different scales.

Antecedents (all must be IN):

OUT superposition-as-compositional-basis — Superposition is the fundamental compositional mechanism in LLMs: the 10–200× over-complete expansion (SAE), the key-value memory structure (ROME's W_fc/W_proj), and the direct-sum space decomposition (Park's polytope+orthogonality) are three independent geometric consequences of the same over-completeness.
IN sae-feature-granularity-scales-with-run-size — Larger SAE runs yield more specific features (e.g., 34M run: 'Golden Gate Bridge') while smaller runs yield broader categories (e.g., 1M run: 'Monuments and tourist attractions').
IN sae-expansion-ratio-10-to-200x — Sparse Autoencoders decompose dense d-dimensional activations into a typically 10× to 200× larger set of mostly-sparse, near-orthogonal latent features.

Dependents

These beliefs depend on this one:

OUT sae-resolution-of-space-decomposition — SAE's expansion-ratio scaling (broad→specific features) is the operational resolution mechanism for the algebraic space decomposition: coarse SAEs resolve the polytope hulls (categorical structure), fine SAEs resolve individual vertices (entity-level features), and the neighborhood adjacency graph is the polytope edge structure visible at whichever resolution is chosen.
OUT sae-shrinkage-as-finite-resolution-limit — The SAE shrinkage problem (under-reconstruction with finite expansion ratios) is the operational signature of finite resolution in the superposition→whitening framework: any finite dictionary size leaves irreducible reconstruction loss because the superposed structure is fundamentally over-complete, and the power-law decrease in loss with compute is the scaling signature of approaching (but never reaching) full resolution.