superposition-as-compositional-basis
OUT derived (depth 1)
Created 2026-08-25T03:02:14+00:00 · Reviewed 2026-08-25T04:02:18+00:00
Superposition is the fundamental compositional mechanism in LLMs: the 10–200× over-complete expansion (SAE), the key-value memory structure (ROME's W_fc/W_proj), and the direct-sum space decomposition (Park's polytope+orthogonality) are three independent geometric consequences of the same over-completeness.
Justifications
This belief has 3 justifications — it is IN if any one holds.
SL — Convergent evidence from three independent research programs. SAE quantifies the over-completeness ratio; ROME exploits its functional consequence (addressable KV pairs); Park derives its geometric consequence (direct-sum decomposition). Each independently supports "superposition is the compositional basis," so the conclusion survives if any one holds.
Antecedents (all must be IN):
- IN sae-expansion-ratio-10-to-200x — Sparse Autoencoders decompose dense d-dimensional activations into a typically 10× to 200× larger set of mostly-sparse, near-orthogonal latent features.
Unless (any of these IN defeats this justification):
- IN superposition-as-compositional-basis-v2 — Superposition appears to be a significant structural principle in LLMs: the 10–200× over-complete expansion observed in SAEs, the key-value associative-memory structure identified in ROME (W_fc as key projection, W_proj as value projection), and the direct-sum decomposition of hierarchical orthogonal subspaces shown by Park et al. are related findings that are mutually consistent with over-complete, superposed representations, though the antecedents do not establish them as independent consequences of a single shared geometric mechanism.
SL — Convergent evidence from three independent research programs. SAE quantifies the over-completeness ratio; ROME exploits its functional consequence (addressable KV pairs); Park derives its geometric consequence (direct-sum decomposition). Each independently supports "superposition is the compositional basis," so the conclusion survives if any one holds.
Antecedents (all must be IN):
- IN rome-mlp-key-value-interpretation — In the ROME framework, W_fc in the MLP acts as the key projection and W_proj acts as the value projection, modeling the MLP as a linear associative memory where Wk ≈ v.
Unless (any of these IN defeats this justification):
- IN superposition-as-compositional-basis-v2 — Superposition appears to be a significant structural principle in LLMs: the 10–200× over-complete expansion observed in SAEs, the key-value associative-memory structure identified in ROME (W_fc as key projection, W_proj as value projection), and the direct-sum decomposition of hierarchical orthogonal subspaces shown by Park et al. are related findings that are mutually consistent with over-complete, superposed representations, though the antecedents do not establish them as independent consequences of a single shared geometric mechanism.
SL — Convergent evidence from three independent research programs. SAE quantifies the over-completeness ratio; ROME exploits its functional consequence (addressable KV pairs); Park derives its geometric consequence (direct-sum decomposition). Each independently supports "superposition is the compositional basis," so the conclusion survives if any one holds.
Antecedents (all must be IN):
- IN park-2025-direct-sum-space-decomposition — The combination of polytope representations and hierarchical orthogonality (Theorem 8) implies the full representation space decomposes as a direct sum of orthogonal subspaces, one per level of the hierarchy.
Unless (any of these IN defeats this justification):
- IN superposition-as-compositional-basis-v2 — Superposition appears to be a significant structural principle in LLMs: the 10–200× over-complete expansion observed in SAEs, the key-value associative-memory structure identified in ROME (W_fc as key projection, W_proj as value projection), and the direct-sum decomposition of hierarchical orthogonal subspaces shown by Park et al. are related findings that are mutually consistent with over-complete, superposed representations, though the antecedents do not establish them as independent consequences of a single shared geometric mechanism.
Dependents
These beliefs depend on this one:
- OUT geometric-editing-addressability-bound — The covariance-geometry framework defines a precise and minimal addressable space for knowledge editing (rank-one updates to a single MLP value projection), but the combination of superposition and distributed corpus acquisition structurally bounds this to single-fact local corrections—edits cannot create novel multi-hop associations because the target knowledge was never locally consolidated in the first place.
- OUT read-write-asymmetry — A structural asymmetry governs LLM knowledge: the read channel (feature-level interpretation via SAE, Park polytope analysis, embedding evaluation) operates in a rich, model-independent geometric space, while the write channel (editing) is impoverished to parametric rank-one MLP updates in a single layer—superposition enriches the feature space for reading but makes feature-level writing intractable.
- OUT sae-granularity-as-superposition-resolution — SAE's empirical granularity scaling (broad categorical features at small run sizes → specific entity features at large run sizes) is the operational resolution of superposition: the 10–200× expansion ratio creates discrete "zoom levels" at which the same underlying covariance geometry manifests as features of different semantic specificity.
- OUT superposition-geometry-explains-universality — The cross-model universality of feature geometry (SAE features more similar across architectures than within, Park orthogonality validated on both Gemma and LLaMA) is a consequence of superposition: the over-complete compositional basis is determined by the shared semantic grammar of language, making geometric structure an architectural invariant rather than a model-specific artifact.
- OUT superposition-necessitates-covariance-whitening — Over-complete superposition in the shared residual-stream substrate is the precise structural condition that necessitates covariance/whitening (second-moment projection) as the canonical tool for isolating individual features and performing targeted rank-one edits; without superposition, raw Euclidean geometry would suffice and the entire covariance-geometry framework would be unnecessary.