superposition-as-compositional-basis-v2
IN premise
Created 2026-08-25T04:09:09+00:00
Superposition appears to be a significant structural principle in LLMs: the 10–200× over-complete expansion observed in SAEs, the key-value associative-memory structure identified in ROME (W_fc as key projection, W_proj as value projection), and the direct-sum decomposition of hierarchical orthogonal subspaces shown by Park et al. are related findings that are mutually consistent with over-complete, superposed representations, though the antecedents do not establish them as independent consequences of a single shared geometric mechanism.
Summary
Three separate lines of LLM research all point in the same direction: LLMs pack far more concepts into their internal activations than they have dimensions to represent independently, layering them on top of each other. The built-in caution is that while these findings hang together, the system should not assume they are three views of one single geometric mechanism; they may be distinct tricks that merely look consistent.
Dependents
These beliefs depend on this one:
- OUT superposition-as-compositional-basis — Superposition is the fundamental compositional mechanism in LLMs: the 10–200× over-complete expansion (SAE), the key-value memory structure (ROME's W_fc/W_proj), and the direct-sum space decomposition (Park's polytope+orthogonality) are three independent geometric consequences of the same over-completeness.