superposition-covariance-editability-triangle
OUT derived (depth 4)
Created 2026-08-25T03:07:18+00:00 · Reviewed 2026-08-25T03:24:16+00:00
Superposition, covariance whitening, and rank-one editability form a closed logical triangle in which each property necessitates the others: over-complete superposition requires covariance separation for feature addressability, covariance separation defines the geometric space in which rank-one updates are well-defined, and the boundedness of rank-one editing confirms the addressable space is finite.
Justifications
SL — The three depth-2/3 geometric properties are individually stated but their mutual dependency is never made explicit; together they form a closed triangle where retracting any vertex collapses the others.
Antecedents (all must be IN):
- OUT superposition-necessitates-covariance-whitening — Over-complete superposition in the shared residual-stream substrate is the precise structural condition that necessitates covariance/whitening (second-moment projection) as the canonical tool for isolating individual features and performing targeted rank-one edits; without superposition, raw Euclidean geometry would suffice and the entire covariance-geometry framework would be unnecessary.
- OUT geometric-editing-addressability-bound — The covariance-geometry framework defines a precise and minimal addressable space for knowledge editing (rank-one updates to a single MLP value projection), but the combination of superposition and distributed corpus acquisition structurally bounds this to single-fact local corrections—edits cannot create novel multi-hop associations because the target knowledge was never locally consolidated in the first place.
- OUT covariance-geometry-as-operational-semantic-space — The covariance/whitening geometry (second-moment matrices) is the operational definition of semantic coordinate space in LLMs: it simultaneously parameterises feature interpretation (SAE decoder space, Park polytopes), similarity evaluation (cosine→Spearman pipeline), and knowledge modification (ROME rank-one updates), and this structure converges across model families.
Unless (any of these IN defeats this justification):
- IN superposition-covariance-editability-triangle-v2 — Superposition, covariance geometry, and rank-one editability form an interdependent framework in which each concept provides structural context for the others: over-complete superposition in the residual stream is the primary structural condition that makes covariance/whitening the canonical tool for feature isolation and targeted rank-one edits, covariance geometry serves as the operational coordinate space in which those rank-one updates are formulated, and the structural bounds on such edits—limited to single-fact local corrections rather than novel multi-hop associations—reflect the distributed, locally non-consolidated nature of the underlying superposed representation.
Dependents
These beliefs depend on this one:
- OUT context-as-infinite-dimension-complement — The context window is the operational realization of the "infinite" or "unbounded" dimension in the superposition framework: while the d-dimensional residual stream is over-compressed to store N≫d features (necessitating covariance whitening for well-defined read/write operations), the context window provides an unbounded, interference-free channel where knowledge is injected without superposition, without the need for C⁻¹k* addressability, and without the rank-one constraint—making it the architectural complement to the finite, superposed parametric memory.
- OUT read-write-asymmetry-superposition-consequence — The read-broad/write-narrow asymmetry is the direct operational consequence of over-complete superposition: reading is a simultaneous linear projection over all active features, but writing must be rank-one to avoid cross-talk in the over-complete basis, making the write channel inherently and necessarily narrower.
- OUT read-write-geometric-asymmetry-topology — The LLM is architecturally a read-broad/write-narrow system: the read channel (SAE feature extraction, evaluation, retrieval) spans the full d-dimensional covariance-whitened space, while the write channel (ROME rank-one, SAE feature ablation) is restricted to a 1-dimensional key-direction subspace, making knowledge correction fundamentally more constrained than knowledge interrogation.
- OUT sae-guided-feature-space-editing — SAE-identified features provide a semantically-interpretable coordinate system for specifying knowledge edits—enabling edits to be expressed as feature-space operations (e.g., "suppress feature 34M/31164353 and amplify its neighborhood") that the covariance geometry guarantees map to valid rank-one weight-space operations—thereby bridging the interpretability and editing literatures through the shared second-moment structure.
- OUT space-decomposition-under-superposition — The full LLM semantic space admits a clean algebraic decomposition into categorical polytope subspaces (discrete concepts) and hierarchical orthogonality subspaces (graded taxonomic structure) within the covariance-geometric framework, providing a complete account of how discrete and graded meaning coexist in a single over-complete vector space
- OUT superposition-as-single-root-cause — Superposition is the unique root cause from which the full read/write/editing logical structure follows: over-complete representation necessitates covariance whitening (making geometry well-defined), which in turn explains both the broad-read/narrow-write asymmetry (projection vs. rank-one injection) and the closed triangle linking all three properties—there is no independent second principle needed.