Superposition

42 beliefs (42 IN, 0 OUT)

Superposition refers to the phenomenon in which a neural network encodes more features than it has activation dimensions, multiplexing information into a fixed-size weight space by storing features in non-orthogonal, over-complete directions (eh2022-superposition-overcomplete-features, superposition-more-features-than-dimensions, superposition-features-exceed-dimensionality). This matters because it is arguably the single most consequential structural property of modern networks: it explains why individual neurons are polysemantic, why standard disentanglement tools like PCA fail, and why interpretability research must recover features rather than simply reading them off the network. The topic is carefully distinguished from the disentanglement literature, which imposes a privileged basis on a latent space, whereas superposition research starts from the existing neuron basis and attempts to recover the hidden features that are superimposed within it (disentanglement-vs-superposition-research-distinction). Two prerequisites make the phenomenon possible and beneficial: features must be sparse, active on only a small fraction of inputs, so that the dimensionality savings are real rather than illusory (eh2022-sparsity-prerequisite-for-superposition, superposition-requires-feature-sparsity), and a hidden-layer nonlinearity such as ReLU must be present to both enable computation within the superposed representation and to allow individual overlapping feature signals to be resolved (eh2022-hidden-nonlinearity-essential-for-superposition, superposition-requ-rel-relu-necessary). Without sparsity the model collapses to an orthogonal PCA-like basis capturing only the top-k features (superposition-dense-regime-pca-behavior), and without the nonlinearity the superposed layer cannot implement the non-linear mappings it is trained to perform.

The Elhage et al. 2022 toy model, a 5-feature-to-2-dimension ReLU network trained with MSE loss and per-feature importance multipliers (superposition-toy-model-architecture, superposition-experiment-hyperparameter-sweep), provided the canonical demonstration that superposition is not a training artifact but a structural attractor. In the sparse regime the model packs all five features into two dimensions with ReLU resolving the interference (superposition-sparse-regime-overcapacity), while the antipodal solution for a 2-feature/1-dimension variant shows that superposition can specifically destroy the joint-activation pattern [1,1] while preserving marginal statistics (antipodal-solution-loses-joint-activation). Training dynamics are notably non-smooth: feature dimensionalities jump discretely like energy levels rather than transitioning gradually, even under continuous gradient descent (feature-dimensionality-discrete-jumps-training), and superposition itself emerges via a sharp phase transition with a hard regime boundary beyond which it is entirely absent (superposition-sharp-phase-transition). The maximum number of recoverable features scales linearly, not exponentially, with embedding dimension, with the constant set by sparsity density (superposition-capacity-linear-in-dimension). In the uniform case (equal importance, equal sparsity), optimal feature directions converge to the vertices of regular polytopes—a pentagon for 5 features in 2D, a tetrahedron for 4 in 3D—giving rise to a geometric regularity that is model-independent (uniform-superposition-regular-polytopes, uniform-superposition-converges-digons, isotropic-superposition-model-uniform-spacing). This geometric convergence, together with the covariance/whitening structure it induces, functions as a structural attractor determined by the shared semantic grammar of language rather than being a quirk of any single model (geometric-convergence-as-mathematical-attractor-v2). A tightly interlocked "triangle" emerges: over-complete superposition in the residual stream makes covariance geometry the canonical tool for feature isolation, covariance geometry provides the coordinate space in which rank-one edits are formulated, and the boundedness of those edits reflects the distributed, locally non-consolidated nature of the underlying superposed representation (superposition-covariance-editability-triangle-v2, superposition-as-single-root-cause-v2). The Sachan follow-up at Redwood Research formalizes the whole picture as a constrained optimization problem—minimize reconstruction error subject to a sparsity or orthogonality budget—yielding principled predictions of when polysemanticity will emerge (superposition-as-constrained-optimization).

Several downstream consequences follow from the interference that superposition necessarily introduces. The small non-zero off-diagonal entries (epsilon) that appear in W^T W when features are packed into shared dimensions (superposition-creates-interference-entries) constitute the direct mechanism of adversarial vulnerability, which increases by more than 3x as superposition forms and tracks the number of features per dimension (superposition-increases-vulnerability-3x). This vulnerability persists in the infinite-data limit, confirming it is a property of the optimal sparse representation rather than a finite-sample artifact (vulnerability-persists-infinite-data-limit). Positive dot products between semantically unrelated features make naive cosine-similarity probes unreliable (superposition-introduces-positive-dot-products). The Templeton 2024 paper further complicates the picture by identifying three distinct types—activation superposition, attention superposition across heads, and weight superposition in learned weights (three-superposition-types)—each requiring different tools to resolve. Architectural choices can modulate the incentive for superposition: L1 regularization kills below-threshold features and reduces polysemanticity (l1-regularization-reduces-superposition), and Mixture-of-Experts architectures decouple neuron count from compute, potentially eliminating the efficiency gain that makes superposition attractive in the first place (moe-reduces-superposition-incentive). On the positive side, superposition is not merely a storage trick; neural networks can perform full non-linear computation within the superposed hidden layer (eh2022-superposition-supports-computation), and the two-model decomposition that assigns input/output layers to a disentangled basis while keeping the hidden layer superposed provides a practical framework for study (eh2022-two-model-disentangled-superposed). The Gaussian-latent assumption in VAEs and GANs may even force superposition of inherently sparse variables, since the superposition of many sparse components approaches a Gaussian by the Central Limit Theorem (gaussian-latent-assumption-drives-superposition).

Empirically, the theory aligns with observations across architectures: early InceptionV1 layers are monosemantic while later layers become polysemantic as features grow sparser (inceptionv1-early-layers-monosemantic), and the first MLP layers in Transformers are predicted and confirmed to be extremely polysemantic (transformer-early-mlp-extremely-polysemantic). The "solving superposition" program is defined as enumerating all features in a model, equivalently unfolding superposed activations into a larger space via compressed sensing, formalized as finding sparse A and mixing B matrices such that H approximately equals A times B (solving-superposition-defined-as-feature-enumeration, overcomplete-basis-sparse-coding-formulation). Independent replication by Redwood Research confirmed the key results in the superposition, phase-change, and uniform-superposition sections of the original paper (independent-replication-confirmed-key-results), strengthening confidence in the geometric claims. At the broadest level, superposition appears as a unifying structural principle connecting the 10-to-200x over-complete expansion seen in SAEs, the key-value associative-memory structure identified in ROME, and the direct-sum decomposition of hierarchical orthogonal subspaces, with all three mutually consistent with over-complete superposed representations (superposition-as-compositional-basis-v2). The same over-completeness is also consistent with data-side redundancy (correlated corpora yielding diminishing accuracy gains) and representation-side redundancy (SAE dead features rising from 2% to 65% as dimensionality grows), though these are established as descriptive correlations rather than isolated causal claims (superposition-unifies-data-and-representation-redundancy-v2). Equal-loss minima in compressed-sensing tasks—some polysemantic, some monosemantic—caution that the loss landscape alone does not determine the degree of superposition (equal-loss-minima-compressed-sensing), and the memorization-versus-generalization phase transition shows that small datasets are stored in superposed, entangled form rather than as separate features (memorization-superposition-phase-transition). All beliefs in this topic are currently IN; none have been retracted, indicating a stable and internally consistent cluster of claims in the knowledge base.

IN antipodal-solution-loses-joint-activation
The antipodal solution W=[1,-1] for two binary features in one dimension stores features in superposition but specifically loses the ability to represent the joint-activation pattern [1,1].
IN disentanglement-vs-superposition-research-distinction
Disentanglement research aims to impose a privileged basis on a latent space, whereas superposition research starts from an existing (neuron) basis and attempts to recover features that are superimposed within that basis.
IN eh2022-hidden-nonlinearity-essential-for-superposition
The hidden-layer non-linearity is essential for computation in superposition; without it, the superposed hidden layer cannot implement the non-linear mapping
IN eh2022-sparsity-prerequisite-for-superposition
Sparsity of features (active on only a small subset of inputs) is the prerequisite that makes superposition beneficial; if features were dense, the dimensionality savings would vanish
IN eh2022-superposition-overcomplete-features
Superposition in neural networks means linear features are overcomplete (more features than dimensions), so the model multiplexes information into the same weight space
IN eh2022-superposition-supports-computation
Neural networks can perform non-linear computation entirely within a superposed (lower-dimensional) hidden-layer representation, not merely store features in superposition
IN eh2022-two-model-disentangled-superposed
The two-model decomposition for studying superposition assigns input and output layers to a disentangled (privileged-basis) representation and the hidden layer to a lower-dimensional superposed representation
IN equal-loss-minima-compressed-sensing
MLP layers trained on compressed sensing tasks exhibit multiple equal-loss minima, some polysemantic and some monosemantic, suggesting the loss landscape alone does not determine the degree of superposition
IN feature-dimensionality-discrete-jumps-training
Feature dimensionalities in toy superposition models exhibit discrete jumps ('energy level jumps') during training rather than smooth transitions, despite continuous gradient-based optimization.
IN gaussian-latent-assumption-drives-superposition
VAEs and GANs assume Gaussian latent distributions; since sparse latent variables are non-Gaussian but their superposition approaches Gaussian by the Central Limit Theorem, the architecture may force superposition of sparse variables.
IN geometric-convergence-as-mathematical-attractor-v2
The cross-model universality of feature geometry, combined with its ontological status as a model-independent semantic structure, indicates that LLMs sharing over-complete superposition architectures converge toward a shared geometric regularity: the covariance/whitening geometry functions as a convergent structural attractor determined by the shared semantic grammar of language, rather than being merely a model-specific artifact.
IN inceptionv1-early-layers-monosemantic
In InceptionV1, early-layer neurons are monosemantic (aligned with a privileged basis, no superposition), while later-layer neurons become polysemantic as features become sparser.
IN independent-replication-confirmed-key-results
Independent replication by Redwood Research confirmed the key results in the 'Demonstrating Superposition,' 'Superposition as a Phase Change,' and 'Geometry of Superposition – Uniform Superposition' sections of the Elhage et al. 2022 paper.
IN isotropic-superposition-model-uniform-spacing
The isotropic superposition model assumes features are discrete, one-dimensional directions in activation space that repel each other via interference, producing roughly uniform angular spacing
IN l1-regularization-reduces-superposition
Adding L1 regularization (λ||h||₁) to the loss function in the superposition toy model kills features below an importance threshold, particularly non-basis-aligned ones, thereby reducing superposition.
IN memorization-superposition-phase-transition
Small datasets are memorized in superposed (entangled) form rather than generalized as separate features, with a sharp phase transition separating memorization from generalization regimes (Henighan et al.)
IN moe-reduces-superposition-incentive
Mixture-of-Experts (MoE) architectures decouple neuron count from FLOPs by activating only a fraction of neurons per example, potentially eliminating the efficiency gain that makes superposition attractive and thereby reducing superposition.
IN overcomplete-basis-sparse-coding-formulation
The overcomplete basis formulation for solving superposition is: given activation matrix H ~ [d, m] and n underlying features (n >> m), find sparse matrix A ~ [d, n] and mixing matrix B ~ [n, m] such that H ≈ A · B.
IN solving-superposition-defined-as-feature-enumeration
'Solving superposition' is defined in the Elhage et al. 2022 paper as achieving the ability to enumerate all features in a model, equivalently 'unfolding' superposition activations into a larger non-superposition space via compressed sensing.
IN superposition-as-compositional-basis-v2
Superposition appears to be a significant structural principle in LLMs: the 10–200× over-complete expansion observed in SAEs, the key-value associative-memory structure identified in ROME (W_fc as key projection, W_proj as value projection), and the direct-sum decomposition of hierarchical orthogonal subspaces shown by Park et al. are related findings that are mutually consistent with over-complete, superposed representations, though the antecedents do not establish them as independent consequences of a single shared geometric mechanism.
IN superposition-as-constrained-optimization
The Sachan follow-up (Redwood Research) formalizes superposition as a constrained optimization problem: minimize reconstruction error subject to a sparsity/orthogonality budget, providing a principled prediction of when polysemanticity emerges.
IN superposition-as-single-root-cause-v2
Superposition is a central unifying node in a tightly interlocked logical structure linking over-complete representation, covariance whitening, and rank-one editability. The read-broad/write-narrow asymmetry is a direct operational consequence of the over-complete basis (simultaneous linear projection for reading, rank-one injection for writing), covariance separation supports the geometric well-definedness in which rank-one updates operate, and the boundedness of rank-one editing is consistent with a finite addressable space. These three properties form a closed triangle of mutual dependence, so that none must be treated as independent of the others, and the structure is accounted for without invoking additional free parameters among them.
IN superposition-capacity-linear-in-dimension
In the Elhage et al. 2022 toy model, the maximum number of recoverable features is linear (not exponential) in the embedding dimension m, with the proportionality constant depending on sparsity density S: m = Ω(-n*(1-S) log(1-S)).
IN superposition-covariance-editability-triangle-v2
Superposition, covariance geometry, and rank-one editability form an interdependent framework in which each concept provides structural context for the others: over-complete superposition in the residual stream is the primary structural condition that makes covariance/whitening the canonical tool for feature isolation and targeted rank-one edits, covariance geometry serves as the operational coordinate space in which those rank-one updates are formulated, and the structural bounds on such edits—limited to single-fact local corrections rather than novel multi-hop associations—reflect the distributed, locally non-consolidated nature of the underlying superposed representation.
IN superposition-creates-interference-entries
Superposition introduces small non-zero interference entries (ε) in W^TW, changing it from (1, 0, 0, 0, ...) to (1, ε, −ε, ε, ...), and these ε terms constitute the direct mechanism of adversarial vulnerability.
IN superposition-dense-regime-pca-behavior
In the dense feature activation regime, the toy model learns an orthogonal basis representing only the top-k most important features, analogous to PCA, leaving remaining features unrepresented.
IN superposition-experiment-hyperparameter-sweep
The 2-feature/1-dimension superposition experiment used ReLU(W^T W x - b) architecture, varied feature-2 importance from 0.1 to 10, sparsity from 1.0 to 0.01, trained 10 models per point, and discarded the highest-loss run.
IN superposition-features-exceed-dimensionality
Superposition hypothesis posits that neural networks encode more features than the dimensionality of their activation space, whereas disentanglement seeks features equal to or fewer than the dimensionality
IN superposition-increases-vulnerability-3x
Adversarial vulnerability increases by more than 3x as superposition forms and closely tracks the number of features per dimension (reciprocal of feature dimensionality).
IN superposition-introduces-positive-dot-products
Superposition introduces positive dot products between unrelated features, making naive cosine-similarity probes (e.g., to 'deceptive behavior' embeddings) unreliable for detecting feature presence.
IN superposition-more-features-than-dimensions
Superposition enables a network to represent more features than it has activation dimensions by encoding features in non-orthogonal directions, effectively simulating a larger network within a fixed dimensionality budget.
IN superposition-requ-rel-relu-necessary
The Elhage et al. 2022 paper demonstrates that ReLU (or a similar nonlinear filter) is necessary to recover individual superposed features; without it, the model cannot separate overlapping feature signals in shared dimensions.
IN superposition-requires-feature-sparsity
With dense features, a model collapses to an orthogonal basis of top-k features (PCA-like); with sparse features, the model exploits rare co-occurrence to pack more than d features into d dimensions (superposition).
IN superposition-sharp-phase-transition
Superposition and polysemanticity exhibit a sharp phase transition (not a gradual continuum): there exists a regime boundary beyond which superposition is entirely absent for any given model architecture and sparsity level.
IN superposition-sparse-regime-overcapacity
In the sparse feature activation regime, the toy model represents all 5 features superposed in 2 dimensions, with ReLU resolving the resulting interference.
IN superposition-toy-model-architecture
The Elhage et al. 2022 toy model uses a 5 input features → 2-dimensional linear embedding → ReLU → output architecture trained with MSE loss and per-feature scalar importance multipliers.
IN superposition-unifies-data-and-representation-redundancy-v2
Over-complete superposition is a primary structural factor consistent with both the data-side redundancy (5× correlated corpora at Spearman 0.87–0.97 yielding only marginal accuracy gains, suggesting 'relevant document count' overcounts unique information) and the representation-side redundancy (SAE dead features ranging from ~2% at 1M to ~65% at 34M, alongside systematic under-reconstruction): in both cases the observed patterns are consistent with available capacity exceeding the unique information to be encoded, though the antecedents establish these as descriptive correlations rather than isolating over-completeness as the sole causal mechanism.
IN three-superposition-types
The Templeton 2024 paper identifies three distinct types of superposition that complicate mechanistic understanding: activation superposition (addressed by SAEs), attention superposition (features packed across attention heads), and weight superposition (interference in learned weights).
IN transformer-early-mlp-extremely-polysemantic
The first MLP layers in Transformer models are predicted by the superposition theory and confirmed empirically to be extremely polysemantic.
IN uniform-superposition-converges-digons
In the uniform-superposition setup (equal importance, equal sparsity), features converge into a small number of regular polytopes (e.g., digons) with fractional dimensionalities.
IN uniform-superposition-regular-polytopes
In uniform superposition (equal importance and sparsity), optimal feature directions organize into vertices of regular polytopes (e.g., pentagon for 5 features in 2D, tetrahedron for 4 in 3D).
IN vulnerability-persists-infinite-data-limit
Superposition-induced adversarial vulnerability is present in the infinite-data limit and is a property of the optimal sparse representation, not a finite-sample or training artifact.