Sparse Autoencoders

83 beliefs (83 IN, 0 OUT)

Sparse Autoencoders (SAEs) are a central tool in mechanistic interpretability, decomposing a model's dense d-dimensional residual stream activations into a typically 10x to 200x larger set of mostly-sparse, near-orthogonal latent features (sae-expansion-ratio-10-to-200x, sae-expansion-ratio-definition). The standard architecture uses a ReLU nonlinear encoder paired with a purely linear decoder (sae-architecture-relu-encoder-linear-decoder), trained to minimize reconstruction error plus an L1 sparsity penalty weighted by decoder weight norm (sae-training-objective-mse-plus-l1, sae-loss-function-decoder-norm-penalty). Sparsity is enforced either through this soft L1 penalty or, in Anthropic's scaling monosemanticity work, through a harder top-k constraint that retains only the top-K activations per token (sae-topk-architecture-used, eh2022-sae-top-k-sparsity-requirement). The SAE is applied specifically to middle-layer residual stream activations to reduce dimensionality versus MLP outputs and mitigate cross-layer superposition (sae-middle-layer-residual-stream-target). A key algorithmic improvement, proposed by Bricken, multiplies the sparsity penalty by the decoder norm and removes the unit-norm constraint on decoder vectors (sae-algorithmic-improvement-bricken). Losses across sizes (1M, 4M, 34M features) follow an approximate power law in compute, and all three achieve at least 65% variance reconstruction with fewer than 300 active features per token (sae-scaling-law-power-law-compute, sae-reconstruction-variance-and-sparsity). The Bricken et al. paper explicitly rejects the "sparse architecture" training approach as insufficient to eliminate polysemanticity (saey-paper-rejects-sparse-architecture-approach), and known pathologies include the shrinkage problem (sae-shrinkage-problem) and feature splitting, where a single coarse feature decomposes into finer sub-features as width increases (sae-jcr-prevents-feature-splitting, saey-feature-splitting-base64-three-subfeatures, feature-splitting-sae-width).

The interpretability payoff is substantial: autointerpretability analysis showed SAE-identified features are substantially more interpretable than individual transformer neurons (sae-features-more-interpretable-than-neurons), and feature granularity scales with SAE size, with larger runs yielding specific entities like "Golden Gate Bridge" while smaller runs yield broad categories like "Monuments and tourist attractions" (sae-feature-granularity-scales-with-run-size). Feature neighborhoods in decoder space, defined by cosine similarity, map onto semantic relatedness (sae-cosine-similarity-defines-neighborhoods) and exhibit a general-to-specific gradient: the Golden Gate Bridge feature is flanked by San Francisco locations, California landmarks, and further-out tourist attractions (sae-golden-gate-bridge-neighborhood-structure), while a transit-infrastructure feature activates on trains, ferries, and even wormholes, encoding a higher-level transport abstraction (sae-transit-feature-abstract-generalization, feature-hierarchy-empirical-validation-v2). This neighborhood structure has been interpreted as an operational navigation of the categorical polytope geometry identified by Park et al., with cross-model feature similarity suggesting the geometric structure reflects shared data properties rather than a model-specific artifact (sae-neighborhood-as-polytope-navigation-v2, feature-neighborhood-as-geometric-theorem-instantiation-v2). More broadly, cross-model geometric convergence in SAE features and Park orthogonality is strongly predicted by the superposition framework's over-complete representation requirement (m > d) in a shared substrate, with empirical validation on Gemma-2B and LLaMA-3-8B arguing against a pure contingent coincidence, though the evidence establishes a well-motivated expectation rather than a proven logical necessity (convergence-is-necessary-not-contingent-v2). Evaluation results reinforce a distinction between output-space task-specificity and internal representational convergence: no single embedding model dominates all MTEB task categories, yet SAE-extracted features tend to be more similar across architectures than to their own model's neurons (task-specificity-vs-feature-universality-v2). SAEs trained purely on text also generalize zero-shot to image activations, indicating shared cross-modal representations (text-sae-generalizes-to-image-activations, sae-cross-modal-text-trained-image-activation), and days of the week and months appear as circular structures in SAE-reconstructed activations across GPT-2, Mistral 7B, and Llama 3 8B (three-models-circular-pca-structure). The Engels et al. ICLR 2025 paper extends this line by adding a clustering step over dictionary elements to recover multi-dimensional irreducible subspaces, with toy experiments showing that a 2-feature SAE must activate both features on every point of a circular structure while a 10-feature SAE spreads around the circle with no canonical direction, and a lower bound on the number of pairwise orthogonal subspaces (engels-2025-sae-clustering-extension, sae-circular-feature-ambiguity, engels-2024-theorem1-lower-bound, engels-2024-sae-m2-circle-both-features-fire, engels-2024-sae-m10-sparse-subset).

Causal analysis of SAE features relies on ablation by clamping a feature's activation to zero at a specific token position (sae-ablation-clamping-zero-method) and logit-difference attribution computed as the dot product of the gradient with the feature vector times activation (sae-attribution-linear-formula), with the two methods showing 0.8 correlation in worked examples (sae-attribution-ablation-0-8-correlation). Crucially, ranking features by raw activation magnitude does not reliably identify causally important features (sae-raw-activation-not-causal-importance), and steering experiments demonstrate that safety-relevant SAE features causally influence generated output rather than merely correlating with unsafe content (safety-feature-steering-causal-not-correlational). SAE-based features outperformed few-shot probe steering vectors in 5 of 7 test cases (sae-vs-fewshot-steering-comparison). Anthropic's feature search employs four methods—single-prompt activation, multi-prompt filtering, cosine-similarity nearest-neighbor search, and logit-difference attribution (sae-feature-search-four-methods)—and uses Claude 3 Opus as an interpretability judge distinct from the analyzed model to reduce self-reference bias (sae-claude-opus-as-interpretability-judge). Specific training configurations vary across models: Mistral 7B SAEs use 65,536 dictionary elements at 16x expansion, an L_{1/2} half-norm penalty, AdamW with dead-feature resampling leaving roughly 1000 dead features out of 65,536, and are trained on layers 8, 16, and 24 (mistral-sae-65536-dict-elements, mistral-sae-l1-2-sparse-penalty, mistral-sae-optimizer-dead-features, mistral-sae-layers-8-16-24); the Bricken et al. A/1 run produces 4,096 features of which 168 are dead and 292 ultralow-density (saey-a1-autoencoder-4096-features-dead-ultralow); and GPT-2-small layer 7 has approximately 25,000 features (gpt2-small-sae-25k-features-layer7).

All beliefs in this topic are currently IN (active); none have been retracted. The knowledge base distinguishes between foundational premises—architectural specifications, training hyperparameters, and observed empirical facts such as dead-feature proportions and reconstruction rates—and higher-level derived claims about geometric structure, cross-model convergence, and the interpretability superiority of SAE features over neurons. The derived geometric and convergence claims (convergence-is-necessary-not-contingent-v2, sae-neighborhood-as-directsum-navigation-v2, feature-neighborhood-as-geometric-theorem-instantiation-v2) are explicitly framed as well-motivated expectations or parallel evidence rather than proven theorems, and the neighborhood-as-polytope-navigation claim notes that the antecedents confirm categorical-polytope navigation and imply structural decomposition separately without establishing that the same traversal navigates both subspaces simultaneously (sae-neighborhood-as-directsum-navigation-v2). The Engels et al. results, validated across GPT-2, Mistral 7B, and Llama 3 8B with references to GPT-4 and Claude 3 indicating intended generality (engels-2025-models-validated), add a formal lower-bound theorem and a clustering-based extension that moves the discussion from "SAE features look interpretable" toward "SAE features recover irreducible multi-dimensional subspaces whose combinatorial geometry is bounded from below" (engels-2024-theorem1-lower-bound, engels-2025-sae-clustering-extension). Together, these beliefs paint a picture in which SAEs serve as a bridge between low-level linear features and high-level semantic geometry, with their value resting on the conjunction of sparsity-driven interpretability, causal (not merely correlational) feature effects, and a shared geometric structure that transcends individual architectures.