sae-expansion-ratio-10-to-200x
IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity.md
Created 2026-08-25T02:58:39+00:00
Sparse Autoencoders decompose dense d-dimensional activations into a typically 10× to 200× larger set of mostly-sparse, near-orthogonal latent features.
Summary
Instead of working with a single compressed signal, the system unpacks neural network activity into roughly 10 to 200 times as many mostly-silent, independent features. This means every downstream calculation operates on a much larger but far more separable representation, trading raw density for clarity at the cost of significant expansion in dimensionality.
Dependents
These beliefs depend on this one:
- OUT sae-granularity-as-superposition-resolution — SAE's empirical granularity scaling (broad categorical features at small run sizes → specific entity features at large run sizes) is the operational resolution of superposition: the 10–200× expansion ratio creates discrete "zoom levels" at which the same underlying covariance geometry manifests as features of different semantic specificity.
- OUT superposition-as-compositional-basis — Superposition is the fundamental compositional mechanism in LLMs: the 10–200× over-complete expansion (SAE), the key-value memory structure (ROME's W_fc/W_proj), and the direct-sum space decomposition (Park's polytope+orthogonality) are three independent geometric consequences of the same over-completeness.