sae-reconstruction-variance-and-sparsity
IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity-chunk-3.md
Created 2026-08-25T02:58:37+00:00
All three SAEs (1M, 4M, 34M features) achieve ≥65% variance reconstruction with fewer than 300 active features per token.
Summary
Across all three SAE sizes tested, the sparse encoders can explain at least 65% of the variation in model activations while keeping most of their features switched off for any given token. This matters because it shows the features that do light up are genuinely carrying signal rather than being a diffuse spread of noise, which gives downstream interpretability work a solid empirical foundation to rely on.