sae-feature-granularity-scales-with-run-size
IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity-chunk-4.md
Created 2026-08-25T02:58:38+00:00
Larger SAE runs yield more specific features (e.g., 34M run: 'Golden Gate Bridge') while smaller runs yield broader categories (e.g., 1M run: 'Monuments and tourist attractions').
Summary
The size of a sparse autoencoder run determines how sharply you can describe what the model is tracking: a large run surfaces narrow, named concepts, while a small run only gives you general categories. Practically, this means the resolution of any interpretability claim you draw depends on the compute budget behind it, so conclusions from a small run should not be treated as the finest-grained explanation available.
Dependents
These beliefs depend on this one:
- OUT sae-granularity-as-superposition-resolution — SAE's empirical granularity scaling (broad categorical features at small run sizes → specific entity features at large run sizes) is the operational resolution of superposition: the 10–200× expansion ratio creates discrete "zoom levels" at which the same underlying covariance geometry manifests as features of different semantic specificity.