sae-scaling-law-power-law-compute
IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity-chunk-3.md
Created 2026-08-25T02:58:37+00:00
SAE loss decreases approximately as a power law with respect to compute (features × training steps), and training runs for exactly one epoch so steps map linearly to data volume.
Summary
SAE training follows a predictable scaling curve: the loss goes down in a smooth, power-law way as you throw more compute at it (more features or more steps), and because each training run sees the data exactly once, the number of steps is just a direct proxy for how much data has been processed. This matters because it lets you forecast performance and trade off feature count against dataset size with confidence, rather than treating each architecture choice as a guess.
Dependents
These beliefs depend on this one:
- OUT sae-shrinkage-as-finite-resolution-limit — The SAE shrinkage problem (under-reconstruction with finite expansion ratios) is the operational signature of finite resolution in the superposition→whitening framework: any finite dictionary size leaves irreducible reconstruction loss because the superposed structure is fundamentally over-complete, and the power-law decrease in loss with compute is the scaling signature of approaching (but never reaching) full resolution.