sae-topk-sparsity-constraint
IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity-chunk-2.md
Created 2026-08-25T02:58:37+00:00
Top-k SAE variant enforces a hard sparsity constraint by allowing only the top k encoder features to fire per token, improving interpretability over L1-regularized vanilla SAEs.
Summary
Instead of softly penalizing a large number of active features, the top-k version draws a hard line: only the k strongest features get to contribute for each input token. This makes the resulting feature set easier to inspect and name, because you get a clean, fixed-size group of explanations rather than a fuzzy tail of weakly-activated ones that are hard to interpret.