sae-loss-function-decoder-norm-penalty
IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity-chunk-3.md
Created 2026-08-25T02:58:37+00:00
The SAE loss includes an L1 penalty on feature activations weighted by decoder weight norm ‖W^{dec}_{·,i}‖₂ with coefficient λ=5, preventing the SAE from decoupling small activations from large weights.
Summary
This rule prevents the sparse autoencoder from cheating its own sparsity by assigning a feature a huge decoder weight while keeping its activation near zero, effectively hiding that feature in the background. In practice, it forces any feature that carries a large reconstruction responsibility to actually fire regularly, which keeps the resulting feature dictionary honest and interpretable rather than full of silent, oversized entries.