sae-architecture-relu-encoder-linear-decoder
IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity-chunk-3.md
Created 2026-08-25T02:58:37+00:00
The SAE encoder uses a ReLU nonlinearity (f_i(x) = ReLU(W^{enc}_{i,·} · x + b^{enc}_i)) while the decoder is purely linear (x̂ = b^{dec} + Σᵢ f_i(x) · W^{dec}_{·,i}).
Summary
The sparse features are activated by a threshold (ReLU) so that each one is either zero or a positive amount, and the reconstruction is just a straight weighted sum of whichever features fired. This means every active feature contributes only in a fixed, interpretable direction to the output, and no feature can "undo" another, which keeps the system's behavior easy to audit feature by feature.