mistral-sae-65536-dict-elements
IN premise — summaries/2026/08/24/engels-2024-not-all-features-linear-sR-references-chunk-2.md
Created 2026-08-25T02:58:02+00:00
Mistral 7B SAEs use a 16× expansion factor over the model dimension, yielding 65,536 dictionary elements per layer.
Summary
When we break down what Mistral 7B is doing internally at each layer, the interpretability tool expands the model's 4,096-dimensional activity into 65,536 distinct, individually inspectable features. This sets the resolution for any analysis we do on the model's representations, meaning we can attribute behavior to specific features rather than seeing only a compressed, entangled signal.