sae-expansion-ratio-definition
IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity-chunk-1.md
Created 2026-08-25T02:58:36+00:00
The SAE expansion ratio is defined as SAE latent dimension divided by model hidden dimension; a 32× expansion on a 4096-dimensional model yields a 131,072-dimensional sparse code.
Summary
This sets the vocabulary: the "expansion ratio" is simply how many more sparse features the autoencoder uses to describe the model's state compared to the model's own internal width. It matters because every downstream result — sparsity counts, feature interpretability, pruning decisions — is scaled relative to this number, so getting the definition wrong would silently distort the entire analysis pipeline.