mistral-sae-activation-normalization

Summary

When running a superposed autoencoder on Mistral 7B, the input activations are rescaled so they all sit at the same distance from the origin, which keeps individual feature directions on equal footing and prevents any one direction from dominating purely because of its raw magnitude. The lack of a learned offset before encoding means the features must capture the signal from the normalized geometry alone, simplifying the architecture but tying feature quality more tightly to the normalization choice.