transformer-mlp-privileged-basis-correspondence

IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-9.md

Created 2026-08-25T02:58:01+00:00

Transformer MLP layers and convolutional network neurons correspond to the privileged-basis model, while word embeddings and the transformer residual stream correspond to the no-privileged-basis model.

Summary

Different parts of a transformer have fundamentally different geometry in how they store information. The feed-forward layers and CNN neurons come with a built-in preferred coordinate system that shapes what they can represent, while the word-embedding space and the main information highway (residual stream) have no such fixed preference and must learn their structure from data. This matters because interpretability techniques that work on one part of the network may be the wrong tool for another.