three-models-circular-pca-structure
IN premise — summaries/2026/08/24/engels-2024-not-all-features-linear-s4-s-parse-autoencoders-find-multi-d-imensional-features.md
Created 2026-08-25T02:58:01+00:00
In GPT-2, Mistral 7B, and Llama 3 8B, days of the week and months of the year appear as circular (conical) structures in SAE-reconstructed activations, with the first PCA dimension encoding radius ('intensity') and the circle living in PCA dimensions 2–3.
Summary
Across three very different large language models, cyclical concepts like days of the week and months of the year get encoded as a little loop in the model's internal activation space, with how far out on the loop you are (the radius) tracking how strongly the concept is being used. This matters because it shows these models share a geometric trick for representing "wraps-around" information, meaning that any tool or probe that treats those dimensions as a simple line will miss the periodic structure the model actually relies on.