mistral-layer-5-circular-patch-equivalence
IN premise — summaries/2026/08/24/engels-2024-not-all-features-linear-s8-we-then-evaluate-interventions-using-this.md
Created 2026-08-25T02:58:02+00:00
Patching the 2D circular subspace in Mistral 7B layer 5 achieves nearly the same causal effect as replacing the entire layer activation and outperforms patching only the top-5 PCA components.
Summary
You can reproduce almost the entire causal impact of a full layer in Mistral 7B by touching only a tiny two-dimensional circular region of that layer's activations, which is far more effective than the more intuitive approach of tweaking the five most important linear directions. This means the layer's real work is compressed into a very small geometric structure, so efficient interventions and interpretability tools can target that sliver instead of the whole high-dimensional space.