sae-cosine-similarity-defines-neighborhoods
IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity-chunk-5.md
Created 2026-08-25T02:58:38+00:00
Feature neighborhoods are defined by cosine similarity of feature vectors in SAE decoder space, and distance in decoder space approximately maps onto semantic relatedness in concept space.
Summary
In the SAE decoder space, features that sit close to each other geometrically are treating nearby concepts as related in meaning, so the system can use that spatial layout to build a map of which ideas cluster together. This matters because it gives the system a principled, measurable way to say "these concepts go together" rather than relying on arbitrary or hand-picked groupings.