park2025-hierarchy-orthogonality-subspaces

IN premise — summaries/2026/08/24/park-2024-categorical-hierarchical-concepts-s1-we-show-how-to-move-from-representations-of-binary-concepts-.md

Created 2026-08-25T02:58:24+00:00

Hierarchically related concepts (e.g., animal ⊃ mammal ⊃ dog) occupy orthogonal subspaces in LLM representation space under a suitable inner product.

Summary

Even though concepts like "dog" and "mammal" are linked by an "is-a" hierarchy, the model stores each level in its own independent direction in internal space rather than nesting one inside the other. This matters because it means the model treats each rung of a category as a separable feature, so probing or perturbing one concept does not automatically drag along its parents or children, which has direct consequences for how the model reasons about categories and how it might be steered or attacked.