park-2024-llama3-hierarchy-orthogonality
IN premise — summaries/2026/08/24/park-2024-categorical-hierarchical-concepts-sR-references-chunk-2.md
Created 2026-08-25T02:58:26+00:00
In LLaMA-3's representation space, the vector (child − parent) is approximately orthogonal to the parent vector for WordNet taxonomic relations, with cosine similarity approaching zero.
Summary
In LLaMA-3's internal word representations, the extra features that make a specific concept (like "dog") differ from its general category (like "animal") are stored in a direction that is essentially independent of how the general category itself is encoded. This means the model treats "what it is to be a dog" and "what it is to be an animal" as separate, non-overlapping dimensions, which is a clean compositional structure that makes the hierarchy easier to reason about and manipulate geometrically.
Dependents
These beliefs depend on this one:
- OUT feature-hierarchy-empirical-validation — The hierarchical feature structure predicted geometrically by Park's orthogonality theorem (parent⊥child−parent) is empirically instantiated in SAE decoder-space neighborhoods: abstract/general features (transit infrastructure) subsume concrete/specific features (Golden Gate Bridge, Alcatraz), and this neighborhood hierarchy mirrors the WordNet synset hierarchy that Park validates across Gemma-2B and LLaMA-3-8B.