park-2025-colinearity-binary-features

IN premise — summaries/2026/08/24/park-2024-categorical-hierarchical-concepts-s2-using-this-result-we-show-that-semantic-hierarchy-between-co-chunk-2.md

Created 2026-08-25T02:58:25+00:00

Binary feature vectors for hierarchical concepts (e.g., 'animal,' 'mammal,' 'bird') are strongly co-linear in the standard representation space, making them difficult to disentangle via sparse autoencoder approaches.

Summary

When related concepts in a hierarchy, like "animal," "mammal," and "bird," are encoded as simple on/off feature lists, their representations all point in nearly the same mathematical direction. This means standard tools that try to find clean, separate features, such as sparse autoencoders, will struggle to pull the levels apart, so the system cannot easily distinguish a parent category from its children.