disentanglement-vs-dictionary-learning

Summary

This observation highlights a key fork in the road for how we think about reading individual concepts out of a model's weights: disentangling methods assume each concept gets its own tidy, separate direction (fewer concepts than there are dimensions), while dictionary-learning methods assume concepts are packed on top of each other and shared across directions (more concepts than dimensions, like superposition). The choice between those two pictures determines whether you can expect to isolate one feature at a time or whether you must reconstruct it from a tangled mix of overlapping patterns.