superposition-sparse-regime-overcapacity

Summary

This is an observed fact about a simplified model: even when five distinct features are squeezed into just two dimensions so they overlap and interfere with each other, the ReLU activation function is enough to untangle them and keep the representations workable. It matters because it gives a concrete, testable example of how superposition can survive in a very tight, sparse setting without needing extra machinery, supporting the broader claim that sparse activation naturally limits interference.