moe-reduces-superposition-incentive
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-11.md
Created 2026-08-25T02:57:59+00:00
Mixture-of-Experts (MoE) architectures decouple neuron count from FLOPs by activating only a fraction of neurons per example, potentially eliminating the efficiency gain that makes superposition attractive and thereby reducing superposition.
Summary
Mixture-of-Experts networks let a model have a very large number of neurons without paying the full compute cost, because only a small slice is active at any time. This undercuts the main efficiency reason to cram multiple features into a single neuron, so the pressure that drives superposition weakens or disappears.