gaussian-latent-assumption-drives-superposition
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-12.md
Created 2026-08-25T02:57:59+00:00
VAEs and GANs assume Gaussian latent distributions; since sparse latent variables are non-Gaussian but their superposition approaches Gaussian by the Central Limit Theorem, the architecture may force superposition of sparse variables.
Summary
VAEs and GANs are built on the assumption that their hidden representations look like a bell curve, but the actual useful information might live in sparse, mostly-zero signals. The concern is that these architectures may be forcing the network to juggle and blend many sparse pieces together just to make the output look Gaussian, which could distort what the model is actually learning about the data.