superposition-toy-model-architecture
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-2.md
Created 2026-08-25T02:57:58+00:00
The Elhage et al. 2022 toy model uses a 5 input features → 2-dimensional linear embedding → ReLU → output architecture trained with MSE loss and per-feature scalar importance multipliers.
Summary
This nails down the exact recipe of the Elhage et al. 2022 toy model: five inputs get projected into a two-dimensional space, passed through a single ReLU gate, and then read out, with each input feature carrying its own tunable weight so you can dial importance up or down. It matters because every later claim in the system about what that model demonstrates, how features interact, or where its conclusions hold or break, has to be judged against this specific and quite minimal setup.