dai-2023-350m-model-architecture-specs
IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-sA-appendix.md
Created 2026-08-24T17:10:54+00:00
The Dai et al. 2023 from-scratch 350M-parameter model uses 1024 hidden dimension, 4096 FFN inner dimension, 16 attention heads, 24 layers, and 1024 maximum sequence length.
Summary
These architectural numbers define exactly how much a small 350M-parameter model can "think" per token, how many steps of processing it takes, and roughly 700 words of context it can handle at once. They set the baseline for what this class of model can and cannot do, and they're the reference point any comparison against larger or differently-shaped models has to start from.