swiglu-rope-rmsnorm-not-meta-inventions
IN premise — entries/2026/06/21/wiki-LLaMA-chunk-3.md
Created 2026-06-21T09:50:09+00:00
SwiGLU (Shazeer 2020), RoPE (Su et al. 2021), and RMSNorm (Zhang & Sennrich 2019) used in Llama are adopted from prior research, not Meta inventions
Summary
Llama's three core architectural pieces — its gating function, its positional encoding scheme, and its normalization layer — were each taken from other researchers' published papers rather than invented by Meta. This matters because it frames Meta's actual contribution as one of integration, recipe design, and large-scale training, not as the source of the underlying algorithms.
Dependents
These beliefs depend on this one:
- IN llama-optimized-architecture-for-data-scaling-strategy — Llama's architecture adopts proven refinements from prior independent research (SwiGLU from Shazeer 2020, RoPE from Su et al. 2021, RMSNorm from Zhang & Sennrich 2019) rather than introducing novel components, while its training data volume scaled dramatically from 1.4T to 40T tokens across four generations — a pattern consistent with concentrating effort on data scaling over architectural novelty.