ffn-original-relu-modern-geglu-swiglu

Summary

The 2017 Transformer paper used a basic ReLU activation in its feed-forward layers, but every major model since, including GPT-3 and LLaMA, has swapped that for more sophisticated gating functions like GeGLU or SwiGLU. This matters because it means the gap between the original architecture and modern ones is not just a matter of scale; the core math of how each layer processes information has fundamentally changed, so results from the 2017 paper should not be extrapolated directly to state-of-the-art systems.