original-transformer-relu-gpt-bert-gelu

IN premise — summaries/2026/08/24/wiki-Transformer_deep_learning_architecture-chunk-4-chunk-1.md

Created 2026-08-24T17:11:25+00:00

The original 2017 transformer used ReLU as its feedforward activation function; GPT-1 and BERT use GELU; Llama and PaLM use SwiGLU (a gated linear unit variant)

Summary

Activation functions have evolved across the transformer lineage, from ReLU in the original 2017 paper to GELU in GPT-1 and BERT, to the gated SwiGLU variant in modern large models like Llama and PaLM. This matters because the activation choice is a concrete architectural detail that shapes how each model processes intermediate representations, and it is a fixed reference point that any comparison, reproduction, or debugging of these systems depends on getting right.