swiGLU-llama-palm-gelu-gpt1-bert

Summary

Across the major transformer architectures, teams have settled on different feed-forward activation functions: Llama and PaLM chose SwiGLU, while GPT-1 and BERT went with GELU, and all of these are upgrades over the ReLU that appeared in the original 2017 Transformer. This matters because the activation choice is a low-level but real design decision that affects how each model processes information, so anyone comparing, fine-tuning, or reimplementing these architectures needs to know exactly which one is in play.