gptj-gptneox-parallel-attn-mlp-swiglu

IN premise — summaries/2026/08/24/meng-2022-memit-s3-p-reliminaries-l-anguage-modeling-and-memory-editing-chunk-1.md

Created 2026-08-25T02:58:13+00:00

GPT-J and GPT-NeoX use parallel attention+MLP block architecture with SwiGLU as the MLP nonlinearity

Summary

GPT-J and GPT-NeoX both route each transformer layer's computation through two parallel branches — one for attention, one for the feed-forward network — rather than stacking them in sequence, and they use the gated SwiGLU activation inside that feed-forward branch. This matters because it places both models in a shared architectural family (similar to PaLM) that differs from the standard sequential transformer block used in models like LLaMA or GPT-2, which affects how they scale, how they are implemented, and how their parameters should be compared.