gelu-approximates-multiplication-via-third-order-expansion
IN premise — summaries/2026/08/24/akyurek-2023-icl-algorithm-sR-references-chunk-1.md
Created 2026-08-24T17:10:49+00:00
The identity xy ≈ √(π/2)·(GeLU(x+y) − GeLU(x) − GeLU(y)) holds for small x, y with error O(x³+y³), allowing a transformer MLP to perform element-wise multiplication using three GeLU evaluations.
Summary
A transformer's MLP layer can compute the product of two small numbers using just three GeLU activation evaluations, with the error staying very small for modest input values. This gives a concrete reason why GeLU is useful beyond being a smooth alternative to ReLU: it lets the network perform element-wise multiplication natively inside its feedforward blocks, which is a foundational operation underlying how attention scores and other computations combine values.