gelu-multiplication-identity-enables-mlp-arithmetic

IN premise — summaries/2026/08/24/akyurek-2023-icl-algorithm-sR-references.md

Created 2026-08-24T17:10:49+00:00

The identity √(π/2)·(GeLU(x+y) − GeLU(x) − GeLU(y)) = xy + O(x³+y³) enables elementwise multiplication inside a transformer's MLP, forming the key mechanism for performing nonlinear arithmetic within the model.

Summary

A particular combination of GeLU activation outputs can closely approximate the product of two inputs, meaning a standard transformer MLP layer can compute multiplication without any dedicated arithmetic hardware. This reveals that the nonlinear arithmetic capability of transformers is baked into the shape of the activation function itself, rather than being a separate operation layered on top.