attention-is-mathematical-dual-of-one-gradient-descent-step

IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s0-abstract.md

Created 2026-08-24T17:10:53+00:00

Dai et al. (ACL 2023) establish that the computation inside a Transformer attention module is the dual form of one step of gradient descent, making ICL (forward-pass meta-gradients) structurally identical to fine-tuning (backward-pass standard gradients).

Summary

Dai et al. show that the computation inside a single Transformer attention layer is the mathematical mirror of one gradient-descent update, meaning in-context learning and fine-tuning are two views of the same optimization process — one computed in the forward pass, the other in the backward pass. This unifies two seemingly different ways a model learns into one framework and explains, at a structural level, why a prompt can steer behavior the way weight updates do.