gd-simulating-transformer-sparsity-requirements
IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s3-a-b-in-4-5.md
Created 2026-08-25T02:58:31+00:00
Theoretical GD-simulating Transformer constructions require sparsity ratios >99.99% in W_K and W_Q, and approximately 75% in W_V, whereas real LLMs such as LLaMA-7B and GPT-J exhibit far lower (denser) weight sparsity.
Summary
The mathematical results showing that a transformer can be made to mimic one step of gradient descent depend on the attention weight matrices being almost entirely zero, a condition that real models like LLaMA-7B or GPT-J do not come close to satisfying. This gap means those theoretical equivalences do not describe what deployed LLMs are actually doing, so any system reasoning that "LLMs perform gradient descent" on the strength of those papers is resting on an assumption that fails in practice.