sgd-implies-on-layers-and-on-plus-d-hidden-space

Summary

Simulating a full gradient-descent step as a transformer computation has a hard scaling cost: you cannot collapse the per-example work, because each additional example in the batch forces one more sequential layer, and the internal memory must grow in proportion to both the batch size and the model dimension. This sets a concrete lower bound on how deep and wide a transformer must be if it is to "do the math" of optimization on its own rather than relying on an external optimizer.