akyurek-2022-phase-shift-gd-to-bayes

Summary

This suggests that the way Transformers learn changes qualitatively with scale: small models behave like a crude gradient-descent learner, iteratively nudging weights to reduce error, while larger models appear to approximate the mathematically optimal answer more directly. The practical implication is that "learning" in a large Transformer may not be the same process as in a small one, just bigger, so intuitions drawn from small-model training don't straightforwardly transfer to the frontier-scale systems we actually deploy.