von-oswald-2023-dispute-phase-shift
IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s6-related-work.md
Created 2026-08-25T02:58:33+00:00
von Oswald et al. (2023) disputed Akyürek et al.'s phase-shift claim, arguing that larger Transformers still implement gradient descent but with data transformations.
Summary
A 2023 paper by von Oswald and colleagues pushed back on the idea that Transformers undergo a dramatic change in how they compute as they get bigger, arguing instead that the core mechanism stays the same throughout — they are still performing gradient descent, just on slightly reorganized inputs. This matters because it reframes the debate from "Transformers switch strategies" to "Transformers tweak their inputs," which has different implications for how we understand their learning and generalization.