gd-order-stable-averages-gradients
IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s4-icl-is-likely-not-equivalent-to-order-stable.md
Created 2026-08-25T02:58:32+00:00
Standard GD is order-stable because it computes the gradient as the average loss across a batch, so reordering samples does not change the resulting parameter update.
Summary
Because standard gradient descent averages the loss across the whole batch before computing the step, the resulting parameter update is identical no matter how the individual samples are sequenced. This means the algorithm's trajectory within a single step is fully determined by which examples are present, not their arrangement, so shuffling data inside a batch has no effect on the learning path.