transformer-gd-ood-parity

IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-sR-references.md

Created 2026-08-24T17:11:05+00:00

Trained Transformers and GD/GD++ exhibit nearly identical loss curves when tested on rescaled inputs (α·x, x∼U(−α,α)), rescaled teacher weights (αW, W∼N(0,I)), and entirely novel distributions (Normal, Exponential, Laplace), with the match degrading for deep non-recurrent Transformers far from the training range.

Summary

Transformers trained with GD/GD++ behave almost identically to the GD/GD++ reference when fed stretched inputs, stretched weights, or completely unfamiliar data distributions, meaning the two can be treated as interchangeable for out-of-distribution robustness testing. However, this equivalence breaks down for deep, non-recurrent architectures when inputs are pushed far outside the training range, so those models need separate scrutiny under extreme conditions.