dai-2023-optimizer-training-schedule
IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-sA-appendix.md
Created 2026-08-24T17:10:54+00:00
Dai et al. 2023 from-scratch training uses Adam (β₁=0.9, β₂=0.98, ε=1e-6), polynomial decay LR schedule with max LR 3e-4, 500K total steps, 20K warm-up, 512K-token batch size, and gradient clip norm 2.0.
Summary
This records the exact training recipe — optimizer choice, learning rate curve, batch size, and total step count — that Dai et al. 2023 used to build their model from scratch. It matters because every downstream comparison or reproduction claim in the system that references this work is anchored to these specific settings, so changing any single value (like the warm-up length or clip norm) could invalidate results built on top of them.