original-warmup-linear-scale-2pct-training-steps
IN premise — summaries/2026-08-24/wiki-Transformer_deep_learning_architecture-chunk-1.md
Created 2026-08-24T17:11:24+00:00
The original 2017 Transformer paper recommended linearly scaling the learning rate from 0 to its maximum over the first approximately 2% of total training steps, then decaying.
Summary
The original 2017 Transformer paper specified a training recipe where the learning rate starts near zero, ramps up gently over roughly the first two percent of training steps, and then tapers down. This warmup schedule became a near-universal practice for training attention-based models, so it stands as a baseline assumption whenever results are compared, reproduced, or built upon.