lora-gpt3-training-config-adamw-128batch-2epoch
IN premise — summaries/2026/08/24/hu-2021-lora-sR-references.md
Created 2026-08-24T17:10:55+00:00
GPT-3 LoRA training uses AdamW optimizer, batch size 128, 2 epochs, weight decay 0.1, 250,000-token warmup, linear LR schedule, and learning rates of 1.6E-3 (WikiSQL) or 2.00E-4 (MNLI)
Summary
These are the exact hyperparameters locked in for the GPT-3 LoRA fine-tuning runs, meaning any performance claims, comparisons, or reproducibility checks in the system are only valid under this specific recipe. If someone swaps the optimizer, changes the learning rate, or extends training beyond two epochs, the results no longer directly support or contradict the conclusions drawn from this configuration.