training-costs-scaled-exponentially-across-generations
IN derived (depth 1)
Created 2026-06-21T09:57:37+00:00 · Reviewed 2026-06-21T14:41:08+00:00
Training costs scaled by orders of magnitude — from ~$500 for BERT (4 TPUs, 4 days) through ~$50K for GPT-2 to millions for PaLM — while the per-parameter cost formula (6 FLOPs per parameter per token) remained consistent, meaning cost growth is driven by scale ambition, not inefficiency.
Summary
The dramatic rise in training costs from hundreds of dollars to millions of dollars across model generations is not a symptom of growing inefficiency; the fundamental cost per unit of computation has stayed essentially flat. What's driving the expense is simply that each generation trains far more parameters on far more data, so budgeting and planning should assume cost scales directly with model ambition rather than expecting a technological breakthrough to make larger runs cheaper.
Justifications
SL — consistent per-parameter costs with exponentially growing scale ambitions
Antecedents (all must be IN):
- IN bert-training-cost-500-usd-4-tpus-4-days — BERT_BASE training cost approximately $500 USD on 4 Cloud TPUs (16 chips) over 4 days
- IN training-cost-scaling-gpt2-palm — Training cost scaled dramatically: GPT-2 (1.5B params, 2019) cost approximately $50K while PaLM (540B params, 2022) cost approximately $8M
- IN training-cost-6-flops-per-param-per-token — LLM training costs approximately 6 FLOPs per parameter per token, while inference costs 1-2 FLOPs per parameter per token.
Dependents
These beliefs depend on this one:
- IN economic-pressure-drives-capability-cost-decoupling — Exponential training cost scaling across model generations creates economic pressure that drives architectural innovations systematically decoupling capability from cost: MoE decouples parameter count from inference compute, knowledge distillation decouples model size from output quality, and compute routing selects granularity per task — all validated across the full scale spectrum from 110M to 2T parameters.
- IN training-pipeline-is-mature-engineering-discipline — The LLM training pipeline has matured into a fully specified engineering discipline with standardized stages (pretraining → SFT → alignment), complete mathematical specification of the alignment phase (Bradley-Terry loss, KL penalty, PPO clipping), and calibrated cost models (6 FLOPs/param/token) enabling predictable budgeting from BERT's $500 to PaLM's millions.