training-cost-scaling-gpt2-palm

IN premise — entries/2026/06/21/wiki-Large_language_model.md

Created 2026-06-21T09:50:09+00:00

Training cost scaled dramatically: GPT-2 (1.5B params, 2019) cost approximately $50K while PaLM (540B params, 2022) cost approximately $8M

Summary

Building a state-of-the-art AI model went from a small-budget project to a nine-figure endeavor in just three years: a team in 2019 trained a 1.5-billion-parameter model for roughly fifty thousand dollars, but by 2022 the equivalent frontier push required about eight million dollars for a 540-billion-parameter model. This matters because it means the economic barrier to competing at the frontier has climbed by two orders of magnitude, concentrating the ability to train leading models in a shrinking set of well-funded organizations.

Dependents

These beliefs depend on this one:

IN training-costs-scaled-exponentially-across-generations — Training costs scaled by orders of magnitude — from ~$500 for BERT (4 TPUs, 4 days) through ~$50K for GPT-2 to millions for PaLM — while the per-parameter cost formula (6 FLOPs per parameter per token) remained consistent, meaning cost growth is driven by scale ambition, not inefficiency.