training-cost-6-flops-per-param-per-token

IN premise — entries/2026/06/21/wiki-Large_language_model-chunk-2.md

Created 2026-06-21T09:50:09+00:00

LLM training costs approximately 6 FLOPs per parameter per token, while inference costs 1-2 FLOPs per parameter per token.

Summary

Training a large language model is roughly three to six times more computationally expensive per token than simply using it, because every parameter must be updated during training whereas only a forward pass is needed at inference. This cost ratio is the baseline that justifies doing training once and paying for many cheap inference calls, and it anchors any decision about whether a proposed change is worth the compute it would require.

Dependents

These beliefs depend on this one:

IN llm-resource-consumption-quantifiable-across-three-dimensions — LLM resource consumption is precisely quantifiable across three independent dimensions — memory (2 bytes per float16 parameter), compute (6 FLOPs per parameter per token for training, 1-2 for inference), and energy (0.002-0.05 Wh per prompt depending on task type) — enabling principled cost modeling for deployment decisions.
IN training-costs-scaled-exponentially-across-generations — Training costs scaled by orders of magnitude — from ~$500 for BERT (4 TPUs, 4 days) through ~$50K for GPT-2 to millions for PaLM — while the per-parameter cost formula (6 FLOPs per parameter per token) remained consistent, meaning cost growth is driven by scale ambition, not inefficiency.