gpt3-lora-parameter-budget-4-7m-to-37-7m

IN premise — summaries/2026/08/24/hu-2021-lora-sR-references-chunk-1.md

Created 2026-08-24T17:10:55+00:00

GPT-3 175B LoRA uses 4.7M trainable parameters at r=1 and 37.7M at r=8, representing up to 10,000× fewer trainable parameters than full fine-tuning of the 175B model

Summary

Fine-tuning the full 175-billion-parameter GPT-3 model would require updating every single weight, but LoRA shows you can get a useful adaptation by only training 4.7 to 37.7 million of them — roughly ten thousand times fewer. In practice, this means you can customize a model the size of GPT-3 on a fraction of the hardware and cost that full fine-tuning would demand.