lora-gpt3-175b-resource-reduction

IN premise — summaries/2026/08/24/hu-2021-lora-s0-abstract-chunk-1.md

Created 2026-08-24T17:10:54+00:00

LoRA on GPT-3 175B with r=4 adapting only Wq and Wv reduces checkpoint size from ~350 GB to ~35 MB (~10,000×) and reduces Adam VRAM from ~1.2 TB to ~350 GB, while improving training throughput from 32.5 to 43.1 tokens/s per V100 GPU (~25% speedup).

Summary

Fine-tuning a 175-billion-parameter model normally requires hundreds of gigabytes of storage for checkpoints and over a terabyte of GPU memory for the optimizer, making it impractical outside large datacenters. By freezing most of the network and training only small rank-4 adjustment matrices on the attention weights, this approach makes that same fine-tuning job feasible on a single GPU while actually training about a quarter faster, turning a multi-node infrastructure problem into a workstation-scale one.