lora-rank1-suffices-gpt3-175b

IN premise — summaries/2026/08/24/hu-2021-lora-s7-u-nderstanding-the-low-rank-updates.md

Created 2026-08-24T17:10:55+00:00

Rank r=1 already performs competitively when adapting {Wq, Wv} on GPT-3 175B for tasks like WikiSQL and MultiNLI, indicating a very low intrinsic rank of the weight update

Summary

Fine-tuning GPT-3's 175 billion parameters for tasks like SQL generation or natural-language inference barely requires any new information at all — a single-number adjustment to the query and value weights already gets close to full fine-tuning performance. This implies the model already contains nearly all the task-relevant knowledge, and what's missing is only a tiny directional nudge rather than a substantial new capability.