lora-amplification-factor-21x-at-rank4

Summary

When LoRA is run at a very low rank (4) on GPT-3's 48th layer, the resulting weight changes are roughly 21 times larger than the underlying low-rank decomposition would predict, meaning the training process acts as a powerful magnifier on a small set of internal feature directions rather than a gentle nudge. This matters because it implies that low-rank fine-tuning is far more aggressive and targeted than its simplicity suggests, which has real consequences for how much a model's representations shift and how easily that shift generalizes beyond the training task.