lora-delta-w-amplifies-underemphasized-directions

IN premise — summaries/2026/08/24/hu-2021-lora-s7-u-nderstanding-the-low-rank-updates.md

Created 2026-08-24T17:10:55+00:00

ΔW correlates more strongly with pretrained W than a random matrix, but specifically amplifies directions in W that are NOT in W's top singular vectors, activating underemphasized features relevant to the downstream task

Summary

LoRA updates are not random tweaks to the model weights; they specifically reach for subtle, less-dominant capabilities that are already latent in the pretrained model and bring them forward for the target task. In practice, this means fine-tuning via LoRA works more like selectively activating dormant strengths than like learning something brand new, which explains why it can be so effective with so few parameters.