lora-applied-wq-wv-primary-experiments
IN premise — summaries/2026/08/24/hu-2021-lora-s0-abstract-chunk-1.md
Created 2026-08-24T17:10:54+00:00
In LoRA's primary experiments, low-rank adaptation is applied only to the Wq (query) and Wv (value) weight matrices in the self-attention module; MLP/feed-forward layer weights are frozen and not adapted.
Summary
LoRA's headline results come from a very narrow intervention: only the query and value projections inside the attention mechanism get any adaptation, while the model's feed-forward layers stay completely untouched. This means LoRA's reported performance gains reflect what you get by tweaking just one slice of the architecture, not the full set of trainable weights, which sets the baseline for any broader or deeper adaptation strategy.