lora-subspace-overlap-r8-r64-noise-evidence
IN premise — summaries/2026/08/24/hu-2021-lora-sR-references-chunk-2.md
Created 2026-08-24T17:10:55+00:00
The top singular-vector direction of LoRA's r=8 update overlaps with the r=64 update at normalized similarity > 0.5, while higher-index directions do not, indicating that extra rank beyond the top direction captures training noise rather than signal
Summary
Most of the useful change a LoRA adapter learns fits into a single dominant direction, and the extra dimensions beyond that are largely just memorizing training-specific noise rather than capturing new signal. This means bumping the rank from 8 to 64 does not meaningfully expand what the adapter can learn, so much of the added capacity is wasted on fitting quirks of the training set.