lora-generalization-of-full-finetuning
IN premise — summaries/2026/08/24/hu-2021-lora-s0-abstract.md
Created 2026-08-24T17:10:54+00:00
LoRA is a strict generalization of full fine-tuning: setting rank r equal to the full matrix rank and applying LoRA to all weight matrices recovers full fine-tuning expressiveness, whereas adapter layers converge to an MLP as parameters increase.
Summary
LoRA is architecturally broader than full fine-tuning rather than a simplified version of it: at maximum rank applied to every weight matrix, LoRA becomes full fine-tuning, meaning it hits no expressiveness ceiling below the full method. This also sets it apart from adapter layers, which are structurally different added modules that simply reduce to a standard MLP as their parameters grow.