lora-replaces-weight-updates-with-lowrank-matrix-additions
IN premise — summaries/2026/08/24/wiki-Fine-tuning_deep_learning.md
Created 2026-08-24T17:11:10+00:00
LoRA (Low-Rank Adaptation) adds a low-rank decomposition (B ∈ ℝ^{d×r}, A ∈ ℝ^{r×d}, r ≪ d) to selected weight matrices (ΔW = BA), reducing trainable parameters from billions to a few million.
Summary
Instead of retraining billions of parameters to adapt a large model, LoRA tacks on a thin, low-dimensional correction to just a handful of weight matrices, cutting the number of learnable parameters down to a few million. This makes it possible to fine-tune very large models on modest hardware without having to store and update the full set of weights, which is what keeps the approach practical for real-world deployment.