lora-initialization-b-zero-a-gaussian-zero-update

IN premise — summaries/2026/08/24/hu-2021-lora-s0-abstract-chunk-1.md

Created 2026-08-24T17:10:54+00:00

LoRA initializes matrix B to zero and matrix A to random Gaussian N(0, σ²), ensuring ΔW = BA = 0 at training start so the model behaves identically to the pre-trained model initially; a scaling factor α/r is applied to stabilize training across different ranks.

Summary

LoRA is set up so that at the very start of fine-tuning, the adaptation layers contribute nothing to the model's output, meaning the model behaves exactly as the original pre-trained version did before any new training begins. A built-in scaling factor then keeps the learning stable no matter how large or small the chosen rank is, so practitioners can tune capacity without risking a destabilized start.