lora-initialization-kaiming-uniform-zero
IN premise — summaries/2026/08/24/hu-2021-lora-sR-references-chunk-1.md
Created 2026-08-24T17:10:55+00:00
LoRA initializes matrix A with Kaiming uniform and matrix B with zeros, so ΔW = 0 at initialization and the model behaves identically to the frozen base model before training begins
Summary
LoRA is set up so that the added adaptation layer produces exactly zero change at the start, meaning the model's predictions are identical to the unmodified base model before any training step. This guarantees that introducing LoRA never accidentally perturbs a working model; the adapter only begins to shift behavior once training starts pushing its second matrix away from zero.