lora-zero-inference-latency-merge

IN premise — summaries/2026/08/24/hu-2021-lora-s0-abstract-chunk-2.md

Created 2026-08-24T17:10:55+00:00

LoRA introduces zero additional inference latency because at deployment the low-rank update is merged into the base weight matrix (W = W₀ + BA computed once and stored as a single d×k matrix), making inference identical to a fully fine-tuned model.

Summary

Once LoRA's small adjustment is folded into the model's original weights as a one-time setup step, generating responses is exactly as fast as running a fully fine-tuned model, with no extra computation per request. This means you get fine-tuning quality without any user-facing speed penalty, so the efficiency trick used during training doesn't become a bottleneck at deployment.