lora-matches-finetuning-dart-47-1-bleu
IN premise — summaries/2026/08/24/hu-2021-lora-sR-references.md
Created 2026-08-24T17:10:55+00:00
LoRA with 0.35M trainable parameters matches or exceeds full fine-tuning (354M parameters) on DART (47.1 vs. 46.2 BLEU, GPT-2 Medium) and WebNLG, demonstrating comparable generation quality at ~1000× fewer parameters
Summary
A tiny adapter with just 350,000 trainable parameters can produce text generation quality that matches or slightly beats retraining all 354 million parameters of the underlying model, meaning you get nearly the same output while updating roughly one thousandth of the weights. This makes it far cheaper and more practical to customize large language models for specific tasks without the compute cost of full fine-tuning.