lora-outperforms-prefix-tuning-low-data-mnli100

IN premise — summaries/2026/08/24/hu-2021-lora-sR-references-chunk-2.md

Created 2026-08-24T17:10:55+00:00

On MNLI-100 with GPT-3, LoRA achieves 63.8% accuracy vs. PrefixEmbed's 37.6% (near 33.3% random chance baseline) and Full Fine-Tune's 60.2%, establishing LoRA's superior sample efficiency in the extreme low-data regime

Summary

When you only have 100 training examples, LoRA adapts GPT-3 far better than prefix tuning, which barely beats random guessing, and even edges out full fine-tuning. This matters because it shows LoRA is the most sample-efficient way to specialize a large model when data is scarce, and it does so without the heavy computational cost of retraining every parameter.