lora-plus-prefixlayer-degrades-performance
IN premise — summaries/2026/08/24/hu-2021-lora-sR-references-chunk-2.md
Created 2026-08-24T17:10:55+00:00
Combining LoRA with prefix-layer tuning (PL) performs worse than LoRA alone on GPT-3, attributed to PL's sensitivity to learning rate destabilizing joint optimization, while LoRA + prefix-embedding tuning is somewhat additive
Summary
Mixing LoRA with full prefix-layer tuning on GPT-3 actually hurts performance compared to just using LoRA by itself, because the prefix-layer part is too sensitive to learning-rate changes and throws off the combined training process. The practical takeaway is that if you want to layer additional tuning on top of LoRA, a milder approach like prefix-embedding tuning is a safer bet than full prefix-layer tuning.