adapter-sequential-insertion-latency-exceeds-30pct
IN premise — summaries/2026/08/24/hu-2021-lora-sR-references-chunk-1.md
Created 2026-08-24T17:10:55+00:00
Adapter layers inserted sequentially (in series with the base model) cause up to >30% latency overhead in small-batch, short-sequence online inference scenarios, a penalty mitigated only by large batch sizes and long sequences via hardware parallelism
Summary
When you stack adapter modules one after another in front of the base model, each real-time request for a short prompt or small batch pays a latency tax of more than 30% compared to the unmodified model, because the hardware cannot overlap the extra compute effectively. This matters for any deployment that needs fast per-user responses, since the penalty only shrinks when you shift to batch-heavy, long-sequence workloads where GPU parallelism absorbs the extra sequential steps.