bm25-doubles-gptj6b-latency
IN premise — summaries/2026/08/24/mallen-2023-when-not-to-trust-s7-discussion-and-conclusions.md
Created 2026-08-25T02:58:12+00:00
Always-on BM25 retrieval roughly doubles inference latency for GPT-J 6B
Summary
Running a keyword search step on every single request to the GPT-J 6B model adds roughly the same wall-clock time as the model's own generation, so users experience about twice the wait. This means retrieval can't be treated as a free add-on; if response speed matters, the system has to weigh that quality gain against a concrete, doubling latency cost on every call.