adaptive-retrieval-gpt3-retrieves-40pct

IN premise — summaries/2026/08/24/mallen-2023-when-not-to-trust-s6-adaptive-retrieval-using-retrieval.md

Created 2026-08-25T02:58:11+00:00

With Adaptive Retrieval, GPT-3 davinci-003 retrieves for only approximately 40% of POPQA questions, while GPT-Neo 2.7B retrieves for nearly all.

Summary

Under Adaptive Retrieval, the two models behave very differently: GPT-3 davinci-003 chooses to skip the lookup step for about 60% of POPQA questions, while the smaller GPT-Neo 2.7B relies on it almost every time. This matters because it means the retrieval-augmented pipeline is doing essentially different work for each model, so any performance comparison between them under this setup isn't measuring the same thing.