gptneo-27b-plus-contriever-beats-gpt3

IN premise — summaries/2026/08/24/mallen-2023-when-not-to-trust-s4-memorization-depends-on-popularity.md

Created 2026-08-25T02:58:11+00:00

A GPT-Neo 2.7B model augmented with Contriever retrieval outperforms vanilla GPT-3 (~175B parameters) on POPQA factual QA.

Summary

A small 2.7B-parameter model that can pull in outside information at question time answers factual questions better than GPT-3 with roughly 65x more parameters trying to recall everything from memory. This matters because it suggests that for knowledge-heavy tasks, giving a model access to retrieval can be more effective than simply scaling up model size, which has cost and compute implications for how the system should be designed.