Retrieval Augmentation

123 beliefs (123 IN, 0 OUT)

The Retrieval Augmentation topic centers on a well-established problem in large language models: the long-tail knowledge gap. LLMs retain factual knowledge in a way that is strongly frequency-dependent, so questions about rare entities yield dramatically lower accuracy than questions about popular ones. Kandpal et al. (ICML 2023) demonstrated that accuracy on rare Natural Questions instances follows a log-linear trend with model parameter size at R-squared of 0.98 (kandpal-2023-loglinear-r2-098-natural-questions), yet even the 176B-parameter BLOOM model struggles on questions with few supporting documents (kandpal-176b-struggles-long-tail), and they estimated that roughly one quadrillion parameters would be needed for competitive long-tail recall (kandpal-2023-scaling-estimate-10-15-params). GPT-3 davinci-003 scores only about 19% on the 4,000 least-popular POPQA questions (gpt3-davinci-003-popqa-longtail-19pct), and an order-of-magnitude parameter increase to GPT-J 6B gains just 3 percentage points (gptj-6b-vs-gpt3-longtail-scaling-insufficient). Critically, humans show the inverse trend, performing best on questions with few relevant pre-training documents, which isolates the cause as exposure frequency rather than intrinsic question difficulty (kandpal-2023-human-inverse-trend-rare-facts). This is why retrieval augmentation matters: it decouples knowledge storage from parametric capacity.

The foundational RAG architecture (Lewis et al., 2020, NeurIPS) pairs a parametric generator with a non-parametric retriever that fetches external documents at inference time, operating as a prompting-layer technique that does not modify model weights (rag-lewis-2020-neurips-arxiv, rag-prompting-layer-not-architecture). The practical pipeline chunks documents, generates embeddings, indexes them, retrieves top-k passages, and prepends them to the generation prompt (rag-pipeline-embed-index-retrieve-generate). Mallen et al. (ACL 2023) advanced this with Adaptive Retrieval, the first system to make per-query retrieval decisions without fine-tuning, letting GPT-3 skip retrieval for roughly 60% of POPQA questions it can answer from parametric memory (mallen-2023-first-query-level-retrieval, adaptive-retrieval-gpt3-retrieves-40pct). This yields up to 10% accuracy improvement on POPQA while cutting GPT-3 API costs by about 50% (adaptive-retrieval-10pct-improvement-50pct-cost-reduction), and the best reported POPQA accuracy of 46.5% is 5.3 points above any non-adaptive method (best-popqa-result-46pct). The counterintuitive finding that a 2.7B GPT-Neo model augmented with a dense retriever outperforms vanilla 175B GPT-3 on long-tail questions (mallen-2023-2-7b-retriever-beats-davinci-longtail, gptneo-27b-plus-contriever-beats-gpt3) is perhaps the strongest evidence that retrieval is a substitute for infeasible parametric scaling. However, retrieval is not uniformly beneficial: for roughly 10% of questions where vanilla GPT-3 was correct, augmentation introduced a wrong answer, with Recall@1 dropping to 0.14 in those cases (retrieval-harm-rate-10pct, retrieval-hurts-recall-at-1-0.14), because retrieved passages can inject misleading context for entities the model already knew (retrieval-harms-popular-entity-accuracy).

On the infrastructure side, the retrieval community relies on benchmarks such as MTEB, whose retrieval task comprises 15 BEIR datasets evaluated primarily on nDCG@10 (mteb-retrieval-15-beir-datasets, mteb-retrieval-main-metric), though the aggregate ranking carries structural bias toward breadth of dataset coverage rather than semantic quality (mteb-aggregate-structural-bias). Hybrid search fusing dense and sparse retrieval has emerged as a practical standard (rag-hybrid-search-definition, rag-dense-vs-sparse-vectors), and retrieval quality itself is a bottleneck in multi-step settings: MeLLo's multi-hop accuracy drops from 52% to 31% as retrieval accuracy falls from 88% to 60% with increasing k, but recovers to 73% when all associated facts are correctly retrieved (mello-retrieval-accuracy-bottleneck, mello-73-1-percent-conditional-accuracy). The field is also evolving from flat document retrieval toward structured knowledge traversal, exemplified by Microsoft's GraphRAG, which layers knowledge graphs over the retrieval process to synthesize insights across large corpora (graphrag-microsoft-knowledge-graphs, retrieval-augmentation-evolved-from-flat-to-structured-knowledge). A unifying principle runs through much of this work: rare knowledge is more efficiently externalized into context windows or retrieval stores than encoded parametrically, a view operationalized by 200K-token context windows and near-perfect synthetic key-value retrieval (context-externalization-principle, context-window-externalization-validation-v2, claude-near-perfect-kv-retrieval). All beliefs in this topic are currently IN with no retracted or OUT beliefs, indicating a coherent and internally consistent set of claims spanning the problem diagnosis, the adaptive-retrieval solution, the benchmarking infrastructure, and the architectural evolution from RAG to GraphRAG.