rag-ict-retriever-pretraining
IN premise — summaries/2026/08/24/wiki-Retrieval-augmented_generation-chunk-1.md
Created 2026-08-25T02:58:51+00:00
The Inverse Cloze Task (ICT) is a pre-training technique for the retriever (not the generator), where the model predicts masked text within documents.
Summary
The retriever is trained before deployment by being asked to fill in missing words inside documents, which gives it a working understanding of how text content relates to itself. This matters because it tells us the retriever's sense of relevance is shaped early on by this gap-filling exercise, rather than by any retrieval-specific training signal.