gpt2-context-window-1k-tokens

IN premise — entries/2026/06/21/wiki-Large_language_model.md

Created 2026-06-21T09:50:09+00:00

GPT-2 had 12 attention heads and a 1,024-token context window

Summary

GPT-2 could only keep about 750 words in its working memory at once and processed each passage through 12 parallel "lenses" that tracked different kinds of relationships in the text. This sets a hard ceiling on what the model could reason about in a single pass, since anything outside that short window was simply invisible to it.

Dependents

These beliefs depend on this one:

IN context-windows-expanded-10000x-in-seven-years — Context windows expanded approximately 10,000x from GPT-2's 1,024 tokens (2019) to Llama 4 Scout's 10 million tokens (2025), with Claude 2.1's 200K window as a major intermediate milestone in late 2023.