inference-memory-management-recapitulates-os-evolution

IN derived (depth 2)

Created 2026-06-21T13:28:05+00:00 · Reviewed 2026-06-21T14:41:08+00:00

The approximately 10,000x context window expansion (from GPT-2's 1,024 tokens to Llama 4 Scout's 10 million) coincided with inference memory management evolving from basic KV caching (storing key/value vectors to avoid recomputation) to virtual-memory-style paging (PagedAttention partitioning KV cache into non-contiguous blocks), a progression that parallels the operating system's evolution from simple buffers to demand-paged virtual memory.

Summary

As context windows grew ten thousand-fold, the way GPUs manage inference memory followed the same arc that operating systems took decades to develop: from a simple "remember what you just used" cache to a demand-paged scheme that allocates memory in non-contiguous blocks only when needed. The takeaway is that LLM serving infrastructure is converging on well-understood OS patterns, so we should expect further familiar tricks — swapping, memory-mapping, overcommit — to appear as contexts keep scaling.

Justifications

SL — KV cache evolved from caching to virtual-memory paging, paralleling OS memory architecture evolution

Antecedents (all must be IN):

  • IN kv-caching-avoids-recomputation-autoregressive — KV caching stores previously computed key and value vectors during autoregressive inference to avoid recomputation; prefilling is the initial forward pass that populates the cache.
  • IN pagedattention-kv-cache-paging — PagedAttention (used in vLLM) applies virtual memory-style paging to the KV cache, reducing GPU memory waste during inference.
  • IN context-windows-expanded-10000x-in-seven-years — Context windows expanded approximately 10,000x from GPT-2's 1,024 tokens (2019) to Llama 4 Scout's 10 million tokens (2025), with Claude 2.1's 200K window as a major intermediate milestone in late 2023.