kv-caching-avoids-recomputation-autoregressive
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-4.md
Created 2026-06-21T09:50:11+00:00
KV caching stores previously computed key and value vectors during autoregressive inference to avoid recomputation; prefilling is the initial forward pass that populates the cache.
Summary
When generating text one token at a time, the model can save the attention computations it already performed for earlier tokens instead of redoing them from scratch at every step. This means the first pass over the prompt does the heavy lifting, and each subsequent token only needs a small incremental computation, making generation dramatically faster than it would be otherwise.
Dependents
These beliefs depend on this one:
- IN inference-memory-management-recapitulates-os-evolution — The approximately 10,000x context window expansion (from GPT-2's 1,024 tokens to Llama 4 Scout's 10 million) coincided with inference memory management evolving from basic KV caching (storing key/value vectors to avoid recomputation) to virtual-memory-style paging (PagedAttention partitioning KV cache into non-contiguous blocks), a progression that parallels the operating system's evolution from simple buffers to demand-paged virtual memory.
- IN inference-optimization-is-multi-layer-discipline — LLM inference optimization operates at multiple layers simultaneously: algorithmic (speculative decoding with draft models), memory management (PagedAttention's KV cache paging), compute (Flash Attention's IO-aware tiling), and caching (KV reuse across tokens).
- IN kv-caching-and-paging-recapitulate-os-memory-management — KV caching (storing computed key/value vectors to avoid recomputation) and PagedAttention (applying virtual memory-style paging to the KV cache) recapitulate operating system memory management abstractions at the neural network inference layer — the same cache-hierarchy and virtual-memory techniques that optimized CPU performance now optimize attention performance, demonstrating a cross-domain transfer from systems engineering to ML inference that mirrors the broader pattern of NLP importing techniques from outside the field.