pagedattention-kv-cache-paging

IN premise — entries/2026/06/21/wiki-LLaMA-chunk-4.md

Created 2026-06-21T09:50:09+00:00

PagedAttention (used in vLLM) applies virtual memory-style paging to the KV cache, reducing GPU memory waste during inference.

Summary

vLLM manages the memory used for tracking attention history by breaking it into small, independently allocated blocks rather than reserving one fixed chunk per request. This means GPU memory is used more tightly, letting the same hardware handle more concurrent users or longer generations without wasting space on unused allocations.

Dependents

These beliefs depend on this one:

IN hardware-aware-optimization-displaced-algorithmic-attention-alternatives — Hardware-aware optimization of standard quadratic attention offers substantial practical efficiency gains: FlashAttention-2 achieves up to 230 TFLOPs/s on A100 GPUs via SRAM-tiled computation, and PagedAttention reduces GPU memory waste through virtual-memory-style KV cache paging. Meanwhile, sub-quadratic algorithmic alternatives exist — Reformer (O(N log N) via LSH), BigBird (O(N) via sparse attention), and Random Feature Attention (O(N) via kernel approximation) — though the antecedents do not establish their relative production adoption rates. The evidence suggests that implementation engineering targeting hardware constraints can yield large efficiency improvements independent of theoretical complexity reduction.
IN inference-memory-management-recapitulates-os-evolution — The approximately 10,000x context window expansion (from GPT-2's 1,024 tokens to Llama 4 Scout's 10 million) coincided with inference memory management evolving from basic KV caching (storing key/value vectors to avoid recomputation) to virtual-memory-style paging (PagedAttention partitioning KV cache into non-contiguous blocks), a progression that parallels the operating system's evolution from simple buffers to demand-paged virtual memory.
IN inference-optimization-is-multi-layer-discipline — LLM inference optimization operates at multiple layers simultaneously: algorithmic (speculative decoding with draft models), memory management (PagedAttention's KV cache paging), compute (Flash Attention's IO-aware tiling), and caching (KV reuse across tokens).
IN kv-caching-and-paging-recapitulate-os-memory-management — KV caching (storing computed key/value vectors to avoid recomputation) and PagedAttention (applying virtual memory-style paging to the KV cache) recapitulate operating system memory management abstractions at the neural network inference layer — the same cache-hierarchy and virtual-memory techniques that optimized CPU performance now optimize attention performance, demonstrating a cross-domain transfer from systems engineering to ML inference that mirrors the broader pattern of NLP importing techniques from outside the field.