inference-optimization-is-multi-layer-discipline

IN derived (depth 1)

Created 2026-06-21T09:52:15+00:00 · Reviewed 2026-06-21T14:41:08+00:00

LLM inference optimization operates at multiple layers simultaneously: algorithmic (speculative decoding with draft models), memory management (PagedAttention's KV cache paging), compute (Flash Attention's IO-aware tiling), and caching (KV reuse across tokens).

Summary

Speeding up LLM inference is not a single trick but a stack of complementary techniques, each targeting a different bottleneck — how tokens are proposed, how GPU memory is allocated, how compute is scheduled on hardware, and how to reuse work already done. This means real performance gains require coordinating across all of these layers, and no single optimization on its own gets you close to the throughput you need.

Justifications

SL — Four orthogonal optimization techniques operating at different abstraction levels, composable rather than competing

Antecedents (all must be IN):

IN speculative-decoding-draft-model-verify — Speculative decoding uses a smaller draft model to propose tokens that a larger model then verifies, accelerating inference by generating multiple tokens per forward pass of the large model.
IN pagedattention-kv-cache-paging — PagedAttention (used in vLLM) applies virtual memory-style paging to the KV cache, reducing GPU memory waste during inference.
IN flash-attention-dominant-efficiency-technique — Flash Attention is the dominant efficiency technique for practical Transformer deployment, using hardware-aware IO-optimized computation
IN kv-caching-avoids-recomputation-autoregressive — KV caching stores previously computed key and value vectors during autoregressive inference to avoid recomputation; prefilling is the initial forward pass that populates the cache.

Dependents

These beliefs depend on this one:

IN production-deployment-requires-full-optimization-stack — Frontier model deployment benefits from optimization across multiple levels simultaneously: architectural (MoE decoupling parameters from compute, as in Llama 4), algorithmic (speculative decoding), memory management (PagedAttention's KV cache paging), and compute-aware (Flash Attention's IO-aware tiling), suggesting these layers form a complementary deployment stack.
IN quadratic-attention-spawned-comprehensive-efficiency-stack — Self-attention's O(n²) complexity is a primary driver of efficiency research, motivating both sub-quadratic algorithmic alternatives (sparse/linear variants, Performers) and a multi-layer practical optimization stack spanning algorithmic (speculative decoding), memory (PagedAttention), and compute (FlashAttention) approaches.