flashattention-2-230-tflops-a100

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-4.md

Created 2026-06-21T09:50:11+00:00

FlashAttention-2 achieves up to 230 TFLOPs/s on A100 GPUs (2x over v1, 9x over standard PyTorch) by performing attention in GPU-cache-sized blocks to minimize data movement.

Summary

FlashAttention-2 nearly doubles the speed of its predecessor and runs about nine times faster than standard PyTorch attention on A100 GPUs, hitting 230 TFLOPs/s, because it breaks the attention computation into small chunks that fit in the GPU's fast on-chip memory instead of constantly shuttling data to and from slow DRAM. This matters because the bottleneck in attention is data movement, not raw arithmetic, so staying in cache is the key to unlocking most of the GPU's actual compute throughput.

Dependents

These beliefs depend on this one:

IN hardware-aware-optimization-displaced-algorithmic-attention-alternatives — Hardware-aware optimization of standard quadratic attention offers substantial practical efficiency gains: FlashAttention-2 achieves up to 230 TFLOPs/s on A100 GPUs via SRAM-tiled computation, and PagedAttention reduces GPU memory waste through virtual-memory-style KV cache paging. Meanwhile, sub-quadratic algorithmic alternatives exist — Reformer (O(N log N) via LSH), BigBird (O(N) via sparse attention), and Random Feature Attention (O(N) via kernel approximation) — though the antecedents do not establish their relative production adoption rates. The evidence suggests that implementation engineering targeting hardware constraints can yield large efficiency improvements independent of theoretical complexity reduction.