cad-two-forward-passes-per-token

IN premise — summaries/2026/08/24/shi-2024-context-aware-decoding-s3-experimental-setup.md

Created 2026-08-25T02:58:35+00:00

CAD requires two forward passes per generated token: one with context (c, x, y_

Summary

Every token CAD generates costs twice as much compute as standard decoding, because the model must run a full forward pass both with the auxiliary context and without it before it can commit to the next word. In practice, this doubles the time and energy required for text generation, making CAD a meaningful efficiency tradeoff for any deployment that needs to scale output.