ghost-attention-multi-turn-consistency
IN premise — entries/2026/06/21/wiki-LLaMA-chunk-1.md
Created 2026-06-21T09:50:09+00:00
Ghost Attention is a technique for multi-turn consistency that concatenates system instructions to each user message but zeros out loss on prompt tokens
Summary
Ghost Attention is a training trick that keeps a model's instructions consistent across long, multi-turn conversations by repeating the system prompt alongside every user message, while telling the model to ignore those repeated tokens when computing its learning updates. The practical upshot is that the model stays on-task over many turns without wasting its learning capacity on tokens it is merely re-reading rather than generating.
Dependents
These beliefs depend on this one:
- IN agentic-memory-spans-three-temporal-levels — Several techniques address memory coherence at different temporal scales in agentic LLM systems: Ghost Attention supports intra-turn consistency by concatenating system instructions to each user message (with loss zeroed on prompt tokens), Reflexion accumulates episodic 'lessons learned' across multiple attempts, and Dreaming (a May 2026 research preview) consolidates persistent memory between sessions by merging duplicates and removing stale entries — suggesting a pattern where different temporal horizons of agent continuity may benefit from distinct mechanisms.