Attention Mechanisms

95 beliefs (95 IN, 0 OUT)

• IN agentic-memory-spans-three-temporal-levels
  Several techniques address memory coherence at different temporal scales in agentic LLM systems: Ghost Attention supports intra-turn consistency by concatenating system instructions to each user message (with loss zeroed on prompt tokens), Reflexion accumulates episodic 'lessons learned' across multiple attempts, and Dreaming (a May 2026 research preview) consolidates persistent memory between sessions by merging duplicates and removing stale entries — suggesting a pattern where different temporal horizons of agent continuity may benefit from distinct mechanisms.
• IN alibi-bias-inside-attention-not-input-embedding
  ALiBi adds a linear bias matrix B (where B_{i,j} = j - i) directly into attention scores, not at the input embedding level; it enables short-context pretraining with long-context fine-tuning.
• IN attention-dimension-equality-constraints
  Required dimension equalities for valid attention computation are d_query = d_key, d_value = d_head, and sequence lengths ℓ_seq,key = ℓ_seq,value; violating any of these makes the matrix multiplication ill-defined.
• IN attention-efficiency-enables-the-security-risk-it-cannot-address-v2
  The attention efficiency breakthroughs that serve as existential prerequisites for the agentic paradigm coexist within the same technical landscape as the persistent memory capabilities that amplify compounding security challenges — meaning the technical foundations that opened the agentic frontier accompany some of its most difficult security problems, though the antecedents do not establish a direct causal pathway from efficiency gains to memory capabilities, nor whether architectural mechanisms could decouple the enabling efficiency from the resulting risk.
• IN attention-entropy-mechanism-correlational
  Attention entropy H(a) = −Σ aᵢ log aᵢ over input positions shows a negative Pearson correlation (r = −0.41 to −0.48) with problem difficulty across 6 models, proposed as the mechanism by which hard problems produce diffuse, model-agnostic attention that homogenizes cross-model representations (correlational, not causal).
• IN attention-evolved-from-augmentation-to-architecture
  Attention evolved from an add-on mechanism augmenting RNNs (Bahdanau 2014) through standalone viability without recurrence (2016) to becoming the entire architecture (Transformer 2017).
• IN attention-head-sharing-is-tunable-efficiency-spectrum
  Attention head parameter sharing operates along a continuous spectrum — from no sharing (standard multi-head attention) through grouped sharing (GQA, with Llama 3's 8 KV heads serving 32 query heads) to complete sharing (MQA, single KV set across all heads) — representing a tunable efficiency-quality tradeoff within the attention mechanism itself, independent of other architectural choices.
• IN attention-is-non-symmetric
  Attention is non-symmetric: token i attending to token j does not imply j attends to i, because the query (W^Q) and key (W^K) weight matrices are different.
• IN attention-is-structured-directed-computation
  Transformer attention is a structured directed computation — asymmetric (i→j ≠ j→i), position-dependent (requiring explicit positional encoding), and gradient-stabilized (√d_k scaling) — not a symmetric similarity metric.
• IN attention-matrix-sparsity-pattern
  W^K and W^Q matrices show ~99% sparsity at all layers in LLaMa-7B and GPT-J, while W^V shows ~75% sparsity with slight decay across depth.
• IN attention-mechanism-predates-transformer
  The attention mechanism predates the Transformer: Bahdanau et al. (2014) introduced additive neural attention for machine translation, and Luong et al. (2015) refined it with multiplicative attention.
• IN attention-nmt-origin-bahdanau-2014-luong-2015
  The attention mechanism for neural machine translation was first introduced by Bahdanau, Cho & Bengio (2014, arXiv:1409.0473) and subsequently formalized by Luong, Pham & Manning (2015), predating the 2017 Transformer paper.
• IN attention-scaled-by-sqrt-dk
  Scaled dot-product attention divides by sqrt(d_k) to stabilize gradients: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) * V.
• IN attention-universality-couples-capability-and-vulnerability-expansion
  Attention's modality-agnostic universality — extending through tokenization to process any domain's data as token sequences — suggests a structural coupling between capability expansion and vulnerability expansion: each new modality enabled by the universal computation primitive may simultaneously create new attack surfaces. Evidence from dual-use properties like memorization indicates these security surfaces may scale superlinearly with capability, though this pattern is characterized at limited scales rather than confirmed as a universal structural law.
• IN attention-universality-grounded-in-structural-richness
  Attention's validated universality across domains — language, protein folding, chess, reinforcement learning — is grounded in its structural computational richness: asymmetry (i attending to j does not imply j attends to i), mandatory position-dependence (requiring explicit positional encoding), and learned scaling (sqrt(d_k) stabilization) create a primitive expressive enough to serve as the sole computational mechanism for diverse sequence-processing tasks.
• IN attention-validated-as-universal-computation-primitive
  Attention's evolution from a task-specific augmentation (Bahdanau 2014) to the sole computational primitive of the Transformer, combined with cross-domain success in protein folding, chess, and reinforcement learning, validates dot-product attention as a general-purpose sequence computation mechanism beyond NLP.
• IN bert-attention-heads-h-div-64-ffn-4h
  BERT's number of attention heads equals H/64 and feed-forward layer size equals 4H, where H is the hidden size — these are fixed ratios, not free parameters
• IN bert-base-config-12l-768h-110m-params
  BERT-BASE has 12 Transformer layers, 768 hidden size, 12 attention heads, 3072 feed-forward size, and 110M parameters.
• IN bert-base-large-layer-count
  BERT base has 12 transformer layers and BERT large has 24 layers, both using multi-head self-attention with [CLS] and [SEP] special tokens.
• IN bert-interpretability-lower-syntactic-higher-semantic
  BERT interpretability research (Clark et al. 2019, Khandelwal et al. 2018) found that lower layers encode local/syntactic features while higher layers encode semantic and task-specific features, and that attention weights do not reliably correspond to syntactic dependencies.
• IN bert-large-config-24l-1024h-340m-params
  BERT-LARGE has 24 Transformer layers, 1024 hidden size, 16 attention heads, 4096 feed-forward size, and 340M parameters; it was trained on 16 TPUs (64 chips) for 4 days.
• IN causal-mask-added-before-softmax-not-after
  The causal mask matrix (lower-triangular zeros, upper-triangular −∞) is added to the attention scores before softmax is applied, not after, ensuring blocked positions receive exactly zero probability mass.
• IN context-expansion-enabled-agentic-application-paradigm-v2
  The 10,000x context window expansion (1K to 10M tokens), made computationally tractable by efficiency research (Flash Attention, PagedAttention, linear attention variants) addressing quadratic attention costs, provided a foundational capability that supported Claude's evolution from a chatbot to an agentic platform — a progression from conversation to autonomous task execution across CLI coding, GUI office automation, and visual design — with large-context processing plausibly serving as an enabling substrate for such stateful agentic workloads.
• IN context-expansion-enabled-by-efficiency-breakthroughs-v2
  The 10,000x context window expansion (1K to 10M tokens over seven years) was facilitated in part by the proliferation of efficiency techniques (Flash Attention, sparse/linear attention variants, Performers) that address self-attention's quadratic O(n²) cost, which is identified as the root constraint driving both context window limitations and this line of research.
• IN core-llm-techniques-transferred-from-outside-nlp-v2
  Two significant LLM techniques — attention (which evolved from an RNN-augmentation mechanism in NMT to a standalone architecture) and RLHF (which transferred from Atari/robotics to NLP, enabled by a domain-independent property: tasks that are hard to specify but easy to judge) — each reached prominence through a notable methodological shift. RLHF's case involves a clear cross-domain transfer from non-NLP settings into NLP; attention's involves an architectural reconfiguration within NLP's own trajectory. Together, these two cases suggest that both technique evolution and cross-domain transfer have each contributed to important LLM advances, though two examples alone do not establish their frequency or relative importance compared to NLP-native innovation.
• IN cross-attention-queries-decoder-keys-values-encoder
  In cross-attention, queries come from the decoder and keys/values come from the encoder output.
• IN cross-attention-uses-decoder-query-encoder-keyvalue
  Cross-attention in the transformer decoder uses queries (Q) from the decoder representation and keys (K) and values (V) from the encoder output, making it the only sublayer unique to the decoder block
• IN dai2023-duality-requires-relaxed-linear-attention
  The equivalence between Transformer attention and gradient descent holds specifically in the relaxed linear attention form (no softmax, no √d scaling); standard softmax attention is an approximation to this form.
• IN dai2023-linear-attention-equals-gradient-descent-on-linear-layer
  A linear layer updated by one step of gradient descent, F(x) = W₀x + Σᵢ(eᵢ⊗x′ᵢ)x, is mathematically equivalent to a linear attention operation where error signals are values, historic inputs are keys, and the current input is the query.
• IN dai2023-moattn-applies-ema-to-value-vectors-not-keys-or-queries
  Momentum-Based Attention (MoAttn) applies Exponential Moving Average to attention value vectors specifically (not keys or queries), adding Σ_{i=1}^{t-1} η^{t-i}·v_i to the standard softmax attention output, where η∈(0,1) is the decay factor.
• IN deberta-disentangled-attention-three-matrices
  DeBERTa (2020) uses disentangled attention that keeps position and token encodings as separate tuples and computes three attention matrices (content-to-content, content-to-position, position-to-content), adding absolute position only in the final layer
• IN decoder-only-has-2-sublayers-no-cross-attention
  Decoder-only transformers (e.g., GPT) have only 2 sublayers per layer (masked self-attention + FFN) because without an encoder there is no cross-attention.
• IN decomposable-attention-2016-no-recurrence-sota
  Decomposable attention (2016) first demonstrated that attention without recurrence could achieve state-of-the-art results, directly inspiring the transformer.
• IN dual-form-derivation-requires-relaxed-linear-attention-no-softmax
  The dual-form derivation between attention and gradient descent holds under relaxed linear attention (no softmax, no √d scaling); standard Transformer attention is the softmax-weighted generalization of this linear form.
• IN efficient-attention-variants
  Efficient attention variants include sparse attention (BigBird), hash-based attention (Reformer), and hardware-optimized computation (FlashAttention) to reduce quadratic cost.
• IN eh2022-induction-head-is-a-pair
  An induction head consists of a pair of attention heads where the first copies a previously seen token (prefix) and the second uses that prefix to attend to a later occurrence
• IN encoder-2-sublayers-decoder-3-sublayers
  Each encoder layer has exactly 2 sublayers (self-attention + FFN); each decoder layer has exactly 3 sublayers (masked self-attention + cross-attention + FFN).
• IN ffn-dimensions-follow-fixed-ratios-across-architectures
  The transformer's feed-forward layers contain the majority of model parameters. A common dimensioning convention sets FFN size to 4x the hidden size (as seen in GPT-2 and BERT), and BERT specifically uses a fixed ratio of H/64 for attention heads — though these ratios are architectural conventions of specific models rather than universal structural constants across all architectures.
• IN ffn-layers-contain-most-transformer-parameters
  Feed-forward layers (not attention layers) contain the majority of parameters in a transformer model.
• IN flamingo-cross-attention-over-frozen-llm
  Flamingo achieves few-shot visual instruction following by adding cross-attention layers over a frozen language model backbone, without modifying the LLM weights
• IN flash-attention-dominant-efficiency-technique
  Flash Attention is the dominant efficiency technique for practical Transformer deployment, using hardware-aware IO-optimized computation
• IN flashattention-2-230-tflops-a100
  FlashAttention-2 achieves up to 230 TFLOPs/s on A100 GPUs (2x over v1, 9x over standard PyTorch) by performing attention in GPU-cache-sized blocks to minimize data movement.
• IN flashattention-exact-computation-not-approximation
  FlashAttention computes exactly equivalent attention results (not an approximation) using block-level tiling designed to fit within GPU shared memory (SRAM), minimizing slow HBM reads
• IN full-nlp-paradigm-shift-from-rules-to-attention-architecture
  The complete NLP paradigm shift spans from overcoming institutional resistance to neural methods (Bengio 2003 → 2015 dominance), through attention evolving from RNN add-on (2014) to standalone architecture (2017), to transformers replacing LSTMs — a multi-decade transition from rule-based to attention-based processing.
• IN gdpp-data-transformation-acceleration
  Stacking K linear self-attention layers outperforms K plain GD steps by implicitly learning a data transformation H(X) = I − γ·XXᵀ applied to inputs before each GD step (the GD++ variant), where γ is a learned scalar that adapts the update to the data geometry.
• IN ghost-attention-multi-turn-consistency
  Ghost Attention is a technique for multi-turn consistency that concatenates system instructions to each user message but zeros out loss on prompt tokens
• IN gpt2-117m-parameters-12-heads-1k-context
  GPT-2 (2019) had 117M parameters, 12 attention heads, and a 1,000-token context window.
• IN gpt2-context-window-1k-tokens
  GPT-2 had 12 attention heads and a 1,024-token context window
• IN gpt4o-extends-modality-independence-to-audio
  GPT-4o's tri-modal capability (text, image, and audio processing and generation) extends modality independence validation beyond vision-only (ViT processing image patches as tokens) and protein-only (AlphaFold) demonstrations into a third sensory domain, strengthening the evidence that transformer attention is genuinely modality-agnostic.
• IN hardware-aware-optimization-displaced-algorithmic-attention-alternatives
  Hardware-aware optimization of standard quadratic attention offers substantial practical efficiency gains: FlashAttention-2 achieves up to 230 TFLOPs/s on A100 GPUs via SRAM-tiled computation, and PagedAttention reduces GPU memory waste through virtual-memory-style KV cache paging. Meanwhile, sub-quadratic algorithmic alternatives exist — Reformer (O(N log N) via LSH), BigBird (O(N) via sparse attention), and Random Feature Attention (O(N) via kernel approximation) — though the antecedents do not establish their relative production adoption rates. The evidence suggests that implementation engineering targeting hardware constraints can yield large efficiency improvements independent of theoretical complexity reduction.
• IN iti-truthfulqa-attention-head-probing
  ITI (Li et al., 2023c) identifies sparse attention heads with high TruthfulQA linear-probing accuracy, then shifts activations along the truth-correlated direction at each autoregressive step.
• IN kv-caching-and-paging-recapitulate-os-memory-management
  KV caching (storing computed key/value vectors to avoid recomputation) and PagedAttention (applying virtual memory-style paging to the KV cache) recapitulate operating system memory management abstractions at the neural network inference layer — the same cache-hierarchy and virtual-memory techniques that optimized CPU performance now optimize attention performance, demonstrating a cross-domain transfer from systems engineering to ML inference that mirrors the broader pattern of NLP importing techniques from outside the field.
• IN linear-attention-equivalence-to-fast-weight-rnn
  Katharopoulos et al. (ICML 2020) demonstrated that Transformers with linear attention are equivalent to RNNs, and Schlag, Irie & Schmidhuber (ICML 2021) proved linear-attention Transformers compute the same function as fast-weight programmers, with lineage tracing to Schmidhuber (1992) and Hinton & Plaut (1987).
• IN llama3-405b-architectural-parameters
  Llama 3 405B has 126 layers, 16,384 model dimension, 53,248 FFN dimension, 128 attention heads, 8 key/value heads (GQA), RoPE(θ=500,000), SwiGLU activation, and 128,000 vocabulary size.
• IN llama3-8-kv-heads-32-attention-heads-gqa
  Llama 3 8B uses grouped-query attention with 8 key/value heads and 32 attention heads.
• IN lsa-layer-w0-zero-equivalent-to-nadaraya-watson-kernel-smoothing
  A single linear self-attention layer in a Transformer block with W₀=0 computes Nadaraya-Watson kernel smoothing ŷ = Σᵢ yᵢk(xᵢ, x_test), where k(xᵢ,x_test) = m(xᵢ)ᵀm(x_test) is implicitly defined by the MLP residual connection m(x) = x + ẽ(x).
• IN lstm-multiplicative-gating-distinct-dot-product-attention
  LSTM's multiplicative gating mechanism is conceptually distinct from the additive/dot-product attention mechanism used in transformers, despite both addressing sequential information flow.
• IN mamba-state-space-model-linear-complexity
  Mamba is a state-space model alternative to attention-based sequence processing that achieves linear computational complexity.
• IN memit-mlp-not-attention-mediation
  Path-dependent ablation experiments confirm that mid-layer MLPs (not Attention modules) at the last subject token are the causal mediators of factual recall in transformers.
• IN mha-vs-gqa-head-level-ablation-divergence
  Maximum per-head causal ablation flip rates differ by attention architecture: multi-head attention (MHA) models show 43%–63% while grouped-query attention (GQA) models show 20%, a difference invisible to full-hidden-state CKA measurement.
• IN mlp-computes-attention-copies-modular-addition
  In modular addition tasks (e.g., days of the week), patching experiments show MLPs in late layers compute the output value γ on input tokens, while attention heads only copy γ to the prediction token position.
• IN modality-is-tokenization-choice-not-architectural-decision
  The coexistence of multiple multimodal fusion strategies (early vs intermediate) alongside attention's modality-agnostic universality suggests that modality functions primarily as a tokenization design choice — how to convert domain data into token sequences — rather than an architectural constraint, since the attention mechanism can process all token sequences through a common abstraction regardless of whether they originated as text, image patches, or other domain data.
• IN mqa-shares-single-kv-across-all-heads
  Multi-Query Attention (MQA) shares a single set of key/value projections across all attention heads; Grouped-Query Attention (GQA) generalizes this to groups of heads sharing KV projections.
• IN nlp-revolution-culminated-in-data-driven-decoder-dominance
  The full NLP paradigm shift — from rules through neural methods to attention architectures — preceded and contextualizes the subsequent rise of decoder-only dominance, which coincided with evidence that data scaling matters more than parameter count. Autoregressive training's ability to exploit massive unsupervised corpora may have contributed to this architectural convergence, but the evidence does not establish the paradigm shift's endpoint as inseparable from the training data insight.
• IN performer-gram-schmidt-orthonormalizes-random-features
  The Performer refines Random Feature Attention by sampling w₁,…,w_D from N(0, σ²I) and then applying the Gram-Schmidt process to orthonormalize them, yielding a more precise approximation of softmax attention as Q(KᵀV/√dₖ)
• IN performers-favor-random-feature-attention
  Performers (Choromanski et al., 2022, arXiv:2009.14794) use a random-feature (FAVOR+) approximation of softmax attention to reduce the quadratic complexity of standard self-attention.
• IN performers-linear-attention-on-complexity
  Performers (Choromanski et al., 2020) use kernel-based linear attention to achieve O(n) complexity versus the standard Transformer's O(n²) self-attention.
• IN positional-encoding-is-a-diverse-unsettled-design-choice
  Multiple competing positional encoding strategies (absolute sinusoidal, RoPE, ALiBi) exist because self-attention is inherently permutation-invariant — each approach trades off different properties.
• IN post-transformer-refinements-stabilize-without-changing-fundamentals
  Post-2017 architectural refinements — Pre-LN normalization (eliminating warmup), SwiGLU activation, RMSNorm, grouped-query attention — all improve training stability and inference efficiency without altering the fundamental self-attention mechanism, representing evolutionary optimization of a settled architecture.
• IN pre-ln-vs-post-ln-transformer
  Pre-LN Transformers apply layer normalization before attention and feedforward layers, which stabilizes training and removes the need for learning rate warmup, unlike the original post-LN design.
• IN prefixlm-hybrid-mask-bidirectional-prefix-causal-generation
  PrefixLM is a decoder-only variant that uses bidirectional attention for prefix tokens and causal masking for generated tokens, resembling encoder-decoder behavior.
• IN proposition-3-next-token-selection-mechanism
  In Proposition 3's single-head construction, the attention mechanism selects the next token's value via KᵀW_Q e_j = p_{j+1}, where W_V contains a lower-diagonal identity I_x,off that copies y_{j+1} into token j, enabling the dot-product structure required for gradient descent.
• IN pruning-head-path-patching-no-weight-update
  Pruning Head via Path Patching (Jin et al., 2024b) identifies attention heads with opposite effects in later layers and prunes conflicting heads without updating model weights.
• IN quadratic-attention-drives-efficiency-research
  Self-attention's O(n²) complexity is the root constraint driving both context window limitations and a proliferation of efficiency techniques (Flash Attention, sparse/linear variants, Performers).
• IN quadratic-attention-spawned-comprehensive-efficiency-stack-v2
  Self-attention's O(n²) complexity is a root constraint driving context window limitations and a proliferation of attention-specific efficiency techniques (Flash Attention, sparse/linear variants, Performers). LLM inference optimization more broadly operates as a multi-layer discipline spanning algorithmic (speculative decoding), memory management (PagedAttention), compute (Flash Attention's IO-aware tiling), and caching (KV reuse) approaches, with attention-efficiency being one concern addressed across these layers.
• IN rome-aie-mlp-vs-attention
  Causal Tracing measures the Average Indirect Effect (AIE) of MLP at the last subject token as approximately 6.6%, compared to approximately 1.6% for attention at the same position.
• IN rome-attnedit-layer-33-fails-generalization
  AttnEdit (constrained fine-tuning of W_Q, W_K, W_V at layer 33) succeeds on direct prompts but fails paraphrase generalization, supporting the hypothesis that middle-layer MLPs store factual recall while late-layer attention handles word prediction.
• IN rome-factual-knowledge-location
  In GPT-2 XL, factual associations are stored in mid-layer feed-forward MLP modules, not in attention layers, as identified by causal mediation analysis.
• IN rome-mlp-run-10-recovery-23.6pct
  Restoring runs of 10 consecutive MLP lookup values at the early causal site achieves up to 23.6% average max score recovery, while attention at the late site achieves up to 19.4% in GPT-2 XL.
• IN rome-two-site-causal-architecture
  Causal Tracing identifies two critical sites for factual recall in GPT-2 XL: an early site (~layers 15–17) at the last subject token where MLP lookups are decisive, and a late site (~layer 32) at the last token before prediction where attention modules dominate.
• IN rome-two-site-pattern-gpt2xl-layers
  In GPT-2 XL, the two-site causal pattern places the early MLP lookup site at approximately layer 17 (last subject token) and the late attention delivery site at approximately layer 32 (last token before prediction).
• IN rwkv-rnn-with-transformer-training
  RWKV (Peng et al., 2023) reinvents RNNs with Transformer-era training techniques, avoiding the quadratic attention complexity while attempting comparable performance.
• IN sbert-inference-cosine-similarity-o-n
  At inference, SBERT computes similarity via cosine-similarity between two fixed-size embeddings, which is O(n) per pair, unlike poly-encoders requiring O(n²) attention-based scoring.
• IN self-attention-permutation-invariant-needs-positional-encoding
  Self-attention is permutation-invariant; without positional encodings or learned positional embeddings, token order does not affect the output.
• IN self-attention-quadratic-complexity
  Standard self-attention has quadratic complexity with respect to input sequence length
• IN seq2seq-fixed-size-encoder-bottleneck-2014
  Seq2seq models (2014, Sutskever & Cho) used fixed-size encoder output vectors creating an information bottleneck; attention (Bahdanau et al. 2014; Luong et al. 2015) solved this by allowing the decoder to attend over all encoder states rather than a single compressed vector.
• IN single-head-phase-transitions-eliminated-by-two-heads
  Single-head self-attention layers exhibit seed-dependent long training plateaus before sudden loss drops; adding a second head eliminates these phase transitions. With limited training data (8192 tasks), grokking-like behavior where test loss spikes then collapses is reproducible.
• IN softmax-additive-offset-cancelled-by-two-heads
  Softmax in attention introduces an additive offset ε relative to linear attention, making a single-head softmax layer suboptimal for gradient descent; two attention heads with sign-reversed off-diagonal W_KW_Q matrices cancel this offset, recovering PVKᵀq_j dynamics.
• IN softmax-applied-row-wise-over-key-positions
  In the attention formula Attention(Q,K,V) = softmax(QK^T/√d_k)·V, softmax is applied row-wise (over the key positions for each query), not column-wise or globally.
• IN softmax-required-for-attention-copying
  The copying mechanism (merging input and target tokens into concatenated form) requires softmax attention; the authors could not train the two-layer circuit with linear self-attention in the first layer, consistent with Olsson et al. (2022) showing softmax attention easily learns to copy.
• IN subquadratic-attention-reformer-bigbird
  Sub-quadratic attention methods: Reformer uses locality-sensitive hashing for O(N log N), BigBird uses sparse random graphs for O(N), Random Feature Attention uses kernel approximation for O(N).
• IN swin-transformer-shifted-window-linear-complexity
  Swin Transformer is a hierarchical vision transformer that uses shifted-window self-attention to achieve linear complexity in image resolution and enable multi-scale feature maps analogous to CNNs.
• IN transformer-generality-validated-within-and-beyond-nlp
  The Transformer's practical value is independently validated at two levels: within NLP through architectural maturity and known over-parameterization, and beyond NLP through successful application to protein folding, chess, and reinforcement learning — suggesting the attention mechanism captures a domain-general computation pattern.
• IN transformer-quadratic-cost-context-window
  Transformer attention computation scales quadratically O(n²) with context window size, unlike RNNs which scale linearly O(n) but cannot parallelize.
• IN transformer-replaces-recurrence-with-parallel-attention
  Transformers replaced recurrent mechanisms (RNN/LSTM) with parallel multi-head self-attention, eliminating sequential bottlenecks but introducing O(n²) computation cost in context window size.