alibi-bias-inside-attention-not-input-embedding

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-4.md

Created 2026-06-21T09:50:11+00:00

ALiBi adds a linear bias matrix B (where B_{i,j} = j - i) directly into attention scores, not at the input embedding level; it enables short-context pretraining with long-context fine-tuning.

Summary

The positional signal in ALiBi is injected at the very moment the model decides how much to "look" from one word to another, rather than being baked into each word's identity from the start. This separation is what lets you train cheaply on short passages and then stretch the model to handle much longer ones, because the content representations never got tangled up with a fixed window length.

Dependents

These beliefs depend on this one:

IN positional-encoding-is-a-diverse-unsettled-design-choice — Multiple competing positional encoding strategies (absolute sinusoidal, RoPE, ALiBi) exist because self-attention is inherently permutation-invariant — each approach trades off different properties.