alibi-linear-bias-attention

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-5.md

Created 2026-06-21T09:55:55+00:00

ALiBi adds a linear bias matrix B (where B_{i,j} = j-i) scaled by scalar s directly to attention logits, enabling short-context pretraining followed by long-context fine-tuning.