bert-embeddings-sum-of-three-types
IN premise — entries/2026/06/21/wiki-BERT_language_model-chunk-1.md
Created 2026-06-21T09:50:09+00:00
BERT's input embedding is the sum (not concatenation) of three components: token type embedding, absolute sinusoidal position embedding, and segment type embedding (0 or 1)
Summary
BERT builds each input representation by adding three separate vectors together — one for the word, one for its position, and one flagging which of two sentences it belongs to — rather than stacking them side by side. This means all three signals live in the same space and get blended into a single vector, so the model has to learn to interpret a mixed signal rather than reading cleanly separated channels.
Dependents
These beliefs depend on this one:
- IN positional-encoding-is-a-diverse-unsettled-design-choice — Multiple competing positional encoding strategies (absolute sinusoidal, RoPE, ALiBi) exist because self-attention is inherently permutation-invariant — each approach trades off different properties.