bert-position-embedding-absolute-sinusoidal

IN premisesummaries/2026/08/24/wiki-BERT_language_model-chunk-1.md

Created 2026-08-24T17:11:05+00:00

BERT uses absolute sinusoidal position embeddings (not learned parameters), summed with token embeddings and binary segment type embeddings (0 or 1, split at [SEP]) before LayerNorm.

Summary

This locks in a fixed, non-learned sense of word order and a simple two-sided "which sentence" label at the input stage, meaning the model's ability to handle position and pairing is inherited from a hard-coded formula rather than discovered from data. Any downstream reasoning that depends on where words sit or how two passages relate starts from this rigid scaffold before a single layer of computation has happened.