bert-mlm-15pct-80-10-10-split

IN premise — entries/2026/06/21/wiki-BERT_language_model-chunk-1.md

Created 2026-06-21T09:50:09+00:00

BERT's masked language modeling selects 15% of tokens; of those, 80% are replaced with [MASK], 10% with a random token, and 10% left unchanged to prevent dataset shift

Summary

When BERT trains, it doesn't just mask every token it picks; it also swaps in a few random words and leaves a handful of real words exactly as they appear, so the model gets a taste of what its inputs look like once it's actually deployed. That 80/10/10 split keeps the training experience close to real usage, meaning the model can still track context from words it already sees rather than only reacting to blanks, which is what makes it useful on downstream tasks like question-answering and summarization.