bert-masked-language-modeling-2018

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-6.md

Created 2026-06-21T09:50:10+00:00

BERT (Devlin et al., 2018) is an encoder-only Transformer that uses masked language modeling (MLM) and next sentence prediction for bidirectional pre-training.

Summary

BERT reads text in both directions at once by training on fill-in-the-blank questions and predicting whether two sentences follow each other, rather than generating text token by token. This made it a strong foundation for understanding and classifying existing text, which is why so many downstream NLP tasks were built on top of it.

Dependents

These beliefs depend on this one:

IN bert-established-pretrain-finetune-paradigm — BERT (Devlin et al., 2018) is an encoder-only Transformer pre-trained on BookCorpus and English Wikipedia using two simultaneous self-supervised objectives — masked language modeling (MLM) and next sentence prediction (NSP) — to achieve bidirectional pre-training. Its [CLS] token output serves as a sequence-level representation for classification tasks, while individual token outputs support token-level tasks, enabling a single pre-trained model to be adapted to diverse downstream tasks.