bert-two-pretraining-tasks-mlm-nsp

IN premise — entries/2026/06/21/wiki-BERT_language_model-chunk-1.md

Created 2026-06-21T09:50:09+00:00

BERT is pre-trained on two simultaneous self-supervised tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP)

Summary

BERT learns language by simultaneously filling in randomly hidden words and figuring out whether two sentences go together, which gives it both word-level and passage-level understanding in a single training phase. This dual objective is a core architectural choice that shapes how the model represents context, so any downstream task inherits whatever strengths or blind spots come from balancing those two signals.

Dependents

These beliefs depend on this one:

IN bert-established-pretrain-finetune-paradigm — BERT (Devlin et al., 2018) is an encoder-only Transformer pre-trained on BookCorpus and English Wikipedia using two simultaneous self-supervised objectives — masked language modeling (MLM) and next sentence prediction (NSP) — to achieve bidirectional pre-training. Its [CLS] token output serves as a sequence-level representation for classification tasks, while individual token outputs support token-level tasks, enabling a single pre-trained model to be adapted to diverse downstream tasks.