bert-trained-on-bookcorpus-and-wikipedia

IN premise — entries/2026/06/21/wiki-BERT_language_model-chunk-2.md

Created 2026-06-21T09:50:09+00:00

BERT was trained on Toronto BookCorpus (800M words) and English Wikipedia (2,500M words, filtered to exclude lists, tables, and headers)

Summary

This is the factual starting point for where BERT's language knowledge came from: roughly 3.3 billion words of books and encyclopedic prose, stripped of lists and tables. It matters because it tells us BERT's understanding of English is shaped by published, formal writing rather than casual conversation, which sets realistic limits on what the model can be expected to know or express naturally.

Dependents

These beliefs depend on this one:

IN bert-established-pretrain-finetune-paradigm — BERT (Devlin et al., 2018) is an encoder-only Transformer pre-trained on BookCorpus and English Wikipedia using two simultaneous self-supervised objectives — masked language modeling (MLM) and next sentence prediction (NSP) — to achieve bidirectional pre-training. Its [CLS] token output serves as a sequence-level representation for classification tasks, while individual token outputs support token-level tasks, enabling a single pre-trained model to be adapted to diverse downstream tasks.