BERT

32 beliefs (32 IN, 0 OUT)

This topic centers on BERT, the bidirectional transformer pre-trained by Devlin, Chang, Lee, and Toutanova and published as arXiv:1810.04805 in October 2018, with code released under the google-research GitHub organization one month later (bert-paper-devlin-2018-arxiv-1810-04805, bert-open-sourced-november-2018, bert-code-under-google-research-org). The beliefs collectively describe BERT's internal mechanics: a WordPiece sub-word tokenizer with a 30,000-token vocabulary (bert-wordpiece-tokenizer-30k-vocab); input embeddings formed by summing token, absolute sinusoidal position, and binary segment-type embeddings rather than concatenating them (bert-embeddings-sum-of-three-types, bert-position-embedding-absolute-sinusoidal); and GELU activations in place of the original transformer's ReLU (swiGLU-llama-palm-gelu-gpt1-bert). Pre-training runs two simultaneous self-supervised objectives, Masked Language Modeling and Next Sentence Prediction, over BookCorpus (800M words) and filtered English Wikipedia (2,500M words), with the MLM step masking 15 percent of tokens in an 80-10-10 split between [MASK], random, and unchanged tokens to reduce dataset shift (bert-two-pretraining-tasks-mlm-nsp, bert-pretraining-corpora-bookcorpus-wikipedia, bert-trained-on-bookcorpus-and-wikipedia, bert-mlm-15pct-80-10-10-split). Fine-tuning for classification reads the [CLS] token through a linear-softmax head while updating all parameters end-to-end, whereas token-level tasks use individual token outputs (bert-classification-finetune-cls-linear-softmax, bert-cls-token-for-classification).

The broader significance of these architectural details is that they underwrote a set of 2018 state-of-the-art results on GLUE, SQuAD v1.1 and v2.0, and SWAG (bert-sota-2018-glue-squad-swag, bert-evaluated-on-squad-and-swag), and they proved that the pretrain-finetune paradigm was viable at production scale. Google deployed BERT to Search for English queries in October 2019, expanded to 70-plus languages by December 2019, and brought it to nearly all English queries by October 2020 (bert-google-search-deployment-2019-2020, bert-deployment-validated-pretrain-finetune-at-scale-v2). A strikingly low training cost of roughly 500 USD on 4 Cloud TPUs over 4 days (bert-training-cost-500-usd-4-tpus-4-days, bert-implemented-in-tensorflow-trained-on-tpus) anchors the broader economic narrative that training costs have scaled by orders of magnitude through GPT-2 and PaLM while per-parameter compute stayed consistent, meaning growth is driven by scale ambition rather than inefficiency (training-costs-scaled-exponentially-across-generations, economic-pressure-drives-capability-cost-decoupling-v2). Turc et al. further demonstrated that pre-training small BERT models directly outperforms distilling from larger ones, validating accessible small-scale pre-training as a performance-optimal methodology rather than merely an approximation (bert-pretraining-small-models-better-than-distilling, bert-accessible-pretraining-validates-efficient-small-scale-research). Distillation itself has been validated across the full scale spectrum, from DistilBERT retaining 95 percent of benchmark performance at 60 percent of parameters to frontier-scale codistillation (distillation-validated-across-full-scale-spectrum-v2).

On the successor and derivative side, the beliefs note that BERT's raw [CLS] embeddings score only 29.19 average STS Spearman correlation, well below GloVe's 61.32, which motivated the Sentence-BERT project: a siamese BERT architecture trained with triplet loss and cosine-similarity regression to produce fixed-size sentence vectors, published at EMNLP 2019 by Reimers and Gurevych and funded by DFG and BMBF (raw-bert-cls-sts-correlation-29-19, sentence-bert-base-architecture-bert, sentence-bert-siamese-triplet-loss-architecture, sentence-bert-venue-emnlp-2019, sentence-bert-funding-dfg-bmbf). Architectural variants such as RoBERTa (removing NSP, using larger batches), ALBERT (cross-layer parameter sharing with SOP), and ELECTRA (replaced-token detection) each modified a specific pretraining design choice while preserving BERT's core architecture, collectively suggesting that BERT's particular design decisions were improvable without discarding the underlying framework (bert-variants-systematically-ablated-pretraining-design, roberta-removed-nsp-larger-batches). Organizationally, Google Brain and DeepMind have since merged into Google DeepMind, which now houses BERT and successor research (google-brain-merged-into-deepmind). All beliefs in this topic carry the IN status; none have been retracted, so the knowledge base presents a fully coherent and unrevised picture of BERT's architecture, training regime, benchmarks, deployment history, and derivative work.

IN bert-accessible-pretraining-validates-efficient-small-scale-research
BERT's $500 training cost (4 TPUs, 4 days) combined with evidence that pre-training small models directly outperforms distilling from large ones validates that accessible small-scale pre-training remains a viable and performance-optimal research methodology — direct pretraining doesn't merely approximate large-model capability but exceeds what distillation from large models can achieve.
IN bert-classification-finetune-cls-linear-softmax
For classification fine-tuning, BERT feeds the [CLS] token's output vector to a linear-softmax layer while updating all model parameters end-to-end.
IN bert-cls-token-for-classification
BERT's [CLS] token output is used for sequence-level classification tasks; for token-level tasks, individual token outputs are used
IN bert-code-under-google-research-org
BERT's official source code is maintained under the 'google-research' GitHub organization (github.com/google-research/bert), not directly under the 'google' organization
IN bert-deployment-validated-pretrain-finetune-at-scale-v2
Google's deployment of BERT to Search — covering English queries from October 2019, expanding to 70+ languages by December 2019, and reaching nearly all English queries by October 2020 — provided a large-scale production demonstration that the pretrain-finetune paradigm could be applied in real-world search settings beyond controlled benchmarks.
IN bert-embeddings-sum-of-three-types
BERT's input embedding is the sum (not concatenation) of three components: token type embedding, absolute sinusoidal position embedding, and segment type embedding (0 or 1)
IN bert-evaluated-on-squad-and-swag
BERT was evaluated on the SQuAD benchmark (reading comprehension) and SWAG benchmark (commonsense inference)
IN bert-google-search-deployment-2019-2020
Google deployed BERT to Search for English queries in October 2019, expanded to 70+ languages by December 2019, and used it for nearly all English queries by October 2020
IN bert-implemented-in-tensorflow-trained-on-tpus
BERT was originally implemented in TensorFlow and trained on Google's Tensor Processing Units (TPUs)
IN bert-mlm-15pct-80-10-10-split
BERT's masked language modeling selects 15% of tokens; of those, 80% are replaced with [MASK], 10% with a random token, and 10% left unchanged to prevent dataset shift
IN bert-open-sourced-november-2018
BERT was open-sourced by Google in November 2018, one month after the paper was published in October 2018
IN bert-paper-devlin-2018-arxiv-1810-04805
BERT's foundational paper is Devlin, Chang, Lee, Toutanova (2018), 'Pre-training of Deep Bidirectional Transformers for Language Understanding', arXiv:1810.04805
IN bert-position-embedding-absolute-sinusoidal
BERT uses absolute sinusoidal position embeddings (not learned parameters), summed with token embeddings and binary segment type embeddings (0 or 1, split at [SEP]) before LayerNorm.
IN bert-pretraining-corpora-bookcorpus-wikipedia
BERT was pre-trained on BookCorpus (800M words) and filtered English Wikipedia (2,500M words).
IN bert-pretraining-small-models-better-than-distilling
Turc et al. (2019) showed that pre-training small BERT models directly is more effective than only distilling from larger models — 'well-read students learn better'
IN bert-sota-2018-glue-squad-swag
BERT set new state-of-the-art results on GLUE (9 tasks), SQuAD v1.1, SQuAD v2.0, and SWAG benchmarks at its 2018 release.
IN bert-trained-on-bookcorpus-and-wikipedia
BERT was trained on Toronto BookCorpus (800M words) and English Wikipedia (2,500M words, filtered to exclude lists, tables, and headers)
IN bert-training-cost-500-usd-4-tpus-4-days
BERT_BASE training cost approximately $500 USD on 4 Cloud TPUs (16 chips) over 4 days
IN bert-two-pretraining-tasks-mlm-nsp
BERT is pre-trained on two simultaneous self-supervised tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP)
IN bert-variants-systematically-ablated-pretraining-design
BERT's immediate successors (RoBERTa, ALBERT, ELECTRA) each modified different pretraining design choices — NSP removal with larger batches, cross-layer parameter sharing with SOP, and replaced-token detection — suggesting that BERT's specific design decisions could be improved upon while its core architecture remained a viable foundation for alternative pretraining objectives.
IN bert-wordpiece-tokenizer-30k-vocab
BERT uses a WordPiece sub-word tokenizer with a 30,000-token vocabulary; unknown tokens are mapped to [UNK]
IN distillation-validated-across-full-scale-spectrum-v2
Knowledge distillation has been demonstrated at widely separated model scales: from BERT-scale (DistilBERT retaining 95% of benchmark performance at 60% of parameters) to frontier-scale (Llama 4 Maverick codistilled from the unreleased ~2T-parameter Behemoth), suggesting the technique is not confined to a narrow range of model sizes, though these two data points do not establish it as universally scale-invariant.
IN economic-pressure-drives-capability-cost-decoupling-v2
Training costs have scaled by orders of magnitude across model generations (from ~$500 for BERT to millions for PaLM) while per-parameter compute requirements remained consistent, indicating growth is driven by scale ambition rather than inefficiency. In response, architectural approaches along two complementary axes decouple capability from inference cost: knowledge distillation compresses capability into smaller models (validated across the full scale spectrum from 110M to 2T parameters), and multi-granularity compute routing—encompassing intra-model MoE and inter-model task routing—allocates compute only where needed. Together, these mechanisms support deployment of frontier-class capability at a fraction of nominal inference cost.
IN google-brain-merged-into-deepmind
Google Brain and DeepMind merged to form Google DeepMind, which is now the organizational home for BERT and successor model research
IN raw-bert-cls-sts-correlation-29-19
Raw BERT CLS-token embeddings score 29.19 average STS Spearman correlation, worse than GloVe (61.32).
IN roberta-removed-nsp-larger-batches
RoBERTa (2019) uses the same architecture as BERT but removes Next Sentence Prediction and uses larger mini-batch sizes and more training data
IN sentence-bert-base-architecture-bert
Sentence-BERT fine-tunes the BERT architecture (Devlin et al., 2018) with a siamese/triplet architecture and custom training objectives (triplet loss, cosine-similarity regression) on NLI data.
IN sentence-bert-funding-dfg-bmbf
The Sentence-BERT work was funded by the German Research Foundation (DFG) via German-Israeli Project Cooperation grants and the German Federal Ministry of Education and Research (BMBF, ArgumenText project).
IN sentence-bert-siamese-triplet-loss-architecture
Sentence-BERT (Reimers & Gurevych, 2019) uses a siamese BERT architecture trained with triplet loss to produce fixed-size sentence vectors suitable for semantic similarity and natural language inference
IN sentence-bert-venue-emnlp-2019
The Sentence-BERT paper by Nils Reimers and Iryna Gurevych was published at EMNLP 2019.
IN swiGLU-llama-palm-gelu-gpt1-bert
SwiGLU activation is used by Llama and PaLM; GELU is used by GPT-1 and BERT; both replace ReLU from the original 2017 transformer.
IN training-costs-scaled-exponentially-across-generations
Training costs scaled by orders of magnitude — from ~$500 for BERT (4 TPUs, 4 days) through ~$50K for GPT-2 to millions for PaLM — while the per-parameter cost formula (6 FLOPs per parameter per token) remained consistent, meaning cost growth is driven by scale ambition, not inefficiency.