bert-pretraining-small-models-better-than-distilling

IN premise — entries/2026/06/21/wiki-BERT_language_model-chunk-2.md

Created 2026-06-21T09:50:09+00:00

Turc et al. (2019) showed that pre-training small BERT models directly is more effective than only distilling from larger models — 'well-read students learn better'

Summary

Building a small language model from scratch on a large text corpus outperforms the approach of training a big model first and then squeezing its knowledge into the smaller one. For system design, this means compute is better spent training the actual target-sized model directly rather than maintaining a separate oversized teacher model just to distill from.

Dependents

These beliefs depend on this one:

IN bert-accessible-pretraining-validates-efficient-small-scale-research — BERT's $500 training cost (4 TPUs, 4 days) combined with evidence that pre-training small models directly outperforms distilling from large ones validates that accessible small-scale pre-training remains a viable and performance-optimal research methodology — direct pretraining doesn't merely approximate large-model capability but exceeds what distillation from large models can achieve.