pretraining-universally-beneficial

OUT derived (depth 1)

Created 2026-06-21T10:06:02+00:00

The pretrain-then-finetune paradigm universally improves downstream task performance, as demonstrated by its adoption across BERT, GPT, and all modern LLMs as the standard training pipeline.

Justifications

SL — Pretraining is beneficial by default but not universally — strong augmentation can make self-training competitive

Antecedents (all must be IN):

IN pretrain-finetune-paradigm — The dominant transformer training paradigm is self-supervised pretraining on unlabeled data followed by supervised fine-tuning on labeled task-specific data.
IN self-supervised-learning-dominant-pretraining-paradigm — Self-supervised learning is the dominant pre-training paradigm for modern deep learning, as opposed to supervised or unsupervised learning.

Unless (any of these IN defeats this justification):

IN pretraining-can-hurt-zoph-2020 — Zoph et al. (2020) showed pre-training can reduce accuracy in some cases, finding self-training can outperform transfer learning when strong data augmentation is available.