pretraining-finetune-dominant-but-fragile

IN derived (depth 1)

Created 2026-06-21T10:06:02+00:00 · Reviewed 2026-06-21T15:37:01+00:00

The pretrain-then-finetune paradigm is dominant for modern deep learning — underpinning both BERT and GPT — but is empirically fragile, as pretraining can actually hurt performance when strong data augmentation or self-training alternatives are available.

Justifications

SL — Paradigm dominance coexists with documented failure cases

Antecedents (all must be IN):

IN pretrain-finetune-paradigm — The dominant transformer training paradigm is self-supervised pretraining on unlabeled data followed by supervised fine-tuning on labeled task-specific data.
IN generative-pretraining-paradigm-unsupervised-then-finetune — The generative pretraining paradigm (train unsupervised, then fine-tune supervised) is the foundation of modern LLMs
IN pretraining-can-hurt-zoph-2020 — Zoph et al. (2020) showed pre-training can reduce accuracy in some cases, finding self-training can outperform transfer learning when strong data augmentation is available.

Dependents

These beliefs depend on this one:

IN dominant-paradigms-empirically-fragile-and-transient — The most successful ML paradigms are simultaneously dominant and fragile — pretrain-then-finetune is standard practice yet empirically hurtful in some transfer settings, GANs dominated generative modeling for years yet were displaced by diffusion — suggesting that current best practices are locally optimal recipes liable to succession rather than fundamental principles.
IN pretraining-is-transfer-learning-at-scale — Modern self-supervised pretraining is transfer learning at industrial scale — the formal transfer learning framework (source domain D_S → target domain D_T) exactly describes the pretrain-then-finetune pipeline, unifying a 50-year-old theoretical concept with the dominant modern training methodology.