pretraining-can-hurt-zoph-2020

IN premise — entries/2026/06/21/wiki-Transfer_learning-chunk-1.md

Created 2026-06-21T09:55:54+00:00

Zoph et al. (2020) showed pre-training can reduce accuracy in some cases, finding self-training can outperform transfer learning when strong data augmentation is available.

Dependents

These beliefs depend on this one:

• IN pretraining-finetune-dominant-but-fragile — The pretrain-then-finetune paradigm is dominant for modern deep learning — underpinning both BERT and GPT — but is empirically fragile, as pretraining can actually hurt performance when strong data augmentation or self-training alternatives are available.
• OUT pretraining-universally-beneficial — The pretrain-then-finetune paradigm universally improves downstream task performance, as demonstrated by its adoption across BERT, GPT, and all modern LLMs as the standard training pipeline.
• OUT transfer-learning-universal-data-efficiency — Transfer learning provides a universal solution for data-limited ML problems — the formal framework (source→target domain adaptation) validated at industrial scale through self-supervised pretraining makes it the standard approach for any domain with insufficient labeled data.