transfer-learning-universal-data-efficiency

OUT derived (depth 3)

Created 2026-06-21T10:23:12+00:00

Transfer learning provides a universal solution for data-limited ML problems — the formal framework (source→target domain adaptation) validated at industrial scale through self-supervised pretraining makes it the standard approach for any domain with insufficient labeled data.

Justifications

SL — universality claim is gated because Zoph (2020) showed pretraining can hurt — the dominant instantiation of transfer learning is not reliably beneficial

Antecedents (all must be IN):

IN pretraining-is-transfer-learning-at-scale — Modern self-supervised pretraining is transfer learning at industrial scale — the formal transfer learning framework (source domain D_S → target domain D_T) exactly describes the pretrain-then-finetune pipeline, unifying a 50-year-old theoretical concept with the dominant modern training methodology.
IN transfer-learning-standard-for-small-data — Transfer learning (pretraining on a large dataset then fine-tuning on a small target dataset) is the standard technique when training data is limited, preventing overfitting

Unless (any of these IN defeats this justification):

IN pretraining-can-hurt-zoph-2020 — Zoph et al. (2020) showed pre-training can reduce accuracy in some cases, finding self-training can outperform transfer learning when strong data augmentation is available.