pretrain-finetune-paradigm

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-2.md

Created 2026-06-21T09:55:55+00:00

The dominant transformer training paradigm is self-supervised pretraining on unlabeled data followed by supervised fine-tuning on labeled task-specific data.

Dependents

These beliefs depend on this one:

• IN pretraining-finetune-dominant-but-fragile — The pretrain-then-finetune paradigm is dominant for modern deep learning — underpinning both BERT and GPT — but is empirically fragile, as pretraining can actually hurt performance when strong data augmentation or self-training alternatives are available.
• OUT pretraining-universally-beneficial — The pretrain-then-finetune paradigm universally improves downstream task performance, as demonstrated by its adoption across BERT, GPT, and all modern LLMs as the standard training pipeline.