generative-pretraining-paradigm-unsupervised-then-finetune

IN premise — entries/2026/06/21/wiki-Unsupervised_learning-chunk-1.md

Created 2026-06-21T09:55:55+00:00

The generative pretraining paradigm (train unsupervised, then fine-tune supervised) is the foundation of modern LLMs

Dependents

These beliefs depend on this one:

• IN pretraining-finetune-dominant-but-fragile — The pretrain-then-finetune paradigm is dominant for modern deep learning — underpinning both BERT and GPT — but is empirically fragile, as pretraining can actually hurt performance when strong data augmentation or self-training alternatives are available.