gpt-pretraining-objective-next-token

IN premise — entries/2026/06/21/wiki-Generative_pre-trained_transformer-chunk-1.md

Created 2026-06-21T09:50:09+00:00

The GPT pre-training objective is next token prediction using self-supervised learning on unlabeled data.

Summary

GPT models are trained by simply reading large amounts of raw text and practicing at guessing what word comes next, with no human labels or task-specific instructions guiding the learning. This single objective is what gives the model its broad, general-purpose language ability and also explains why its behavior is shaped by whatever statistical patterns appeared in the training text, rather than by any explicit reasoning or fact-checking mechanism.

Dependents

These beliefs depend on this one:

OUT decision-transformer-validates-rl-subsumed-by-sequence-modeling — Decision Transformer's successful recasting of reinforcement learning as autoregressive sequence modeling validates that the transformer's next-token prediction objective can subsume entire computational paradigms, reducing RL problems to the same sequence completion framework underlying all language modeling.