gpt-architecture-uses-decoder-only-for-next-token-prediction

IN premise — summaries/2026/08/24/wiki-Generative_pre-trained_transformer-chunk-1.md

Created 2026-08-24T17:11:10+00:00

GPT models use only the decoder portion of the transformer architecture, trained to predict the next token in a sequence, distinguishing them from encoder-only (BERT) or encoder-decoder (T5) variants.

Summary

GPT models read text strictly left-to-right, building context as they go and predicting the next piece of language one token at a time, rather than processing the whole input in both directions like BERT or doing a separate understand-then-generate pass like T5. This makes them naturally strong at open-ended generation but means they can't peek ahead in the input to reinterpret earlier parts of a sentence.