GPT Series

63 beliefs (63 IN, 0 OUT)

The GPT Series topic in this knowledge base traces the full lineage of OpenAI's generative pre-trained transformer models from GPT-1 (117M parameters, 2018) through GPT-5 (August 2025), alongside open-source derivatives like GPT-J (gpt-j-6b-parameters-eleutherai-2021) and GPT-NeoX (c2023-models-tested-gpt2-neox-llama-gpt3), and the consumer product layer that built on them. The series matters because it established the dominant architectural and scaling paradigm for large language models: next-token prediction via self-supervised learning (gpt-pretraining-objective-next-token), with each generation introducing a qualitatively new capability tier. GPT-2 demonstrated zero-shot multitask learning at 1.5 billion parameters trained on WebText (gpt2-15b-params-webtext-40gb-feb-2019, zero-shot-multitask-demonstrated-gpt2), GPT-3 showed that few-shot in-context learning emerges at 175 billion parameters without task-specific fine-tuning (gpt3-few-shot-2020), GPT-4 added multimodal input (gpt-4-released-march-2023-multimodal, gpt4-multimodal-input-text-only-output-gpt4o-full-multimodal), GPT-4o extended to full bidirectional multimodality including audio (gpt4o-text-image-audio), and GPT-5 introduced a routing mechanism that selects between fast and reasoning models per task (gpt5-router-fast-vs-reasoning, gpt-5-router-aug-2025). ChatGPT, launched November 30, 2022 on GPT-3.5 (chatgpt-launched-nov-30-2022), sits in OpenAI's layered product stack above the foundation models alongside specialized tools like DALL-E, Codex, and Sora (openai-product-stack-foundation-specialized-agents-consumer).

Several threads connect across these beliefs. The U-shaped performance curve in multi-document question answering, where accuracy peaks at the beginning and end of context and dips in the middle, is a robust finding across GPT-3.5-Turbo, GPT-4, and non-GPT models alike (liu-2023-u-shaped-curve-finding, llm-u-shaped-performance-curve-universal, gpt35-30doc-mid-context-penalty, gpt4-20doc-accuracy-range), and in the worst case GPT-3.5-Turbo actually scores below its closed-book baseline when the relevant document is mid-context (liu-2023-closed-book-below-mid-context). Fine-tuning remains essential even at 175B scale, lifting MNLI accuracy from 40.6% to 89.5% (gpt3-finetune-mnli-40-6-to-89-5), while GPT-4 still trails human fact-checkers at roughly 71% accuracy (gpt4-fact-checking-71pct-below-human, gpt4-factchecking-accuracy-71-percent-below-human) and exhibits a 13% self-inconsistency rate on Commonsense QA (gpt4-13pct-inconsistency-commonsense-qa) and 32% on FaVIQ (xu2024-gpt4-faviq-32pct-inconsistency). Context windows have expanded roughly 10,000-fold from GPT-2's 1,024 tokens to Llama 4 Scout's 10 million (context-windows-expanded-10000x-in-seven-years), a growth that paralleled the evolution of inference memory management from basic KV caching to PagedAttention-style virtual memory paging (inference-memory-management-recapitulates-os-evolution). Open-source variants like GPT-J and GPT-NeoX share architectural details such as parallel attention-plus-MLP blocks with SwiGLU activations (gptj-gptneox-parallel-attn-mlp-swiglu), and their performance in downstream tasks scales with base-model instruction-following ability, as MeLLo scores show (melllo-gpt35-vs-gptj-scores).

Most beliefs in this group are factual premises: parameter counts, release dates, training data specifications, benchmark scores, and architectural details (gpt2-768-dim-12-heads-64-dhead, gpt-3-175b-few-shot-zero-shot, gpt4-technical-report-arxiv-2303-08774). A smaller set is interpretive or derived. The claim that the GPT series demonstrated exponential capability emergence across four generations (gpt-series-demonstrated-exponential-capability-emergence) is a synthesis over individual model facts. The observation that frontier models are converging on multimodal agentic capabilities (frontier-models-converging-on-multimodal-agentic-capabilities) and that inference memory management recapitulates operating-system evolution (inference-memory-management-recapitulates-os-evolution) are analogical inferences drawn from the factual premises. The memorization-as-attack-surface belief (memorization-creates-attack-surface-for-training-poisoning) extends a measured 1-7% duplication rate in GPT-2 into a plausible but unproven security mechanism. The trademark belief (gpt-trademark-generic-descriptive-us) is a legal-status fact rather than a technical claim. The belief that Claude is architecturally and organizationally distinct from the GPT lineage (claude-classified-distinct-from-gpt-lineage) and the note that Claude and GPT-3.5 share a tokenizer (claude-gpt35-same-tokenizer) serve as boundary markers, clarifying what falls inside and outside the GPT series proper. All beliefs in this topic are currently marked IN; none have been retracted, so there are no withdrawn claims whose absence would require explanation.

IN c2023-gpt3-ice-abstain-49-vs-llama-28
GPT-3 with ICE abstains on 49% of LG (logical generalization) queries, compared to 28% for LLAMA with ICE
IN c2023-models-tested-gpt2-neox-llama-gpt3
Cohen et al. (2023) evaluated KE methods across four model architectures: GPT-2, GPT-NeoX, LLAMA, and GPT-3
IN chatgpt-launched-nov-30-2022
ChatGPT launched on November 30, 2022, initially running on GPT-3.5.
IN chatgpt-release-date-november-30-2022
ChatGPT was released by OpenAI on November 30, 2022, as a productized application of the GPT lineage.
IN claude-classified-distinct-from-gpt-lineage
In Wikipedia's LLM taxonomy, Claude is classified under 'Other model families' alongside Llama, Mistral, DeepSeek, and Qwen—distinct from the GPT series lineage—confirming its independent architectural and organizational provenance (Anthropic vs. OpenAI).
IN claude-gpt35-same-tokenizer
Claude-1.3 and GPT-3.5-Turbo use the same tokenizer (modulo special tokens), yielding identical token counts in the Lost in the Middle experiments.
IN closed-source-models-used-in-experiments
The three closed-source models used in the experiments are ChatGPT-0301, GPT-4-0314, and Chat-Bison-001 (PaLM 2).
IN context-windows-expanded-10000x-in-seven-years
Context windows expanded approximately 10,000x from GPT-2's 1,024 tokens (2019) to Llama 4 Scout's 10 million tokens (2025), with Claude 2.1's 200K window as a major intermediate milestone in late 2023.
IN fine-tuning-layer-and-norm-config
Fine-tuning uses layer 21 for GPT-J and layer 31 for Vicuna-7B with a weight-norm constraint of 5×10⁻⁵ (following Zhu et al., 2021).
IN frontier-models-converging-on-multimodal-agentic-capabilities
Both GPT (text → zero-shot → few-shot → multimodal) and Claude (chatbot → CLI agent → GUI agent → design tool) show parallel trajectories toward multimodal agentic capabilities, which may suggest this direction is a common pattern in frontier model development rather than a design choice specific to any single lab.
IN genread-inference-cost-70s
GenRead (elicitive prompting) incurs approximately 70 seconds per query on GPT-NeoX 20B.
IN gpt-2-1-5b-params-webtext
GPT-2 had 1.5 billion parameters and was trained on WebText (40GB, 8 million web pages). Its release was staged due to misuse concerns.
IN gpt-3-175b-few-shot-zero-shot
GPT-3 had 175 billion parameters and was a breakthrough in few-shot and zero-shot learning.
IN gpt-4-released-march-2023-multimodal
GPT-4 was released March 14, 2023, with multimodal capabilities (text + image input, text output only).
IN gpt-5-router-aug-2025
GPT-5 (August 7, 2025) introduced a router that selects between fast and reasoning models per task.
IN gpt-j-6b-parameters-eleutherai-2021
GPT-J is a 6-billion-parameter open-source model produced by EleutherAI in 2021, reproducing the GPT-3 architecture.
IN gpt-naming-o-prefix-reasoning-4o-omni
In OpenAI's model naming convention, the 'o' prefix (o1, o3, o4-mini) denotes reasoning models using extended chain-of-thought, while '4o' (GPT-4o) denotes natively multimodal ('omni') capability
IN gpt-pretraining-objective-next-token
The GPT pre-training objective is next token prediction using self-supervised learning on unlabeled data.
IN gpt-series-demonstrated-exponential-capability-emergence
The GPT series demonstrated exponential capability emergence across four generations: basic language modeling (GPT-1, 117M params, 2018) → zero-shot multitask (GPT-2, 1.5B, 2019) → few-shot in-context learning (GPT-3, 175B, 2020) → multimodal reasoning (GPT-4, 2023).
IN gpt-trademark-generic-descriptive-us
The USPTO issued a nonfinal office action in May 2023 indicating 'GPT' may be too descriptive/generic to trademark; OpenAI secured the trademark in the EU (EUIPO) and Switzerland but not in the US
IN gpt2-1-5b-parameters-restricted-release-nov-2019
GPT-2 (Radford et al., 2019) has 1.5 billion parameters; it was initially released under restricted terms and fully released in November 2019.
IN gpt2-15b-params-webtext-40gb-feb-2019
GPT-2 (February 2019) had 1.5 billion parameters and was trained on WebText (40 GB, 8 million web pages) as an unsupervised multitask learner.
IN gpt2-1point5b-params-staged-release
GPT-2 had 1.5 billion parameters, was released February 2019 with a staged release citing misuse risks, and the full model was released November 2019
IN gpt2-768-dim-12-heads-64-dhead
GPT-2 (124M parameters) uses d_emb=768, n_heads=12, d_head=64, satisfying the constraint d_emb = n_heads × d_head (12 × 64 = 768), making W_O a square 768×768 matrix.
IN gpt2-circular-rep-fails-downstream
GPT-2 exhibits the circular representation for days/months in its hidden states but fails the downstream modular arithmetic task, indicating the representation alone is insufficient without appropriate downstream circuitry.
IN gpt2-initially-withheld-released-november-2019
GPT-2 was initially withheld from public release due to misuse concerns and was ultimately released in stages in November 2019.
IN gpt3-69-of-178-incorrect-dois-cureus-2023
In a 2023 Cureus study, GPT-3 produced 69 out of 178 (approximately 39%) incorrect or nonexistent DOIs.
IN gpt3-few-shot-2020
GPT-3 (Brown et al. 2020) demonstrated that few-shot learning emerges from scale, using 175B parameters without task-specific fine-tuning
IN gpt3-finetune-mnli-40-6-to-89-5
On GPT-3, fine-tuning improves MNLI accuracy from 40.6% (few-shot prompting) to 89.5% and RTE from 69.0% to 85.4%, demonstrating parameter updates remain essential even at 175B scale
IN gpt35-30doc-mid-context-penalty
In the 30-document setting, GPT-3.5-Turbo (16K) drops from 73.4% accuracy at index 0 to 50.5% at index 9, then recovers to 63.7% at index 29—a >20-point mid-context penalty.
IN gpt4-13pct-inconsistency-commonsense-qa
GPT-4 shows approximately a 13% self-inconsistency rate on Commonsense QA tasks (Zhao et al., 2023b).
IN gpt4-20doc-accuracy-range
In the 20-document QA setting, GPT-4 (8K) achieves near 90% accuracy at positions 1 and 20 but drops to approximately 70% at middle positions.
IN gpt4-fact-checking-71pct-below-human
GPT-4 achieved 71% fact-checking accuracy in 2023, below the accuracy of human fact-checkers.
IN gpt4-factchecking-accuracy-71-percent-below-human
A 2023 study measured GPT-4's fact-checking accuracy at approximately 71%, below the accuracy of human fact-checkers at PolitiFact and Snopes
IN gpt4-multimodal-input-text-only-output-gpt4o-full-multimodal
GPT-4 (2023) accepts text and image input but produces text-only output; GPT-4o (2024) processes and generates text, images, and audio in both directions.
IN gpt4-technical-report-arxiv-2303-08774
The GPT-4 technical report (OpenAI, March 2023) is published as arXiv:2303.08774.
IN gpt4o-text-image-audio
GPT-4o (released May 2024) can process and generate text, images, and audio
IN gpt5-released-august-7-2025-to-all-chatgpt-users
GPT-5 was released to all ChatGPT users on August 7, 2025, representing the current-generation reasoning model.
IN gpt5-router-fast-vs-reasoning
GPT-5 (August 7, 2025) introduced a router that automatically selects between a fast model and a slower reasoning model based on task complexity
IN gptj-eleutherai-6b-params-open-source-july-2021
GPT-J (EleutherAI) is a 6-billion-parameter open-source GPT variant released in July 2021.
IN gptj-gptneox-parallel-attn-mlp-swiglu
GPT-J and GPT-NeoX use parallel attention+MLP block architecture with SwiGLU as the MLP nonlinearity
IN inference-memory-management-recapitulates-os-evolution
The approximately 10,000x context window expansion (from GPT-2's 1,024 tokens to Llama 4 Scout's 10 million) coincided with inference memory management evolving from basic KV caching (storing key/value vectors to avoid recomputation) to virtual-memory-style paging (PagedAttention partitioning KV cache into non-contiguous blocks), a progression that parallels the operating system's evolution from simple buffers to demand-paged virtual memory.
IN int8bit-quantization-v100-32gb
OPT-13B and GPT-Neo-20B were run on a single V100 Volta GPU (32 GB VRAM) using int8bit quantization with no notable performance drop
IN liu-2023-closed-book-below-mid-context
When the relevant document is placed mid-context among 20 documents, GPT-3.5-Turbo scores below its closed-book (no-document) accuracy of approximately 56.1%, indicating distractors actively harm performance.
IN liu-2023-gpt35-turbo-position-swing
GPT-3.5-Turbo accuracy swings from approximately 70% to approximately 55% depending on whether the answer-containing document is at the edges or middle of 20 retrieved documents (~4K tokens).
IN liu-2023-models-evaluated
The lost-in-the-middle study evaluates GPT-3.5-Turbo, Claude-1.3, MPT-30B-Instruct, and LongChat-13B 16K, with extended-context variants GPT-3.5-Turbo (16K) and Claude-1.3 (100K).
IN liu-2023-u-shaped-curve-finding
Liu et al. (2023) demonstrate that LLMs exhibit a U-shaped performance curve where accuracy is highest when relevant information is at the beginning or end of context and lowest in the middle, across GPT-3.5-Turbo and Claude-1.3.
IN llama-arch-swiglu-rope-rmsnorm
Llama uses SwiGLU activation (not GeLU), rotary positional embeddings/RoPE (not absolute), and RMSNorm (not LayerNorm), differing from GPT-3's architecture
IN llm-contradiction-detection-subpar
GPT-4, PaLM-2, and Llama 2 show subpar accuracy in detecting contradictions within documents, with subjective-emotion contradictions being especially difficult.
IN llm-u-shaped-performance-curve-universal
LLMs exhibit a U-shaped performance curve in multi-document QA where accuracy is highest when the relevant document is at the beginning or end of context and lowest at the middle, across all tested models including GPT-4, Claude-1.3, GPT-3.5-Turbo, MPT-30B, LongChat-13B, and Llama-2.
IN melllo-gpt35-vs-gptj-scores
MeLLo achieves 91.1/85.5 (CF/T at 1 edit) with GPT-3.5 versus 38.9/30.7 with GPT-J, indicating performance scales with base model instruction-following ability.
IN memorization-creates-attack-surface-for-training-poisoning
LLM training data memorization (measured at 1-7% exact duplicates in GPT-2) is a plausible mechanism through which deliberate data poisoning (grooming) could materialize in model outputs — if poisoned content is memorized, it may be retrievable at inference time, suggesting that memorization rates could serve as one indicator of susceptibility to poisoning attacks.
IN model-capacity-ranking-shen-2023
The model capacity ranking used in Shen et al. (2023) experiments is LLaMA (7B) > GPT-J (6B) > GPT-Neo (2.7B) > GPT2-XL (1.5B).
IN openai-o1-83pct-imo-vs-gpt4o-13pct
OpenAI o1 scored 83% on IMO qualifying problems compared to GPT-4o's 13%.
IN openai-product-stack-foundation-specialized-agents-consumer
OpenAI's product architecture is layered: foundation models (GPT series) → specialized models (DALL-E, Codex, Sora, Whisper) → intelligent agents (ChatGPT agent, Operator) → consumer platforms (ChatGPT with Atlas, Deep Research, GPT Store, Search).
IN popularity-accuracy-correlation-by-model-size
The correlation between subject entity popularity and model accuracy is approximately 0.4 for GPT-3 (davinci-003) versus approximately 0.1 for GPT-Neo-1.3B.
IN query-aware-contextualization-kv-gain
Placing the query both before and after the data (query-aware contextualization) improves GPT-3.5-Turbo worst-case accuracy on 300 key-value pairs from 45.6% to 100%, but has minimal effect on multi-document QA performance.
IN reasoning-models-represent-distinct-capability-tier
Reasoning-specialized models — OpenAI o1 scoring 83% vs GPT-4o's 13% on IMO qualifying problems, DeepSeek R1 matching proprietary models at lower cost — represent a distinct capability tier above standard LLMs, achievable through both proprietary and open-weight approaches.
IN xu-2024-gpt4turbo-enumeration-failure
GPT-4-turbo (≥175B parameters, 128K context window) fails to enumerate all 7-character strings over the alphabet {a,b} (128 strings, ~896 characters total), demonstrating the failure is computational/logical rather than a memory or context-length limitation.
IN xu-2024-linear-order-universal-model-failure
All tested models in Xu et al. 2024 (Llama 2/3 70B, GPT-3.5, GPT-4, GPT-4-turbo) fail linear-order reasoning tasks ω(m) at m=1000₂, exhibiting specific errors including inability to apply transitivity and inconsistent answers for 'x$y' versus 'y$x'.
IN xu2024-gpt4-contradiction-detection-advantage
On the CONTRADOC dataset, GPT-4 detects inter-context contradictions above 70% while ChatGPT, PaLM2, and Llama2 remain below 50% (Xu et al. 2024 Table 2).
IN xu2024-gpt4-faviq-32pct-inconsistency
GPT-4 exhibits a 32% inconsistency rate on the FaVIQ benchmark (Zhao et al. 2023b), indicating no model is fully immune to intra-memory conflicts.
IN zero-shot-multitask-demonstrated-gpt2
Zero-shot multitask learning was demonstrated with GPT-2 by Radford et al. (2019), showing language models can perform downstream tasks without parameter modification