rlhf-applied-after-gpt3-instructgpt-chatgpt

IN premise — entries/2026/06/21/wiki-Generative_pre-trained_transformer.md

Created 2026-06-21T09:50:09+00:00

RLHF was applied after GPT-3 to create InstructGPT, then ChatGPT (launched November 30, 2022)

Summary

ChatGPT (launched November 30, 2022) was not built from scratch; it emerged from layering a human-feedback training technique on top of the GPT-3 base model, with InstructGPT as the intermediate product. This matters because it tells us ChatGPT's distinctive conversational behavior traces back to that specific training recipe applied to GPT-3, rather than to a fundamentally new architecture, which constrains what improvements or fixes can realistically target that lineage.

Dependents

These beliefs depend on this one:

IN chatgpt-demonstrated-alignment-enables-adoption — ChatGPT's November 2022 launch — applying RLHF to GPT-3.5 — illustrated that alignment techniques like RLHF, applied on top of capable base models, can play a significant role in transforming raw capability into widely adopted products.
OUT rlhf-is-reproducible-engineering-discipline — RLHF is a reproducible engineering discipline with a complete mathematical specification (Bradley-Terry loss, KL penalty, PPO clipping) and proven production outcomes (InstructGPT, ChatGPT).
OUT rlhf-three-stage-pipeline-is-standard-alignment — RLHF's three-stage pipeline (SFT → reward model → PPO) remains the standard proven alignment method, having produced both InstructGPT and ChatGPT.