rlhf-seminal-papers-lineage

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-4.md

Created 2026-06-21T09:50:10+00:00

The canonical RLHF lineage consists of three papers: Christiano et al. (2017) introduced deep RL from human preferences, Stiennon et al. (2020) applied it to summarization, and Ouyang et al. (2022) scaled it to InstructGPT.

Summary

This pins down the three-paper chain — the original deep learning formulation, the first application to text summarization, and the large-scale InstructGPT deployment — as the accepted foundation for how RLHF actually got built. Any argument in the system about where RLHF came from, who is credited for what, or whether a newer paper is a "first" in this space will be checked against this three-step lineage.

Dependents

These beliefs depend on this one:

IN rlhf-transferred-from-games-via-universal-evaluation-property — RLHF transferred from Atari/robotics (Christiano 2017) to NLP because its motivating property — tasks that are hard to specify but easy to judge — is domain-independent, enabling the same preference-comparison framework to align game bots and language models alike.