rlhf-alternatives-dpo-ipo-kto

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-4.md

Created 2026-06-21T09:50:10+00:00

DPO (Rafailov et al. 2023), IPO (Azar et al. 2023), and KTO (Ethayarajh et al. 2024) are direct alignment algorithms that optimize from preferences without explicit reward models, representing a shift away from PPO-based RLHF.

Summary

Several newer alignment methods, including DPO, IPO, and KTO, let you tune a language model directly from human preference data without training a separate reward model or running a full reinforcement learning loop. This matters because it strips out some of the most brittle and expensive components of the traditional RLHF pipeline, making alignment simpler to implement and less prone to the instability problems that come with PPO-based training.