ipo-quadratic-loss-no-bradley-terry

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-3.md

Created 2026-06-21T09:50:10+00:00

Identity Preference Optimization (IPO) modifies DPO by using a quadratic loss instead of log-sigmoid and does not assume Bradley-Terry preferences, reducing overfitting on noisy preference data

Summary

IPO is a modified version of DPO that replaces the standard loss function with a simpler squared-error one and stops assuming human preferences follow a strict probability model. This matters in practice because real-world preference feedback is often noisy, and the change makes training less likely to memorize that noise, resulting in more robust model updates.

Dependents

These beliefs depend on this one:

IN direct-alignment-eliminated-rlhf-complexity — DPO, IPO, and KTO independently eliminated RLHF's separate reward model and RL loop while sharing a common β-controlled KL regularization structure, each varying only the loss function and data requirements.