kto-binary-feedback-prospect-theory

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-3.md

Created 2026-06-21T09:50:10+00:00

Kahneman-Tversky Optimization (KTO) requires only binary feedback (desirable/undesirable) rather than pairwise comparisons, uses separate scaling factors for desirable vs. undesirable outputs reflecting loss aversion from prospect theory

Summary

This optimization method works off simple thumbs-up or thumbs-down signals instead of requiring a full side-by-side comparison of two outputs, which cuts annotation cost significantly. It also penalizes bad outputs more harshly than it rewards good ones, mirroring the human tendency to feel losses more acutely than gains, so the system becomes more sensitive to avoiding errors than it is to maximizing positives.

Dependents

These beliefs depend on this one:

IN direct-alignment-eliminated-rlhf-complexity — DPO, IPO, and KTO independently eliminated RLHF's separate reward model and RL loop while sharing a common β-controlled KL regularization structure, each varying only the loss function and data requirements.