rlhf-offline-pessimistic-online-optimistic
IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback.md
Created 2026-06-21T09:50:10+00:00
Offline RLHF uses pessimistic MLE (lower confidence bound) while online RLHF uses optimistic MLE (upper confidence bound) to minimize regret.
Summary
When you can't collect new human feedback (offline), you assume your reward model is overstating how good an action looks, so you plan conservatively around the lower end of your uncertainty. When you can still query new feedback (online), you assume your model is understating an action's potential, so you explore boldly around the upper end. In both cases the goal is the same — keep the gap between what you achieved and the best possible outcome as small as possible — but the direction you bias depends entirely on whether you can still learn more.
Dependents
These beliefs depend on this one:
- IN rlhf-has-irreducible-theoretical-complexity — RLHF exhibits irreducible theoretical complexity beyond its practical engineering challenges: optimal policies from pairwise comparisons are inherently non-Markovian (memory-dependent), and the optimal estimation strategy differs qualitatively between offline (pessimistic lower-bound) and online (optimistic upper-bound) settings — implying no single RLHF implementation can be universally optimal across deployment contexts.