rlhf-pairwise-optimal-policies-non-markovian
IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-1.md
Created 2026-06-21T09:50:10+00:00
Optimal RLHF policies from pairwise/dueling comparisons are inherently non-Markovian (memory-dependent) — the best action depends on previous events, unlike standard RL settings
Summary
Because human feedback in RLHF comes as head-to-head comparisons rather than absolute scores, the best action at any step depends on what happened earlier, not just the current state. This means a simple memoryless policy is insufficient; the system must track its own history to behave optimally, which makes the control problem harder than in standard RL settings where the present state is enough.
Dependents
These beliefs depend on this one:
- OUT decision-transformer-validates-rl-subsumed-by-sequence-modeling — Decision Transformer's successful recasting of reinforcement learning as autoregressive sequence modeling validates that the transformer's next-token prediction objective can subsume entire computational paradigms, reducing RL problems to the same sequence completion framework underlying all language modeling.
- IN rlhf-has-irreducible-theoretical-complexity — RLHF exhibits irreducible theoretical complexity beyond its practical engineering challenges: optimal policies from pairwise comparisons are inherently non-Markovian (memory-dependent), and the optimal estimation strategy differs qualitatively between offline (pessimistic lower-bound) and online (optimistic upper-bound) settings — implying no single RLHF implementation can be universally optimal across deployment contexts.