rlhf-has-irreducible-theoretical-complexity
IN derived (depth 1)
Created 2026-06-21T10:20:46+00:00 · Reviewed 2026-06-21T14:41:08+00:00
RLHF exhibits irreducible theoretical complexity beyond its practical engineering challenges: optimal policies from pairwise comparisons are inherently non-Markovian (memory-dependent), and the optimal estimation strategy differs qualitatively between offline (pessimistic lower-bound) and online (optimistic upper-bound) settings — implying no single RLHF implementation can be universally optimal across deployment contexts.
Summary
RLHF does not have a single universally optimal recipe because the best strategy shifts fundamentally depending on whether the system is learning from a frozen dataset or interacting in real time, and because the optimal policy derived from pairwise human comparisons inherently needs to remember past context rather than reacting only to the present moment. In practice, every RLHF implementation is a context-specific compromise, and changing the deployment setting will break what made it work before.
Justifications
SL — Non-Markovian optimality and offline/online divergence together establish fundamental limits on RLHF unification
Antecedents (all must be IN):
- IN rlhf-pairwise-optimal-policies-non-markovian — Optimal RLHF policies from pairwise/dueling comparisons are inherently non-Markovian (memory-dependent) — the best action depends on previous events, unlike standard RL settings
- IN rlhf-offline-pessimistic-online-optimistic — Offline RLHF uses pessimistic MLE (lower confidence bound) while online RLHF uses optimistic MLE (upper confidence bound) to minimize regret.
Dependents
These beliefs depend on this one:
- OUT constitutional-ai-scales-beyond-rlhf-complexity — Constitutional AI provides a complete alignment path that bypasses RLHF's irreducible theoretical complexity (non-Markovian optimal policies, divergent online/offline formulations) by using written principles and AI-generated feedback instead of per-example human preferences — but only if training data memorization does not create attack surfaces that corrupt the base model's capacity to follow constitutional principles faithfully.
- IN rlhf-irreducible-complexity-validates-alignment-diversification — RLHF's irreducible theoretical complexity — optimal policies are inherently non-Markovian, and online and offline formulations diverge fundamentally — independently validates the field's diversification into simpler alignment alternatives (DPO, KTO, Constitutional AI): the complexity is a theoretical ceiling, not merely an engineering inconvenience, making alternatives necessary rather than just convenient.