constitutional-ai-scales-beyond-rlhf-complexity
OUT derived (depth 2)
Created 2026-06-21T13:01:36+00:00
Constitutional AI provides a complete alignment path that bypasses RLHF's irreducible theoretical complexity (non-Markovian optimal policies, divergent online/offline formulations) by using written principles and AI-generated feedback instead of per-example human preferences — but only if training data memorization does not create attack surfaces that corrupt the base model's capacity to follow constitutional principles faithfully.
Justifications
SL — Constitutional AI bypasses RLHF complexity unless training poisoning undermines principle-following
Antecedents (all must be IN):
- IN constitutional-ai-is-complete-alternative-alignment-path — Constitutional AI, developed by Anthropic, uses written principles rather than per-example human feedback and employs AI-generated feedback (RLAIF) based on those principles in place of human preference labels, representing a principle-driven approach to alignment that differs from standard RLHF in its feedback mechanism.
- IN rlhf-has-irreducible-theoretical-complexity — RLHF exhibits irreducible theoretical complexity beyond its practical engineering challenges: optimal policies from pairwise comparisons are inherently non-Markovian (memory-dependent), and the optimal estimation strategy differs qualitatively between offline (pessimistic lower-bound) and online (optimistic upper-bound) settings — implying no single RLHF implementation can be universally optimal across deployment contexts.
Unless (any of these IN defeats this justification):
- IN memorization-creates-attack-surface-for-training-poisoning — LLM training data memorization (measured at 1-7% exact duplicates in GPT-2) is a plausible mechanism through which deliberate data poisoning (grooming) could materialize in model outputs — if poisoned content is memorized, it may be retrievable at inference time, suggesting that memorization rates could serve as one indicator of susceptibility to poisoning attacks.