decision-transformer-validates-rl-subsumed-by-sequence-modeling

OUT derived (depth 1)

Created 2026-06-21T13:22:52+00:00

Decision Transformer's successful recasting of reinforcement learning as autoregressive sequence modeling validates that the transformer's next-token prediction objective can subsume entire computational paradigms, reducing RL problems to the same sequence completion framework underlying all language modeling.

Justifications

SL — RL-as-sequence-modeling succeeds UNLESS optimal RL policies are inherently non-Markovian — meaning autoregressive (memoryless) generation fundamentally cannot represent the full structure of preference-optimal behavior

Antecedents (all must be IN):

IN decision-transformer-rl-as-sequence-modeling — Decision Transformer (Chen et al., 2021) recasts reinforcement learning as sequence modeling, applying the Transformer architecture to RL problems.
IN gpt-pretraining-objective-next-token — The GPT pre-training objective is next token prediction using self-supervised learning on unlabeled data.

Unless (any of these IN defeats this justification):

IN rlhf-pairwise-optimal-policies-non-markovian — Optimal RLHF policies from pairwise/dueling comparisons are inherently non-Markovian (memory-dependent) — the best action depends on previous events, unlike standard RL settings