rlhf-pairwise-comparisons-form-one-batch

Summary

When a prompt has K ranked responses, every possible pair of those responses is grouped together and fed to the reward model as a single training step, rather than being split up or mixed with comparisons from other prompts. This means the model learns the full relative ordering from one prompt as a coherent unit, and the data pipeline must keep those pairs bundled together — you cannot shuffle them across batches without breaking the intended training signal.