ripple-edits-success-criterion

IN premise — summaries/2026/08/24/cohen-2023-ripple-effects-s5-experiments.md

Created 2026-08-25T02:57:57+00:00

In RIPPLE EDITS evaluation, a generation is scored as correct if one of the aliases of the target object appears within the first 20 generated tokens.

Summary

This defines the pass/fail rule for the RIPPLE EDITS benchmark: a model's answer counts as right if it names the target object (under any recognized alias) somewhere in its first 20 tokens of output. In practice, this means the evaluation rewards early commitment to the correct entity while tolerating variation in naming, and it effectively caps how much "thinking out loud" a model can do before it must land on the answer.