t5-excluded-due-to-qa-pretraining

IN premise — summaries/2026/08/24/mallen-2023-when-not-to-trust-s4-memorization-depends-on-popularity.md

Created 2026-08-25T02:58:11+00:00

Encoder-decoder models such as T5 are excluded from Mallen et al. (2023) evaluation because their supervised pretraining already includes QA, confounding the memorization analysis.

Summary

T5 and similar encoder-decoder models are left out of the Mallen et al. memorization study because their pretraining was already supervised and included question-answering, making it impossible to isolate how much of their output comes from memorizing training data versus learning the task. For the system, this means T5 cannot serve as a clean test case when evaluating raw memorization effects, so any conclusions drawn from that paper don't apply to it.