kandpal-2023-counterfactual-removes-30pct-c4
IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-s3-lm-accuracy-depends-on-relevant.md
Created 2026-08-25T02:58:06+00:00
The counterfactual re-training experiment in Kandpal et al. (2023) removes approximately 30% of the C4 corpus (all relevant documents for the sampled questions) before re-training.
Summary
In the Kandpal et al. (2023) counterfactual experiment, the authors wiped out roughly 30% of the C4 training documents that correspond to the sampled questions and then retrained the model from scratch. This matters because it sets the exact scale of the data-removal intervention that every downstream conclusion about memorization or dependence is measured against; if that 30% figure were wrong, the quantitative findings built on top of it would need to be re-evaluated.