rome-prompt-extraction-learn-vs-recall-risk

Summary

When you fine-tune a model on prompts designed to pull out information, the model can sneakily absorb new or incorrect facts instead of simply recalling what it already knows, which means the fine-tuning step itself can corrupt the model's knowledge base. This is the core reason paraphrase-based evaluation benchmarks exist: they test whether the model genuinely retrieves a fact or just generates one because the prompt shape nudged it, so you can trust what the model "knows."