rome-second-moment-100k-all-wikipedia-tokens

Summary

The covariance matrix that ROME uses to find the minimal edit is built from 100,000 hidden-state vectors drawn across the entire Wikipedia corpus rather than a narrow set of subject-related tokens, so the geometry it captures reflects how the model represents all of language generally. This means the edit will be stable and well-conditioned, but it is anchored to broad distributional patterns rather than being specialized to the particular fact being rewritten, which trades some targeted precision for robustness against collinearity and rank-deficiency problems.