memit-core-update-equation
IN premise — summaries/2026/08/24/meng-2022-memit-s3-p-reliminaries-l-anguage-modeling-and-memory-editing-chunk-2.md
Created 2026-08-25T02:58:13+00:00
MEMIT's weight update is computed as Δ = R·K₁ᵀ · (C₀ + K₁K₁ᵀ)⁻¹, where C₀ = λ·E_k[kkᵀ] is an uncentered covariance of pre-existing keys.
Summary
When MEMIT inserts a new fact into a model's weights, the update is not computed in isolation; it is regularized by the raw second-moment statistics of the model's existing keys, treating the origin (zero vector) as the reference point rather than the mean. This means the strength and direction of the edit depend on how spread out the pre-existing key patterns are near zero, which implicitly couples the new fact to the model's overall key geometry and makes the edit sensitive to the scale of existing representations.