memit-norm-clamp-fractions
IN premise — summaries/2026/08/24/meng-2022-memit-sR-references-chunk-1.md
Created 2026-08-25T02:58:13+00:00
MEMIT clamps the L2 norm of δ_i to ¾ ‖hⁱ_L‖ for GPT-J and 3/10 ‖hⁱ_L‖ for GPT-NeoX-20B to prevent destabilizing updates.
Summary
When MEMIT edits a model's weights to change a specific fact, it caps how large each individual update can be relative to the model's existing internal activations, so the edit doesn't wreck everything else the model knows. The cap is tighter for GPT-NeoX-20B (three-tenths of the activation size) than for GPT-J (three-quarters), meaning the larger model needs more careful handling and the system must tune this guardrail per architecture to avoid destabilizing the weights.