memit-gptj-critical-layer-range
IN premise — summaries/2026/08/24/meng-2022-memit-s3-p-reliminaries-l-anguage-modeling-and-memory-editing-chunk-2.md
Created 2026-08-25T02:58:13+00:00
For GPT-J (6B), MEMIT's critical MLP layer range R is {3, 4, 5, 6, 7, 8}, and for GPT-NeoX-20B it is {6, 7, 8, 9, 10}.
Summary
When editing a model's stored facts with MEMIT, you need to target the right internal layers or the edit won't take. These ranges pin down exactly which MLP layers hold the relevant memories for each model size, meaning that for the 6B model you work in layers 3 through 8, while the 20B model requires going deeper, to layers 6 through 10, reflecting that larger models push critical knowledge further back in the stack.