rome-model-layer-counts

IN premise — summaries/2026/08/24/meng-2022-rome-sR-references.md

Created 2026-08-25T02:58:17+00:00

GPT-2 XL has 48 layers, GPT-J 6B has 28 layers, and GPT-NeoX 20B has 44 layers; GPT-NeoX uses noise scale ν = 0.03 and GPT-J uses ν = 0.025 for corruption.

Summary

This locks in the exact architectural sizes and corruption noise settings for the three models being compared, so that every downstream calculation or comparison has a consistent factual foundation. Without these reference values pinned down, any analysis of how the models behave or how their parameters affect performance would be built on shifting ground.