fine-tuning-layer-and-norm-config
IN premise — summaries/2026/08/24/zhong-2023-mquake-sR-references-chunk-1.md
Created 2026-08-25T02:59:07+00:00
Fine-tuning uses layer 21 for GPT-J and layer 31 for Vicuna-7B with a weight-norm constraint of 5×10⁻⁵ (following Zhu et al., 2021).
Summary
This pins down exactly where and how strongly the model weights are adjusted during fine-tuning: GPT-J gets its 21st layer touched and Vicuna-7B gets its 31st, each with a tight cap (5×10⁻⁵) on how much those layer weights can shift, following the Zhu et al. recipe. It matters because any downstream claim about model behavior after fine-tuning implicitly depends on this specific layer choice and constraint — change either one and the expected results shift.