finetuning-ood-robustness-mitigated-by-linear-weight-interpolation
IN premise — summaries/2026/08/24/wiki-Fine-tuning_deep_learning.md
Created 2026-08-24T17:11:10+00:00
Fine-tuning can distort pre-trained features and reduce out-of-distribution performance; the standard mitigation is linearly interpolating fine-tuned weights with original weights (w_final = α·w_finetuned + (1−α)·w_original).
Summary
Adjusting a pre-trained model to excel at one specific task can quietly erode its ability to handle unfamiliar or out-of-scope inputs. The practical fix is to partially roll back the changes by blending the fine-tuned weights with the original ones, striking a balance between task specialization and general robustness rather than fully committing to the specialized version.