sgpt-bias-only-tuning
IN premise — summaries/2026/08/24/muennighoff-2022-mteb-s4-results.md
Created 2026-08-25T02:58:21+00:00
SGPT uses fewer than 0.1% of pre-trained parameters for contrastive fine-tuning (bias-only tuning)
Summary
When SGPT adjusts itself for a specific task, it only tweaks the tiny offset values (biases) in its network, leaving over 99.9% of its learned weights completely untouched. This means the fine-tuning step is extremely low-risk: the system can adapt its decision boundaries without overwriting the general knowledge it built up during pre-training, which keeps the process cheap and safe from catastrophic forgetting.