sbert-smart-batching-speedup-89-cpu-48-gpu
IN premise — summaries/2026/08/24/reimers-2019-sentence-bert-s6-ablation-study.md
Created 2026-08-25T02:58:29+00:00
Smart batching (grouping by sentence length, padding to max within batch) achieves ~89% speedup on CPU and ~48% speedup on GPU for SBERT.
Summary
Grouping sentences of similar length into the same batch, rather than padding everything to the longest sentence in the whole set, cuts embedding time by roughly half on GPU and nearly 90% on CPU. This means batch ordering is a free, high-impact optimization the system should apply by default, especially when running on CPU where the gain is enormous.