bert-cls-token-poor-sentence-embedding
IN premise — summaries/2026/08/24/wiki-Sentence_embedding.md
Created 2026-08-25T02:58:54+00:00
BERT's raw [CLS] token embedding performs poorly as a sentence representation, often worse than averaging non-contextual word embeddings, without further fine-tuning.
Summary
Simply grabbing BERT's special [CLS] token as a ready-made summary of a sentence's meaning tends to underperform compared to the much simpler approach of averaging basic word vectors. This means the system cannot rely on that token out of the box as a sentence representation without investing in task-specific training, and for quick-and-dirty embedding needs, a naive average may actually be the stronger choice.