Sentence Embeddings

41 beliefs (41 IN, 0 OUT)

The Sentence Embeddings topic in this knowledge base traces the arc from distributional semantics as a theoretical foundation (distributional-semantics-foundation-embeddings) through the early development of word-level vectors and into the modern sentence-level paradigm. Mikolov's 2010 RNN-based language modeling work established the basis for distributed word representations (mikolov-2010-rnn-language-modeling-word-embeddings), which matured into Word2Vec with its CBOW and Skip-gram architectures (word2vec-two-architectures-cbow-skipgram, word2vec-mikolov-embeddings). Word2Vec and GloVe are explicitly distinguished as learning similar embeddings from fundamentally different training signals: a predictive next-word objective versus factorization of a global co-occurrence matrix (word2vec-vs-glove-distinct-training-objectives). ALBERT extended the transformer lineage by sharing parameters across layers and replacing the NSP objective with Sentence-Order Prediction (albert-shared-params-sop-replaces-nsp). A recurring finding across several beliefs is that naive extraction of sentence representations from these models fails: BERT's raw [CLS] token performs worse than averaging non-contextual word vectors (bert-cls-token-poor-sentence-embedding), and simply averaging BERT's output layer or reading [CLS] yields results worse than averaging GloVe vectors (sbert-naive-bert-worse-than-glove). The stated quality progression runs CBOW < Skip-Thought < InferSent < SBERT, positioning SBERT as state of the art at the time of writing (sentence-embedding-quality-ordering). Bias concerns are also noted, with word2vec vectors trained on Google News reproducing and even amplifying gender and racial stereotypes (bolukbasi-2016-gender-analogies-word2vec-google-news).

SBERT, authored by Reimers and Gurevych at TU Darmstadt and published at EMNLP 2019 (sbert-emnlp-2019-authors), is the central subject of the belief set, and its key contribution is demonstrating that additional NLI fine-tuning of a pretrained transformer beyond BERT's pre-training significantly improves sentence embedding quality (mteb-sbert-finetuning-contribution). Architecturally, SBERT uses a siamese structure where two BERT encoders share tied weights (sbert-siamese-tied-weights), with MEAN pooling over all token vectors as the default strategy (sbert-default-pooling-mean). Ablation on STSb confirms MEAN (80.78) outperforms CLS (79.80) and MAX (79.07) (sbert-pooling-ablation-mean-best), and on raw BERT the gap is stark: MEAN achieves 77.03 Spearman correlation versus 29.19 for [CLS] (sbert-mean-vs-cls-stsb-77-03-vs-29-19). Training uses three distinct objectives—cross-entropy classification, MSE regression, and triplet margin loss (sbert-three-objectives)—with the specific NLI configuration drawing on SNLI and MultiNLI for roughly 1 million pairs (sbert-training-config-snli-multinli). The triplet loss uses Euclidean distance with margin 1 (sbert-triplet-loss-euclidean-margin-1), and Wikipedia section triplet training yields 80.78 accuracy, beating a BiLSTM baseline at 74.00 (sbert-wikipedia-triplet-80-78-vs-bilstm-74, sbert-wiki-sec-triplet-data-size). At inference time, only cosine similarity on raw embeddings is used; the richer combination vector (u, v, |u-v|) serves exclusively during classifier training (sbert-inference-no-combination-vector), and adding the element-wise product u*v actually hurts performance (sbert-uv-product-hurts-performance). Speed is a major advantage: finding the most similar pair among 10,000 sentences takes roughly 65 hours with a BERT cross-encoder but about 5 seconds with SBERT embeddings (sbert-10k-sentence-speed-comparison), and smart batching achieves roughly 89% speedup on CPU and 48% on GPU (sbert-smart-batching-speedup-89-cpu-48-gpu), pushing throughput to 2042 sentences/second on GPU (sbert-smart-batching-throughput-83-cpu-2042-gpu). On GPU, SBERT with smart batching is about 9% faster than InferSent and 55% faster than Universal Sentence Encoder, reversing InferSent's 65% CPU advantage (sbert-gpu-faster-than-infersent-and-use, inferent-benchmark-137-cpu-1876-gpu). Backbone choice matters less than expected: RoBERTa yields only minor differences (roberta-no-significant-gain-over-bert-sbert), XLNet underperforms BERT (sbert-xlnet-underperformed-bert), and poly-encoders are unsuitable for symmetric O(n squared) tasks due to their asymmetric scoring and computational cost (sbert-poly-encoders-unsuitable-on2).

Evaluation methodology and downstream use round out the topic. The SemEval STS shared task series from 2012 through 2017 serves as the standard benchmark (sem-eval-sts-tasks-2012-2017), and Spearman rank correlation is the canonical metric, with Pearson correlation deemed "badly suited" for STS evaluation (sbert-spearman-not-pearson-metric). The SBERT pipeline of NLI fine-tuning, mean pooling, cosine similarity, and Spearman correlation is validated as the de-facto standard in both the original paper and the MTEB benchmark's STS protocol (embedding-evaluation-pipeline-consistency). Ablation experiments report averages over 10 random seeds per configuration (sbert-10-random-seeds-ablation), and speed benchmarks were run on specific hardware: Intel i7-5820K, Nvidia Tesla V100, CUDA 9.2 (sbert-benchmark-hardware-specs). On cross-topic AFS, a notable gap appears: BERT-AFS-large achieves 62.02 Pearson r while SBERT-AFS-large reaches 53.82, an roughly 8-point difference (sbert-afss-cross-topic-pearson-gap). In practical systems, Sentence-BERT embeddings from the sentence-transformers/multi-qa-mpnet-base-dot-v1 model are used to retrieve irrelevant evidence passages in the Chameleon distractor experiment (chameleon-sentence-bert-irrelevant-retrieval), and MeLLo embeds fact statements using the Contriever model for retrieval indexing (mello-uses-contriever-embedding). The reference implementation is the sentence-transformers Python library (sbert-sentence-transformers-repo).

In terms of belief status, every belief in this group carries an IN designation; none have been retracted or marked OUT. This means the knowledge base currently holds a fully coherent and uncontradicted account of sentence embeddings, spanning theoretical foundations, architectural details, training procedures, evaluation protocols, and downstream applications, with no internal conflicts or withdrawn claims to reconcile.

IN albert-shared-params-sop-replaces-nsp
ALBERT (2019) shares parameters across transformer layers, replaces NSP with Sentence-Order Prediction (SOP), and decouples hidden size from embedding size
IN bert-cls-token-poor-sentence-embedding
BERT's raw [CLS] token embedding performs poorly as a sentence representation, often worse than averaging non-contextual word embeddings, without further fine-tuning.
IN bolukbasi-2016-gender-analogies-word2vec-google-news
Bolukbasi et al. (2016) demonstrated that word2vec vectors trained on Google News reproduce gender/racial stereotype analogies such as 'man : programmer :: woman : homemaker', and Zhao et al. (2017) showed embeddings can amplify societal biases beyond what is present in training data
IN chameleon-sentence-bert-irrelevant-retrieval
Irrelevant evidence passages for distractor experiments are retrieved using Sentence-BERT embeddings from the model sentence-transformers/multi-qa-mpnet-base-dot-v1.
IN distributional-semantics-foundation-embeddings
Distributional semantics is the theoretical foundation for word embeddings — 'you shall know a word by the company it keeps' — encompassing models from Word2Vec and GloVe through BERT and transformers.
IN embedding-evaluation-pipeline-consistency
The SBERT pipeline (NLI fine-tuning → mean pooling → cosine similarity → Spearman correlation) is the de-facto standard validated across both the original SBERT paper and the MTEB benchmark's STS evaluation protocol.
IN inferent-benchmark-137-cpu-1876-gpu
InferSent achieves 137 sentences/second on CPU and 1876 sentences/second on GPU in the SBERT paper's benchmark.
IN mello-uses-contriever-embedding
MeLLo converts edited facts to sentence statements via manual templates, embeds them using the pretrained Contriever model (Izacard et al., 2021), and stores embeddings in a retrieval index.
IN mikolov-2010-rnn-language-modeling-word-embeddings
In 2010, Mikolov's RNN-based language modeling approach led to the development of distributed word representations (word embeddings), establishing the foundation for the word2vec model
IN mteb-sbert-finetuning-contribution
SBERT's key contribution was demonstrating that additional fine-tuning of a pretrained transformer (beyond BERT's pre-training) improves sentence embedding performance.
IN roberta-no-significant-gain-over-bert-sbert
Replacing BERT with RoBERTa as the SBERT backbone yields only minor differences for sentence embedding generation.
IN sbert-10-random-seeds-ablation
SBERT ablation experiments use 10 random seeds per configuration and report averaged performances.
IN sbert-10k-sentence-speed-comparison
Finding the most similar pair among 10,000 sentences requires ~49,995,000 forward passes (~65 hours) with a BERT cross-encoder on a single V100 GPU, versus ~5 seconds with SBERT embeddings plus cosine similarity.
IN sbert-afss-cross-topic-pearson-gap
On cross-topic AFS, BERT-AFS-large achieves 62.02 Pearson r while SBERT-AFS-large achieves 53.82 Pearson r, a ~8-point gap.
IN sbert-benchmark-hardware-specs
SBERT speed benchmarks were run on Intel i7-5820K @ 3.30GHz, Nvidia Tesla V100, CUDA 9.2, cuDNN; SBERT and InferSent use PyTorch, USE uses TensorFlow.
IN sbert-concatenation-uv-diff-80-78
The concatenation (u, v, |u−v|) scores 80.78 on STSb, while simple concatenation (u, v) scores only 66.04 and (u*v) alone scores 70.54.
IN sbert-default-pooling-mean
SBERT's default sentence pooling strategy is MEAN over all token vectors, with CLS-token and MAX-over-time as alternatives.
IN sbert-emnlp-2019-authors
Sentence-BERT was authored by Nils Reimers and Iryna Gurevych (UKP Lab, TU Darmstadt) and published at EMNLP 2019 (arXiv:1908.10084).
IN sbert-gpu-faster-than-infersent-and-use
On GPU, SBERT with smart batching is ~9% faster than InferSent and ~55% faster than Universal Sentence Encoder, reversing InferSent's 65% CPU advantage.
IN sbert-inference-no-combination-vector
At inference time, SBERT uses only cosine similarity on the two raw sentence embeddings; the combination vector (u, v, |u−v|) is used exclusively during classifier training.
IN sbert-mean-vs-cls-stsb-77-03-vs-29-19
On STSb, MEAN pooling achieves 77.03 average Spearman correlation versus 29.19 for the CLS-token vector in raw BERT.
IN sbert-naive-bert-worse-than-glove
Simply averaging BERT's output layer or reading the [CLS] token produces sentence embeddings that perform worse than averaging GloVe word vectors.
IN sbert-poly-encoders-unsuitable-on2
Poly-encoders (Humeau et al., 2019) use an asymmetric score function and high computational cost, making them unsuitable for symmetric O(n²) tasks like clustering where SBERT's symmetric cosine approach is preferred.
IN sbert-pooling-ablation-mean-best
Pooling ablation on STSb with NLI classification: MEAN (80.78) > CLS (79.80) > MAX (79.07).
IN sbert-sentence-transformers-repo
The SBERT implementation is available as the Python library 'sentence-transformers' at https://github.com/UKPLab/sentence-transformers.
IN sbert-siamese-tied-weights
In SBERT's siamese network structure, the two BERT encoders share (are tied to) the same weights rather than being separate models.
IN sbert-smart-batching-speedup-89-cpu-48-gpu
Smart batching (grouping by sentence length, padding to max within batch) achieves ~89% speedup on CPU and ~48% speedup on GPU for SBERT.
IN sbert-smart-batching-throughput-83-cpu-2042-gpu
SBERT-base with smart batching processes 83 sentences/second on CPU and 2042 sentences/second on GPU; without smart batching it is 44 CPU and 1378 GPU.
IN sbert-spearman-not-pearson-metric
Spearman rank correlation (ρ) is the standard STS evaluation metric in the SBERT paper; Pearson correlation is deemed 'badly suited' for STS evaluation.
IN sbert-three-objectives
SBERT uses three distinct training objectives: classification with cross-entropy (for NLI fine-tuning), regression with MSE (for STS scoring), and triplet with margin loss (for contrastive learning).
IN sbert-training-config-snli-multinli
SBERT is trained on SNLI (570K pairs) + MultiNLI (430K pairs) with 3-way softmax classification, 1 epoch, batch size 16, Adam optimizer at lr=2e-5, linear warm-up over 10% of steps, MEAN pooling.
IN sbert-triplet-loss-euclidean-margin-1
SBERT's triplet loss uses Euclidean distance with margin ε = 1: Loss = max(||s_a − s_p|| − ||s_a − s_n|| + 1, 0).
IN sbert-uv-product-hurts-performance
Adding the element-wise product u*v to the siamese concatenation decreased SBERT performance (80.44 with u*v vs. 80.78 without), unlike InferSent/USE which benefit from it.
IN sbert-wiki-sec-triplet-data-size
Wikipedia section triplet training uses ~1.8M training triplets with 222,957 held-out test triplets from distinct articles, evaluated by rank-order accuracy.
IN sbert-wikipedia-triplet-80-78-vs-bilstm-74
SBERT with triplet loss achieves 80.78 accuracy on Wikipedia section triplets, outperforming Dor et al.'s BiLSTM at 74.00.
IN sbert-xlnet-underperformed-bert
When used as the SBERT backbone, XLNet yielded worse results than BERT in the SBERT experiments.
IN sem-eval-sts-tasks-2012-2017
The SemEval Semantic Textual Similarity (STS) shared task series spans 2012 through 2017 and is the standard benchmark for evaluating sentence-embedding models.
IN sentence-embedding-quality-ordering
The stated quality progression of sentence embedding methods is CBOW < Skip-Thought < InferSent < SBERT, with SBERT being state of the art at time of writing.
IN word2vec-mikolov-embeddings
Word2vec was a word embedding technique developed by Tomáš Mikolov (circa 2010–2013), who also applied RNNs to language modeling
IN word2vec-two-architectures-cbow-skipgram
Word2vec (Mikolov et al., 2013, Google) provides two training architectures: CBOW (predict target word from surrounding context) and Skip-gram (predict context words from target word), with Skip-gram generally preferred for lower-frequency words
IN word2vec-vs-glove-distinct-training-objectives
Word2Vec learns word vectors via a predictive neural-network objective (next-word prediction) while GloVe derives them via factorization of a global word co-occurrence matrix, producing similar embeddings from fundamentally different training signals