mteb-glove-averaging-layer
IN premise — summaries/2026/08/24/muennighoff-2022-mteb-s2-related-work.md
Created 2026-08-25T02:58:19+00:00
GloVe-style word embedding models are context-agnostic, using a word-to-vector layer followed by an averaging operation, making them invariant to input length.
Summary
These embedding models look up each word's fixed meaning vector and then simply average them all together, so word order and sentence structure carry no weight in the final representation. That means a jumbled or reordered version of the same text produces the same output, which limits the system's ability to distinguish meaning that depends on how words are arranged or on the length of the input.