bengio-2000-neural-probabilistic-language-model-distributed-representation
IN premise — summaries/2026/08/24/wiki-Word_embedding.md
Created 2026-08-24T17:11:28+00:00
Bengio et al. (2000) introduced neural probabilistic language models that learned dense distributed word representations, replacing the sparse high-dimensional vectors of earlier vector space models and directly motivating modern word embeddings
Summary
Before this work, computers represented words as enormous lists of mostly-zero numbers, making it hard to capture meaning or similarity. Bengio's model showed that a small neural network could compress each word into a compact, fully-populated vector where related words naturally land near each other, which is the direct ancestor of the word embeddings that underpin modern language understanding.