bpe-ulm-vocabularization-plus-segmentation-algorithms

IN premise — summaries/2026-08-24/wiki-Transformer_deep_learning_architecture-chunk-2.md

Created 2026-08-24T17:11:25+00:00

BPE and ULM each include both a vocabularization algorithm (building the vocabulary) AND a segmentation algorithm (splitting text into tokens); an alternative is rule-based greedy left-to-right segmentation over a pre-trained vocabulary.

Summary

BPE and ULM are two-part systems: they first build a vocabulary from scratch, then use that vocabulary to chop up text into tokens, and the two steps are tightly coupled. This matters because if you already have a vocabulary from another source, you don't need the first half at all — a simpler greedy left-to-right scan over that fixed vocabulary can do the tokenizing job on its own.