bpe-and-ulm-subword-tokenization-algorithms

Summary

In practice, the way modern language models split text into subword units is dominated by just two algorithms, and the ecosystem for using them is concentrated in two libraries (HuggingFace tokenizers and sentencepiece). This makes the tokenization layer of NLP fairly standardized and easy to adopt, but it also means most models inherit whatever design choices and limitations those two implementations bake in.