bert-wordpiece-tokenizer-30k-vocab

IN premise — entries/2026/06/21/wiki-BERT_language_model-chunk-1.md

Created 2026-06-21T09:50:09+00:00

BERT uses a WordPiece sub-word tokenizer with a 30,000-token vocabulary; unknown tokens are mapped to [UNK]

Summary

BERT handles text by splitting it into sub-word fragments chosen from a fixed pool of 30,000 pieces, and anything it cannot reassemble from those pieces gets flattened into a single "unknown" placeholder. This means the model's view of any input is capped by that vocabulary — genuinely novel terms, rare characters, or out-of-scope languages become lossy and carry almost no signal into the network.

Dependents

These beliefs depend on this one:

IN subword-tokenization-bridges-lexical-and-neural-worlds — Subword tokenization (BPE, WordPiece) occupies a unique position between classical lexical analysis and neural processing — it uses statistical merging rules rather than linguistic grammars, producing variable-granularity tokens that differ fundamentally from both compiler tokens and word-level NLP tokens.