lexer-vs-llm-tokenization

IN premise — entries/2026/06/21/wiki-Tokenization_lexical_analysis-chunk-1.md

Created 2026-06-21T09:50:10+00:00

Lexical tokenization (compiler) differs from LLM tokenization in two ways: it is rule-based using lexical grammars rather than probability-based (e.g., BPE), and it does not convert tokens to numerical embeddings.

Summary

A compiler's tokenizer splits code into named symbols using explicit, deterministic rules and leaves them as text labels, never converting them to numbers. This matters because it means compiler pipelines and LLM pipelines work with fundamentally different kinds of tokens, and treating one as a drop-in replacement for the other will break assumptions about determinism, syntax structure, and how downstream stages consume the output.

Dependents

These beliefs depend on this one:

IN subword-tokenization-bridges-lexical-and-neural-worlds — Subword tokenization (BPE, WordPiece) occupies a unique position between classical lexical analysis and neural processing — it uses statistical merging rules rather than linguistic grammars, producing variable-granularity tokens that differ fundamentally from both compiler tokens and word-level NLP tokens.