bilingual-pair-single-token-filter-llama2-vocab
IN premise — summaries/2026/08/24/park-2023-linear-representation-s13-every-kingdom-has-its-ruler-typically-a.md
Created 2026-08-24T17:11:02+00:00
Bilingual translation pairs for the measurement notion are restricted to top-1 mutual correspondences in a publicly available lexicon that are also single tokens in LLaMA-2's BPE vocabulary; English⇒French yields 209 contexts for Y⁰ and 231 for Y¹.
Summary
For measuring translation behavior, only English-French word pairs where each word has one unambiguous best translation and fits into a single token in LLaMA-2's tokenizer were kept, yielding 209 examples in one condition and 231 in the other. This means any result they report reflects the model's behavior on clean, whole-word vocabulary rather than subword fragments, which simplifies interpretation but also excludes rarer or longer words that the tokenizer would split apart.