park-2024-token-space-prefix-convention

IN premise — summaries/2026/08/24/park-2024-categorical-hierarchical-concepts-sR-references.md

Created 2026-08-25T02:58:26+00:00

In Park et al. 2024, tokens are prefixed with a space character to represent full-word next-token prediction, and both plural/singular and upper/lower-case variants are included.

Summary

In the Park et al. 2024 paper, every token that represents a whole word is marked with a leading space, and the vocabulary deliberately keeps both singular/plural and upper/lower-case forms as separate entries rather than merging them. This matters because anyone reproducing or building on that work needs to treat the token space as case- and number-sensitive, with the space prefix as the signal for word boundaries rather than subword continuations.