compact-estimate-tokens-chars-div-4
IN premise
`estimate_tokens` uses `len(text) // 4` with a floor of 1, not word count or any external tokenizer.
Summary
The token counting function uses a simple character-based heuristic — dividing the text length by 4 — rather than splitting on words or calling a real tokenizer. This means token estimates are fast but approximate, and they will be least accurate for languages or content where the character-to-token ratio diverges significantly from 4:1.
Dependents
These beliefs depend on this one:
- compact-budget-tracking-is-efficient-and-approximate — The compact module tracks token budgets efficiently through an approximate but computationally fast strategy: O(1) per-line budget checks via a running character count, with token estimation based on chars/4 — a lightweight approximation avoiding external tokenizer dependencies while maintaining accuracy sufficient for budget enforcement.
Details
| Source | entries/2026/04/29/tests-test_compact.md |