claude-gpt35-same-tokenizer

IN premise — summaries/2026/08/24/liu-2023-lost-in-middle-s20-t-otal-retrieved-documents.md

Created 2026-08-25T02:58:09+00:00

Claude-1.3 and GPT-3.5-Turbo use the same tokenizer (modulo special tokens), yielding identical token counts in the Lost in the Middle experiments.

Summary

Claude and GPT-3.5 both chop text into the same sequence of tokens (ignoring a few internal control symbols), so any benchmark that feeds them the same passage is giving them identical inputs rather than slightly different ones. This means performance comparisons in the "Lost in the Middle" study are fair and not distorted by one model seeing more or fewer tokens than the other.