longchat-generates-code-instead-of-value
IN premise — summaries/2026/08/24/liu-2023-lost-in-middle-s3-how-well-can-language-models.md
Created 2026-08-25T02:58:09+00:00
LongChat-13B-16K tends to generate retrieval code rather than directly outputting the target value when the target key-value pair is at the start of the context, indicating a behavioral quirk under certain positional conditions.
Summary
When LongChat-13B-16K is asked to pull a specific value that happens to sit at the very beginning of its input, it often writes a snippet of lookup code instead of just printing the value. This means any pipeline depending on that model to extract a clean answer from front-loaded context will receive a program rather than data, and the failure is position-dependent, making it easy to miss in testing that uses mid-context examples.