icl-lstm-outperforms-transformer-ginc-hmm

IN premise — summaries/2026/08/24/xie-2021-icl-bayesian-s5-discussion-and-related-work.md

Created 2026-08-25T02:58:56+00:00

On GINC at vocabulary sizes 50 and 100, a 28M-parameter LSTM achieves ~95.8% in-context accuracy while a 4-layer (29M-parameter) Transformer achieves only 60–67%, attributed to structural similarity between HMMs and LSTM recurrence.

Summary

On a structured next-item prediction task with small vocabularies, a recurrent LSTM matches the data-generating process far more naturally than a Transformer, scoring nearly 96% versus only 60-67% despite similar parameter counts. This matters because it shows that for tasks with strong sequential dependencies, choosing an architecture whose internal structure mirrors the task's structure can beat a more general-purpose model by a wide margin, and parameter count alone is not a reliable predictor of performance.