gpt2-117m-parameters-12-heads-1k-context

Summary

These are the size and shape numbers for GPT-2, and they set the hard ceiling on what the model can do: it can only track 12 different kinds of word-relationships simultaneously, and it can only hold roughly 700 words in its working memory before earlier input is lost. In practice, this means any pipeline built on GPT-2 is capped in both reasoning breadth and context length, which is exactly the limitation that pushed later models to grow bigger and look further back.