saey-mlp-n-token-conjunctions-vs-attention-3-point

Summary

In a transformer model, the attention mechanism is really only good at linking roughly three nearby positions together, so it handles local, short-range interactions. The heavier lifting of combining information from any number of scattered tokens across a whole sequence falls to the feed-forward layers, which is where the model builds up its global, many-part reasoning. This matters because it tells you which part of the architecture to examine or intervene on when you care about how distant pieces of a sentence get put together into a unified idea.