softmax-required-for-attention-copying

IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s5-discussion.md

Created 2026-08-24T17:11:05+00:00

The copying mechanism (merging input and target tokens into concatenated form) requires softmax attention; the authors could not train the two-layer circuit with linear self-attention in the first layer, consistent with Olsson et al. (2022) showing softmax attention easily learns to copy.

Summary

The copying trick of merging input and target tokens into a single concatenated form simply cannot be learned with linear self-attention; softmax attention is a hard requirement for the circuit to work at all. This matters because it rules out the lighter, faster linear-attention alternative for any architecture that needs to copy, meaning the design is locked into the more expensive softmax path rather than being a free choice.