lstm-multiplicative-gating-distinct-dot-product-attention

Summary

They both manage how information moves through a sequence, but one works by multiplying to decide what to keep or discard at each step, while the other scores relevance across all positions simultaneously. Treating them as the same kind of tool leads to wrong conclusions about which architecture suits which problem and how improvements in one transfer to the other.