multimodal-attention-safely-generalizes-across-domains
OUT derived (depth 5)
Created 2026-06-21T13:10:23+00:00
Attention's modality-agnostic universality and validated cross-modal deployment (ViT for images, AlphaFold for proteins) suggest that the Transformer safely generalizes as a universal computation substrate across new domains through tokenization alone.
Justifications
SL — Tokenization universality enables cross-modal deployment, but cross-modal semantic gaps (word order bias, negation blindness) show tokenization doesn't guarantee understanding
Antecedents (all must be IN):
- IN attention-universality-extends-through-modality-agnostic-tokenization — Attention's validated universality — grounded in its structural computational richness (asymmetric, position-dependent, learned scaling) — extends to arbitrary modalities through a common tokenization abstraction: ViT converting image patches to token sequences, AlphaFold processing protein residues, and Decision Transformer treating RL trajectories as sequences all demonstrate that attention's universality is not domain-specific but architecture-inherent, requiring only a tokenization bridge to apply.
- IN multimodal-transformers-validate-modality-independence — The Transformer's successful deployment across text (GPT/BERT), images (ViT treating 16x16 patches as token sequences), protein sequences (AlphaFold), and game states (Decision Transformer) validates that the architecture's computational power is independent of input modality — any domain expressible as a token sequence can leverage attention's structured directed computation, with ViT's explicit patch-to-token mapping providing the clearest evidence of modality-agnosticism.
Unless (any of these IN defeats this justification):
- IN text-to-image-word-order-matters-no-negation — In text-to-image models, word order matters (earlier words are weighted more heavily), negation is often not understood, and these models do not process grammar the same way LLMs do