modality-is-tokenization-choice-not-architectural-decision

IN derived (depth 5)

Created 2026-06-21T13:28:05+00:00 · Reviewed 2026-06-21T14:41:08+00:00

The coexistence of multiple multimodal fusion strategies (early vs intermediate) alongside attention's modality-agnostic universality suggests that modality functions primarily as a tokenization design choice — how to convert domain data into token sequences — rather than an architectural constraint, since the attention mechanism can process all token sequences through a common abstraction regardless of whether they originated as text, image patches, or other domain data.

Summary

The core insight is that the engine doing the reasoning (attention) doesn't care what kind of data it's processing, as long as that data has been chopped into a token sequence first. This means adding a new type of input to the system is mostly a problem of figuring out how to slice the data into tokens, not redesigning the architecture, which dramatically simplifies how new modalities get integrated.

Justifications

SL — Attention treats all modalities identically; only the tokenization strategy varies by domain

Antecedents (all must be IN):

IN multimodal-fusion-early-vs-intermediate — Two multimodal fusion strategies exist: early fusion (combine modality embeddings before prediction) and intermediate fusion (process modalities independently, then fuse via cross-attention, e.g., Flamingo).
IN attention-universality-extends-through-modality-agnostic-tokenization — Attention's validated universality — grounded in its structural computational richness (asymmetric, position-dependent, learned scaling) — extends to arbitrary modalities through a common tokenization abstraction: ViT converting image patches to token sequences, AlphaFold processing protein residues, and Decision Transformer treating RL trajectories as sequences all demonstrate that attention's universality is not domain-specific but architecture-inherent, requiring only a tokenization bridge to apply.