multimodal-fusion-early-vs-intermediate
IN premise — entries/2026/06/21/wiki-Large_language_model-chunk-2.md
Created 2026-06-21T09:50:09+00:00
Two multimodal fusion strategies exist: early fusion (combine modality embeddings before prediction) and intermediate fusion (process modalities independently, then fuse via cross-attention, e.g., Flamingo).
Summary
When a system takes in multiple kinds of information (like text and images), there are two fundamentally different ways to mix them: slam everything together at the start, or let each type do its own processing first and then merge the results partway through. This choice shapes everything downstream, because it determines how much each modality can inform the other before a final decision is made.
Dependents
These beliefs depend on this one:
- IN modality-is-tokenization-choice-not-architectural-decision — The coexistence of multiple multimodal fusion strategies (early vs intermediate) alongside attention's modality-agnostic universality suggests that modality functions primarily as a tokenization design choice — how to convert domain data into token sequences — rather than an architectural constraint, since the attention mechanism can process all token sequences through a common abstraction regardless of whether they originated as text, image patches, or other domain data.