compute-routing-operates-at-multiple-architectural-granularities
IN derived (depth 1)
Created 2026-06-21T12:50:29+00:00 · Reviewed 2026-06-21T14:41:08+00:00
Compute routing in LLMs operates at two distinct architectural granularities: intra-model routing (MoE activating expert subnetwork subsets per token, as in Llama 4) and inter-model routing (GPT-5 selecting between fast and reasoning models per task), demonstrating that conditional computation is a general efficiency principle applicable at both the parameter level and the model-selection level.
Summary
The same efficiency idea — don't run everything, just run what the input actually needs — shows up at two completely different levels: picking which internal subnetworks to activate for a single token, and picking which entire model to call for a given task. Treating this as one unified principle rather than a single-architecture detail means the system should think about compute allocation as a multi-level design problem, where the routing logic at the parameter layer and the model-selection layer are two expressions of the same underlying constraint.
Justifications
SL — Routing at two granularities (within-model experts and between-model selection) suggests conditional computation is a fundamental architectural pattern, not a specific technique
Antecedents (all must be IN):
- IN moe-activates-subset-of-parameters — Mixture of Experts (MoE) architecture activates only a subset of parameters per input token via a gating mechanism, reducing inference cost relative to dense models of equivalent total parameter count.
- IN gpt5-router-fast-vs-reasoning — GPT-5 (August 7, 2025) introduced a router that automatically selects between a fast model and a slower reasoning model based on task complexity
Dependents
These beliefs depend on this one:
- IN distillation-and-routing-jointly-decouple-capability-from-cost — Knowledge distillation (validated across the full scale spectrum, from 110M to 2T parameters) and multi-granularity compute routing (intra-model MoE and inter-model task routing) together decouple capability from inference cost along two independent axes: distillation compresses capability into smaller models, while routing allocates compute only where needed — together enabling deployment of frontier-class capability at a fraction of nominal cost.