moe-activates-subset-of-parameters
IN premise — entries/2026/06/21/wiki-Large_language_model-chunk-1.md
Created 2026-06-21T09:50:09+00:00
Mixture of Experts (MoE) architecture activates only a subset of parameters per input token via a gating mechanism, reducing inference cost relative to dense models of equivalent total parameter count.
Summary
Instead of processing each token through the entire model, MoE routes it through only a small group of specialized sub-networks, so the system can store a very large model while computing just a slice of its weights per step. The practical upshot is that you get near-large-model quality at a fraction of the inference cost you would pay for a dense model with the same total size.
Dependents
These beliefs depend on this one:
- IN compute-routing-operates-at-multiple-architectural-granularities — Compute routing in LLMs operates at two distinct architectural granularities: intra-model routing (MoE activating expert subnetwork subsets per token, as in Llama 4) and inter-model routing (GPT-5 selecting between fast and reasoning models per task), demonstrating that conditional computation is a general efficiency principle applicable at both the parameter level and the model-selection level.
- IN moe-enables-parameter-scaling-without-inference-cost — Mixture-of-Experts decouples total parameter count from inference cost by activating only a subset per token, as demonstrated by Llama 4's architecture where Scout activates 17B of 109B total and Maverick activates 17B of 400B total.