llama4-moe-scout-maverick-params

IN premise — entries/2026/06/21/wiki-LLaMA-chunk-1.md

Created 2026-06-21T09:50:09+00:00

Llama 4 uses mixture-of-experts: Scout has 109B total / 17B active parameters with 16 experts; Maverick has 400B total / 17B active with 128 experts

Summary

Llama 4 models are much larger than they look when you're actually running them: each token only activates a small slice of the total parameters (17B out of 109B for Scout, 17B out of 400B for Maverick), routed through a pool of specialist sub-networks. For the system, this means headline parameter counts massively overstate inference cost, and the two models share the same per-token compute budget despite Maverick being nearly four times larger in total capacity.

Dependents

These beliefs depend on this one:

IN moe-enables-parameter-scaling-without-inference-cost — Mixture-of-Experts decouples total parameter count from inference cost by activating only a subset per token, as demonstrated by Llama 4's architecture where Scout activates 17B of 109B total and Maverick activates 17B of 400B total.