llama-3-1-400b-moe-model-jul-2024

Summary

Meta released an open-weights model in July 2024 with 400 billion parameters total but a Mixture-of-Experts design, meaning only a fraction of those parameters activate for any given word, so it delivers near-frontier quality at a fraction of the compute cost of a same-size dense model. This matters because it set a new bar for what an open, locally-runnable LLM could achieve and made that capability widely accessible rather than locked behind a single API provider.