distillation-validated-across-full-scale-spectrum

IN derived (depth 1)

Created 2026-06-21T12:50:29+00:00 · Reviewed 2026-06-21T14:41:08+00:00

Knowledge distillation is validated as a scale-invariant capability across the full spectrum of language model sizes: from BERT-scale (DistilBERT retaining 95% performance at 60% of parameters) to frontier-scale (Llama 4 Maverick codistilled from the unreleased ~2T-parameter Behemoth), demonstrating that larger models reliably compress their capability into smaller ones regardless of absolute scale.

Justifications

SL — Distillation working at both ends of the scale spectrum (110M→66M and ~2T→400B) validates it as a general property of neural language models, not a small-model artifact

Antecedents (all must be IN):

IN distilbert-95pct-performance-60pct-params — DistilBERT (2019) retains 95% of BERT's benchmark performance with 60% of its parameters (66M vs 110M) via knowledge distillation
IN maverick-codistilled-from-behemoth — Llama 4 Maverick was codistilled from the unreleased Behemoth model (~2T total parameters), while Scout was trained from scratch

Dependents

These beliefs depend on this one:

IN distillation-and-routing-jointly-decouple-capability-from-cost — Knowledge distillation (validated across the full scale spectrum, from 110M to 2T parameters) and multi-granularity compute routing (intra-model MoE and inter-model task routing) together decouple capability from inference cost along two independent axes: distillation compresses capability into smaller models, while routing allocates compute only where needed — together enabling deployment of frontier-class capability at a fraction of nominal cost.
OUT distillation-enables-safe-capability-cascading — Knowledge distillation's scale-invariant validation (from 110M DistilBERT to 2T Behemoth codistillation) combined with parameter redundancy evidence (95% performance at 60% parameters) enables safe capability cascading — frontier capabilities can systematically flow to smaller, more deployable models while maintaining quality.
OUT inference-democratization-enables-safe-distributed-deployment — Democratized inference — CPU-only execution and single-executable distribution eliminating hardware and installation barriers — enables safe distributed deployment of capable models, lowering the expertise threshold for responsible use.