evaluation-landscape-spans-intrinsic-and-extrinsic-paradigms

IN derived (depth 1)

Created 2026-06-21T13:22:51+00:00 · Reviewed 2026-06-21T14:41:08+00:00

LLM evaluation spans from intrinsic information-theoretic metrics (perplexity as the exponential of average negative log-likelihood) through multi-dimensional benchmarks (HELM evaluating accuracy, calibration, robustness, fairness, and other dimensions) to task-specific benchmarks and rankings (MMLU, Humanity's Last Exam, LMArena) — reflecting a landscape where multiple evaluation paradigms coexist, though the relationship between intrinsic quality metrics and extrinsic task performance is not explicitly characterized by these benchmarks alone.

Summary

LLM evaluation is fragmented across many incompatible frameworks — from raw statistical fit (perplexity), to broad multi-axis audits (HELM), to competitive leaderboards (MMLU, LMArena) — and none of these frameworks tells you how a score on one maps to a score on another. The practical implication is that "model quality" is not a single dial; a system reasoning about model capabilities must treat these as separate, only loosely correlated axes rather than assuming one number captures the whole picture.

Justifications

SL — Evaluation has diversified across paradigms without theoretical unification of intrinsic and extrinsic measures

Antecedents (all must be IN):

  • IN perplexity-exponential-avg-neg-loglik — Perplexity is the exponential of the average negative log-likelihood per token; lower perplexity indicates better prediction of a text corpus.
  • IN helm-multi-metric-benchmark — HELM (Holistic Evaluation of Language Models) from Stanford CRFM evaluates models across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency
  • IN llm-benchmarks-mmlu-hle-lmarena — Key LLM benchmarks and evaluation methods include MMLU, Humanity's Last Exam, LMArena, LLM-as-a-Judge, and perplexity.