AI Benchmark Decoder

What It Actually Tests

MMLU presents 14,042 multiple-choice questions spanning 57 subjects — from abstract algebra and anatomy to US history and moral scenarios. It measures world knowledge breadth encoded in the model's weights during pre-training. A model scoring 90% correctly answers obscure academic trivia across disciplines.

Why It's Now Misleading

• ✗Contamination risk: Questions are publicly available. Models trained on web data may have memorized answers rather than reasoning to them.
• ✗Saturation: Frontier models cluster around 85–90%. Differences within that band don't predict real task quality.
• ✗No reasoning required: Multiple-choice eliminates the need for generation — a poor proxy for real use cases.
• ✓Still useful for: Comparing small vs. large models, evaluating knowledge acquisition from pre-training data.

What It Actually Tests

GPQA questions are authored by domain experts (PhDs) and designed to be Google-proof — searching online cannot easily surface the answer. They require multi-step reasoning across graduate-level biology, physics, and chemistry. Non-expert humans (including PhDs outside the domain) score ~34%. Most frontier models as of mid-2024 score 50–60%.

Why It Matters

• ✓Contamination-resistant: Novel reasoning required — can't be memorized from the web.
• ✓Differentiating: Only the strongest reasoning models exceed 60%. Meaningful spread between frontier models.
• ✓Chain-of-thought proxy: Models need multi-step reasoning chains to score well — a real capability signal.
• ✗Narrow domain: Science only. Not predictive of code, enterprise language, or instruction-following quality.

What It Actually Tests

SWE-bench provides real GitHub issue descriptions from 12 popular Python repositories (Django, scikit-learn, matplotlib, etc.) and asks models to generate a code patch that resolves the issue. Success is determined by running the repository's existing test suite — the patch must make the failing tests pass without breaking others. This is not code completion — it's autonomous software engineering.

Why It's The Most Practical Coding Benchmark

• ✓Real-world tasks: Issues are from production codebases. Success on SWE-bench correlates with real engineering productivity gains.
• ✓Objective scoring: Test suites either pass or they don't. No human evaluation bias.
• ✓Differentiating: Huge spread between models (10%–50%). Clearly ranks coding ability.
• ✗Python-only: Doesn't cover TypeScript, Go, Java, or other languages used in enterprise stacks.
• ✗Scaffold-dependent: Results vary wildly based on agentic scaffolding (tools, context window usage).

What It Actually Tests

HumanEval presents 164 Python function stubs (docstring + signature) and asks the model to complete the function body. Correctness is verified by running hidden unit tests. Problems range from string manipulation to simple algorithms. It was revolutionary in 2021 when GPT-3 scored ~0% and Codex scored 28.8% — the first benchmark to measure code generation at all.

def has_close_elements(numbers, threshold):
    """ Check if list has any two numbers
    closer than the given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0], 0.3)
    True
    """
    # Model must complete this function...

Why It's Now Limited

• ✗Saturated: Modern models score 85–95%+. There is almost no signal between frontier models.
• ✗Too simple: 164 problems, single functions, no multi-file context, no dependencies. Not representative of real engineering work.
• ✗Contamination: Problems are public and widely discussed in model training data.
• ✓Still useful for: Ranking small/fine-tuned models and quick sanity checks on code generation capability.

What It Actually Tests

MATH uses 12,500 competition math problems sourced from AMC, AIME, and other olympiad sources — levels 1 (easy) through 5 (extremely hard). Answers are in LaTeX format and require exact match. Unlike MMLU's multiple choice, MATH requires the model to derive answers from scratch through multi-step algebraic, geometric, or number-theoretic reasoning chains.

Why Researchers Use It

• ✓Exact-match grading: No ambiguity. LaTeX answer matches or it doesn't.
• ✓Scales difficulty: Level 1–5 allows fine-grained measurement across model capability tiers.
• ✓Chain-of-thought sensitive: Models that reason step-by-step score dramatically higher — a clear CoT signal.
• ✗Domain narrow: Strong MATH score doesn't predict quality on language, code, or instruction following tasks.