MMODELYST
Evals/MMLU
EVL

MMLU

Massive Multitask Language Understanding

Source ↗
benchmarkKnowledge & QAAccuracy (% of multiple-choice questions answered correctly)14K items
What this measures

How much a model simply *knows* — broad academic knowledge across 57 subjects, from history and law to physics, medicine, and economics.

Sample
Which body cavity contains the pituitary gland? (A) Abdominal (B) Cranial (C) Pleural (D) Spinal → Answer: B
Leaderboard · 17 models
Accuracy (% of multiple-choice questions answered correctly) — higher is better
1o1
91.8
2Claude 3.5 Sonnet
88.7
3GPT-4o
88.7
4DeepSeek-V3
88.5
5Grok-2
87.5
6Claude 3 Opus
86.8
7Qwen2.5-72B-Instruct
86.1
8Llama-3.1-70B-Instruct
86
9Llama-3.3-70B-Instruct
86
10Gemini 1.5 Pro
85.9
11GPT-4o mini
82
12gemma-2-27b-it
75.2
13Qwen2.5-7B-Instruct
74.2
14gemma-2-9b-it
71.3
15Llama-3.1-8B-Instruct
69.4
16Phi-3.5-mini-instruct
69
17Mistral-7B-Instruct-v0.3
62.5
Benchmark health
active

Discriminating but climbing — the frontier is moving up this benchmark.

87.6
top-10 mean
91.8
best
17
models
What improves this score

Datasets & environments shown to raise it.

—+5.0
FineWeb-Edu+4.0
OpenHermes 2.5+2.5
Tülu 3 SFT Mixture+2.0
UltraFeedback+1.5
Research using this benchmark
all →
NITP: Next Implicit Token Prediction for LLM Pre-trainingHolistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement LearningWho Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMsAC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM PretrainingThe Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure

Matched by name in title/abstract.