Contamination MapText / LLM / Dataset
MMLU (Measuring Massive Multitask Language Understanding)
57-subject 4-way multiple-choice exam questions (STEM, humanities, social sciences, other), collected by students from freely available online sources such as GRE and USMLE practice questions. The paper reports 15,908 questions: 5 per subject few-shot dev set, 1,540 validation and 14,079 test questions; the cais/mmlu HF release has dev 285 / validation 1,531 / test 14,042 (plus a 99,842-question auxiliary_train split drawn from ARC, MCTest, OBQA, RACE).
- Also known as
- MMLU, Massive Multitask Language Understanding, Measuring Massive Multitask Language Understanding, hendrycksTest
- Domain
- knowledge
- License
- MIT (cais/mmlu HF dataset card)
- Homepage
- github.com
Exposure by model
| Benchmark | Model | Level | Reason |
|---|---|---|---|
| Classic LLM benchmarks | LLM360 Amber | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | BLOOM | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude 3.5 Sonnet | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude 3.7 Sonnet | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude 3 (Opus, Sonnet, Haiku) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Opus 4 and Claude Sonnet 4 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Fable 5 / Claude Mythos 5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Mythos Preview | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Opus 4.5 | Not seen | the authors state its test portion was excluded from training |
| Classic LLM benchmarks | Claude Opus 5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Sonnet 5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DCLM-7B-IT (DCLM-IT-7B) | Test data seen | a contamination analysis found test examples in the pre-training data (DCLM's MMLU overlap rule (question last sentence + one answer option on the same page; recall-oriented, likely overestimate) flagged 0.007% of DCLM-Baseline documents (vs 0.001% Dolma-V1.7, 0.009% FineWeb-Edu). Removing the flagged overlaps did not lower MMLU accuracy at 7B-2x scale (51.8 -> 52.7).) (via DCLM-Baseline 1.0) |
| Classic LLM benchmarks | DCLM-7B (DCLM-Baseline-7B) | Test data seen | a contamination analysis found test examples in the pre-training data (DCLM's MMLU overlap rule (question last sentence + one answer option on the same page; recall-oriented, likely overestimate) flagged 0.007% of DCLM-Baseline documents (vs 0.001% Dolma-V1.7, 0.009% FineWeb-Edu). Removing the flagged overlaps did not lower MMLU accuracy at 7B-2x scale (51.8 -> 52.7).) (via DCLM-Baseline 1.0) |
| Classic LLM benchmarks | DeepSeek-R1-0528 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-R1-Distill | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-R1 | Not seen | the authors state its test portion was excluded from training |
| Classic LLM benchmarks | DeepSeek-V3-0324 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V3.1 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V3.2-Exp | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V3.2 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V4.1-Flash | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V4 (Pro / Flash) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemini 1.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemini 2.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemini 3 Pro | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemini 4 Argon | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemma 2 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemma 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemma 3n | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemma 4 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | GPT-4 | Test data seen | a contamination analysis found test examples in the pre-training data (~0.6% of 1000 sampled MMLU test examples matched GPT-4 pre-training data (50-character substring match)) |
| Classic LLM benchmarks | GPT-4o | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | GPT-5.2 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | GPT-5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | GPT-6 Astra | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | gpt-oss | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | InternVL3.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Classic LLM benchmarks | InternVL3 | Unknown | unknown for its base model Qwen2.5: training data not disclosed in enough detail |
| Classic LLM benchmarks | LLM360 K2-65B (K2 Diamond) | Test data seen | a contamination analysis found test examples in the pre-training data (MMLU test questions are also substantially 'dirty' in Pile-train: DCLM-baseline (27.70%) has only 1.1x more dirty entries than Pile-train. Exact Pile-train rate is in Table 2 (not reliably extracted from the PDF text).) [third-party study] (via The Pile) |
| Classic LLM benchmarks | K2-Horizon | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | K2-V2 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Llama 2 | Test data seen | a contamination analysis found test examples in the pre-training data (MMLU-Humanities (L=50): 709 of 4,705 samples (15%) 'not clean' (>=20% token contamination) and 520 (11%) 'dirty' (>=80%); flagged as boosted by contamination. MMLU-Overall for 70B: 2,180 of 14,042 (15.5%) not clean, 1,536 (10.9%) dirty, with a small effect on the overall score (-0.9 clean-vs-mean delta).) |
| Classic LLM benchmarks | Llama 3.1 | Test data seen | a contamination analysis found test examples in the pre-training data (8-gram overlap gave contamination scores too high to estimate a performance gain, even at higher thresholds; percentage not reported (dash in Table 15).) |
| Classic LLM benchmarks | Llama 3.2-Vision (11B, 90B) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Llama 3.2 (1B, 3B) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Llama 3.3 70B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Meta Llama 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Llama 4 (Scout, Maverick) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | LLaVA-OneVision-1.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Classic LLM benchmarks | LLaVA-OneVision | Unknown | unknown for its base model SigLIP: training data not disclosed in enough detail |
| Classic LLM benchmarks | Ministral 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mistral 7B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mistral Large 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mistral Medium 3.5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mistral Small 4 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mixtral 8x7B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Molmo (0924) | Unknown | unknown for its base model OpenAI CLIP: training data not disclosed in enough detail |
| Classic LLM benchmarks | Molmo2 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Classic LLM benchmarks | OpenAI o1 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | OpenAI o3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | OLMo (February 2024) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | OLMo 2 (November 2024) | Test data seen | a contamination analysis found test examples in the pre-training data (DCLM's MMLU overlap rule (question last sentence + one answer option on the same page; recall-oriented, likely overestimate) flagged 0.007% of DCLM-Baseline documents (vs 0.001% Dolma-V1.7, 0.009% FineWeb-Edu). Removing the flagged overlaps did not lower MMLU accuracy at 7B-2x scale (51.8 -> 52.7).) (via DCLM-Baseline 1.0) |
| Classic LLM benchmarks | Olmo 3.1 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Olmo 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Olmo Hybrid 7B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-3.5 (mini / MoE) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-3 (mini / small / medium) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4-mini-flash-reasoning | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4-mini-reasoning | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4-mini | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4-reasoning / Phi-4-reasoning-plus | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4 | Not seen | the authors state its test portion was excluded from training |
| Classic LLM benchmarks | Phi-mini-MoE / Phi-tiny-MoE (SlimMoE) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Pixtral 12B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Pixtral Large | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Pythia (deduplicated) | Test data seen | a contamination analysis found test examples in the pre-training data (MMLU test questions are also substantially 'dirty' in Pile-train: DCLM-baseline (27.70%) has only 1.1x more dirty entries than Pile-train. Exact Pile-train rate is in Table 2 (not reliably extracted from the PDF text).) [third-party study] (via The Pile) |
| Classic LLM benchmarks | Pythia | Test data seen | a contamination analysis found test examples in the pre-training data (MMLU test questions are also substantially 'dirty' in Pile-train: DCLM-baseline (27.70%) has only 1.1x more dirty entries than Pile-train. Exact Pile-train rate is in Table 2 (not reliably extracted from the PDF text).) [third-party study] (via The Pile) |
| Classic LLM benchmarks | Qwen2.5-VL | Unknown | unknown for its base model Qwen2.5: training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen2.5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3-2507 (Instruct-2507 / Thinking-2507) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3-Next | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3-VL | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3.5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3.6 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3.8-Flash-Next | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3.8 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | SmolLM2 | Test data seen | a contamination analysis found test examples in the pre-training data (DCLM's MMLU overlap rule (question last sentence + one answer option on the same page; recall-oriented, likely overestimate) flagged 0.007% of DCLM-Baseline documents (vs 0.001% Dolma-V1.7, 0.009% FineWeb-Edu). Removing the flagged overlaps did not lower MMLU accuracy at 7B-2x scale (51.8 -> 52.7).) (via DCLM-Baseline 1.0); this contradicts a statement that the data was excluded |
| Classic LLM benchmarks | SmolLM3 | Unknown | training data not disclosed in enough detail |
Notes
Test-set size differs between the paper (14,079) and the HF release (14,042). The README asks users to also cite ETHICS, which the test draws from (moral scenarios); ETHICS is not a registry dataset. Copies found in public web data by third-party studies: Common Crawl: 4077 of 13987 MMLU test examples (29.1%) flagged as present in Common Crawl (Oct 2020-Oct 2023 window); 3399 (24.3%) with question and answer.; Llama 1 pre-training corpus: Substantial share of MMLU marked contaminated in the Llama 1 pre-training corpus (Figure 3a), with significant estimated performance gain.; FLAN CoT: 76 MMLU test questions (0.543%) have rephrased counterparts in the CoT subset of FLAN.
Sources
- Measuring Massive Multitask Language Understanding (arXiv v3) · paper · retrieved 2026-10-15
- hendrycks/test README · readme · retrieved 2026-10-15
- cais/mmlu dataset card · dataset card · retrieved 2026-10-15