Contamination MapText / LLM / Dataset
MMMU-Pro
Harder re-filtering of MMMU: questions answerable by text-only LLMs were removed, candidate options were augmented from 4 to 10, and a 'vision' setting embeds each question in a screenshot/photo. 1,730 questions, each in 'standard (10 options)' and 'vision' form (3,460 total), plus a 'standard (4 options)' config; on HF all configs have a single 'test' split of 1,730 with the answer field included (answers public). 577 of the questions come from the MMMU validation set (the rest from other MMMU questions).
- Also known as
- MMMU-Pro, MMMU Pro, MMMU_Pro, MMMU-Pro (standard), MMMU-Pro (vision), MMMU/MMMU_Pro
- Domain
- multi-discipline college-level knowledge and reasoning (image + text)
- License
- Apache-2.0 (MMMU/MMMU_Pro HF dataset card metadata)
- Homepage
- mmmu-benchmark.github.io
Related datasets
-
derived from MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning)
Built from MMMU questions (filtered, options augmented to 10, plus screenshot/photo renderings). The paper states 577 of its questions come from the MMMU validation set; it does not say outright that the rest come from the MMMU test set.
Evidence
“we introduce MMMU-Pro, a more robust and challenging version of the MMMU benchmark.”
arxiv.org · paper · Section 1 Introduction · retrieved 2026-10-15 “Given that the 577 questions in MMMU-Pro are sourced from the MMMU validation set”
arxiv.org · paper · Appendix (human performance) · retrieved 2026-10-15 “It builds upon the original MMMU benchmark by introducing several key improvements that make it more challenging and realistic”
huggingface.co · dataset card · Introduction · retrieved 2026-10-15
Exposure by model
| Benchmark | Model | Level | Reason |
|---|---|---|---|
| Multimodal LLM benchmarks | Claude 3.5 Sonnet | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Claude 3.7 Sonnet | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Claude 3 (Opus, Sonnet, Haiku) | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Claude Opus 4 and Claude Sonnet 4 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Claude Fable 5 / Claude Mythos 5 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Claude Mythos Preview | Test data seen | a contamination analysis found test examples in the training data (A large fraction of MMMU-Pro images were found in the training data, mainly via textbooks, homework-help sites and document crawls. The developer omitted MMMU-Pro results from the system card.) |
| Multimodal LLM benchmarks | Claude Opus 4.5 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Claude Opus 5 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Claude Sonnet 5 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Gemini 1.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Gemini 2.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Gemini 3 Pro | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Gemini 4 Argon | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Gemma 3 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Gemma 3n | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Gemma 4 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | GPT-4 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | GPT-4o | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | GPT-5.2 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | GPT-5 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | InternVL3.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Multimodal LLM benchmarks | InternVL3 | Unknown | unknown for its base model Qwen2.5: training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Llama 3.2-Vision (11B, 90B) | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Llama 4 (Scout, Maverick) | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | LLaVA-OneVision-1.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Multimodal LLM benchmarks | LLaVA-OneVision | Unknown | unknown for its base model SigLIP: training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Mistral Large 3 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Mistral Medium 3.5 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Mistral Small 4 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Molmo (0924) | Unknown | unknown for its base model OpenAI CLIP: training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Molmo2 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Multimodal LLM benchmarks | OpenAI o1 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | OpenAI o3 | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Pixtral 12B | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Pixtral Large | Unknown | training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Qwen2.5-VL | Unknown | unknown for its base model Qwen2.5: training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Qwen3-VL | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Multimodal LLM benchmarks | Qwen3.5 | Unknown | training data not disclosed in enough detail |
Notes
Question stems are MMMU items (Nov 2023 or earlier sources), so published_from is null even though the 10-option and vision versions date from Sep 2024. Overall MMMU-Pro scores are usually reported as standard (10 options) and vision; lmms-eval has separate mmmu_pro_standard and mmmu_pro_vision tasks.
Sources
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (arXiv v3) · paper · retrieved 2026-10-15
- MMMU/MMMU_Pro dataset card · dataset card · retrieved 2026-10-15
- MMMU-Benchmark/MMMU README · readme · retrieved 2026-10-15