Contamination Map Methodology

Text / LLM / Dataset

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning)

11.5K multimodal college-level questions (multiple-choice and open) collected by ~50 college students from textbooks, online sources and lecture materials, across 6 disciplines, 30 subjects, 183 subfields and 30+ image types. Splits: dev 150 (5 per subject, few-shot), validation 900, test 10,500. The validation set (900) is what most reports use. Test answers were withheld behind an EvalAI server at release (Nov 2023), but the authors released them on 2026-02-12, so all splits now have public answers.

Also known as
MMMU, MMMU (val), MMMU-Val, MMMU val, mmmu_val, MMMU/MMMU
Domain
multi-discipline college-level knowledge and reasoning (image + text)
License
Apache-2.0 (MMMU/MMMU HF dataset card metadata)
Homepage
mmmu-benchmark.github.io

Exposure by model

BenchmarkModelLevelReason
Multimodal LLM benchmarks Claude 3.5 Sonnet Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude 3.7 Sonnet Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude 3 (Opus, Sonnet, Haiku) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 4 and Claude Sonnet 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Fable 5 / Claude Mythos 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Mythos Preview Related data seen a contamination analysis found test examples of the related dataset MMMU-Pro (A large fraction of MMMU-Pro images were found in the training data, mainly via textbooks, homework-help sites and document crawls. The developer omitted MMMU-Pro results from the system card.)
Multimodal LLM benchmarks Claude Opus 4.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Sonnet 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 1.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 2.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 3 Pro Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 4 Argon Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3n Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-4o Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-5.2 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks InternVL3.5 Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks InternVL3 Unknown unknown for its base model Qwen2.5: training data not disclosed in enough detail
Multimodal LLM benchmarks Llama 3.2-Vision (11B, 90B) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Llama 4 (Scout, Maverick) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks LLaVA-OneVision-1.5 Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks LLaVA-OneVision Unknown unknown for its base model SigLIP: training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Large 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Medium 3.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Small 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Molmo (0924) Unknown unknown for its base model OpenAI CLIP: training data not disclosed in enough detail
Multimodal LLM benchmarks Molmo2 Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks OpenAI o1 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks OpenAI o3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Pixtral 12B Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Pixtral Large Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen2.5-VL Unknown unknown for its base model Qwen2.5: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3-VL Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3.5 Unknown training data not disclosed in enough detail

Notes

Questions were collected from existing textbooks/online material (annotators were told to prefer questions whose answers are not immediately available online), so individual items can predate the Nov 2023 release; published_from is therefore left null. Test answers public since 2026-02-12 (README news: 'We have released the answers for the MMMU test set!').

Sources