Contamination Map Methodology

Text / LLM / Benchmark

Multimodal LLM benchmarks

Commonly reported multimodal LLM benchmarks; grouping used on this site, not an official suite. Covers multi-discipline expert QA (MMMU, MMMU-Pro), visual math (MathVista), charts (ChartQA, CharXiv), documents (DocVQA) and science diagrams (AI2D).

Test split: Split usually evaluated (as configured in lmms-eval): MMMU validation (900; mmmu_val; test 10,500 answers public since 2026-02-12); MMMU-Pro the single 1,730-question test split in 'standard (10 options)' and 'vision' configs (answers public); MathVista testmini (1,000; answers public; test 5,141 answers private); ChartQA test (2,500 = 1,250 human + 1,250 augmented; public); DocVQA test (5,188; answers private, scored by submission to the RRC server) or validation (5,349; public); AI2D test (3,088 in lmms-lab/ai2d; all AI2D answers are in the public download); CharXiv validation (1,000 charts; reasoning and descriptive tasks; test answers private).

Contamination matrix

Dataset Claude 3 (Opus, Sonnet, Haiku)Claude 3.5 SonnetClaude 3.7 SonnetClaude Fable 5 / Claude Mythos 5Claude Mythos PreviewClaude Opus 4 and Claude Sonnet 4Claude Opus 4.5Claude Opus 5Claude Sonnet 5Gemini 1.5 (Pro, Flash)Gemini 2.5 (Pro, Flash)Gemini 3 ProGemini 4 ArgonGemma 3Gemma 3nGemma 4GPT-4GPT-4oGPT-5GPT-5.2InternVL3InternVL3.5Llama 3.2-Vision (11B, 90B)Llama 4 (Scout, Maverick)LLaVA-OneVisionLLaVA-OneVision-1.5Mistral Large 3Mistral Medium 3.5Mistral Small 4Molmo (0924)Molmo2OpenAI o1OpenAI o3Pixtral 12BPixtral LargeQwen2.5-VLQwen3-VLQwen3.5
MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) ? ? ? ? Rel. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
MMMU-Pro ? ? ? ? Test ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
MathVista ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Rel. Rel. ? ? Rel. Rel. ? ? ? Rel. Rel. ? ? ? ? ? ? ?
ChartQA ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Seen Seen ? ? Train Seen ? ? ? Train Train ? ? ? ? ? ? ?
DocVQA ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Seen Seen ? ? Train Train ? ? ? Train Train ? ? ? ? ? ? ?
AI2D (AI2 Diagrams) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Seen Seen ? ? Train Seen ? ? ? Train Train ? ? ? ? ? ? ?
CharXiv No No ? ? Test ? ? ? ? No ? ? ? ? No ? No No ? ? ? ? No ? ? ? ? ? ? ? ? No No ? ? ? ? ?

Hover a cell for the reason; click it for the evidence.

Notes

lmms-eval (EvolvingLMMs-Lab) is used as the reference for 'usual' splits; individual model reports may differ (e.g., DocVQA val vs test, MMMU val vs test). The 'input_types' field is not in src/cmap/schemas.py Benchmark yet (extra='forbid'); added on request.

Sources