Contamination Map Methodology

Text / LLM / Dataset

MathVista

6,141 examples drawn from 28 existing multimodal datasets (9 math-targeted QA, 19 VQA; up to 400 examples each) plus 3 newly created datasets (IQTest, FunctionQA, PaperQA; 736 new examples). Splits: testmini 1,000 (answers public; the split almost all reports use) and test 5,141 (answer labels not publicly released; leaderboard submission by email).

Also known as
MathVista, MathVista (testmini), MathVista-Mini, MathVista testmini, mathvista_testmini, AI4Math/MathVista
Domain
mathematical reasoning in visual contexts
License
CC BY-SA 4.0 for the new contributions; copyright of images and questions stays with the original sources; the card says it may be used as a test set but not as a training set
Homepage
mathvista.github.io

Related datasets

Exposure by model

BenchmarkModelLevelReason
Multimodal LLM benchmarks Claude 3.5 Sonnet Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude 3.7 Sonnet Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude 3 (Opus, Sonnet, Haiku) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 4 and Claude Sonnet 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Fable 5 / Claude Mythos 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Mythos Preview Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 4.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Sonnet 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 1.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 2.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 3 Pro Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 4 Argon Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3n Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-4o Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-5.2 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks InternVL3.5 Related data seen in fine-tuning data: related dataset ChartQA was used
Multimodal LLM benchmarks InternVL3 Related data seen related dataset ChartQA was used
Multimodal LLM benchmarks Llama 3.2-Vision (11B, 90B) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Llama 4 (Scout, Maverick) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks LLaVA-OneVision-1.5 Related data seen in fine-tuning data: related dataset DocVQA was used
Multimodal LLM benchmarks LLaVA-OneVision Related data seen in fine-tuning data: related dataset ChartQA was used
Multimodal LLM benchmarks Mistral Large 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Medium 3.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Small 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Molmo (0924) Related data seen in fine-tuning data: related dataset ChartQA was used
Multimodal LLM benchmarks Molmo2 Related data seen in fine-tuning data: related dataset ChartQA was used
Multimodal LLM benchmarks OpenAI o1 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks OpenAI o3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Pixtral 12B Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Pixtral Large Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen2.5-VL Unknown unknown for its base model Qwen2.5: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3-VL Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3.5 Unknown training data not disclosed in enough detail

Notes

Most items come from older public datasets (several with public train/val/test files), so published_from is null. Each source is capped at 400 examples: 'Each source dataset is limited to up to 400 examples to ensure a balanced representation of each source in our final compiled benchmark.' (paper §2). A 2024 study found that at least 6.6% of MathVista questions use images from academic training sets often used in SFT (see open_questions).

Sources