Contamination Map Methodology

Text / LLM / Dataset

AI2D (AI2 Diagrams)

Over 5,000 grade-school science diagrams (scraped from Google Image Search with seed terms from Grade 1-6 science textbook chapter titles) with rich annotations and over 15,000 multiple-choice questions written via Amazon Mechanical Turk. The paper used a 4,000-image train / 1,000-image 'blind test set' split, but the public download (ai2d-all.zip, v1.0.0, Sept 2016) contains the correct answer for every question. LMM evaluations use a 3,088-question test split (lmms-lab/ai2d, used by lmms-eval).

Also known as
AI2D, AI2 Diagrams, AI2D (test), AI2D-Test, ai2d, lmms-lab/ai2d
Domain
grade-school science diagram question answering
License
AI2 non-commercial license (license.txt in ai2d-all.zip: academic/non-profit/government research use; may also be used for evaluation; no redistribution)

Exposure by model

BenchmarkModelLevelReason
Multimodal LLM benchmarks Claude 3.5 Sonnet Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude 3.7 Sonnet Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude 3 (Opus, Sonnet, Haiku) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 4 and Claude Sonnet 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Fable 5 / Claude Mythos 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Mythos Preview Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 4.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Sonnet 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 1.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 2.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 3 Pro Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 4 Argon Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3n Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-4o Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-5.2 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks InternVL3.5 Seen, split unclear in fine-tuning data: the dataset was used; the source does not say which part
Multimodal LLM benchmarks InternVL3 Seen, split unclear the dataset was used; the source does not say which part
Multimodal LLM benchmarks Llama 3.2-Vision (11B, 90B) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Llama 4 (Scout, Maverick) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks LLaVA-OneVision-1.5 Seen, split unclear in fine-tuning data: the dataset was used; the source does not say which part
Multimodal LLM benchmarks LLaVA-OneVision Training data seen in fine-tuning data: only the training split was used, not the benchmark's test data
Multimodal LLM benchmarks Mistral Large 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Medium 3.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Small 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Molmo (0924) Training data seen in fine-tuning data: only the training split was used, not the benchmark's test data; the test portion was explicitly excluded
Multimodal LLM benchmarks Molmo2 Training data seen in fine-tuning data: only the training split was used, not the benchmark's test data
Multimodal LLM benchmarks OpenAI o1 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks OpenAI o3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Pixtral 12B Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Pixtral Large Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen2.5-VL Unknown unknown for its base model Qwen2.5: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3-VL Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3.5 Unknown training data not disclosed in enough detail

Notes

The origin of the 3,088-question test split used in LMM evaluations is not documented in the sources read (see open_questions). published_from null (2016 benchmark).

Sources