Contamination Map Methodology

Text / LLM / Dataset

CharXiv

2,323 real charts from arXiv preprints (Jan 2020 - Sep 2023, eight subjects), each with 4 descriptive questions and 1 reasoning question written and verified by human experts (10K+ questions). Split: validation 1,000 charts (QA pairs public) and test 1,323 charts (answers kept private; null in the public data). Reported 'CharXiv reasoning/descriptive' scores are normally on the validation split.

Also known as
CharXiv, CharXiv (reasoning), CharXiv-R, CharXiv reasoning, CharXiv (descriptive), CharXiv-D, charxiv_val_reasoning, princeton-nlp/CharXiv
Domain
scientific chart understanding
License
Questions CC BY-SA 4.0; chart copyright stays with the original authors; the card says the data is for evaluation only and must not be used for training
Test items from
2024-06-26
Evidence
“arXiv:2406.18521v1 [cs.CL] 26 Jun 2024”
arxiv.org · paper · arXiv v1 header · retrieved 2026-10-15
“we release all QA pairs for the validation set and keep the answers to the test set private to prevent data leakage.”
arxiv.org · paper · Section 3, footnote 4 · retrieved 2026-10-15
Homepage
charxiv.github.io

Exposure by model

BenchmarkModelLevelReason
Multimodal LLM benchmarks Claude 3.5 Sonnet Not seen its test items were published from 2024-06-26, after the model's stated training-data cutoff (2024-04-01)
Multimodal LLM benchmarks Claude 3.7 Sonnet Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude 3 (Opus, Sonnet, Haiku) Not seen its test items were published from 2024-06-26, after the model's stated training-data cutoff (2023-08-01)
Multimodal LLM benchmarks Claude Opus 4 and Claude Sonnet 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Fable 5 / Claude Mythos 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Mythos Preview Test data seen a contamination analysis found test examples in the pre-training data (The developer found the majority of CharXiv Reasoning question-answer text pairs in the full pretraining mix by exact-match search, even though evaluation images were filtered at image level. The evaluated items are 1,000 validation-split questions. On a 100-item manually perturbed remix, the developer judged memorisation effects limited.); this contradicts a statement that the data was excluded
Multimodal LLM benchmarks Claude Opus 4.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Sonnet 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 1.5 (Pro, Flash) Not seen its test items were published from 2024-06-26, after the model's stated training-data cutoff (2023-11-01)
Multimodal LLM benchmarks Gemini 2.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 3 Pro Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 4 Argon Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3n Not seen its test items were published from 2024-06-26, after the model's stated training-data cutoff (2024-06-01)
Multimodal LLM benchmarks Gemma 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-4 Not seen its test items were published from 2024-06-26, after the model's stated training-data cutoff (2021-09-01)
Multimodal LLM benchmarks GPT-4o Not seen its test items were published from 2024-06-26, after the model's stated training-data cutoff (2023-10-01)
Multimodal LLM benchmarks GPT-5.2 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks InternVL3.5 Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks InternVL3 Unknown unknown for its base model Qwen2.5: training data not disclosed in enough detail
Multimodal LLM benchmarks Llama 3.2-Vision (11B, 90B) Not seen its test items were published from 2024-06-26, after the model's stated training-data cutoff (2023-12-01)
Multimodal LLM benchmarks Llama 4 (Scout, Maverick) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks LLaVA-OneVision-1.5 Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks LLaVA-OneVision Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Large 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Medium 3.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Small 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Molmo (0924) Unknown unknown for its base model OpenAI CLIP: training data not disclosed in enough detail
Multimodal LLM benchmarks Molmo2 Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks OpenAI o1 Not seen its test items were published from 2024-06-26, after the model's stated training-data cutoff (2023-10-01)
Multimodal LLM benchmarks OpenAI o3 Not seen its test items were published from 2024-06-26, after the model's stated training-data cutoff (2024-06-01)
Multimodal LLM benchmarks Pixtral 12B Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Pixtral Large Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen2.5-VL Unknown unknown for its base model Qwen2.5: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3-VL Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3.5 Unknown training data not disclosed in enough detail

Notes

published_from is the arXiv v1 date: the question-answer pairs were newly written for CharXiv, but the chart images come from arXiv preprints published 2020-2023 and were public before that date.

Sources