Contamination Map Methodology

Text / LLM / Dataset

DocVQA

50,000 extractive questions on 12,767 scanned document images (industry documents from the UCSF Industry Documents Library, dated 1900-2018, mostly 1960-2000); questions and answers written by annotators. Random 80/10/10 split: train 39,463 questions, validation 5,349, test 5,188. Metric ANLS. Test answers are not in public copies (lmms-lab HF copy has answers: null for test; lmms-eval scores docvqa_test with a 'submission' metric for the RRC evaluation server); reports cite either test (server) or validation.

Also known as
DocVQA, DocVQA (test), DocVQA (val), SP-DocVQA, Single Page Document VQA, docvqa_test, docvqa_val
Domain
document image question answering
License
Not stated on docvqa.org or in the paper; the lmms-lab HF copy declares apache-2.0 (copy metadata only)
Homepage
www.docvqa.org

Exposure by model

BenchmarkModelLevelReason
Multimodal LLM benchmarks Claude 3.5 Sonnet Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude 3.7 Sonnet Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude 3 (Opus, Sonnet, Haiku) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 4 and Claude Sonnet 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Fable 5 / Claude Mythos 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Mythos Preview Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 4.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Opus 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Claude Sonnet 5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 1.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 2.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 3 Pro Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemini 4 Argon Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 3n Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Gemma 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-4o Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-5.2 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks GPT-5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks InternVL3.5 Seen, split unclear in fine-tuning data: the dataset was used; the source does not say which part
Multimodal LLM benchmarks InternVL3 Seen, split unclear the dataset was used; the source does not say which part
Multimodal LLM benchmarks Llama 3.2-Vision (11B, 90B) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Llama 4 (Scout, Maverick) Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks LLaVA-OneVision-1.5 Training data seen in fine-tuning data: only the training split was used, not the benchmark's test data
Multimodal LLM benchmarks LLaVA-OneVision Training data seen in fine-tuning data: only the training split was used, not the benchmark's test data
Multimodal LLM benchmarks Mistral Large 3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Medium 3.5 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Mistral Small 4 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Molmo (0924) Training data seen in fine-tuning data: only the training split was used, not the benchmark's test data; the test portion was explicitly excluded
Multimodal LLM benchmarks Molmo2 Training data seen in fine-tuning data: only the training split was used, not the benchmark's test data
Multimodal LLM benchmarks OpenAI o1 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks OpenAI o3 Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Pixtral 12B Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Pixtral Large Unknown training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen2.5-VL Unknown unknown for its base model Qwen2.5: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3-VL Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Multimodal LLM benchmarks Qwen3.5 Unknown training data not disclosed in enough detail

Notes

First released as the 2020 DocVQA challenge Task 1 dataset (docvqa.org: 'DocVQA dataset (2020 Challenge task 1 dataset)'); exact first public release date not documented in the sources read, so published_from is null. The underlying documents are old public archive scans.

Sources