Contamination MapText / LLM / Dataset
GPQA Diamond
Highest-quality GPQA subset: 198 questions where both expert validators answered correctly (or the second made a demonstrable mistake) and the majority of non-experts answered incorrectly.
- Also known as
- GPQA Diamond, GPQA-Diamond, GPQA-D, gpqa_diamond
- Domain
- science
- License
- CC BY 4.0 (HF dataset metadata)
- Homepage
- github.com
Related datasets
-
subset of GPQA (Graduate-Level Google-Proof Q&A)
Curated subset of the released GPQA (Extended) questions.
Evidence
“In addition to the full set of 546 questions (which we refer to as GPQA Extended), we release two curated subsets”
arxiv.org · paper · Section 2.2 Dataset Splits · retrieved 2026-10-15 “GPQA Diamond We create GPQA Diamond (the diamond set, composed of 198 questions) is our highest quality subset which includes only questions where both experts answer correctly and the majority of non-experts answer incorrectly.”
arxiv.org · paper · Section 2.2 Dataset Splits · retrieved 2026-10-15
Exposure by model
| Benchmark | Model | Level | Reason |
|---|---|---|---|
| Recent LLM benchmarks | LLM360 Amber | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | BLOOM | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude 3.5 Sonnet | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude 3.7 Sonnet | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude 3 (Opus, Sonnet, Haiku) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Opus 4 and Claude Sonnet 4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Fable 5 / Claude Mythos 5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Mythos Preview | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Opus 4.5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Opus 5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Sonnet 5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DCLM-7B-IT (DCLM-IT-7B) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DCLM-7B (DCLM-Baseline-7B) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-R1-0528 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-R1-Distill | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-R1 | Not seen | the authors state its test portion was excluded from training |
| Recent LLM benchmarks | DeepSeek-V3-0324 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V3.1 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V3.2-Exp | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V3.2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V4.1-Flash | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V4 (Pro / Flash) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemini 1.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemini 2.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemini 3 Pro | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemini 4 Argon | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemma 2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemma 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemma 3n | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemma 4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | GPT-4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | GPT-4o | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | GPT-5.2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | GPT-5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | GPT-6 Astra | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | gpt-oss | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | InternVL3.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Recent LLM benchmarks | InternVL3 | Unknown | unknown for its base model Qwen2.5: training data not disclosed in enough detail |
| Recent LLM benchmarks | LLM360 K2-65B (K2 Diamond) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | K2-Horizon | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | K2-V2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 3.1 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 3.2-Vision (11B, 90B) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 3.2 (1B, 3B) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 3.3 70B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Meta Llama 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 4 (Scout, Maverick) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | LLaVA-OneVision-1.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Recent LLM benchmarks | LLaVA-OneVision | Unknown | unknown for its base model SigLIP: training data not disclosed in enough detail |
| Recent LLM benchmarks | Ministral 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mistral 7B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mistral Large 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mistral Medium 3.5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mistral Small 4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mixtral 8x7B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Molmo (0924) | Unknown | unknown for its base model OpenAI CLIP: training data not disclosed in enough detail |
| Recent LLM benchmarks | Molmo2 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Recent LLM benchmarks | OpenAI o1 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | OpenAI o3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | OLMo (February 2024) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | OLMo 2 (November 2024) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Olmo 3.1 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Olmo 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Olmo Hybrid 7B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-3.5 (mini / MoE) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-3 (mini / small / medium) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4-mini-flash-reasoning | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4-mini-reasoning | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4-mini | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4-reasoning / Phi-4-reasoning-plus | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-mini-MoE / Phi-tiny-MoE (SlimMoE) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Pixtral 12B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Pixtral Large | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Pythia (deduplicated) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Pythia | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen2.5-VL | Unknown | unknown for its base model Qwen2.5: training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen2.5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3-2507 (Instruct-2507 / Thinking-2507) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3-Next | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3-VL | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3.5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3.6 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3.8-Flash-Next | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3.8 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | SmolLM2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | SmolLM3 | Unknown | training data not disclosed in enough detail |
Sources
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark (arXiv v1) · paper · retrieved 2026-10-15
- idavidrein/gpqa README · readme · retrieved 2026-10-15