Contamination MapText / LLM / Dataset
MATH (Mathematics Aptitude Test of Heuristics)
12,500 competition mathematics problems (AMC 10, AMC 12, AIME and others) with step-by-step solutions, split into 7,500 training and 5,000 test problems, 7 subjects, difficulty levels 1-5. The paper notes many of the problems can be collected from aops.com.
- Also known as
- MATH, Hendrycks MATH, MATH dataset, competition_math, Mathematics Aptitude Test of Heuristics
- Domain
- math
- License
- MIT (HF metadata of the disabled hendrycks/competition_math repo)
- Homepage
- github.com
Exposure by model
| Benchmark | Model | Level | Reason |
|---|---|---|---|
| Classic LLM benchmarks | LLM360 Amber | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | BLOOM | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude 3.5 Sonnet | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude 3.7 Sonnet | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude 3 (Opus, Sonnet, Haiku) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Opus 4 and Claude Sonnet 4 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Fable 5 / Claude Mythos 5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Mythos Preview | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Opus 4.5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Opus 5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Claude Sonnet 5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DCLM-7B-IT (DCLM-IT-7B) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DCLM-7B (DCLM-Baseline-7B) | Test data seen | a contamination analysis found test examples in the pre-training data (Via ProofPile2 (Llemma paper): ~7% of MATH test problem statements and 0.6% of MATH test solutions have 30-gram hits in Proof-Pile-2 (OpenWebMath/AlgebraicStack). CONDA 2024 report also lists hendrycks/competition_math and gsm8k as contaminated in EleutherAI/proof-pile-2.) [third-party study] |
| Classic LLM benchmarks | DeepSeek-R1-0528 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-R1-Distill | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-R1 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V3-0324 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V3.1 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V3.2-Exp | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V3.2 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V4.1-Flash | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | DeepSeek-V4 (Pro / Flash) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemini 1.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemini 2.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemini 3 Pro | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemini 4 Argon | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemma 2 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemma 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemma 3n | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Gemma 4 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | GPT-4 | Training data seen | only the training split was used, not the benchmark's test data |
| Classic LLM benchmarks | GPT-4o | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | GPT-5.2 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | GPT-5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | GPT-6 Astra | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | gpt-oss | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | InternVL3.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Classic LLM benchmarks | InternVL3 | via base model Qwen2.5: a contamination analysis found test examples of the related dataset MATH-500 (Exact completion of 21.2% of MATH-500 problems from a 60% prefix (Table 2).) [third-party study] | |
| Classic LLM benchmarks | LLM360 K2-65B (K2 Diamond) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | K2-Horizon | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | K2-V2 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Llama 2 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Llama 3.1 | Test data seen | a contamination analysis found test examples in the pre-training data (1% of MATH examples considered contaminated at the selected threshold; estimated gain 0.0 / -0.1 / -0.2. Authors note low thresholds yield high contamination for MATH but no performance gains.) |
| Classic LLM benchmarks | Llama 3.2-Vision (11B, 90B) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Llama 3.2 (1B, 3B) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Llama 3.3 70B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Meta Llama 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Llama 4 (Scout, Maverick) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | LLaVA-OneVision-1.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Classic LLM benchmarks | LLaVA-OneVision | Unknown | unknown for its base model SigLIP: training data not disclosed in enough detail |
| Classic LLM benchmarks | Ministral 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mistral 7B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mistral Large 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mistral Medium 3.5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mistral Small 4 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Mixtral 8x7B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Molmo (0924) | Unknown | unknown for its base model OpenAI CLIP: training data not disclosed in enough detail |
| Classic LLM benchmarks | Molmo2 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Classic LLM benchmarks | OpenAI o1 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | OpenAI o3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | OLMo (February 2024) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | OLMo 2 (November 2024) | Training data seen | in fine-tuning data: only the training split was used, not the benchmark's test data; the test portion was explicitly excluded |
| Classic LLM benchmarks | Olmo 3.1 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Olmo 3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Olmo Hybrid 7B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-3.5 (mini / MoE) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-3 (mini / small / medium) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4-mini-flash-reasoning | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4-mini-reasoning | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4-mini | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4-reasoning / Phi-4-reasoning-plus | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Phi-4 | Not seen | the authors state its test portion was excluded from training |
| Classic LLM benchmarks | Phi-mini-MoE / Phi-tiny-MoE (SlimMoE) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Pixtral 12B | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Pixtral Large | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Pythia (deduplicated) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Pythia | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen2.5-VL | via base model Qwen2.5: a contamination analysis found test examples of the related dataset MATH-500 (Exact completion of 21.2% of MATH-500 problems from a 60% prefix (Table 2).) [third-party study] | |
| Classic LLM benchmarks | Qwen2.5 | a contamination analysis found test examples of the related dataset MATH-500 (Exact completion of 21.2% of MATH-500 problems from a 60% prefix (Table 2).) [third-party study] | |
| Classic LLM benchmarks | Qwen3-2507 (Instruct-2507 / Thinking-2507) | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3-Next | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3-VL | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3.5 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3.6 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3.8-Flash-Next | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3.8 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | Qwen3 | Unknown | training data not disclosed in enough detail |
| Classic LLM benchmarks | SmolLM2 | Training data seen | in fine-tuning data: only the training split was used, not the benchmark's test data (via SmolTalk) |
| Classic LLM benchmarks | SmolLM3 | Unknown | training data not disclosed in enough detail |
Notes
The original HF copy hendrycks/competition_math is disabled ('DMCA Takedown notice'); the official README now points to the mirror qwedsacf/competition_math. Problems come from older public competitions (AoPS), so they predate the dataset by years. Copies found in public web data by third-party studies: Llama 1 pre-training corpus: Substantial contamination percentages for MATH in the Llama 1 pre-training corpus; significant estimated performance gain seen for Llama 1 models.; MATHInstruct: 769 MATH test problems (15.4%) have rephrased counterparts in MATHInstruct (Table 7).
Sources
- Measuring Mathematical Problem Solving With the MATH Dataset (arXiv v2) · paper · retrieved 2026-10-15
- hendrycks/math README · readme · retrieved 2026-10-15
- hendrycks/competition_math HF metadata (disabled) · dataset card · retrieved 2026-10-15