Contamination Map Methodology

Text / LLM / Dataset

AIME 2024

The 30 problems of the 2024 American Invitational Mathematics Examination (AIME I and AIME II, 15 each); integer answers 000-999. No official dataset release: commonly used HF copies are Maxwell-Jia/AIME_2024 and HuggingFaceH4/aime_2024 (the latter derived from AI-MO/aimo-validation-aime, AIME 2022-2024).

Also known as
AIME 2024, AIME24, AIME'24
Domain
math
Test items from
2024-01-31
Evidence
“2024 AIME I problems and solutions. The test was held on Wednesday, January 31 – Thursday, February 1, 2024.”
web.archive.org · website · retrieved 2026-10-15
“AIME I (main AIME competition date): Thursday, Feb. 1, 2024 between 1:30 pm ET and 5:30 pm ET.”
web.archive.org · website · retrieved 2026-10-15
“AIME II (alternate AIME competition date): Wednesday, Feb. 7, 2024 between 1:30 pm ET and 5:30 pm ET.”
web.archive.org · website · retrieved 2026-10-15

Exposure by model

BenchmarkModelLevelReason
Recent LLM benchmarks LLM360 Amber Unknown training data not disclosed in enough detail
Recent LLM benchmarks BLOOM Unknown training data not disclosed in enough detail
Recent LLM benchmarks Claude 3.5 Sonnet Unknown training data not disclosed in enough detail
Recent LLM benchmarks Claude 3.7 Sonnet Unknown training data not disclosed in enough detail
Recent LLM benchmarks Claude 3 (Opus, Sonnet, Haiku) Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-08-01)
Recent LLM benchmarks Claude Opus 4 and Claude Sonnet 4 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Claude Fable 5 / Claude Mythos 5 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Claude Mythos Preview Unknown training data not disclosed in enough detail
Recent LLM benchmarks Claude Opus 4.5 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Claude Opus 5 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Claude Sonnet 5 Unknown training data not disclosed in enough detail
Recent LLM benchmarks DCLM-7B-IT (DCLM-IT-7B) Unknown training data not disclosed in enough detail
Recent LLM benchmarks DCLM-7B (DCLM-Baseline-7B) Unknown training data not disclosed in enough detail
Recent LLM benchmarks DeepSeek-R1-0528 Unknown training data not disclosed in enough detail
Recent LLM benchmarks DeepSeek-R1-Distill Unknown training data not disclosed in enough detail
Recent LLM benchmarks DeepSeek-R1 Not seen the authors state its test portion was excluded from training
Recent LLM benchmarks DeepSeek-V3-0324 Unknown training data not disclosed in enough detail
Recent LLM benchmarks DeepSeek-V3.1 Unknown training data not disclosed in enough detail
Recent LLM benchmarks DeepSeek-V3.2-Exp Unknown training data not disclosed in enough detail
Recent LLM benchmarks DeepSeek-V3.2 Unknown training data not disclosed in enough detail
Recent LLM benchmarks DeepSeek-V3 Unknown training data not disclosed in enough detail
Recent LLM benchmarks DeepSeek-V4.1-Flash Unknown training data not disclosed in enough detail
Recent LLM benchmarks DeepSeek-V4 (Pro / Flash) Unknown training data not disclosed in enough detail
Recent LLM benchmarks Gemini 1.5 (Pro, Flash) Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-11-01)
Recent LLM benchmarks Gemini 2.5 (Pro, Flash) Unknown training data not disclosed in enough detail
Recent LLM benchmarks Gemini 3 Pro Unknown training data not disclosed in enough detail
Recent LLM benchmarks Gemini 4 Argon Unknown training data not disclosed in enough detail
Recent LLM benchmarks Gemma 2 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Gemma 3 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Gemma 3n Unknown training data not disclosed in enough detail
Recent LLM benchmarks Gemma 4 Unknown training data not disclosed in enough detail
Recent LLM benchmarks GPT-4 Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2021-09-01)
Recent LLM benchmarks GPT-4o Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-10-01)
Recent LLM benchmarks GPT-5.2 Unknown training data not disclosed in enough detail
Recent LLM benchmarks GPT-5 Unknown training data not disclosed in enough detail
Recent LLM benchmarks GPT-6 Astra Unknown training data not disclosed in enough detail
Recent LLM benchmarks gpt-oss Unknown training data not disclosed in enough detail
Recent LLM benchmarks InternVL3.5 Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Recent LLM benchmarks InternVL3 Test data seen via base model Qwen2.5: a contamination analysis found test examples in the pre-training data (Exact completion of 13.33% of AIME 2024 problems from a 60% prefix (Table 2).) [third-party study]
Recent LLM benchmarks LLM360 K2-65B (K2 Diamond) Unknown training data not disclosed in enough detail
Recent LLM benchmarks K2-Horizon Unknown training data not disclosed in enough detail
Recent LLM benchmarks K2-V2 Test data seen a contamination analysis found test examples in the pre-training data (Sentence-level memorisation rate 23.32% for K2 base (beam-search continuation exactly reproduces the next sentence of the question); lower than Qwen2.5-72B (41.96%) and Olmo 3 32B (37.82%); authors attribute AIME performance to reasoning rather than memorisation.)
Recent LLM benchmarks Llama 2 Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2022-09-01)
Recent LLM benchmarks Llama 3.1 Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-12-01)
Recent LLM benchmarks Llama 3.2-Vision (11B, 90B) Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-12-01)
Recent LLM benchmarks Llama 3.2 (1B, 3B) Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-12-01)
Recent LLM benchmarks Llama 3.3 70B Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-12-01)
Recent LLM benchmarks Meta Llama 3 Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-12-01)
Recent LLM benchmarks Llama 4 (Scout, Maverick) Unknown training data not disclosed in enough detail
Recent LLM benchmarks LLaVA-OneVision-1.5 Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Recent LLM benchmarks LLaVA-OneVision Unknown training data not disclosed in enough detail
Recent LLM benchmarks Ministral 3 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Mistral 7B Unknown training data not disclosed in enough detail
Recent LLM benchmarks Mistral Large 3 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Mistral Medium 3.5 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Mistral Small 4 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Mixtral 8x7B Unknown training data not disclosed in enough detail
Recent LLM benchmarks Molmo (0924) Unknown unknown for its base model OpenAI CLIP: training data not disclosed in enough detail
Recent LLM benchmarks Molmo2 Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Recent LLM benchmarks OpenAI o1 Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-10-01)
Recent LLM benchmarks OpenAI o3 Unknown training data not disclosed in enough detail
Recent LLM benchmarks OLMo (February 2024) Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-02-01)
Recent LLM benchmarks OLMo 2 (November 2024) Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-12-01)
Recent LLM benchmarks Olmo 3.1 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Olmo 3 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Olmo Hybrid 7B Unknown training data not disclosed in enough detail
Recent LLM benchmarks Phi-3.5 (mini / MoE) Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-10-01)
Recent LLM benchmarks Phi-3 (mini / small / medium) Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-10-01)
Recent LLM benchmarks Phi-4-mini-flash-reasoning Unknown training data not disclosed in enough detail
Recent LLM benchmarks Phi-4-mini-reasoning Unknown training data not disclosed in enough detail
Recent LLM benchmarks Phi-4-mini Unknown training data not disclosed in enough detail
Recent LLM benchmarks Phi-4-reasoning / Phi-4-reasoning-plus Unknown training data not disclosed in enough detail
Recent LLM benchmarks Phi-4 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Phi-mini-MoE / Phi-tiny-MoE (SlimMoE) Not seen its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-10-01)
Recent LLM benchmarks Pixtral 12B Unknown training data not disclosed in enough detail
Recent LLM benchmarks Pixtral Large Unknown training data not disclosed in enough detail
Recent LLM benchmarks Pythia (deduplicated) Unknown training data not disclosed in enough detail
Recent LLM benchmarks Pythia Unknown training data not disclosed in enough detail
Recent LLM benchmarks Qwen2.5-VL Test data seen via base model Qwen2.5: a contamination analysis found test examples in the pre-training data (Exact completion of 13.33% of AIME 2024 problems from a 60% prefix (Table 2).) [third-party study]
Recent LLM benchmarks Qwen2.5 Test data seen a contamination analysis found test examples in the pre-training data (Exact completion of 13.33% of AIME 2024 problems from a 60% prefix (Table 2).) [third-party study]
Recent LLM benchmarks Qwen3-2507 (Instruct-2507 / Thinking-2507) Unknown training data not disclosed in enough detail
Recent LLM benchmarks Qwen3-Next Unknown training data not disclosed in enough detail
Recent LLM benchmarks Qwen3-VL Unknown unknown for its base model Qwen3: training data not disclosed in enough detail
Recent LLM benchmarks Qwen3.5 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Qwen3.6 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Qwen3.8-Flash-Next Unknown training data not disclosed in enough detail
Recent LLM benchmarks Qwen3.8 Unknown training data not disclosed in enough detail
Recent LLM benchmarks Qwen3 Unknown training data not disclosed in enough detail
Recent LLM benchmarks SmolLM2 Unknown training data not disclosed in enough detail
Recent LLM benchmarks SmolLM3 Unknown training data not disclosed in enough detail

Notes

published_from uses the earliest date in the AoPS wiki ('January 31 - February 1, 2024'); MAA lists AIME I on Feb. 1, 2024 and AIME II on Feb. 7, 2024. Problems and solutions are published on the AoPS wiki after each contest. Copies found in public web data by third-party studies: Common Crawl CC-2025-21: 40.00% of AIME 2024 problems are 'dirty' in the CC-2025-21 crawl (clean in Pile-train and DCLM-baseline).

Sources