Contamination MapText / LLM / Model
OpenAI o3
- Organisation
- OpenAI
- Family
- o-series
- Released
- 2025-04-16 (Date of the o3 and o4-mini System Card ('April 16, 2025'); matches API snapshot o3-2025-04-16)
- Training cutoff
- 2024-06-01
Evidence
“Jun 01, 2024 knowledge cutoff”
developers.openai.com · website · Model page header · retrieved 2026-10-15 - Checkpoints
- o3-2025-04-16
- Weights
- Closed · proprietary
- Links
- Model card
Training data disclosure: Vague
Level determined for 13% of benchmark cells.
- TestTest data seen1
- SeenSeen, split unclear0
- TrainTraining data seen0
- Related data seen0
- NoNot seen2
- ?Unknown19
“OpenAI o3 and o4-mini were trained on diverse datasets, including information that is publicly available on the internet, information that we partner with third parties to access, and information that our users or human trainers and researchers provide”
Other declared training data
-
information that is publicly available on the internet (broad web data: may contain benchmark test sets) · split: unknown
Evidence
“OpenAI o3 and o4-mini were trained on diverse datasets, including information that is publicly available on the internet, information that we partner with third parties to access, and information that our users or human trainers and researchers provide”
cdn.openai.com · model card · Section 2 (Model Data and Training) · retrieved 2026-10-15 -
information that we partner with third parties to access (not yet matched to a dataset) · split: unknown
Evidence
“OpenAI o3 and o4-mini were trained on diverse datasets, including information that is publicly available on the internet, information that we partner with third parties to access, and information that our users or human trainers and researchers provide”
cdn.openai.com · model card · Section 2 (Model Data and Training) · retrieved 2026-10-15 -
information that our users or human trainers and researchers provide or generate (not yet matched to a dataset) · split: unknown
Evidence
“OpenAI o3 and o4-mini were trained on diverse datasets, including information that is publicly available on the internet, information that we partner with third parties to access, and information that our users or human trainers and researchers provide”
cdn.openai.com · model card · Section 2 (Model Data and Training) · retrieved 2026-10-15
Contamination analyses
Analyses that found benchmark test data in this model's training data, or in a corpus it uses.
-
SWE-bench Verified
· training data
Identifies the buggy file path from the issue text alone in up to 76% of SWE-bench Verified tasks, vs <53% on tasks from other repositories; the authors read it as possible memorisation/contamination.
Evidence
“on the file-path identification task, SoTA models like o3 [12] achieve up to 76% accuracy on SWE-Bench-Verified instances, despite lacking the contextual information that should be necessary for this task.”
arxiv.org third-party study· study · Section 1 · retrieved 2026-10-15 “Instead, accuracy is uniformly lower (<53%) than that in SWE-Bench Verified.”
arxiv.org third-party study· study · Section 4.1.2 · retrieved 2026-10-15
Benchmark cells
Classic LLM benchmarks
Unknown MMLU (Measuring Massive Multitask Language Understanding) · training data not disclosed in enough detail
Unknown GSM8K (Grade School Math 8K) · training data not disclosed in enough detail
Unknown MATH (Mathematics Aptitude Test of Heuristics) · training data not disclosed in enough detail
Unknown HumanEval · training data not disclosed in enough detail
Unknown ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · training data not disclosed in enough detail
Unknown HellaSwag · training data not disclosed in enough detail
Unknown TruthfulQA · training data not disclosed in enough detail
Unknown WinoGrande · training data not disclosed in enough detail
Multimodal LLM benchmarks
Unknown MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) · training data not disclosed in enough detail
Unknown MMMU-Pro · training data not disclosed in enough detail
Unknown MathVista · training data not disclosed in enough detail
Unknown ChartQA · training data not disclosed in enough detail
Unknown DocVQA · training data not disclosed in enough detail
Unknown AI2D (AI2 Diagrams) · training data not disclosed in enough detail
Not seen CharXiv · its test items were published from 2024-06-26, after the model's stated training-data cutoff (2024-06-01)
Evidence (3)
“Jun 01, 2024 knowledge cutoff”
“arXiv:2406.18521v1 [cs.CL] 26 Jun 2024”
“we release all QA pairs for the validation set and keep the answers to the test set private to prevent data leakage.”
Recent LLM benchmarks
Unknown GPQA Diamond · training data not disclosed in enough detail
Unknown MMLU-Pro · training data not disclosed in enough detail
Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail
Test data seen SWE-bench Verified · a contamination analysis found test examples in the training data (Identifies the buggy file path from the issue text alone in up to 76% of SWE-bench Verified tasks, vs <53% on tasks from other repositories; the authors read it as possible memorisation/contamination.) [third-party study]
Evidence (2)
“on the file-path identification task, SoTA models like o3 [12] achieve up to 76% accuracy on SWE-Bench-Verified instances, despite lacking the contextual information that should be necessary for this task.”
“Instead, accuracy is uniformly lower (<53%) than that in SWE-Bench Verified.”
Unknown AIME 2024 · training data not disclosed in enough detail
Not seen AIME 2025 · its test items were published from 2025-02-06, after the model's stated training-data cutoff (2024-06-01)
Evidence (3)
“Jun 01, 2024 knowledge cutoff”
“2025 AIME I problems and solutions. The test was held on Thursday, February 6, 2025.”
“This year the AIME I will take place on February 6, 2025, while the alternate AIME II will take place on February 12, 2025.”
Unknown MATH-500 · training data not disclosed in enough detail
Notes
o4-mini shares the system card but is a separate model (not included here). The card's 'contamination' discussion concerns browsing-based retrieval of answers during evaluation, not training data. Accepts image input: “Modalities Text Input and output Image Input only”
Sources
- OpenAI o3 and o4-mini System Card · model card · retrieved 2026-10-15
- OpenAI API docs: o3 model page · website · retrieved 2026-10-15