Contamination MapText / LLM / Model
Phi-4
- Organisation
- Microsoft
- Family
- Phi · version 4
- Released
- 2024-12-12 (Model card 'Release date: December 12, 2024', https://huggingface.co/microsoft/phi-4; arXiv v1 2024-12-12)
- Training cutoff
- 2024-06-01
Evidence
“Static model trained on an offline dataset with cutoff dates of June 2024 and earlier for publicly available data”
huggingface.co · model card · Model Summary / Status · retrieved 2026-10-15 - Checkpoints
- microsoft/phi-4
- Weights
- Open · MIT
- Links
- PaperModel card
Training data disclosure: Vague
Level determined for 60% of benchmark cells.
- TestTest data seen0
- SeenSeen, split unclear0
- TrainTraining data seen0
- Related data seen0
- NoNot seen9
- ?Unknown6
“Table 5: Data mixture for pretraining.”
“`phi-4` is a state-of-the-art open model built upon a blend of synthetic datasets, data from filtered public domain websites, and acquired academic books and Q&A datasets.”
Other declared training data
-
Web (filtered web dumps: forums, blogs, course material, wikis) (broad web data: may contain benchmark test sets) · split: unknown
Evidence
“we took the approach of selecting a small fraction of highest-quality documents from bulk web dumps, using small (non-LLM) classifiers”
arxiv.org · paper · Section 2.3 (Filtering Web Dumps) · retrieved 2026-10-15 “Web”
arxiv.org · paper · Table 5 (15% of training, 1.3T unique tokens) · retrieved 2026-10-15 -
Web rewrites (synthetic rewrites of filtered web content) (not yet matched to a dataset) · split: unknown
Evidence
“Web rewrites”
arxiv.org · paper · Table 5 (15% of training, 290B unique tokens) · retrieved 2026-10-15 -
Synthetic data (50 dataset types, ~400B unweighted tokens) (not yet matched to a dataset) · split: unknown
Evidence
“We created 50 broad types of synthetic datasets, each one relying on a different set of seeds and different multi-stage prompting procedure”
arxiv.org · paper · Section 2.2 · retrieved 2026-10-15 -
Code data (broad web data: may contain benchmark test sets) · split: unknown
Code repositories (GitHub is named among targeted acquisitions); treated as open-ended.
Evidence
“Code data”
arxiv.org · paper · Table 5 (20% of training, 820B unique tokens) · retrieved 2026-10-15 -
Acquired sources / targeted acquisitions (arXiv, PubMed Central, GitHub, licensed books) (broad web data: may contain benchmark test sets) · split: unknown
Evidence
“We included major repositories of reasoning-dense documents that are publicly permissible for use (e.g., arXiv, PubMed Central, GitHub) or explicitly licensed (e.g., licensed books)”
arxiv.org · paper · Section 2.3 (Targeted Acquisitions) · retrieved 2026-10-15 -
organic Q&A datasets (tens of millions of problems and solutions from public websites and existing datasets) (broad web data: may contain benchmark test sets) · split: unknown
'existing datasets' not named.
Evidence
“We collected tens-of-millions high-quality organic problems and solutions by reviewing public websites, relying on existing datasets, and acquiring external datasets.”
arxiv.org · paper · Section 2.3 (Q&A datasets) · retrieved 2026-10-15 “All collected questions and solutions underwent a thorough decontamination process to ensure there is no overlap with test sets”
arxiv.org · paper · Section 2.3 · retrieved 2026-10-15 -
SFT data (~8B tokens: math, coding, reasoning, conversation, identity, safety, multilingual) (not yet matched to a dataset) · split: unknown · fine-tuning
Evidence
“We use around 8B tokens of data in this phase, all formatted in the chatml format.”
arxiv.org · paper · Section 4.1 · retrieved 2026-10-15
Declared exclusions
-
Dataset ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · test portion excluded
Evidence
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
arxiv.org · paper · Section 5 (limitations of benchmarks) · retrieved 2026-10-15 -
Dataset WinoGrande · test portion excluded
Evidence
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
arxiv.org · paper · Section 5 (limitations of benchmarks) · retrieved 2026-10-15 -
Dataset MATH (Mathematics Aptitude Test of Heuristics) · test portion excluded
Evidence
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
arxiv.org · paper · Section 5 (limitations of benchmarks) · retrieved 2026-10-15 -
Dataset HellaSwag · test portion excluded
Evidence
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
arxiv.org · paper · Section 5 (limitations of benchmarks) · retrieved 2026-10-15 -
Dataset GPQA (Graduate-Level Google-Proof Q&A) · test portion excluded
Evidence
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
arxiv.org · paper · Section 5 (limitations of benchmarks) · retrieved 2026-10-15 -
Dataset MMLU-Pro · test portion excluded
Evidence
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
arxiv.org · paper · Section 5 (limitations of benchmarks) · retrieved 2026-10-15 -
Dataset GSM8K (Grade School Math 8K) · test portion excluded
Evidence
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
arxiv.org · paper · Section 5 (limitations of benchmarks) · retrieved 2026-10-15 -
Dataset HumanEval · test portion excluded
Evidence
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
arxiv.org · paper · Section 5 (limitations of benchmarks) · retrieved 2026-10-15 -
Dataset MMLU (Measuring Massive Multitask Language Understanding) · test portion excluded
Evidence
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
arxiv.org · paper · Appendix B.1 · retrieved 2026-10-15 “these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
arxiv.org · paper · Section 5 (limitations of benchmarks) · retrieved 2026-10-15
Benchmark cells
Classic LLM benchmarks
Not seen MMLU (Measuring Massive Multitask Language Understanding) · the authors state its test portion was excluded from training
Evidence (3)
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
“We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
“these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
Not seen GSM8K (Grade School Math 8K) · the authors state its test portion was excluded from training
Evidence (3)
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
“We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
“these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
Not seen MATH (Mathematics Aptitude Test of Heuristics) · the authors state its test portion was excluded from training
Evidence (3)
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
“We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
“these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
Not seen HumanEval · the authors state its test portion was excluded from training
Evidence (3)
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
“We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
“these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
Not seen ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · the authors state its test portion was excluded from training
Evidence (3)
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
“We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
“these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
Not seen HellaSwag · the authors state its test portion was excluded from training
Evidence (3)
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
“We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
“these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
Unknown TruthfulQA · training data not disclosed in enough detail
Not seen WinoGrande · the authors state its test portion was excluded from training
Evidence (3)
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
“We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
“these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
Recent LLM benchmarks
Unknown GPQA Diamond · training data not disclosed in enough detail
Not seen MMLU-Pro · the authors state its test portion was excluded from training
Evidence (3)
“We decontaminate against the ARC-Easy, MBPP, phibench, CommonsenseQA, WinoGrande, mcphi, MedQA, MATH, AGIEval, PIQA, OpenBookQA, HellaSwag, GPQA, mt-bench, MMLUPro, GSM8k, HumanEval, arena hard, ARC-Challenge, and MMLU benchmarks.”
“We apply a hybrid n-gram algorithm for decontamination which uses 13-gram and 7-gram features for removing matches to the test set”
“these methods are not effective against all scenarios, including rephrasing, which leaves some uncertainty about the true extent of generalization.”
Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail
Unknown SWE-bench Verified · training data not disclosed in enough detail
Unknown AIME 2024 · training data not disclosed in enough detail
Not seen AIME 2025 · its test items were published from 2025-02-06, after the model's stated training-data cutoff (2024-06-01)
Evidence (3)
“Static model trained on an offline dataset with cutoff dates of June 2024 and earlier for publicly available data”
“2025 AIME I problems and solutions. The test was held on Thursday, February 6, 2025.”
“This year the AIME I will take place on February 6, 2025, while the alternate AIME II will take place on February 12, 2025.”
Unknown MATH-500 · training data not disclosed in enough detail
Notes
~10T pre-training tokens (card: 9.8T). Report Section 1.1 'Addressing Overfitting and Data Contamination': decontamination improved vs previous Phi models; strong November 2024 AMC-10/12 results are presented as evidence that 'phi-4’s top-tier performance on the MATH benchmark is not due to overfitting or contamination'; GPQA weighted as 'contamination-proof'. Footnote: 'some test benchmark variants can be found on platforms like Hugging Face' and MMLU is 'frequently compiled from web-sourced questions'. TruthfulQA is not in the decontamination list. Decontamination statements not shown as cells (they name no benchmark tracked here, or none at all): AMC-10 / AMC-12, November 2024. Accepts image input: “Inputs | Text, best suited for prompts in the chat format”
Sources
- Phi-4 Technical Report · paper · retrieved 2026-10-15
- phi-4 model card · model card · retrieved 2026-10-15