Contamination MapText / LLM / Model
Phi-3 (mini / small / medium)
- Organisation
- Microsoft
- Family
- Phi · version 3
- Released
- 2024-04-22 (arXiv v1 submission (2404.14219) and HF repo creation of microsoft/Phi-3-mini-4k-instruct, both 2024-04-22; small/medium weights released May 21, 2024 per their cards)
- Training cutoff
- 2023-10-01
Evidence
“This is a static model trained on an offline dataset with cutoff date October 2023.”
huggingface.co · model card · Model / Status · retrieved 2026-10-15 - Checkpoints
- microsoft/Phi-3-mini-4k-instruct, microsoft/Phi-3-mini-128k-instruct, microsoft/Phi-3-small-8k-instruct, microsoft/Phi-3-small-128k-instruct, microsoft/Phi-3-medium-4k-instruct, microsoft/Phi-3-medium-128k-instruct
- Weights
- Open · MIT
- Links
- PaperModel card
Training data disclosure: Vague
Level determined for 13% of benchmark cells.
- TestTest data seen0
- SeenSeen, split unclear0
- TrainTraining data seen0
- Related data seen0
- NoNot seen2
- ?Unknown13
“Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data.”
Other declared training data
-
heavily filtered publicly available web data (broad web data: may contain benchmark test sets) · split: unknown
Evidence
“Our training data of consists of heavily filtered publicly available web data (according to the “educational level”) from various open internet sources, as well as synthetic LLM-generated data.”
arxiv.org · paper · Section 2 (Training Methodology) · retrieved 2026-10-15 -
synthetic LLM-generated data (textbook-like) (not yet matched to a dataset) · split: unknown
Evidence
“Newly created synthetic, “textbook-like” data for the purpose of teaching math, coding, common sense reasoning, general knowledge of the world (science, daily activities, theory of mind, etc.);”
huggingface.co · model card · Datasets · retrieved 2026-10-15 -
SFT data (math, coding, reasoning, conversation, model identity, safety) (not yet matched to a dataset) · split: unknown · fine-tuning
Evidence
“SFT leverages highly curated high-quality data across diverse domains, e.g., math, coding, reasoning, conversation, model identity, and safety.”
arxiv.org · paper · Post-training · retrieved 2026-10-15 -
DPO data (chat format, reasoning, responsible AI) (not yet matched to a dataset) · split: unknown · fine-tuning
Evidence
“DPO data covers chat format data, reasoning, and responsible AI (RAI) efforts.”
arxiv.org · paper · Post-training · retrieved 2026-10-15
Benchmark cells
Classic LLM benchmarks
Unknown MMLU (Measuring Massive Multitask Language Understanding) · training data not disclosed in enough detail
Unknown GSM8K (Grade School Math 8K) · training data not disclosed in enough detail
Unknown MATH (Mathematics Aptitude Test of Heuristics) · training data not disclosed in enough detail
Unknown HumanEval · training data not disclosed in enough detail
Unknown ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · training data not disclosed in enough detail
Unknown HellaSwag · training data not disclosed in enough detail
Unknown TruthfulQA · training data not disclosed in enough detail
Unknown WinoGrande · training data not disclosed in enough detail
Recent LLM benchmarks
Unknown GPQA Diamond · training data not disclosed in enough detail
Unknown MMLU-Pro · training data not disclosed in enough detail
Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail
Unknown SWE-bench Verified · training data not disclosed in enough detail
Not seen AIME 2024 · its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-10-01)
Evidence (4)
“This is a static model trained on an offline dataset with cutoff date October 2023.”
“2024 AIME I problems and solutions. The test was held on Wednesday, January 31 – Thursday, February 1, 2024.”
“AIME I (main AIME competition date): Thursday, Feb. 1, 2024 between 1:30 pm ET and 5:30 pm ET.”
“AIME II (alternate AIME competition date): Wednesday, Feb. 7, 2024 between 1:30 pm ET and 5:30 pm ET.”
Not seen AIME 2025 · its test items were published from 2025-02-06, after the model's stated training-data cutoff (2023-10-01)
Evidence (3)
“This is a static model trained on an offline dataset with cutoff date October 2023.”
“2025 AIME I problems and solutions. The test was held on Thursday, February 6, 2025.”
“This year the AIME I will take place on February 6, 2025, while the alternate AIME II will take place on February 12, 2025.”
Unknown MATH-500 · training data not disclosed in enough detail
Notes
phi-3-mini: 3.3T tokens; phi-3-small/medium: 4.8T tokens of the same data (small adds 10% multilingual). The current Phi-3-mini-4k-instruct card describes the June 2024 update (4.9T tokens). No decontamination statement in the report. Phi-3-vision not covered. Third-party signal on GSM8K (Grade School Math 8K) (indirect, not shown as a cell): Accuracy 4.4 points lower on GSM1k than on GSM8K (p <= 0.003). A GSM8K-GSM1k gap is evidence of overfitting to GSM8K, not proof that GSM8K test items were in the training data; the authors attribute it partly to partial memorisation (correlation with GSM8K log-likelihood) and partly to other causes.
Sources
- Phi-3 Technical Report (v4) · paper · retrieved 2026-10-15
- Phi-3-mini-4k-instruct model card · model card · retrieved 2026-10-15
- Phi-3-small-8k-instruct model card · model card · retrieved 2026-10-15
- Phi-3-medium-4k-instruct model card · model card · retrieved 2026-10-15