Contamination Map Methodology

Text / LLM / Model

SmolLM2

Organisation
Hugging Face
Family
SmolLM · version 135M, 360M, 1.7B
Released
2024-10-30 (HF createdAt of HuggingFaceTB/SmolLM2-1.7B (https://huggingface.co/api/models/HuggingFaceTB/SmolLM2-1.7B); other sizes and Instruct repos created 2024-10-31; paper arXiv v1 2025-02-04)
Checkpoints
HuggingFaceTB/SmolLM2-135M, HuggingFaceTB/SmolLM2-360M, HuggingFaceTB/SmolLM2-1.7B, HuggingFaceTB/SmolLM2-135M-Instruct, HuggingFaceTB/SmolLM2-360M-Instruct, HuggingFaceTB/SmolLM2-1.7B-Instruct
Weights
Open · apache-2.0
Links
PaperModel cardCode

Training data disclosure: Full list

Level determined for 46% of benchmark cells.

“To support future research and development, we release SmolLM2 alongside the datasets and code used in its training.”
arxiv.org · paper · Section 7 · retrieved 2026-10-15
“The data mixtures of the four pretraining phases are available in Figure 2.”
arxiv.org · paper · Section 4 · retrieved 2026-10-15
“Table 9 shows the final composition of SmolTalk dataset.”
arxiv.org · paper · Appendix F · retrieved 2026-10-15

Training corpora

Other declared training data

Declared exclusions

Contamination analyses

Analyses that found benchmark test data in this model's training data, or in a corpus it uses.

Benchmark cells

Classic LLM benchmarks

Test data seen MMLU (Measuring Massive Multitask Language Understanding) · a contamination analysis found test examples in the pre-training data (DCLM's MMLU overlap rule (question last sentence + one answer option on the same page; recall-oriented, likely overestimate) flagged 0.007% of DCLM-Baseline documents (vs 0.001% Dolma-V1.7, 0.009% FineWeb-Edu). Removing the flagged overlaps did not lower MMLU accuracy at 7B-2x scale (51.8 -> 52.7).) (via DCLM-Baseline 1.0); this contradicts a statement that the data was excluded

Evidence (5)
“DCLM comprises 3.8T tokens filtered using a fastText classifier”
arxiv.org · paper · Section 3.2 · retrieved 2026-10-15
“We adopted a 60% FineWeb-Edu and 40% DCLM ratio (discussed in Section 2.2) for web data”
arxiv.org · paper · Section 4.2 · retrieved 2026-10-15
“the percentage of documents in DCLM- BASELINE that are flagged by our MMLU-specific decontamination is roughly similar to that for other high-performing contemporary datasets”
arxiv.org · paper · Appendix O, Table 25 · retrieved 2026-10-15
“Note that because our flagging rule prioritizes recall over precision, these numbers are likely to be overestimates of the true contamination rates.”
arxiv.org · paper · Appendix O, Table 25 caption · retrieved 2026-10-15
“we decontaminate each dataset against GSM8K, MATH and MMLU using 13-gram matching and a minimum overlap ratio with the longest common subsequence of 0.6.”
arxiv.org · paper · Section 3.3 · retrieved 2026-10-15

Training data seen GSM8K (Grade School Math 8K) · only the training split was used, not the benchmark's test data; the test portion was explicitly excluded

Evidence (2)
“0.02% to AugGSM8K (Li et al., 2024a), an augmented version of the GSM8K benchmark's training set”
arxiv.org · paper · Section 4.5 · retrieved 2026-10-15
“we decontaminate each dataset against GSM8K, MATH and MMLU using 13-gram matching and a minimum overlap ratio with the longest common subsequence of 0.6.”
arxiv.org · paper · Section 3.3 · retrieved 2026-10-15

Training data seen MATH (Mathematics Aptitude Test of Heuristics) · in fine-tuning data: only the training split was used, not the benchmark's test data (via SmolTalk)

Evidence (4)
“We performed supervised fine-tuning of our base SmolLM2 on SmolTalk for 2 epochs”
arxiv.org · paper · Section 5.2 · retrieved 2026-10-15
“You can find the SFT dataset here: https://huggingface.co/datasets/HuggingFaceTB/smoltalk.”
huggingface.co · model card · retrieved 2026-10-15
“MetaMathQA: we add this [dataset](https://huggingface.co/datasets/meta-math/MetaMathQA?) to improve the model on mathematics and reasoning, we include 50k random samples.”
huggingface.co · dataset card · Dataset composition · retrieved 2026-10-15
“All MetaMathQA data are augmented from the training sets of GSM8K and MATH.”
huggingface.co · dataset card · MetaMathQA card · retrieved 2026-10-15

Unknown HumanEval · training data not disclosed in enough detail

Test data seen ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · a contamination analysis found test examples in the pre-training data (32.6% of ARC-Challenge test questions are 'dirty' in DCLM-baseline (17x more than in Pile-train).) [third-party study] (via DCLM-Baseline 1.0)

Evidence (4)
“DCLM comprises 3.8T tokens filtered using a fastText classifier”
arxiv.org · paper · Section 3.2 · retrieved 2026-10-15
“We adopted a 60% FineWeb-Edu and 40% DCLM ratio (discussed in Section 2.2) for web data”
arxiv.org · paper · Section 4.2 · retrieved 2026-10-15
“When checking against DCLMbaseline, MMLU has 27.70% dirty entries, MMLUPro has 16.20%, ARC-Challenge has 32.6%, and ARC-Easy has 32.3%.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15
“For example, ARC-Challenge has 17× more and MMLU has 1.1× more dirty entries.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15

Test data seen HellaSwag · a contamination analysis found test examples in the pre-training data (Overlaps between HellaSwag test examples and DCLM-Baseline pages were detected and removed for an ablation (fraction not reported in text); removal did not lower HellaSwag accuracy (77.9 -> 78.4). In the 10-gram token-overlap analysis HellaSwag is the notable exception with a performance gap between full and 'clean' subsets (Figure 12).) (via DCLM-Baseline 1.0)

Evidence (4)
“DCLM comprises 3.8T tokens filtered using a fastText classifier”
arxiv.org · paper · Section 3.2 · retrieved 2026-10-15
“We adopted a 60% FineWeb-Edu and 40% DCLM ratio (discussed in Section 2.2) for web data”
arxiv.org · paper · Section 4.2 · retrieved 2026-10-15
“Specifically, we attempt to remove examples from these two test sets that exist in DCLM- BASELINE.”
arxiv.org · paper · Section 4.6 · retrieved 2026-10-15
“While Hellaswag appears to be a notable exception, we showed in Section 4.6 that removing contaminated Hellaswag examples from our training set does not decrease performance.”
arxiv.org · paper · Appendix O · retrieved 2026-10-15

Unknown TruthfulQA · training data not disclosed in enough detail

Unknown WinoGrande · training data not disclosed in enough detail

Recent LLM benchmarks

Unknown GPQA Diamond · training data not disclosed in enough detail

Test data seen MMLU-Pro · a contamination analysis found test examples in the pre-training data (16.20% of (downsampled) MMLU-Pro test questions are 'dirty' in DCLM-baseline.) [third-party study] (via DCLM-Baseline 1.0)

Evidence (3)
“DCLM comprises 3.8T tokens filtered using a fastText classifier”
arxiv.org · paper · Section 3.2 · retrieved 2026-10-15
“We adopted a 60% FineWeb-Edu and 40% DCLM ratio (discussed in Section 2.2) for web data”
arxiv.org · paper · Section 4.2 · retrieved 2026-10-15
“When checking against DCLMbaseline, MMLU has 27.70% dirty entries, MMLUPro has 16.20%, ARC-Challenge has 32.6%, and ARC-Easy has 32.3%.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15

Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail

Unknown SWE-bench Verified · training data not disclosed in enough detail

Unknown AIME 2024 · training data not disclosed in enough detail

Unknown AIME 2025 · training data not disclosed in enough detail

Related data seen MATH-500 · in fine-tuning data: related dataset MATH (Mathematics Aptitude Test of Heuristics) was used (via SmolTalk)

Evidence (6)
“We performed supervised fine-tuning of our base SmolLM2 on SmolTalk for 2 epochs”
arxiv.org · paper · Section 5.2 · retrieved 2026-10-15
“You can find the SFT dataset here: https://huggingface.co/datasets/HuggingFaceTB/smoltalk.”
huggingface.co · model card · retrieved 2026-10-15
“MetaMathQA: we add this [dataset](https://huggingface.co/datasets/meta-math/MetaMathQA?) to improve the model on mathematics and reasoning, we include 50k random samples.”
huggingface.co · dataset card · Dataset composition · retrieved 2026-10-15
“All MetaMathQA data are augmented from the training sets of GSM8K and MATH.”
huggingface.co · dataset card · MetaMathQA card · retrieved 2026-10-15
“We therefore evaluate our models only on the remaining 500 held-out problems. We selected these 500 test problems uniformly at random, and we believe they are representative of the test set as a whole.”
raw.githubusercontent.com · readme · MATH Splits · retrieved 2026-10-15
“This dataset contains a subset of 500 problems from the MATH benchmark that OpenAI created in their _Let's Verify Step by Step_ paper.”
huggingface.co · dataset card · retrieved 2026-10-15

Notes

1.7B: 11T tokens in 4 stages + 8k context extension; 360M (4T) and 135M (2T) used a single-stage mix of the same source families (filtered DCLM, FineWeb-Edu, Stack-Edu, InfiMM-WebMath, FineMath, Cosmopedia) per paper Section 6. Instruct models: SFT on SmolTalk (1.7B) or Smol-SmolTalk (135M/360M), then DPO on UltraFeedback. SmolTalk includes MetaMathQA (GSM8K/MATH train-derived) and NuminaMath-CoT; AugGSM8K (GSM8K train-derived) was in the pre-training decay mix. Decontamination statements not shown as cells (they name no benchmark tracked here, or none at all): IFEval.

Sources