Contamination Map Methodology

Text / LLM / Model

DCLM-7B (DCLM-Baseline-7B)

Organisation
Apple / DataComp-LM (DCLM) team
Family
DCLM
Released
2024-07-11 (HF repo creation of apple/DCLM-7B (createdAt 2024-07-11, https://huggingface.co/api/models/apple/DCLM-7B). Model card says 'Date: June 2024'; arXiv v1 of 2406.11794 is 2024-06-17.)
Checkpoints
apple/DCLM-7B, apple/DCLM-7B-8k
Weights
Open · Apple Sample Code License (apple-ascl)
Links
PaperModel cardCode

Training data disclosure: Full list

Level determined for 53% of benchmark cells.

“we combine our 3.8T [DCLM-BASELINE](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) with the [StarCoder](https://huggingface.co/datasets/bigcode/starcoderdata) and [ProofPile2](https://huggingface.co/datasets/EleutherAI/proof-pile-2) data to arrive at a 4.1T token dataset.”
huggingface.co · model card · Training Details · retrieved 2026-10-15
“For the cooldown distribution we use a mix of 70% DCLM- BASELINE with a tighter fastText threshold (top 7% rather than top 10%) and 30% ProofPile.”
arxiv.org · paper · Appendix Q · retrieved 2026-10-15
“An additional 100B of training was done on the same dataset using [Dataset Decomposition](https://arxiv.org/abs/2405.13226) to extend context length from 2k -> 8k.”
huggingface.co · model card · Training Details · retrieved 2026-10-15

Training corpora

Other declared training data

Contamination analyses

Analyses that found benchmark test data in this model's training data, or in a corpus it uses.

Benchmark cells

Classic LLM benchmarks

Test data seen MMLU (Measuring Massive Multitask Language Understanding) · a contamination analysis found test examples in the pre-training data (DCLM's MMLU overlap rule (question last sentence + one answer option on the same page; recall-oriented, likely overestimate) flagged 0.007% of DCLM-Baseline documents (vs 0.001% Dolma-V1.7, 0.009% FineWeb-Edu). Removing the flagged overlaps did not lower MMLU accuracy at 7B-2x scale (51.8 -> 52.7).) (via DCLM-Baseline 1.0)

Evidence (4)
“DCLM-Baseline-7B is a 7 billion parameter language model trained on the DCLM-Baseline dataset”
huggingface.co · model card · Model Card for DCLM-Baseline-7B · retrieved 2026-10-15
“we combine our 3.8T DCLM- BASELINE with the StarCoder [96] and ProofPile2 [14] datasets to arrive at a 4.1T token dataset.”
arxiv.org · paper · Section 5 · retrieved 2026-10-15
“the percentage of documents in DCLM- BASELINE that are flagged by our MMLU-specific decontamination is roughly similar to that for other high-performing contemporary datasets”
arxiv.org · paper · Appendix O, Table 25 · retrieved 2026-10-15
“Note that because our flagging rule prioritizes recall over precision, these numbers are likely to be overestimates of the true contamination rates.”
arxiv.org · paper · Appendix O, Table 25 caption · retrieved 2026-10-15

Test data seen GSM8K (Grade School Math 8K) · a contamination analysis found test examples in the pre-training data (Via ProofPile2: Llemma Table 6 lists 30-gram hits between GSM8k test problems and OpenWebMath documents (2 test examples, 3 documents; 0 for solutions); CONDA 2024 report lists gsm8k as contaminated in EleutherAI/proof-pile-2.) [third-party study]

Evidence (2)
“OpenWebMath GSM8k”
arxiv.org third-party study· study · Table 6 · retrieved 2026-10-15
“EleutherAI/proof-pile-2 (Azer- gsm8k (Cobbe et al., 2021), hendrycks/competition_math (Hendrycks et al., 2021b)”
arxiv.org third-party study· study · Table 1 · retrieved 2026-10-15

Test data seen MATH (Mathematics Aptitude Test of Heuristics) · a contamination analysis found test examples in the pre-training data (Via ProofPile2 (Llemma paper): ~7% of MATH test problem statements and 0.6% of MATH test solutions have 30-gram hits in Proof-Pile-2 (OpenWebMath/AlgebraicStack). CONDA 2024 report also lists hendrycks/competition_math and gsm8k as contaminated in EleutherAI/proof-pile-2.) [third-party study]

Evidence (2)
“around 7% of MATH test problem statements and 0.6% of MATH test solutions have hits”
arxiv.org third-party study· study · Section 3.5, Table 6 · retrieved 2026-10-15
“EleutherAI/proof-pile-2 (Azer- gsm8k (Cobbe et al., 2021), hendrycks/competition_math (Hendrycks et al., 2021b)”
arxiv.org third-party study· study · Table 1 · retrieved 2026-10-15

Test data seen HumanEval · a contamination analysis found test examples in the pre-training data (Via StarCoder data: Riddell et al. (2024) find 18.9% of HumanEval gold solutions (and 20.8% of MBPP) seen in the Python split of the Stack training data of StarCoderBase (60.40 GB searched), despite the Stack's string-matching decontamination.) [third-party study]

Evidence (2)
“Results in Fig. 1 show that 12.2% of the solutions in HumanEval have been seen by models trained on the P ILE and 18.9% have been seen by models trained on the S TACK.”
arxiv.org third-party study· study · Section 4.1 · retrieved 2026-10-15
“we only search through 60.40 GB within the Python split of its training dataset”
arxiv.org third-party study· study · Section 3.1 · retrieved 2026-10-15

Test data seen ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · a contamination analysis found test examples in the pre-training data (32.6% of ARC-Challenge test questions are 'dirty' in DCLM-baseline (17x more than in Pile-train).) [third-party study] (via DCLM-Baseline 1.0)

Evidence (4)
“DCLM-Baseline-7B is a 7 billion parameter language model trained on the DCLM-Baseline dataset”
huggingface.co · model card · Model Card for DCLM-Baseline-7B · retrieved 2026-10-15
“we combine our 3.8T DCLM- BASELINE with the StarCoder [96] and ProofPile2 [14] datasets to arrive at a 4.1T token dataset.”
arxiv.org · paper · Section 5 · retrieved 2026-10-15
“When checking against DCLMbaseline, MMLU has 27.70% dirty entries, MMLUPro has 16.20%, ARC-Challenge has 32.6%, and ARC-Easy has 32.3%.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15
“For example, ARC-Challenge has 17× more and MMLU has 1.1× more dirty entries.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15

Test data seen HellaSwag · a contamination analysis found test examples in the pre-training data (Overlaps between HellaSwag test examples and DCLM-Baseline pages were detected and removed for an ablation (fraction not reported in text); removal did not lower HellaSwag accuracy (77.9 -> 78.4). In the 10-gram token-overlap analysis HellaSwag is the notable exception with a performance gap between full and 'clean' subsets (Figure 12).) (via DCLM-Baseline 1.0)

Evidence (4)
“DCLM-Baseline-7B is a 7 billion parameter language model trained on the DCLM-Baseline dataset”
huggingface.co · model card · Model Card for DCLM-Baseline-7B · retrieved 2026-10-15
“we combine our 3.8T DCLM- BASELINE with the StarCoder [96] and ProofPile2 [14] datasets to arrive at a 4.1T token dataset.”
arxiv.org · paper · Section 5 · retrieved 2026-10-15
“Specifically, we attempt to remove examples from these two test sets that exist in DCLM- BASELINE.”
arxiv.org · paper · Section 4.6 · retrieved 2026-10-15
“While Hellaswag appears to be a notable exception, we showed in Section 4.6 that removing contaminated Hellaswag examples from our training set does not decrease performance.”
arxiv.org · paper · Appendix O · retrieved 2026-10-15

Unknown TruthfulQA · training data not disclosed in enough detail

Unknown WinoGrande · training data not disclosed in enough detail

Recent LLM benchmarks

Unknown GPQA Diamond · training data not disclosed in enough detail

Test data seen MMLU-Pro · a contamination analysis found test examples in the pre-training data (16.20% of (downsampled) MMLU-Pro test questions are 'dirty' in DCLM-baseline.) [third-party study] (via DCLM-Baseline 1.0)

Evidence (3)
“DCLM-Baseline-7B is a 7 billion parameter language model trained on the DCLM-Baseline dataset”
huggingface.co · model card · Model Card for DCLM-Baseline-7B · retrieved 2026-10-15
“we combine our 3.8T DCLM- BASELINE with the StarCoder [96] and ProofPile2 [14] datasets to arrive at a 4.1T token dataset.”
arxiv.org · paper · Section 5 · retrieved 2026-10-15
“When checking against DCLMbaseline, MMLU has 27.70% dirty entries, MMLUPro has 16.20%, ARC-Challenge has 32.6%, and ARC-Easy has 32.3%.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15

Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail

Unknown SWE-bench Verified · training data not disclosed in enough detail

Unknown AIME 2024 · training data not disclosed in enough detail

Unknown AIME 2025 · training data not disclosed in enough detail

Related data seen MATH-500 · a contamination analysis found test examples of the related dataset MATH (Mathematics Aptitude Test of Heuristics) (Via ProofPile2 (Llemma paper): ~7% of MATH test problem statements and 0.6% of MATH test solutions have 30-gram hits in Proof-Pile-2 (OpenWebMath/AlgebraicStack). CONDA 2024 report also lists hendrycks/competition_math and gsm8k as contaminated in EleutherAI/proof-pile-2.) [third-party study]

Evidence (4)
“around 7% of MATH test problem statements and 0.6% of MATH test solutions have hits”
arxiv.org third-party study· study · Section 3.5, Table 6 · retrieved 2026-10-15
“EleutherAI/proof-pile-2 (Azer- gsm8k (Cobbe et al., 2021), hendrycks/competition_math (Hendrycks et al., 2021b)”
arxiv.org third-party study· study · Table 1 · retrieved 2026-10-15
“We therefore evaluate our models only on the remaining 500 held-out problems. We selected these 500 test problems uniformly at random, and we believe they are representative of the test set as a whole.”
raw.githubusercontent.com · readme · MATH Splits · retrieved 2026-10-15
“This dataset contains a subset of 500 problems from the MATH benchmark that OpenAI created in their _Let's Verify Step by Step_ paper.”
huggingface.co · dataset card · retrieved 2026-10-15

Notes

Base model; card says it has not undergone alignment or safety fine-tuning. DCLM's MMLU/HellaSwag decontamination was an ablation at 7B-2x scale only; the released model was trained on non-decontaminated DCLM-Baseline (developer findings recorded on corpus dclm-baseline). DCLM-Pool is all Common Crawl data prior to 2023, but no model knowledge cutoff is stated (card: 'limited to its training data cutoff date').

Sources