Contamination Map Methodology

Text / LLM / Model

OLMo 2 (November 2024)

Organisation
Allen Institute for AI (Ai2)
Family
OLMo · version 1124
Released
2024-11-26 (Ai2 blog 'OLMo 2: The best fully open language model to date' dated November 26, 2024 (https://allenai.org/blog/olmo2))
Training cutoff
2023-12-01
Evidence
“Date cutoff:** Dec. 2023.”
huggingface.co · model card · Model Description · retrieved 2026-10-15
Checkpoints
allenai/OLMo-2-1124-7B, allenai/OLMo-2-1124-13B, allenai/OLMo-2-1124-7B-SFT, allenai/OLMo-2-1124-13B-SFT, allenai/OLMo-2-1124-7B-DPO, allenai/OLMo-2-1124-13B-DPO, allenai/OLMo-2-1124-7B-Instruct, allenai/OLMo-2-1124-13B-Instruct
Weights
Open · apache-2.0
Links
PaperModel cardCode

Training data disclosure: Full list

Level determined for 60% of benchmark cells.

“These gains come from training on [OLMo-mix-1124](https://huggingface.co/datasets/allenai/olmo-mix-1124) and [Dolmino-mix-1124](https://huggingface.co/datasets/allenai/dolmino-mix-1124)”
huggingface.co · model card · Model card intro · retrieved 2026-10-15
“which has undergone supervised finetuning on an OLMo-specific variant of the [Tülu 3 dataset](https://huggingface.co/datasets/allenai/tulu-3-sft-olmo-2-mixture) and further DPO training on [this dataset](https://huggingface.co/datasets/allenai/olmo-2-1124-7b-preference-mix), and finally RLVR training using [this data](https://huggingface.co/datasets/allenai/RLVR-GSM).”
huggingface.co · model card · Release Documentation · retrieved 2026-10-15

Training corpora

Other declared training data

Declared exclusions

Contamination analyses

Analyses that found benchmark test data in this model's training data, or in a corpus it uses.

Benchmark cells

Classic LLM benchmarks

Test data seen MMLU (Measuring Massive Multitask Language Understanding) · a contamination analysis found test examples in the pre-training data (DCLM's MMLU overlap rule (question last sentence + one answer option on the same page; recall-oriented, likely overestimate) flagged 0.007% of DCLM-Baseline documents (vs 0.001% Dolma-V1.7, 0.009% FineWeb-Edu). Removing the flagged overlaps did not lower MMLU accuracy at 7B-2x scale (51.8 -> 52.7).) (via DCLM-Baseline 1.0)

Evidence (3)
“The majority of this dataset comes from DCLM-Baseline with no additional filtering”
huggingface.co · dataset card · Dataset card · retrieved 2026-10-15
“the percentage of documents in DCLM- BASELINE that are flagged by our MMLU-specific decontamination is roughly similar to that for other high-performing contemporary datasets”
arxiv.org · paper · Appendix O, Table 25 · retrieved 2026-10-15
“Note that because our flagging rule prioritizes recall over precision, these numbers are likely to be overestimates of the true contamination rates.”
arxiv.org · paper · Appendix O, Table 25 caption · retrieved 2026-10-15

Training data seen GSM8K (Grade School Math 8K) · in fine-tuning data: only the training split was used, not the benchmark's test data; the test portion was explicitly excluded

Evidence (5)
“dataset comprising GSM8K, MATH training sets, and prompts with constraints from Lambert et al. (2024).”
arxiv.org · paper · Section 5 (RLVR) · retrieved 2026-10-15
“first on the GSM8K training set, followed by the MATH training set.”
arxiv.org · paper · Section 5 (RLVR) · retrieved 2026-10-15
“We quantify such overlap as follows and remove instances from our training mix as needed in order to prevent test set contamination.”
arxiv.org · paper · Section 3.2 · retrieved 2026-10-15
“We consider a training set to be contaminated if any number of its instances overlap with more than 2% of the instances in any of the evaluations in our development and unseen suites.”
arxiv.org · paper · Section 3.2 · retrieved 2026-10-15
“GSM8K(em)”
arxiv.org · paper · Table 3 (Tulu 3 Eval suite) · retrieved 2026-10-15

Training data seen MATH (Mathematics Aptitude Test of Heuristics) · in fine-tuning data: only the training split was used, not the benchmark's test data; the test portion was explicitly excluded

Evidence (4)
“dataset comprising GSM8K, MATH training sets, and prompts with constraints from Lambert et al. (2024).”
arxiv.org · paper · Section 5 (RLVR) · retrieved 2026-10-15
“We quantify such overlap as follows and remove instances from our training mix as needed in order to prevent test set contamination.”
arxiv.org · paper · Section 3.2 · retrieved 2026-10-15
“We consider a training set to be contaminated if any number of its instances overlap with more than 2% of the instances in any of the evaluations in our development and unseen suites.”
arxiv.org · paper · Section 3.2 · retrieved 2026-10-15
“MATH(flex em)”
arxiv.org · paper · Table 3 (Tulu 3 Eval suite) · retrieved 2026-10-15

Unknown HumanEval · training data not disclosed in enough detail

Test data seen ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · a contamination analysis found test examples in the pre-training data (32.6% of ARC-Challenge test questions are 'dirty' in DCLM-baseline (17x more than in Pile-train).) [third-party study] (via DCLM-Baseline 1.0)

Evidence (3)
“The majority of this dataset comes from DCLM-Baseline with no additional filtering”
huggingface.co · dataset card · Dataset card · retrieved 2026-10-15
“When checking against DCLMbaseline, MMLU has 27.70% dirty entries, MMLUPro has 16.20%, ARC-Challenge has 32.6%, and ARC-Easy has 32.3%.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15
“For example, ARC-Challenge has 17× more and MMLU has 1.1× more dirty entries.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15

Test data seen HellaSwag · a contamination analysis found test examples in the pre-training data (Overlaps between HellaSwag test examples and DCLM-Baseline pages were detected and removed for an ablation (fraction not reported in text); removal did not lower HellaSwag accuracy (77.9 -> 78.4). In the 10-gram token-overlap analysis HellaSwag is the notable exception with a performance gap between full and 'clean' subsets (Figure 12).) (via DCLM-Baseline 1.0)

Evidence (3)
“The majority of this dataset comes from DCLM-Baseline with no additional filtering”
huggingface.co · dataset card · Dataset card · retrieved 2026-10-15
“Specifically, we attempt to remove examples from these two test sets that exist in DCLM- BASELINE.”
arxiv.org · paper · Section 4.6 · retrieved 2026-10-15
“While Hellaswag appears to be a notable exception, we showed in Section 4.6 that removing contaminated Hellaswag examples from our training set does not decrease performance.”
arxiv.org · paper · Appendix O · retrieved 2026-10-15

Unknown TruthfulQA · training data not disclosed in enough detail

Unknown WinoGrande · training data not disclosed in enough detail

Recent LLM benchmarks

Unknown GPQA Diamond · training data not disclosed in enough detail

Test data seen MMLU-Pro · a contamination analysis found test examples in the pre-training data (16.20% of (downsampled) MMLU-Pro test questions are 'dirty' in DCLM-baseline.) [third-party study] (via DCLM-Baseline 1.0)

Evidence (2)
“The majority of this dataset comes from DCLM-Baseline with no additional filtering”
huggingface.co · dataset card · Dataset card · retrieved 2026-10-15
“When checking against DCLMbaseline, MMLU has 27.70% dirty entries, MMLUPro has 16.20%, ARC-Challenge has 32.6%, and ARC-Easy has 32.3%.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15

Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail

Unknown SWE-bench Verified · training data not disclosed in enough detail

Not seen AIME 2024 · its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-12-01)

Evidence (4)
“Date cutoff:** Dec. 2023.”
huggingface.co · model card · Model Description · retrieved 2026-10-15
“2024 AIME I problems and solutions. The test was held on Wednesday, January 31 – Thursday, February 1, 2024.”
web.archive.org · website · retrieved 2026-10-15
“AIME I (main AIME competition date): Thursday, Feb. 1, 2024 between 1:30 pm ET and 5:30 pm ET.”
web.archive.org · website · retrieved 2026-10-15
“AIME II (alternate AIME competition date): Wednesday, Feb. 7, 2024 between 1:30 pm ET and 5:30 pm ET.”
web.archive.org · website · retrieved 2026-10-15

Not seen AIME 2025 · its test items were published from 2025-02-06, after the model's stated training-data cutoff (2023-12-01)

Evidence (3)
“Date cutoff:** Dec. 2023.”
huggingface.co · model card · Model Description · retrieved 2026-10-15
“2025 AIME I problems and solutions. The test was held on Thursday, February 6, 2025.”
web.archive.org · website · retrieved 2026-10-15
“This year the AIME I will take place on February 6, 2025, while the alternate AIME II will take place on February 12, 2025.”
maa.org · website · retrieved 2026-10-15

Related data seen MATH-500 · in fine-tuning data: related dataset MATH (Mathematics Aptitude Test of Heuristics) was used

Evidence (3)
“dataset comprising GSM8K, MATH training sets, and prompts with constraints from Lambert et al. (2024).”
arxiv.org · paper · Section 5 (RLVR) · retrieved 2026-10-15
“We therefore evaluate our models only on the remaining 500 held-out problems. We selected these 500 test problems uniformly at random, and we believe they are representative of the test set as a whole.”
raw.githubusercontent.com · readme · MATH Splits · retrieved 2026-10-15
“This dataset contains a subset of 500 problems from the MATH benchmark that OpenAI created in their _Let's Verify Step by Step_ paper.”
huggingface.co · dataset card · retrieved 2026-10-15

Notes

Instruct models (SFT, DPO, RLVR) were released with the base models and are recorded as fine_tuning. Mid-training (Dolmino) includes the GSM8K train split and GSM8K-derived synthetic data; only its FLAN component was decontaminated. The paper notes 200 of 1319 GSM8K test examples were used as a development set during mid-training data development (not as training data). Fine-tuning exclusions follow the Tulu 3 recipe the paper says it followed. Later OLMo 2 releases (OLMo-2-0325-32B, OLMo-2-0425-1B) share olmo-mix-1124 and Dolmino-mix-1124 but are separate releases with slightly different post-training (tulu-3-sft-olmo-2-mixture-0225, GRPO RLVR) and are not entered here.

Sources