Contamination Map Methodology

Text / LLM / Model

Pythia

Organisation
EleutherAI
Family
Pythia · version v1 (retrained suite, renamed January 2023)
Released
2023-04-03 (arXiv v1 submission date of 2304.01373 (https://arxiv.org/abs/2304.01373v1). HF repos of the current suite were created Feb 2023 (e.g. EleutherAI/pythia-12b createdAt 2023-02-28, https://huggingface.co/api/models/EleutherAI/pythia-12b).)
Checkpoints
EleutherAI/pythia-14m, EleutherAI/pythia-31m, EleutherAI/pythia-70m, EleutherAI/pythia-160m, EleutherAI/pythia-410m, EleutherAI/pythia-1b, EleutherAI/pythia-1.4b, EleutherAI/pythia-2.8b, EleutherAI/pythia-6.9b, EleutherAI/pythia-12b
Weights
Open · Apache-2.0
Links
PaperModel cardCode

Training data disclosure: Full list

Level determined for 20% of benchmark cells.

“We train our models on the Pile (Gao et al., 2020; Biderman et al., 2022), a curated collection of English language datasets for training large language models”
arxiv.org · paper · Section 2.2 · retrieved 2026-10-15
“The Pile was not deduplicated before being used to train Pythia-12B.”
huggingface.co · model card · Training data · retrieved 2026-10-15
“Pythia-12B has not been fine-tuned for downstream contexts in which language models are commonly deployed”
huggingface.co · model card · Out-of-scope use · retrieved 2026-10-15

Training corpora

Contamination analyses

Analyses that found benchmark test data in this model's training data, or in a corpus it uses.

Benchmark cells

Classic LLM benchmarks

Test data seen MMLU (Measuring Massive Multitask Language Understanding) · a contamination analysis found test examples in the pre-training data (MMLU test questions are also substantially 'dirty' in Pile-train: DCLM-baseline (27.70%) has only 1.1x more dirty entries than Pile-train. Exact Pile-train rate is in Table 2 (not reliably extracted from the PDF text).) [third-party study] (via The Pile)

Evidence (4)
“For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated.”
huggingface.co · model card · Introduction · retrieved 2026-10-15
“Pythia-14M was trained on the Pile.”
huggingface.co · model card · Training data · retrieved 2026-10-15
“Pythia-31M was trained on the Pile.”
huggingface.co · model card · Training data · retrieved 2026-10-15
“Compared with Pile-train, most benchmarks show a higher dirty rate on DCLM-baseline. For example, ARC-Challenge has 17× more and MMLU has 1.1× more dirty entries.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15

Unknown GSM8K (Grade School Math 8K) · training data not disclosed in enough detail

Unknown MATH (Mathematics Aptitude Test of Heuristics) · training data not disclosed in enough detail

Test data seen HumanEval · a contamination analysis found test examples in the pre-training data (Riddell et al. (2024): 12.2% of HumanEval gold solutions have a perfect (surface- or semantic-level) match in the Pile's GitHub split; CONDA 2024 contamination report also lists openai_humaneval as found (>0%) in EleutherAI/pile. (WIMBD's exact-match input-field check reports 0.00% for openai-humaneval in the Pile, Table 16.)) [third-party study] (via The Pile)

Evidence (6)
“For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated.”
huggingface.co · model card · Introduction · retrieved 2026-10-15
“Pythia-14M was trained on the Pile.”
huggingface.co · model card · Training data · retrieved 2026-10-15
“Pythia-31M was trained on the Pile.”
huggingface.co · model card · Training data · retrieved 2026-10-15
“Results in Fig. 1 show that 12.2% of the solutions in HumanEval have been seen by models trained on the P ILE”
arxiv.org third-party study· study · Section 4.1 · retrieved 2026-10-15
“We use the GitHub split of the training dataset, which has a raw size of 95.16 GiB.”
arxiv.org third-party study· study · Section 3.1 · retrieved 2026-10-15
“amazon_reviews_multi (Keung et al., 2020), mbpp, openai_humaneval (Chen et al., 2021), liar”
arxiv.org third-party study· study · Table 1 (row EleutherAI/pile) · retrieved 2026-10-15

Unknown ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · training data not disclosed in enough detail

Unknown HellaSwag · training data not disclosed in enough detail

Unknown TruthfulQA · training data not disclosed in enough detail

Unknown WinoGrande · training data not disclosed in enough detail

Recent LLM benchmarks

Unknown GPQA Diamond · training data not disclosed in enough detail

Related data seen MMLU-Pro · a contamination analysis found test examples of the related dataset MMLU (Measuring Massive Multitask Language Understanding) (MMLU test questions are also substantially 'dirty' in Pile-train: DCLM-baseline (27.70%) has only 1.1x more dirty entries than Pile-train. Exact Pile-train rate is in Table 2 (not reliably extracted from the PDF text).) [third-party study] (via The Pile)

Evidence (6)
“For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated.”
huggingface.co · model card · Introduction · retrieved 2026-10-15
“Pythia-14M was trained on the Pile.”
huggingface.co · model card · Training data · retrieved 2026-10-15
“Pythia-31M was trained on the Pile.”
huggingface.co · model card · Training data · retrieved 2026-10-15
“Compared with Pile-train, most benchmarks show a higher dirty rate on DCLM-baseline. For example, ARC-Challenge has 17× more and MMLU has 1.1× more dirty entries.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15
“**Original MMLU Questions:** Part of the dataset comes from the original MMLU dataset. We remove the trivial and ambiguous questions.”
huggingface.co · dataset card · 2. Dataset Summary · retrieved 2026-10-15
“Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU.”
arxiv.org · paper · Abstract · retrieved 2026-10-15

Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail

Unknown SWE-bench Verified · training data not disclosed in enough detail

Unknown AIME 2024 · training data not disclosed in enough detail

Unknown AIME 2025 · training data not disclosed in enough detail

Unknown MATH-500 · training data not disclosed in enough detail

Notes

Base models only, no fine-tuning. Paper suite is 70M-12B; 14M and 31M were added later (non-deduplicated 14m/31m repos re-hosted Feb 2026 per their cards). Pile-level contamination findings (e.g. HumanEval in the Pile GitHub split, Riddell et al. who evaluated Pythia 1.4B-12B) are recorded on corpus the-pile. Older v0 checkpoints (EleutherAI/pythia-*-v0) not listed.

Sources