Contamination MapText / LLM / Model
Pythia (deduplicated)
- Organisation
- EleutherAI
- Family
- Pythia · version v1 (retrained suite, renamed January 2023)
- Released
- 2023-04-03 (arXiv v1 submission date of 2304.01373 (https://arxiv.org/abs/2304.01373v1). HF repo EleutherAI/pythia-12b-deduped createdAt 2023-02-27 (https://huggingface.co/api/models/EleutherAI/pythia-12b-deduped).)
- Checkpoints
- EleutherAI/pythia-14m-deduped, EleutherAI/pythia-31m-deduped, EleutherAI/pythia-70m-deduped, EleutherAI/pythia-160m-deduped, EleutherAI/pythia-410m-deduped, EleutherAI/pythia-1b-deduped, EleutherAI/pythia-1.4b-deduped, EleutherAI/pythia-2.8b-deduped, EleutherAI/pythia-6.9b-deduped, EleutherAI/pythia-12b-deduped
- Weights
- Open · Apache-2.0
- Links
- PaperModel cardCode
Training data disclosure: Full list
Level determined for 20% of benchmark cells.
- TestTest data seen2
- SeenSeen, split unclear0
- TrainTraining data seen0
- Related data seen1
- NoNot seen0
- ?Unknown12
“We train one suite of 8 models on the Pile, and the other on a copy of the Pile after applying near-deduplication with MinHashLSH and a threshold of 0.87”
“Pythia-12B-deduped was trained on the Pile after the dataset has been globally deduplicated”
“Pythia-12B-deduped has not been fine-tuned for downstream contexts in which language models are commonly deployed”
Training corpora
-
The Pile
· 22 sources (complete list) · pre-training
· homepage
Evidence that Pythia (deduplicated) uses it
“Pythia-12B-deduped was trained on the Pile after the dataset has been globally deduplicated”
huggingface.co · model card · Training data · retrieved 2026-10-15 “Pythia-14M-deduped was only trained on the deduplicated Pile”
huggingface.co · model card · Note · retrieved 2026-10-15 “Pythia-31M-deduped was only trained on the deduplicated Pile”
huggingface.co · model card · Note · retrieved 2026-10-15 Contents of this corpus
-
Pile-CC (broad web data: may contain benchmark test sets) · split: unknown
Filtered subset of Common Crawl.
-
PubMed Central (not yet matched to a dataset) · split: unknown
Evidence
“PubMed Central”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
Books3 (not yet matched to a dataset) · split: unknown
Marked with a dagger in Table 1 (used with minimal preprocessing from prior work).
Evidence
“Books3”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
OpenWebText2 (broad web data: may contain benchmark test sets) · split: unknown
Web scrape of Reddit-upvoted outgoing links, content up until 2020.
-
ArXiv (not yet matched to a dataset) · split: unknown
Evidence
“ArXiv”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
Github (broad web data: may contain benchmark test sets) · split: unknown
GitHub code dump.
-
FreeLaw (not yet matched to a dataset) · split: unknown
Evidence
“FreeLaw”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
Stack Exchange (not yet matched to a dataset) · split: unknown
Evidence
“Stack Exchange”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
USPTO Backgrounds (not yet matched to a dataset) · split: unknown
Evidence
“USPTO Backgrounds”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
PubMed Abstracts (not yet matched to a dataset) · split: unknown
Evidence
“PubMed Abstracts”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
Gutenberg (PG-19) (not yet matched to a dataset) · split: unknown
Evidence
“Gutenberg (PG-19)”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
OpenSubtitles (not yet matched to a dataset) · split: unknown
Evidence
“OpenSubtitles”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
Wikipedia (en) (not yet matched to a dataset) · split: unknown
Evidence
“Wikipedia (en)”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
DM Mathematics (not yet matched to a dataset) · split: unknown
DeepMind Mathematics (Saxton et al., 2019); which of its splits are included is not stated in the quoted text.
-
Ubuntu IRC (not yet matched to a dataset) · split: unknown
Evidence
“Ubuntu IRC”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
BookCorpus2 (not yet matched to a dataset) · split: unknown
Evidence
“BookCorpus2”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
EuroParl (not yet matched to a dataset) · split: unknown
Evidence
“EuroParl”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
HackerNews (not yet matched to a dataset) · split: unknown
Evidence
“HackerNews”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
YoutubeSubtitles (not yet matched to a dataset) · split: unknown
Evidence
“YoutubeSubtitles”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
PhilPapers (not yet matched to a dataset) · split: unknown
Evidence
“PhilPapers”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
NIH ExPorter (not yet matched to a dataset) · split: unknown
Evidence
“NIH ExPorter”
arxiv.org · paper · Table 1 · retrieved 2026-10-15 -
Enron Emails (not yet matched to a dataset) · split: unknown
Evidence
“Enron Emails”
arxiv.org · paper · Table 1 · retrieved 2026-10-15
-
Pile-CC (broad web data: may contain benchmark test sets) · split: unknown
Contamination analyses
Analyses that found benchmark test data in this model's training data, or in a corpus it uses.
-
HumanEval
· pre-training data · in The Pile
Riddell et al. (2024): 12.2% of HumanEval gold solutions have a perfect (surface- or semantic-level) match in the Pile's GitHub split; CONDA 2024 contamination report also lists openai_humaneval as found (>0%) in EleutherAI/pile. (WIMBD's exact-match input-field check reports 0.00% for openai-humaneval in the Pile, Table 16.)
Evidence
“Results in Fig. 1 show that 12.2% of the solutions in HumanEval have been seen by models trained on the P ILE”
arxiv.org third-party study· study · Section 4.1 · retrieved 2026-10-15 “We use the GitHub split of the training dataset, which has a raw size of 95.16 GiB.”
arxiv.org third-party study· study · Section 3.1 · retrieved 2026-10-15 “amazon_reviews_multi (Keung et al., 2020), mbpp, openai_humaneval (Chen et al., 2021), liar”
arxiv.org third-party study· study · Table 1 (row EleutherAI/pile) · retrieved 2026-10-15 -
MMLU (Measuring Massive Multitask Language Understanding)
· pre-training data · in The Pile
MMLU test questions are also substantially 'dirty' in Pile-train: DCLM-baseline (27.70%) has only 1.1x more dirty entries than Pile-train. Exact Pile-train rate is in Table 2 (not reliably extracted from the PDF text).
Evidence
“Compared with Pile-train, most benchmarks show a higher dirty rate on DCLM-baseline. For example, ARC-Challenge has 17× more and MMLU has 1.1× more dirty entries.”
arxiv.org third-party study· study · Section 4.2 · retrieved 2026-10-15
Benchmark cells
Classic LLM benchmarks
Test data seen MMLU (Measuring Massive Multitask Language Understanding) · a contamination analysis found test examples in the pre-training data (MMLU test questions are also substantially 'dirty' in Pile-train: DCLM-baseline (27.70%) has only 1.1x more dirty entries than Pile-train. Exact Pile-train rate is in Table 2 (not reliably extracted from the PDF text).) [third-party study] (via The Pile)
Evidence (4)
“Pythia-12B-deduped was trained on the Pile after the dataset has been globally deduplicated”
“Pythia-14M-deduped was only trained on the deduplicated Pile”
“Pythia-31M-deduped was only trained on the deduplicated Pile”
“Compared with Pile-train, most benchmarks show a higher dirty rate on DCLM-baseline. For example, ARC-Challenge has 17× more and MMLU has 1.1× more dirty entries.”
Unknown GSM8K (Grade School Math 8K) · training data not disclosed in enough detail
Unknown MATH (Mathematics Aptitude Test of Heuristics) · training data not disclosed in enough detail
Test data seen HumanEval · a contamination analysis found test examples in the pre-training data (Riddell et al. (2024): 12.2% of HumanEval gold solutions have a perfect (surface- or semantic-level) match in the Pile's GitHub split; CONDA 2024 contamination report also lists openai_humaneval as found (>0%) in EleutherAI/pile. (WIMBD's exact-match input-field check reports 0.00% for openai-humaneval in the Pile, Table 16.)) [third-party study] (via The Pile)
Evidence (6)
“Pythia-12B-deduped was trained on the Pile after the dataset has been globally deduplicated”
“Pythia-14M-deduped was only trained on the deduplicated Pile”
“Pythia-31M-deduped was only trained on the deduplicated Pile”
“Results in Fig. 1 show that 12.2% of the solutions in HumanEval have been seen by models trained on the P ILE”
“We use the GitHub split of the training dataset, which has a raw size of 95.16 GiB.”
“amazon_reviews_multi (Keung et al., 2020), mbpp, openai_humaneval (Chen et al., 2021), liar”
Unknown ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · training data not disclosed in enough detail
Unknown HellaSwag · training data not disclosed in enough detail
Unknown TruthfulQA · training data not disclosed in enough detail
Unknown WinoGrande · training data not disclosed in enough detail
Recent LLM benchmarks
Unknown GPQA Diamond · training data not disclosed in enough detail
MMLU-Pro · a contamination analysis found test examples of the related dataset MMLU (Measuring Massive Multitask Language Understanding) (MMLU test questions are also substantially 'dirty' in Pile-train: DCLM-baseline (27.70%) has only 1.1x more dirty entries than Pile-train. Exact Pile-train rate is in Table 2 (not reliably extracted from the PDF text).) [third-party study] (via The Pile)
Evidence (6)
“Pythia-12B-deduped was trained on the Pile after the dataset has been globally deduplicated”
“Pythia-14M-deduped was only trained on the deduplicated Pile”
“Pythia-31M-deduped was only trained on the deduplicated Pile”
“Compared with Pile-train, most benchmarks show a higher dirty rate on DCLM-baseline. For example, ARC-Challenge has 17× more and MMLU has 1.1× more dirty entries.”
“**Original MMLU Questions:** Part of the dataset comes from the original MMLU dataset. We remove the trivial and ambiguous questions.”
“Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU.”
Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail
Unknown SWE-bench Verified · training data not disclosed in enough detail
Unknown AIME 2024 · training data not disclosed in enough detail
Unknown AIME 2025 · training data not disclosed in enough detail
Unknown MATH-500 · training data not disclosed in enough detail
Notes
Base models only, no fine-tuning. Same data order and hyperparameters as the non-deduplicated suite.
Sources
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling · paper · retrieved 2026-10-15
- EleutherAI/pythia-12b-deduped model card · model card · retrieved 2026-10-15
- EleutherAI/pythia-14m-deduped model card · model card · retrieved 2026-10-15
- EleutherAI/pythia-31m-deduped model card · model card · retrieved 2026-10-15
- HF API metadata for EleutherAI/pythia-12b-deduped · website · retrieved 2026-10-15