Contamination MapText / LLM / Model
BLOOM
- Organisation
- BigScience
- Family
- BLOOM
- Released
- 2022-07-11 (Model card 'Release Date Estimate: Monday, 11.July.2022' (https://huggingface.co/bigscience/bloom/raw/main/README.md). arXiv v1 of 2211.05100 is 2022-11-09.)
- Checkpoints
- bigscience/bloom, bigscience/bloom-560m, bigscience/bloom-1b1, bigscience/bloom-1b7, bigscience/bloom-3b, bigscience/bloom-7b1
- Weights
- Open · BigScience BLOOM RAIL License v1.0 (bigscience-bloom-rail-1.0)
- Links
- PaperModel cardCode
Training data disclosure: Partial list
Level determined for 0% of benchmark cells.
- TestTest data seen0
- SeenSeen, split unclear0
- TrainTraining data seen0
- Related data seen0
- NoNot seen0
- ?Unknown15
“BLOOM was trained on the ROOTS corpus (Laurençon et al., 2022), a composite collection of 498 Hugging Face datasets (Lhoest et al., 2021) amounting to 1.61 terabytes of text that span 46 natural languages and 13 programming languages.”
“direct access to the 223 (out of 498) components that we are able to distribute taking into account their licensing status, privacy risks, and agreements with their original custodians”
Training corpora
-
ROOTS (Responsible Open-science Open-collaboration Text Sources)
· 16 sources · pre-training
· homepage
Evidence that BLOOM uses it
“BLOOM was trained on the ROOTS corpus (Laurençon et al., 2022), a composite collection of 498 Hugging Face datasets”
arxiv.org · paper · Section 3.1 · retrieved 2026-10-15 “The ROOTS dataset contains around 341 billion tokens of text, so we aimed to train all models for the equivalent amount of tokens.”
arxiv.org · paper · Section 3.5 · retrieved 2026-10-15 Contents of this corpus
-
BigScience Catalogue (crowdsourced language resources) (not yet matched to a dataset) · split: unknown
Includes existing NLP datasets (QA, summarization, dialogue). Individual sources listed in ROOTS Appendix E, Table 2; a few are listed below.
Evidence
“The first part of our corpus, accounting for 62% of the final dataset size (in bytes), was made up of a collection of monolingual and multilingual language resources”
arxiv.org · paper · Section 2 · retrieved 2026-10-15 “downloading and extracting the text field from a variety of NLP datasets in various formats (including e.g. question answering, summarization, or dialogue datasets)”
arxiv.org · paper · Section 3.1.3 · retrieved 2026-10-15 -
Pseudo-Crawled Data (broad web data: may contain benchmark test sets) · split: unknown
Pages from 614 seed domains (539 with text) retrieved from Common Crawl; 147 pseudo-crawl-based datasets after filtering.
Evidence
“we retrieved pages corresponding to the target domain names from 18 snapshots archived by Common Crawl in 2020 and 2021 in Web ARChive (WARC) format”
arxiv.org · paper · Section 2.1 · retrieved 2026-10-15 -
GitHub Code (broad web data: may contain benchmark test sets) · split: unknown
GitHub on BigQuery dump; per ROOTS, filtered to GPL licenses only due to a bug.
Evidence
“using the same language selection as AlphaCode (Li et al., 2022)”
arxiv.org · paper · Section 2.1 · retrieved 2026-10-15 “The catalogue was further complemented with a dataset of programming languages collected from the GitHub data collection on Google’s BigQuery”
arxiv.org · paper · Section 3.1.2 · retrieved 2026-10-15 -
OSCAR (broad web data: may contain benchmark test sets) · split: unknown
Filtered OSCAR 21.09 (Common Crawl February 2021).
Evidence
“we chose OSCAR version 21.09 (Ortiz Suárez et al., 2020), based on the Common Crawl snapshot of February 2021, to make up the remaining 38% of our final dataset.”
arxiv.org · paper · Section 3 · retrieved 2026-10-15 “we further sourced data from OSCAR version 21.09, corresponding to the February 2021 snapshot of the Common Crawl”
arxiv.org · paper · Section 3.1.2 · retrieved 2026-10-15 -
Stack Exchange (not yet matched to a dataset) · split: unknown
Evidence
“Stack Exchange”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
S2ORC (not yet matched to a dataset) · split: unknown
Evidence
“S2ORC”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
The Pile: EuroParl (not yet matched to a dataset) · split: unknown
Evidence
“The Pile: EuroParl”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
The Pile: USPTO (not yet matched to a dataset) · split: unknown
Evidence
“The Pile: USPTO”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
Project Gutenberg (not yet matched to a dataset) · split: unknown
Evidence
“Project Gutenberg”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
Wikipedia (not yet matched to a dataset) · split: unknown
Multilingual Wikipedia.
Evidence
“Wikipedia”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
Opensubtitles2016 (not yet matched to a dataset) · split: unknown
Evidence
“Opensubtitles2016”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
DuReader (not yet matched to a dataset) · split: unknown
Chinese reading-comprehension (QA) dataset; split not stated.
Evidence
“DuReader”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
IndoNLI (not yet matched to a dataset) · split: unknown
Indonesian NLI dataset; split not stated.
Evidence
“IndoNLI”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
XQUAD-ca (not yet matched to a dataset) · split: unknown
Catalan version of XQuAD (an evaluation set); split not stated.
Evidence
“XQUAD-ca”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
ViquiQuAD (not yet matched to a dataset) · split: unknown
Catalan QA dataset; split not stated.
Evidence
“ViquiQuAD”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15 -
VilaQuAD (not yet matched to a dataset) · split: unknown
Catalan QA dataset; split not stated.
Evidence
“VilaQuAD”
arxiv.org · paper · Appendix E, Table 2 (List of datasets used in crowdsourced dataset) · retrieved 2026-10-15
-
BigScience Catalogue (crowdsourced language resources) (not yet matched to a dataset) · split: unknown
Benchmark cells
Classic LLM benchmarks
Unknown MMLU (Measuring Massive Multitask Language Understanding) · training data not disclosed in enough detail
Unknown GSM8K (Grade School Math 8K) · training data not disclosed in enough detail
Unknown MATH (Mathematics Aptitude Test of Heuristics) · training data not disclosed in enough detail
Unknown HumanEval · training data not disclosed in enough detail
Unknown ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · training data not disclosed in enough detail
Unknown HellaSwag · training data not disclosed in enough detail
Unknown TruthfulQA · training data not disclosed in enough detail
Unknown WinoGrande · training data not disclosed in enough detail
Recent LLM benchmarks
Unknown GPQA Diamond · training data not disclosed in enough detail
Unknown MMLU-Pro · training data not disclosed in enough detail
Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail
Unknown SWE-bench Verified · training data not disclosed in enough detail
Unknown AIME 2024 · training data not disclosed in enough detail
Unknown AIME 2025 · training data not disclosed in enough detail
Unknown MATH-500 · training data not disclosed in enough detail
Notes
Pretrained base models only. The multitask-finetuned BLOOMZ models (trained on xP3) are separate releases and not covered here. Model card lists individual data cards / an interactive corpus map for ROOTS; not all components are public. No decontamination statement found in the BLOOM paper or ROOTS paper.
Sources
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model · paper · retrieved 2026-10-15
- bigscience/bloom model card · model card · retrieved 2026-10-15
- bigscience/bloom-560m model card · model card · retrieved 2026-10-15
- The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset · paper · retrieved 2026-10-15
- HF API metadata for bigscience/bloom · website · retrieved 2026-10-15