OLMo (February 2024)

Training data disclosure: Full list

Level determined for 26% of benchmark cells.

• TestTest data seen1
• SeenSeen, split unclear0
• TrainTraining data seen0
• Rel.Related data seen0
• NoNot seen3
• ?Unknown11

Training corpora

• Dolma v1.5 · 7 sources (complete list) · pre-training · homepage
  Evidence that OLMo (February 2024) uses it

  “Used to train [OLMo-1B](https://huggingface.co/allenai/OLMo-1B). Roughly 3 trillion tokens.”

  “A sample of roughly 1.9 trillion tokens used to train [OLMo-7B](https://huggingface.co/allenai/OLMo-7B)”

  Contents of this corpus

  • Common Crawl (broad web data: may contain benchmark test sets) · split: unknown
    Evidence

    “Common Crawl | web pages | 9,022 | 3,370”

  • GitHub (The Stack) (broad web data: may contain benchmark test sets) · split: unknown

    Code from GitHub; the Dolma paper reports test sets of several benchmarks (incl. HumanEval) in this subset.

    Evidence

    “GitHub Ð code 1,043 210 260 411”

    “The Stack | code| 1,043| 210”

  • C4 (broad web data: may contain benchmark test sets) · split: unknown
    Evidence

    “C4 | web pages | 790 | 364”

  • Reddit (broad web data: may contain benchmark test sets) · split: unknown
    Evidence

    “Reddit| social media| 339 | 377”

  • peS2o (Semantic Scholar) (not yet matched to a dataset) · split: unknown
    Evidence

    “PeS2o | STEM papers| 268 | 38.8”

  • Project Gutenberg (not yet matched to a dataset) · split: unknown
    Evidence

    “Project Gutenberg | books | 20.4 | 0.056”

  • Wikipedia, Wikibooks (not yet matched to a dataset) · split: unknown
    Evidence

    “Wikipedia, Wikibooks | encyclopedic | 16.2 | 6.2”

• Tulu V2 SFT mixture · 11 sources (complete list) · fine-tuning · homepage
  Evidence that OLMo (February 2024) uses it

  “| [OLMo 7B SFT](https://huggingface.co/allenai/OLMo-7B-SFT) | SFT | [Tulu 2 SFT Mix](https://huggingface.co/datasets/allenai/tulu-v2-sft-mixture)”

  “tulu-v2-sft-mixture-olmo-2048”

  Contents of this corpus

  • FLAN v2 (not yet matched to a dataset) · split: train · fine-tuning

    Card: subsets use only their training set when splits exist. FLAN v2 is built from many NLP task datasets; which benchmark train sets are included is not listed here.

    Evidence

    “We use 50,000 examples sampled from FLAN v2.”

  • FLAN v2 CoT subset (not yet matched to a dataset) · split: train · fine-tuning
    Evidence

    “we sample another 50,000 examples from the CoT”

  • Open Assistant 1 (not yet matched to a dataset) · split: unknown · fine-tuning
    Evidence

    “We isolate the highest-scoring paths in each conversation tree and use these samples, resulting in 7,708 examples.”

  • ShareGPT (broad web data: may contain benchmark test sets) · split: unknown · fine-tuning

    Real user-ChatGPT conversations. The later Tulu 3 paper (AI2) lists ShareGPT (anon8231489123/ShareGPT_Vicuna_unfiltered, the source linked by this card) in Table 37 among public datasets with >5% overlap with its evaluation suite (per-benchmark values not extracted reliably); overlap of the processed Tulu 2 subset itself was not measured.

    Evidence

    “We use all 114,046 from our processed ShareGPT dataset”

    “Table 37 Public datasets where we found significant (>5% eval overlap) contamination with our evaluation suite.”

  • GPT4-Alpaca (not yet matched to a dataset) · split: unknown · fine-tuning
    Evidence

    “We sample 20,000 samples from GPT-4 Alpaca to further include distilled GPT-4 data.”

  • Code-Alpaca (not yet matched to a dataset) · split: unknown · fine-tuning
    Evidence

    “We use all 20,022 examples from Code Alpaca”

  • LIMA (not yet matched to a dataset) · split: unknown · fine-tuning
    Evidence

    “We use 1,030 examples from LIMA as an example of carefully curated data.”

  • WizardLM Evol Instruct (not yet matched to a dataset) · split: unknown · fine-tuning
    Evidence

    “We subsample 30,000 examples from WizardLM”

  • Open-Orca (not yet matched to a dataset) · split: unknown · fine-tuning
    Evidence

    “We sample 30,000 samples generated by GPT-4 from OpenOrca”

  • Hardcoded (not yet matched to a dataset) · split: unknown · fine-tuning
    Evidence

    “We wrote 14 samples and repeat each sample 10 times in the mixture, resulting in 140 total samples.”

  • Science (not yet matched to a dataset) · split: unknown · fine-tuning
    Evidence

    “7,544 examples from a mixture of scientific document understand tasks”

Other declared training data

• UltraFeedback (cleaned, binarized) (not yet matched to a dataset) · split: unknown · fine-tuning

  DPO data for OLMo-7B-Instruct (Argilla 'fixed' variant).

  Evidence

  “A modified form of UltraFeedback (Cui et al., 2023), with TruthfulQA prompts removed.”

  “but with TruthfulQA prompts removed and source annotations added”

Contamination analyses

Analyses that found benchmark test data in this model's training data, or in a corpus it uses.

• HumanEval · pre-training data · in Dolma v1.5

  WIMBD analysis by the Dolma authors: over 90% of the OpenAI HumanEval test set appears in Dolma documents (mostly via the GitHub code subset); only Paloma was decontaminated, not downstream benchmarks.

  Evidence

  “other datasets are mostly contaminated, with over 90% of their test sets appearing in Dolma documents: OpenAI HumanEval (Chen et al., 2021)”

  “Further analysis reveals that many of these sets are contaminated in our code subset, as public repositories in GitHub often contains copies of these datasets.”

Benchmark cells

Classic LLM benchmarks

Recent LLM benchmarks

Notes

Instruct variants (SFT on Tulu 2 mix, DPO on cleaned UltraFeedback) are recorded as fine_tuning on the same entry; they are described in the same paper (v3/v4). TruthfulQA prompts were removed from UltraFeedback only (the Tulu 2 paper found UltraFeedback used TruthfulQA prompts); the Tulu 2 SFT mixture itself is not stated to be decontaminated. Dolma v1.5 contains >90% of HumanEval test problems per the Dolma paper (see corpus dolma-1.5). Later OLMo 1.x releases (OLMo-7B-0424 / 1.7 on Dolma 1.7, OLMo-7B-0724) are not covered. Decontamination statements not shown as cells (they name no benchmark tracked here, or none at all): Paloma.

Sources

• OLMo: Accelerating the Science of Language Models · paper · retrieved 2026-10-15
• allenai/OLMo-7B model card · model card · retrieved 2026-10-15
• allenai/OLMo-7B-Instruct model card · model card · retrieved 2026-10-15
• allenai/dolma dataset card · dataset card · retrieved 2026-10-15
• Dolma paper · paper · retrieved 2026-10-15
• Tulu 2 paper · paper · retrieved 2026-10-15