Llama 2

Training data disclosure: Vague

Level determined for 40% of benchmark cells.

• TestTest data seen2
• SeenSeen, split unclear0
• TrainTraining data seen0
• Rel.Related data seen1
• NoNot seen3
• ?Unknown9

Other declared training data

• a new mix of data from publicly available sources (broad web data: may contain benchmark test sets) · split: unknown

  2 trillion tokens; no sources named.

  Evidence

  “Our training corpus includes a new mix of data from publicly available sources, which does not include data from Meta’s products or services.”

  “A new mix of publicly available online data”

• publicly available instruction tuning data (Chung et al., 2022) (not yet matched to a dataset) · split: unknown · fine-tuning

  Cited as Chung et al. (2022); the paper does not name the individual datasets. Authors say they then set aside millions of third-party examples in favour of their own annotations.

  Evidence

  “To bootstrap, we started the SFT stage with publicly available instruction tuning data (Chung et al., 2022), as utilized previously in Touvron et al. (2023).”

• vendor-collected SFT annotations (27,540) (not yet matched to a dataset) · split: unknown · fine-tuning
  Evidence

  “We stopped annotating SFT after collecting a total of 27,540 annotations.”

• Meta reward modeling data (not yet matched to a dataset) · split: unknown · fine-tuning

  Human preference data used for RLHF reward models.

  Evidence

  “We collected a large dataset of over 1 million binary comparisons based on humans applying our specified guidelines, which we refer to as Meta reward modeling data.”

• Anthropic Helpful and Harmless (not yet matched to a dataset) · split: unknown · fine-tuning

  Open-source preference dataset used in the reward-model training mixture (RLHF).

  Evidence

  “We combine our newly collected data with existing open-source preference datasets to form a larger training dataset.”

  “against multiple open-source preference datasets including Anthropic Helpful and Harmless (Bai et al., 2022a), OpenAI Summarize (Stiennon et al., 2020), OpenAI WebGPT (Nakano et al., 2021), StackExchange (Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla).”

• OpenAI Summarize (not yet matched to a dataset) · split: unknown · fine-tuning

  Open-source preference dataset used in the reward-model training mixture (RLHF).

  Evidence

  “We combine our newly collected data with existing open-source preference datasets to form a larger training dataset.”

  “against multiple open-source preference datasets including Anthropic Helpful and Harmless (Bai et al., 2022a), OpenAI Summarize (Stiennon et al., 2020), OpenAI WebGPT (Nakano et al., 2021), StackExchange (Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla).”

• OpenAI WebGPT (not yet matched to a dataset) · split: unknown · fine-tuning

  Open-source preference dataset used in the reward-model training mixture (RLHF).

  Evidence

  “We combine our newly collected data with existing open-source preference datasets to form a larger training dataset.”

  “against multiple open-source preference datasets including Anthropic Helpful and Harmless (Bai et al., 2022a), OpenAI Summarize (Stiennon et al., 2020), OpenAI WebGPT (Nakano et al., 2021), StackExchange (Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla).”

• StackExchange (not yet matched to a dataset) · split: unknown · fine-tuning

  Open-source preference dataset used in the reward-model training mixture (RLHF).

  Evidence

  “We combine our newly collected data with existing open-source preference datasets to form a larger training dataset.”

  “against multiple open-source preference datasets including Anthropic Helpful and Harmless (Bai et al., 2022a), OpenAI Summarize (Stiennon et al., 2020), OpenAI WebGPT (Nakano et al., 2021), StackExchange (Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla).”

• Stanford Human Preferences (not yet matched to a dataset) · split: unknown · fine-tuning

  Open-source preference dataset used in the reward-model training mixture (RLHF).

  Evidence

  “We combine our newly collected data with existing open-source preference datasets to form a larger training dataset.”

  “against multiple open-source preference datasets including Anthropic Helpful and Harmless (Bai et al., 2022a), OpenAI Summarize (Stiennon et al., 2020), OpenAI WebGPT (Nakano et al., 2021), StackExchange (Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla).”

• Synthetic GPT-J (not yet matched to a dataset) · split: unknown · fine-tuning

  Open-source preference dataset used in the reward-model training mixture (RLHF).

  Evidence

  “We combine our newly collected data with existing open-source preference datasets to form a larger training dataset.”

  “against multiple open-source preference datasets including Anthropic Helpful and Harmless (Bai et al., 2022a), OpenAI Summarize (Stiennon et al., 2020), OpenAI WebGPT (Nakano et al., 2021), StackExchange (Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla).”

Contamination analyses

Analyses that found benchmark test data in this model's training data, or in a corpus it uses.

• HellaSwag · pre-training data

  With min. match length L=40: 2,651 of 10,042 samples (26%) 'not clean' (>=20% of tokens in 10+-token n-grams found in pre-training data) and 848 (8.4%) 'dirty' (>=80%); dirty samples scored 92.2 vs 82.5 sampling mean for 70B (Z=7.42). Authors conclude HellaSwag scores appear boosted by contamination, 70B more than 7B.

  Evidence

  “We observe that only HellaSwag and MMLU-Humanities appear to have been boosted due to contamination in the training data, with the 70B model appearing to have gained a greater benefit than the 7B model, as one might expect.”

  “We consider a token to be contaminated if it appears in any token n-gram longer than 10 tokens in both the evaluation sample and the training set, and define the contamination percentage of a sample to be the percentage of tokens contaminated.”

  “Not Clean 67.5 2651 89.5 82.4 9.56”

  “Dirty 86.1 848 92.2 82.5 7.42”

• MMLU (Measuring Massive Multitask Language Understanding) · pre-training data

  MMLU-Humanities (L=50): 709 of 4,705 samples (15%) 'not clean' (>=20% token contamination) and 520 (11%) 'dirty' (>=80%); flagged as boosted by contamination. MMLU-Overall for 70B: 2,180 of 14,042 (15.5%) not clean, 1,536 (10.9%) dirty, with a small effect on the overall score (-0.9 clean-vs-mean delta).

  Evidence

  “We observe that only HellaSwag and MMLU-Humanities appear to have been boosted due to contamination in the training data, with the 70B model appearing to have gained a greater benefit than the 7B model, as one might expect.”

  “Furthermore, the impact of this effect on MMLU-Humanities appears to cause a benefit for MMLU-Overall for the 70B model, albeit with only a small delta (-0.9) between the "clean" subset performance and the sampling mean.”

  “Not Clean 85.12 709 82.7 65.3 9.71”

  “Dirty 94.4 1536 78.2 68.9 7.87”

Benchmark cells

Classic LLM benchmarks

Recent LLM benchmarks

Notes

Contamination analysis (Appendix A.6) is on pre-training data with pretrained-only models; only HellaSwag and MMLU(-Humanities) were judged affected. Other benchmarks may have overlap without measurable benefit; the paper omits them. Chat models share pre-training data and add the listed fine-tuning data. Third-party signal on HellaSwag (indirect, not shown as a cell): Sample-specific contamination detected on HellaSwag (p < 0.5%, estimated effect 3-8%); ConStat notes the provider itself reported HellaSwag contamination.

Sources

• Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv v2) · paper · retrieved 2026-10-15
• Llama 2 paper, arXiv HTML v2 (used for Table 51 rows) · paper · retrieved 2026-10-15
• Llama 2 MODEL_CARD.md (meta-llama/llama) · model card · retrieved 2026-10-15
• meta-llama/llama-models README (launch dates) · readme · retrieved 2026-10-15