Llama 3.1

Training data disclosure: Vague

Level determined for 66% of benchmark cells.

• TestTest data seen7
• SeenSeen, split unclear0
• TrainTraining data seen0
• Rel.Related data seen1
• NoNot seen2
• ?Unknown5

Other declared training data

• web data / publicly available online data (broad web data: may contain benchmark test sets) · split: unknown

  ~15T multilingual tokens; no named datasets. Annealing data upsamples high-quality code and math.

  Evidence

  “Much of the data we utilize is obtained from the web and we describe our cleaning process below.”

  “A new mix of publicly available online data.”

• publicly available instruction datasets (broad web data: may contain benchmark test sets) · split: unknown · fine-tuning

  Datasets not named.

  Evidence

  “The fine-tuning data includes publicly available instruction datasets, as well as over 25M synthetically generated examples.”

• over 25M synthetically generated examples / synthetic data targeting specific capabilities (not yet matched to a dataset) · split: unknown · fine-tuning
  Evidence

  “The fine-tuning data includes publicly available instruction datasets, as well as over 25M synthetically generated examples.”

  “Synthetic data targeting specific capabilities (see Section 4.3 for more details).”

• prompts from human annotation collection with rejection-sampled responses (not yet matched to a dataset) · split: unknown · fine-tuning
  Evidence

  “Prompts from our human annotation collection with rejection-sampled responses.”

• small amounts of human-curated data (not yet matched to a dataset) · split: unknown · fine-tuning
  Evidence

  “Small amounts of human-curated data (see Section 4.3 for more details).”

Declared exclusions

• Dataset MMLU (Measuring Massive Multitask Language Understanding) · test portion excluded from fine-tuning data
  Evidence

  “We apply decontamination of the post-training data by running exact match with the prompts from each benchmark.”

  “General MMLU (Hendrycks et al., 2021a), MMLU-Pro (Wang et al., 2024b), IFEval (Zhou et al., 2023)”

• Dataset MMLU-Pro · test portion excluded from fine-tuning data
  Evidence

  “We apply decontamination of the post-training data by running exact match with the prompts from each benchmark.”

  “General MMLU (Hendrycks et al., 2021a), MMLU-Pro (Wang et al., 2024b), IFEval (Zhou et al., 2023)”

• Dataset GSM8K (Grade School Math 8K) · test portion excluded from fine-tuning data
  Evidence

  “We apply decontamination of the post-training data by running exact match with the prompts from each benchmark.”

  “Math and reasoning GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), GPQA (Rein et al., 2023), ARC-Challenge (Clark et al., 2018)”

• Dataset MATH (Mathematics Aptitude Test of Heuristics) · test portion excluded from fine-tuning data
  Evidence

  “We apply decontamination of the post-training data by running exact match with the prompts from each benchmark.”

  “Math and reasoning GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), GPQA (Rein et al., 2023), ARC-Challenge (Clark et al., 2018)”

• Dataset GPQA (Graduate-Level Google-Proof Q&A) · test portion excluded from fine-tuning data
  Evidence

  “We apply decontamination of the post-training data by running exact match with the prompts from each benchmark.”

  “Math and reasoning GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), GPQA (Rein et al., 2023), ARC-Challenge (Clark et al., 2018)”

• Dataset ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · test portion excluded from fine-tuning data
  Evidence

  “We apply decontamination of the post-training data by running exact match with the prompts from each benchmark.”

  “Math and reasoning GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), GPQA (Rein et al., 2023), ARC-Challenge (Clark et al., 2018)”

• Dataset HumanEval · test portion excluded from fine-tuning data
  Evidence

  “We apply decontamination of the post-training data by running exact match with the prompts from each benchmark.”

  “Code HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021)”

Contamination analyses

Analyses that found benchmark test data in this model's training data, or in a corpus it uses.

• HellaSwag · pre-training data

  85% of HellaSwag examples considered contaminated (8-gram overlap with the pre-training corpus at a dataset-specific token-ratio threshold); estimated performance gain 14.8 / 14.8 / 14.3 points for 8B / 70B / 405B.

  Evidence

  “Table 15: Percentage of evaluation sets considered to be contaminated because similar data exists in the training corpus, and the estimated performance gain that may result from that contamination.”

  “HellaSwag 85 14.8 14.8 14.3”

  “For example, for PiQA and HellaSwag, both the estimation of contamination and the estimation of performance gain are high.”

• GSM8K (Grade School Math 8K) · pre-training data

  41% of GSM8K examples considered contaminated (8-gram overlap); estimated performance gain 0.0 / 0.1 / 1.3 points for 8B / 70B / 405B.

  Evidence

  “Table 15: Percentage of evaluation sets considered to be contaminated because similar data exists in the training corpus, and the estimated performance gain that may result from that contamination.”

  “GSM8K 41 0.0 0.1 1.3”

• MATH (Mathematics Aptitude Test of Heuristics) · pre-training data

  1% of MATH examples considered contaminated at the selected threshold; estimated gain 0.0 / -0.1 / -0.2. Authors note low thresholds yield high contamination for MATH but no performance gains.

  Evidence

  “Table 15: Percentage of evaluation sets considered to be contaminated because similar data exists in the training corpus, and the estimated performance gain that may result from that contamination.”

  “MATH 1 0.0 -0.1 -0.2”

  “For SQuAD and MATH, low thresholds yield high levels of contamination, but no performance gains.”

• WinoGrande · pre-training data

  6% of Winogrande examples considered contaminated; estimated gain -0.1 / -0.1 / -0.2 (no benefit).

  Evidence

  “Table 15: Percentage of evaluation sets considered to be contaminated because similar data exists in the training corpus, and the estimated performance gain that may result from that contamination.”

  “Winogrande 6 -0.1 -0.1 -0.2”

• MMLU (Measuring Massive Multitask Language Understanding) · pre-training data

  8-gram overlap gave contamination scores too high to estimate a performance gain, even at higher thresholds; percentage not reported (dash in Table 15).

  Evidence

  “Finally, for MBPP, HumanEval, MMLU and MMLU-Pro, other contamination detection methods may be needed: even with higher thresholds, 8-gram overlap gives such high contamination scores that it is impossible to get a good performance gain estimate.”

• MMLU-Pro · pre-training data

  8-gram overlap gave contamination scores too high to estimate a performance gain, even at higher thresholds; percentage not reported (dash in Table 15).

  Evidence

  “Finally, for MBPP, HumanEval, MMLU and MMLU-Pro, other contamination detection methods may be needed: even with higher thresholds, 8-gram overlap gives such high contamination scores that it is impossible to get a good performance gain estimate.”

• HumanEval · pre-training data

  8-gram overlap gave contamination scores too high to estimate a performance gain, even at higher thresholds; percentage not reported (dash in Table 15).

  Evidence

  “Finally, for MBPP, HumanEval, MMLU and MMLU-Pro, other contamination detection methods may be needed: even with higher thresholds, 8-gram overlap gives such high contamination scores that it is impossible to get a good performance gain estimate.”

Benchmark cells

Classic LLM benchmarks

Recent LLM benchmarks

Notes

'The Llama 3 Herd of Models' covers Llama 3.1 ('All results in this paper are for the Llama 3.1 models.'). Contamination was measured against the pre-training corpus using 8-gram overlap (Singh et al., 2024). Other Table 15 benchmarks not in the shared id list (AGIEval 98%, BIG-Bench Hard 95%, BoolQ 96%, QuaC 99%, WorldSense 73%, SiQA 63%, PiQA 55%, NaturalQuestions 52%, CommonSenseQA 30%, OpenBookQA 21%, SQuAD 0%) are also reported. The paper states 'We do not include any training sets from commonly used benchmarks in our annealing data.' (training sets, not test sets). The authors separately tested annealing on GSM8K/MATH training sets as an experiment.

Sources

• The Llama 3 Herd of Models (arXiv v3) · paper · retrieved 2026-10-15
• The Llama 3 Herd of Models, arXiv HTML v3 (used for Tables 15 and 16) · paper · retrieved 2026-10-15
• Llama 3.1 MODEL_CARD.md (meta-llama/llama-models) · model card · retrieved 2026-10-15
• meta-llama/llama-models README (launch dates) · readme · retrieved 2026-10-15