Contamination MapText / LLM / Model
Meta Llama 3
- Organisation
- Meta
- Family
- Llama · version 3
- Released
- 2024-04-18 (Model card 'Model Release Date April 18, 2024' (https://raw.githubusercontent.com/meta-llama/llama3/main/MODEL_CARD.md))
- Training cutoff
- 2023-12-01
Evidence
“The pretraining data has a cutoff of March 2023 for the 8B and December 2023 for the 70B models respectively.”
raw.githubusercontent.com · model card · Training Data / Data Freshness · retrieved 2026-10-15 - Checkpoints
- meta-llama/Meta-Llama-3-8B, meta-llama/Meta-Llama-3-70B, meta-llama/Meta-Llama-3-8B-Instruct, meta-llama/Meta-Llama-3-70B-Instruct
- Weights
- Open · Meta Llama 3 Community License
- Links
- Model cardCode
Training data disclosure: Vague
Level determined for 13% of benchmark cells.
- TestTest data seen0
- SeenSeen, split unclear0
- TrainTraining data seen0
- Related data seen0
- NoNot seen2
- ?Unknown13
“Llama 3 was pretrained on over 15 trillion tokens of data from publicly available sources.”
Other declared training data
-
A new mix of publicly available online data. (broad web data: may contain benchmark test sets) · split: unknown
Evidence
“A new mix of publicly available online data.”
raw.githubusercontent.com · model card · Model Details table · retrieved 2026-10-15 “Llama 3 was pretrained on over 15 trillion tokens of data from publicly available sources.”
raw.githubusercontent.com · model card · Training Data / Overview · retrieved 2026-10-15 -
publicly available instruction datasets (broad web data: may contain benchmark test sets) · split: unknown · fine-tuning
Datasets not named.
Evidence
“The fine-tuning data includes publicly available instruction datasets, as well as over 10M human-annotated examples.”
raw.githubusercontent.com · model card · Training Data / Overview · retrieved 2026-10-15 -
over 10M human-annotated examples (not yet matched to a dataset) · split: unknown · fine-tuning
Evidence
“The fine-tuning data includes publicly available instruction datasets, as well as over 10M human-annotated examples.”
raw.githubusercontent.com · model card · Training Data / Overview · retrieved 2026-10-15
Benchmark cells
Classic LLM benchmarks
Unknown MMLU (Measuring Massive Multitask Language Understanding) · training data not disclosed in enough detail
Unknown GSM8K (Grade School Math 8K) · training data not disclosed in enough detail
Unknown MATH (Mathematics Aptitude Test of Heuristics) · training data not disclosed in enough detail
Unknown HumanEval · training data not disclosed in enough detail
Unknown ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · training data not disclosed in enough detail
Unknown HellaSwag · training data not disclosed in enough detail
Unknown TruthfulQA · training data not disclosed in enough detail
Unknown WinoGrande · training data not disclosed in enough detail
Recent LLM benchmarks
Unknown GPQA Diamond · training data not disclosed in enough detail
Unknown MMLU-Pro · training data not disclosed in enough detail
Unknown LiveCodeBench (code generation, release_v6) · training data not disclosed in enough detail
Unknown SWE-bench Verified · training data not disclosed in enough detail
Not seen AIME 2024 · its test items were published from 2024-01-31, after the model's stated training-data cutoff (2023-12-01)
Evidence (4)
“The pretraining data has a cutoff of March 2023 for the 8B and December 2023 for the 70B models respectively.”
“2024 AIME I problems and solutions. The test was held on Wednesday, January 31 – Thursday, February 1, 2024.”
“AIME I (main AIME competition date): Thursday, Feb. 1, 2024 between 1:30 pm ET and 5:30 pm ET.”
“AIME II (alternate AIME competition date): Wednesday, Feb. 7, 2024 between 1:30 pm ET and 5:30 pm ET.”
Not seen AIME 2025 · its test items were published from 2025-02-06, after the model's stated training-data cutoff (2023-12-01)
Evidence (3)
“The pretraining data has a cutoff of March 2023 for the 8B and December 2023 for the 70B models respectively.”
“2025 AIME I problems and solutions. The test was held on Thursday, February 6, 2025.”
“This year the AIME I will take place on February 6, 2025, while the alternate AIME II will take place on February 12, 2025.”
Unknown MATH-500 · training data not disclosed in enough detail
Notes
training_cutoff is the 70B's (December 2023); the 8B's pre-training data has a cutoff of March 2023. No technical paper for this April 2024 release; 'The Llama 3 Herd of Models' states that all its results are for the Llama 3.1 models, so its contamination analysis is recorded under llama-3.1. Third-party signal on ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) (indirect, not shown as a cell): Sample-specific contamination detected on ARC-Challenge (p < 0.5%, estimated effect 3-8%). Third-party signal on GSM8K (Grade School Math 8K) (indirect, not shown as a cell): Accuracy 6.2 points lower on GSM1k than on GSM8K (p <= 0.003). A GSM8K-GSM1k gap is evidence of overfitting to GSM8K, not proof that GSM8K test items were in the training data; the authors attribute it partly to partial memorisation (correlation with GSM8K log-likelihood) and partly to other causes. Third-party signal on GSM8K (Grade School Math 8K) (indirect, not shown as a cell): Accuracy 5.4 points lower on GSM1k than on GSM8K (p <= 0.003). A GSM8K-GSM1k gap is evidence of overfitting to GSM8K, not proof that GSM8K test items were in the training data; the authors attribute it partly to partial memorisation (correlation with GSM8K log-likelihood) and partly to other causes.
Sources
- Meta Llama 3 MODEL_CARD.md (meta-llama/llama3) · model card · retrieved 2026-10-15
- meta-llama/llama-models README (launch dates) · readme · retrieved 2026-10-15