GPT-4

Training data disclosure: Partial list

Level determined for 59% of benchmark cells.

• TestTest data seen5
• SeenSeen, split unclear0
• TrainTraining data seen1
• Rel.Related data seen2
• NoNot seen5
• ?Unknown9

Other declared training data

• publicly available data (such as internet data) (broad web data: may contain benchmark test sets) · split: unknown
  Evidence

  “using both publicly available data (such as internet data) and data licensed from third-party providers.”

• data licensed from third-party providers (not yet matched to a dataset) · split: unknown
  Evidence

  “using both publicly available data (such as internet data) and data licensed from third-party providers.”

• MATH (Mathematics Aptitude Test of Heuristics) · split: train of llm-classic

  Only the training set; part of it was held back, so each example may or may not have been seen.

  Evidence

  “To improve GPT-4's ability to do mathematical reasoning, we mixed in data from the training set of MATH and GSM-8K, two commonly studied benchmarks for mathematical reasoning in language models.”

• GSM8K (Grade School Math 8K) · split: train of llm-classic
  Evidence

  “To improve GPT-4's ability to do mathematical reasoning, we mixed in data from the training set of MATH and GSM-8K, two commonly studied benchmarks for mathematical reasoning in language models.”

  “For GSM-8K, we include part of the training set in GPT-4's pre-training mix (see Appendix E for details).”

Declared exclusions

• Dataset HellaSwag · test portion excluded
  Evidence

  “GPT-4's holdout results are close to the results on the validation set (95.6%) which was explicitly masked out during training.”

Contamination analyses

Analyses that found benchmark test data in this model's training data, or in a corpus it uses.

• MMLU (Measuring Massive Multitask Language Understanding) · pre-training data

  ~0.6% of 1000 sampled MMLU test examples matched GPT-4 pre-training data (50-character substring match)

  Evidence

  “MMLU 86.4% 70.0% ~0.6%”

  “For datasets other than HumanEval, we estimated contamination based on 1000 randomly chosen examples against our training data.”

• GSM8K (Grade School Math 8K) · pre-training data

  ~1% of 1000 sampled GSM-8K test examples matched GPT-4 pre-training data

  Evidence

  “GSM-8K 92.0% 57.1% ~1%”

  “For datasets other than HumanEval, we estimated contamination based on 1000 randomly chosen examples against our training data.”

• ARC-Challenge (AI2 Reasoning Challenge, Challenge Set) · pre-training data

  ~3.4% of 1000 sampled ARC (Challenge set) test examples matched GPT-4 pre-training data

  Evidence

  “AI2 96.3% 85.2% ~3.4%”

  “AI2 Reasoning Challenge (ARC) [54] 96.3% 85.2% 85.2% 86.5% Grade-school multiple choice science questions. Challenge-set.”

• WinoGrande · pre-training data

  ~0.9% of 1000 sampled WinoGrande test examples matched GPT-4 pre-training data

  Evidence

  “WinoGrande 87.5% 81.6% ~0.9%”

• HumanEval · pre-training data

  25% of HumanEval problems matched GPT-4 pre-training data (whole benchmark checked); score on the non-contaminated subset was 65.58% vs 67.0% overall

  Evidence

  “HumanEval 67.0% 48.1% 25% 65.58% -2.12%”

• MMLU (Measuring Massive Multitask Language Understanding) · training data

  Guessed masked wrong answer options of MMLU test questions with 57% exact match, suggesting memorisation of test items. Model version not stated.

  Evidence

  “Specifically, in the MMLU benchmark, ChatGPT and GPT-4 demonstrated an exact match rate of 52% and 57%, respectively, in guessing the missing options in benchmark test data.”

• HumanEval · training data

  Estimated contamination level of 55.62-56.71% on the full HumanEval test set (164 problems).

  Evidence

  “Our DCQ (%) [55.62, 56.71] [42.86, 44.00] [78.79, 79.00]”

  “As done by OpenAI (2023), we apply DCQ to the entire test set of the HumanEval dataset with 164 samples.”

  “gpt-4-0613 snapshots, accessed via the OpenAI API.”

Benchmark cells

Classic LLM benchmarks

Multimodal LLM benchmarks

Recent LLM benchmarks

Notes

Technical report gives no dataset construction details apart from the MATH/GSM-8K training-set mix. Contamination checked with 3 random 50-character substrings per evaluation example against processed pre-training data (Appendix C/D). RLHF post-training data was not checked for contamination ('We did not check the RLHF post-training data for contamination with TruthfulQA'). The current API docs page lists a 'Dec 01, 2023 knowledge cutoff' for GPT-4, which conflicts with the report (September 2021). Other reported contamination, not shown as cells: DROP: ~21% of 1000 sampled DROP examples matched GPT-4 pre-training data; BIG-bench: portions of BIG-bench were inadvertently mixed into the training set (extent not quantified); Professional and academic exams (GPT-4 report Tables 9-10): Tables 9-10 report per-exam fractions of contaminated questions for the professional/academic exams (AP, GRE, SAT, LSAT, USABO, sommelier, Leetcode, etc.), ranging from 0% to 100% of questions per exam section Third-party signal on TruthfulQA (indirect, not shown as a cell): 11.1 percentage points higher on TruthfulQA than on the retro-holdout. Performance gap on a fresh holdout is evidence of benchmark inflation/overfitting, not proof of test-set inclusion. Accepts image input: “We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs.”

Sources

• GPT-4 Technical Report (arXiv 2303.08774v6, includes the GPT-4 System Card) · paper · retrieved 2026-10-15
• OpenAI API docs: GPT-4 model page · website · retrieved 2026-10-15