Methodology

Contamination levels

Test Test data seen

The benchmark's test data was in the pre-training data.

Seen Seen, split unclear

The dataset was used, but the source does not say which part, so the test data may have been seen.

Train Training data seen

Only the benchmark's training portion of the dataset was used.

Rel. Related data seen

A related dataset was used: a subset, superset, resampled version or the same underlying source.

No Not seen

The test data was not in the pre-training data: either the full training list is published and does not include it, or the authors state it was excluded.

? Unknown

Not enough information has been disclosed to tell.

What is recorded

• Datasets, with the names they go by in papers, and how they relate to each other (subset, resampled version, same underlying source).
• Benchmarks: the datasets they evaluate on and how their test portion is defined.
• Models: what their authors disclose about pre-training data, the shared corpora they use, and any data they say they excluded.

Each model gets a disclosure level:

Full list

The authors publish the complete list of pre-training datasets.

Partial list

Some datasets are named, but the list is not complete.

Vague

Only a general description (“public datasets from many domains”).

None

Nothing is disclosed.

How a cell's level is assigned

For each benchmark dataset and model, every declared pre-training dataset is compared with the benchmark dataset:

1. The same dataset, used in full or its test split used for training → Test data seen
2. The same dataset, used with an unstated split, or with a training split defined by another benchmark → Seen, split unclear
3. The same dataset, restricted to this benchmark's training split → Training data seen
4. A related dataset (subset, superset, resampled version, same source) → Related data seen

When several declarations match, the strongest exposure wins (the order above). If the authors state that the test portion was held out, “Test data seen” and “Seen, split unclear” become “Training data seen”. If a statement that data was excluded conflicts with a declaration that it was used, the exposure is kept and the conflict is shown with both quotes.

When nothing matches:

• if the authors state the dataset (or its test portion) was excluded → Not seen;
• if the benchmark's test items were all published after the model's stated training-data cutoff → Not seen;
• if the model has a full list of training data, every entry in it has been matched to a known dataset or reviewed as unrelated, none of it is a broad web crawl, and the dataset is not there → Not seen;
• otherwise → Unknown. Absence from a partial or vague disclosure is never read as “not seen”.

Models built on other models

Many models start from a pre-trained base model (for example an anomaly detector built on CLIP, or a distilled language model). They inherit whatever their base model may have seen: a cell shows the base model's exposure when it is stronger, and “Not seen” requires every base model to be “Not seen” as well.

Language models

Language models are mostly trained on web crawls rather than on named datasets, so their pages use three more kinds of evidence:

• Contamination analyses. When the developer or a third party found benchmark test examples in a model's training data (for example by searching the corpus for overlapping text), the cell is Test data seen, with the reported extent. Evidence from anyone other than the developer is labelled third-party study.
• Training stages. Pre-training and fine-tuning (instruction tuning, reinforcement learning, distillation) are tracked separately. Fine-tuning mixes often include benchmark training sets; such cells say “in fine-tuning data”. A decontamination step applied only to fine-tuning data says nothing about pre-training.
• Training-data cutoffs. When a developer states a cutoff date, benchmarks published later cannot be in pre-training. This assumes the stated cutoff is accurate; fine-tuning data collected later is not covered by it.

Image benchmarks (MMMU, ChartQA, …) are only shown for models that accept images; text-only models have an empty slot there.

Broad web crawls (Common Crawl and the like) can contain benchmark test sets copied onto the web. A model trained on one is never marked “Not seen” just because the benchmark is absent from its list of sources.

Limitations

• The map reflects what is declared. Undeclared or mis-declared data cannot be detected from documents alone.
• Dataset names are ambiguous (for example, several “Electricity” datasets exist). Entries are matched by hand, and relations between datasets are only recorded with a source.
• Sources change. Each quote records the date it was retrieved.

Data

The full matrix is available as CSV and JSON, with the source URLs for each cell.