Contamination MapText / LLM / Dataset
SWE-bench Verified
500 SWE-bench test instances screened by human annotators (93 Python developers annotated 1,699 random test samples) and released by OpenAI with the SWE-bench authors in August 2024.
- Also known as
- SWE-bench Verified, SWE-bench_Verified
- Domain
- code
- Homepage
- openai.com
Related datasets
-
subset of SWE-bench
Subset of the SWE-bench test split.
Evidence
“we are releasing SWE-bench Verified: a subset of the original test set from SWE-bench, consisting of 500 samples verified to be non-problematic by our human annotators.”
openai.com · website · retrieved 2026-10-15 “SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality.”
huggingface.co · dataset card · Dataset Summary · retrieved 2026-10-15
Exposure by model
| Benchmark | Model | Level | Reason |
|---|---|---|---|
| Recent LLM benchmarks | LLM360 Amber | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | BLOOM | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude 3.5 Sonnet | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude 3.7 Sonnet | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude 3 (Opus, Sonnet, Haiku) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Opus 4 and Claude Sonnet 4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Fable 5 / Claude Mythos 5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Mythos Preview | Test data seen | a contamination analysis found test examples in the training data (Developer memorisation audit found some signs of memorisation on SWE-bench Verified, Multilingual and Pro despite corpus-level decontamination; a high-recall filter (threshold 0.7) removes 8-15% of each benchmark) |
| Recent LLM benchmarks | Claude Opus 4.5 | Test data seen | a contamination analysis found test examples in the training data (Third-party (OpenAI) elicitation study found strong contamination examples: Claude Opus 4.5 reproduced SWE-bench Verified gold-patch / task-specific details) [third-party study] |
| Recent LLM benchmarks | Claude Opus 5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Claude Sonnet 5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DCLM-7B-IT (DCLM-IT-7B) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DCLM-7B (DCLM-Baseline-7B) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-R1-0528 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-R1-Distill | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-R1 | Not seen | the authors state its test portion was excluded from training |
| Recent LLM benchmarks | DeepSeek-V3-0324 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V3.1 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V3.2-Exp | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V3.2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V4.1-Flash | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | DeepSeek-V4 (Pro / Flash) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemini 1.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemini 2.5 (Pro, Flash) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemini 3 Pro | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemini 4 Argon | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemma 2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemma 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemma 3n | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Gemma 4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | GPT-4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | GPT-4o | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | GPT-5.2 | Test data seen | a contamination analysis found test examples in the training data (Developer analysis found GPT-5.2 (and GPT-5.2-Chat) could reproduce SWE-bench Verified gold patches / task details and solved 31 tasks judged almost impossible without prior exposure; OpenAI stopped reporting SWE-bench Verified) |
| Recent LLM benchmarks | GPT-5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | GPT-6 Astra | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | gpt-oss | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | InternVL3.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Recent LLM benchmarks | InternVL3 | Unknown | unknown for its base model Qwen2.5: training data not disclosed in enough detail |
| Recent LLM benchmarks | LLM360 K2-65B (K2 Diamond) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | K2-Horizon | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | K2-V2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 3.1 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 3.2-Vision (11B, 90B) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 3.2 (1B, 3B) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 3.3 70B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Meta Llama 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Llama 4 (Scout, Maverick) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | LLaVA-OneVision-1.5 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Recent LLM benchmarks | LLaVA-OneVision | Unknown | unknown for its base model SigLIP: training data not disclosed in enough detail |
| Recent LLM benchmarks | Ministral 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mistral 7B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mistral Large 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mistral Medium 3.5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mistral Small 4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Mixtral 8x7B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Molmo (0924) | Unknown | unknown for its base model OpenAI CLIP: training data not disclosed in enough detail |
| Recent LLM benchmarks | Molmo2 | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Recent LLM benchmarks | OpenAI o1 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | OpenAI o3 | Test data seen | a contamination analysis found test examples in the training data (Identifies the buggy file path from the issue text alone in up to 76% of SWE-bench Verified tasks, vs <53% on tasks from other repositories; the authors read it as possible memorisation/contamination.) [third-party study] |
| Recent LLM benchmarks | OLMo (February 2024) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | OLMo 2 (November 2024) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Olmo 3.1 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Olmo 3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Olmo Hybrid 7B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-3.5 (mini / MoE) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-3 (mini / small / medium) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4-mini-flash-reasoning | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4-mini-reasoning | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4-mini | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4-reasoning / Phi-4-reasoning-plus | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-4 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Phi-mini-MoE / Phi-tiny-MoE (SlimMoE) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Pixtral 12B | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Pixtral Large | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Pythia (deduplicated) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Pythia | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen2.5-VL | Unknown | unknown for its base model Qwen2.5: training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen2.5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3-2507 (Instruct-2507 / Thinking-2507) | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3-Next | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3-VL | Unknown | unknown for its base model Qwen3: training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3.5 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3.6 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3.8-Flash-Next | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3.8 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | Qwen3 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | SmolLM2 | Unknown | training data not disclosed in enough detail |
| Recent LLM benchmarks | SmolLM3 | Unknown | training data not disclosed in enough detail |
Notes
On 2026-02-23 OpenAI announced it no longer reports SWE-bench Verified, citing flawed tests and contamination (https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/).
Sources
- Introducing SWE-bench Verified (OpenAI) · website · retrieved 2026-10-15
- princeton-nlp/SWE-bench_Verified dataset card · dataset card · retrieved 2026-10-15