Contamination MapText / LLM / Dataset
SWE-bench
2,294 task instances (test split) built from real GitHub issues and the merged pull requests that resolved them, across 12 popular Python repositories; a model must produce a patch that makes fail-to-pass unit tests pass. HF princeton-nlp/SWE-bench also has dev (225) and train (19,008) splits.
- Also known as
- SWE-bench, SWE-bench Full, SWEbench
- Domain
- code
- Homepage
- www.swebench.com
Exposure by model
This dataset is not part of any benchmark tracked here.
Notes
Gold patches are merged PRs in public GitHub repositories, so they exist in any GitHub crawl predating the benchmark. The paper partitions results by PRs created before/after 2023 but gives no earliest creation date; published_from left null (each instance has a created_at field).
Sources
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv v3) · paper · retrieved 2026-10-15
- SWE-bench README · readme · retrieved 2026-10-15
- princeton-nlp/SWE-bench dataset card · dataset card · retrieved 2026-10-15