SWE-bench A
AI benchmark · Coding
What is SWE-bench?
SWE-bench measures whether language models can resolve real-world GitHub issues by generating patches to codebases that pass tests.
Coding mostly
“SWE-bench is a benchmark for evaluating large language models on real world software issues collected from GitHub.”
github.com/swe-bench/SWE-bench ↗
Lineage
Benchmarks this one is built on and built from: versions, subsets, extensions and the indexes that combine them.
Derived benchmarks
Executable Benchmarking Suite for Tool-Using Agents extension
“An executable benchmarking suite that evaluates closed-loop tool-using agents across web (WebArena Verified, MiniWoB++) and code (SWE-Gym/SWE-bench) workloads under a shared evidence-admission contract, recording latency, invalid actions, and patch-generation cost.”
Senior SWE-bench extension
“Senior SWE-bench evaluates coding agents on senior-level software engineering work: under-specified feature requests, bugs requiring runtime investigation, and code quality (tasteful solves).”
SWE Multilingual extension
“SWE-bench Multilingual measures how well LLM agents resolve real-world GitHub issues across 300 tasks in 42 repositories and 9 programming languages, verified by unit tests.”
SWE-bench Docker extension
SWE-bench Extra extension
SWE-bench Full extension
SWE-bench JavaScript extension
SWE-bench Lite subset
SWE-bench Live extension
SWE-bench Multimodal extension
SWE-bench Pro extension
SWE-Bench Pro - Private Dataset extension
SWE-Bench Pro V2 HARD extension
SWE-bench Verified extension
SWE-bench Verified Extended extension
SWE-bench Verified Mini extension
Tier A
Widely used: a lab reports it, or 50 or more models are scored on it.
How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.
- no usage data
- in curated RL directories (1 pt)
- its family includes SWE-bench Verified (tier S)
On X · mixed, moderate buzz, from 5 posts about SWE-bench in the last month
Still widely cited for coding results, but posts focus on contamination, models searching for answers, and saturation of the Verified subset.
How we got to tier A
- Adoption: no usage data. Tier D.
- On X: mixed, moderate buzz. “Still widely cited for coding results, but posts focus on contamination, models searching for answers, and saturation of the Verified subset.” No move.
- In curated RL directories: 1 point, a floor of none.
- Root of a benchmark family that includes SWE-bench Verified (tier S): raised to A, one step below.
- Confirmed: a line from its own site or paper places it.
A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.
Where SWE-bench sits
Its domain on the map, live. Hover a logo for its name; click to open it. Open it on the full map
Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-11-05 snapshot.