SWE-bench Pro A
AI benchmark · Multi-Domain
- Domains
- Multi-Domain Coding
- Tier
- A why
- Snapshot
What is SWE-bench Pro?
SWE-Bench Pro measures AI agents' ability to resolve long-horizon software engineering tasks (bug fixes and features) in public and proprietary codebases, scored by resolve rate against fail-to-pass and pass-to-pass tests.
Coding mostly
“SWE-Bench Pro is a benchmark designed to provide a rigorous and realistic evaluation of AI agents for software engineering.”
labs.scale.com/leaderboard/swe_bench_pro_public ↗
Lineage
Benchmarks this one is built on and built from: versions, subsets, extensions and the indexes that combine them.
Built on
SWE-bench original
Derived benchmarks
SWE-Bench Pro v1 (2025-09-19) and v2 later version
SWE-Bench Pro V2 Full subset
SWE-Bench Pro V2 HARD extension
Tier A
Widely used: a lab reports it, or 50 or more models are scored on it.
How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.
- 65 models scored
- listed with Scale
- in curated RL directories (7 pts)
On X · unclear, low buzz, from 3 posts about SWE-bench Pro in the last month
SWE-Pro shows up in passing as one of several reference coding benchmarks in model comparisons, with no direct praise or criticism.
How we got to tier A
- Adoption: 65 models scored; listed with Scale. Tier A.
- On X: unclear, low buzz. “SWE-Pro shows up in passing as one of several reference coding benchmarks in model comparisons, with no direct praise or criticism.” No move.
- In curated RL directories: 7 points, a floor of B.
- Confirmed: a line from its own site or paper places it.
A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.
Training methods
Methods SWE-bench Pro uses, offers or builds for, with public evidence: its own website says so, or two independent sites do.
- RLHF Reinforcement Learning from Human Feedback On its own website
“Exploring reinforcement learning from human feedback (RLHF) to align language models with human preferences for safer adoption and use.”
labs.scale.com - RLVR Reinforcement Learning with Verifiable Rewards On its own website
“Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR”
labs.scale.com“- Open-source models fine–tuned through Reinforcement Learning with Verifiable Feedback (RLVR) exhibit significant gains over state-of-the-art, larger general-purpose models. - Text-to-SQL Benchmark: Our RL-tuned agent achieved 46.9% execution accuracy, compared to 21.9% for the best out-of-the-box baseline mode…”
labs.scale.com - GRPO Group Relative Policy Optimization On its own website
“Improving Multi-Turn Tool Use with GRPO: Results and Insights”
labs.scale.com“…nd category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy’s outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins 24 of 30 base-policy/metric comparisons, impro…”
labs.scale.com - SFT Supervised Fine-Tuning On its own website
“Execution accuracy by model and training method (Base / SFT / GRPO) on the T2SQL benchmark. Dashed line marks the Qwen3-Coder-480B-A35B-Instruct baseline.”
labs.scale.com
Where SWE-bench Pro sits
Its domain on the map, live. Hover a logo for its name; click to open it. Open it on the full map
Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-11-05 snapshot.