SWE-Bench Pro v1 (2025-09-19) and v2 C

AI benchmark · Coding

Tier
C why
Snapshot

Open benchmark ↗labs.scale.com/papers

What is SWE-Bench Pro v1 (2025-09-19) and v2?

SWE-Bench Pro measures whether AI agents can solve long-horizon software engineering tasks.

Lineage

Benchmarks this one is built on and built from: versions, subsets, extensions and the indexes that combine them.

Built on

Tier C

C

Niche: a handful of models scored, or built by an RL vendor.

How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.

  • listed with Scale AI
  • in curated RL directories (2 pts)
How we got to tier C
  1. Adoption: listed with Scale AI. Tier C.
  2. On X: unclear, none buzz No move.
  3. In curated RL directories: 2 points, a floor of C.
  4. Confirmed: a line from its own site or paper places it.

A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.

Training methods

Methods SWE-Bench Pro v1 (2025-09-19) and v2 uses, offers or builds for, with public evidence: its own website says so, or two independent sites do.

  • “Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR”

    labs.scale.com

    “- Open-source models fine–tuned through Reinforcement Learning with Verifiable Feedback (RLVR) exhibit significant gains over state-of-the-art, larger general-purpose models. - Text-to-SQL Benchmark: Our RL-tuned agent achieved 46.9% execution accuracy, compared to 21.9% for the best out-of-the-box baseline mode…”

    labs.scale.com
  • “Improving Multi-Turn Tool Use with GRPO: Results and Insights”

    labs.scale.com

    “…nd category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy’s outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins 24 of 30 base-policy/metric comparisons, impro…”

    labs.scale.com
  • “…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”

    labs.scale.com
  • SFT Supervised Fine-Tuning On its own website

    “Execution accuracy by model and training method (Base / SFT / GRPO) on the T2SQL benchmark. Dashed line marks the Qwen3-Coder-480B-A35B-Instruct baseline.”

    labs.scale.com

Where SWE-Bench Pro v1 (2025-09-19) and v2 sits

Its domain on the map, live. Hover a logo for its name; click to open it. Open it on the full map

The galaxy could not load. See every vendor on the board.

Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-11-16 snapshot.