SWE-Pro A
AI benchmark · Software Engineering
- Domains
- Software Engineering
- Tier
- A why
- Snapshot
What is SWE-Pro?
Software performance optimization is a notoriously complex and manual task. Despite the growing use of Large Language Models (LLMs) for code refinement, we still lack benchmarks that capture how optimization actually happens in real-world codebases. Existing frameworks often oversimplify the problem by focusing on isolated functions or a single performance metric, missing the critical trade-offs between execution time and memory footprint, the inherent noise of the measurement environment, and the variability introduced by different input data and execution conditions. We address this by introducing SWE-Pro, a repository-level benchmark derived from 102 expert-written optimizations from open-source projects. Unlike previous benchmarks, SWE-Pro pairs each task with parameterized tests to evaluate runtime, peak memory, and Time-Weighted Memory Usage (TWMU) across varying input data and execution conditions under noise-aware measurement conditions. Our evaluation shows that current LLMs struggle significantly: runtime gains are negligible, and memory optimizations are nearly non-existent. This stands in sharp contrast to expert implementations, which achieve an aggregate speedup of 15.5x and peak memory reduction of 171.3x over benchmark tasks. Expert-written improvements are observed in 91.2% of tasks for runtime and 65.7% for peak memory. Our findings expose a substantial gap between current LLM capabilities and the demands of expert-level engineering.
Tier A
Widely used: a lab reports it, or 50 or more models are scored on it.
How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.
- 62 models scored
- built by Scale
- in curated RL directories (7 pts)
On X · well regarded, low buzz, from 7 posts about SWE-Pro in the last month
SWE-Pro is cited as a coding metric in model announcements like Xiaomi's MiMo distill and in leaderboard posts, with little direct discussion of the benchmark.
How we got to tier A
- Adoption: 62 models scored; built by Scale. Tier A.
- On X: well regarded, low buzz. “SWE-Pro is cited as a coding metric in model announcements like Xiaomi's MiMo distill and in leaderboard posts, with little direct discussion of the benchmark.” No move.
- In curated RL directories: 7 points, a floor of B.
- Confirmed: a line from its own site or paper places it.
A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.
Not on the map yet: no line from its own site or paper places it in a domain.
Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-10-11 snapshot.