Domain on the map · as Coding
Software Engineering: AI benchmarks
- AI benchmarks
- 28
- Shares entries with
- 1 category
Snapshot 2026-11-12. How categories are built · Where the data comes from
In short
- What are the main software engineering benchmarks?
- APEX-SWE, SaaS-Bench, SaaSBench, EvoCode-Bench, From Laboratory to Real-World Applications and SWE-Together, among 26 confirmed below.
Only benchmarks use the Software Engineering tag: the RL environment directories file their vendors under other categories. Browse all RL environments.
AI benchmarks 28
-
APEX-SWE
Tests coding“Measures AI performance on real-world software engineering tasks, from bug fixes to feature builds. Built in collaboration with Cognition.” source ↗
-
SaaS-Bench
Tests coding“uni-m | Software | 31 | Baserow, OpenProject, code-server, Metabase” source ↗
-
SaaSBench
Tests coding“uni-m | Software | 31 | Baserow, OpenProject, code-server, Metabase” source ↗
-
EvoCode-Bench
Tests coding“We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks and 227 evaluated rounds.” source ↗
-
From Laboratory to Real-World Applications
Tests coding“We present RepoReason, a white-box diagnostic benchmark centered on abductive assertion verification.” source ↗
-
SWE-Together
Tests coding“we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes.” source ↗
-
BUILD-AND-FIND
Tests coding“We introduce BUILD-AND-FIND, a protocol for evaluating whether downstream agents can recover those intended choices from generated repositories, and how much inspection that recovery requires.” source ↗
-
LibEvoBench
Tests coding“we introduce LibEvoBench, a multi-task benchmark spanning multiple versions of widely used Python libraries” source ↗
-
LiveFMBench
Tests coding“We introduce LiveFMBench, a continuously evolving benchmark of 630 ACSL (ANSI/ISO C Specification Language)-annotated C programs” source ↗
-
LogDx-CI
Tests coding“compares 11 context-reduction tools (raw, tail, grep, three RTK modes, two real LLM map-reduce summarizers, three hybrid routers) on 35 real GitHub Actions failure cases” source ↗
-
MathlibPR
Tests coding“can LLMs help review Mathlib PRs?” source ↗
-
OmniCode
Tests coding“we propose OmniCode, a novel software engineering benchmark that contains a broader and more diverse set of task categories beyond code or patch generation.” source ↗
-
RepoMirage
Tests coding“we introduce RepoMirage, a two-stage evaluation suite built on SWE-Bench Verified” source ↗
-
RoadmapBench
Tests coding“we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages.” source ↗
-
SAKE
Tests coding“We introduce SAKE (Software Architectural Knowledge Evaluation), a standardized and reproducible benchmark for assessing software architectural knowledge in LLMs.” source ↗
-
SmellBench
Tests coding“We evaluate 11 agent configurations from four model families (GPT, Claude, Gemini, Mistral) on 65 hard-severity architectural smells detected by PyExamine in the Python project scikit-learn” source ↗
-
SWE-Mutation
Tests coding“we introduce SWE-Mutation, a benchmark for evaluating LLM-generated test suites.” source ↗
-
VeriContest
Tests coding“a benchmark of 946 competitive-programming problems from LeetCode and Codeforces for verifiable code generation in Rust with Verus.” source ↗
-
Verus-SpecBench
Tests coding“We introduce Verus-SpecBench, a benchmark of 581 spec-writing tasks derived from Codeforces problems targeting Verus, a verifier for Rust” source ↗
-
WebGameBench
Tests coding“a requirement-to-application benchmark that evaluates whether coding agents can turn a frozen Structured WebGame Specification into a browser-accessible game” source ↗
-
RigorBench
Tests coding“We introduce RigorBench, the first benchmark designed to measure process discipline in AI coding agents.” source ↗
-
SCDBench
Tests coding“We introduce SCDBench, a dataset and benchmark methodology for LLM-based smart contract decompilation.” source ↗
-
Sphinx
Tests coding“a checklist-based evaluation benchmark that assesses review quality based on structured coverage of actionable verification points, moving beyond surface-level metrics like BLEU” source ↗
-
SWE Atlas
Tests coding“SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution” source ↗
-
SWE-Chain
Tests coding“We present SWE-Chain, a benchmark for evaluating agents on chained release-level package upgrades, where each transition builds on the agent's prior codebase.” source ↗
-
Towards Evaluation of Implicit Software World Models in Coding LLMs
Tests coding“we predict peak memory, wall-clock time, and ranked profiler outputs at method and line granularity. We use SWE-bench Verified as the source of data” source ↗
Data 29
- Aviro
- Bespoke Labs
- Datacurve
- Emulated
-
General Reasoning
Sells data“A frozen archive of news, SEC filings, arXiv, Wikipedia and the wider web, with point-in-time search and fetch for forecasting and reproducible evals and training.” source ↗
Covers coding“a benchmark of 89 challenging terminal-based software engineering tasks spanning compilation, cryptography, data processing, machine learning, system administration and scientific computing.” source ↗
- Parsewave
-
pre.dev
Covers coding“Hard, verified coding tasks. From code no model has seen.” source ↗
- Revelo
- Abundant
- AfterQuery
- Anthromind
- Collinear
- Contra Labs
- Handshake
- HUD
- Huzzle Labs
- Mercor
-
micro1
Sells data“We're building the infrastructure for advancing intelligence through expert human data, real-world training environments, and contextual evaluations” source ↗
Covers coding“Source code documentation, engineering workflows, bug tracking, product planning, and technical knowledge bases” source ↗
- Patronus AI
-
ReasonCore
Sells data“We produce exclusive and off-the-shelf (OTS) datasets built around multi-step hierarchical reasoning, tool-augmented workflows, and domain-specific problems” source ↗
Covers coding“including data targeting SWE-Bench (real-world GitHub bug resolution and patch generation) and Terminal-Bench” source ↗
- Refresh
- Rise Data Labs
- Snorkel AI
- Surge AI
-
Theta
Covers coding“Review Pull Request” source ↗
- Trajectory Labs
- Turing
-
Verita AI
Sells data“Our platform solves this by collecting open-ended information about every expert in our community and building on each profile as they work with us.” source ↗
Covers coding“Developed training data for large language models by formulating coding problems that models could not resolve, and subsequently documenting the correct solutions.” source ↗
- Vetto AI
Next: pick another domain. Tags come from the RL Research daily scrape, canonicalized nightly.