RL environments and benchmarks for coding

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Sells RL environments“Snorkel helps frontier labs and AI teams develop specialized training data and environments that set their models and agents apart.” source ↗

Covers coding“DeepSWE long-horizon software engineering tasks from Harbor Hub.” source ↗

Sells RL environments“Turn any task into an RL environment. Init, develop, eval, and push with the Prime CLI.” source ↗

Covers coding“Spans expert-written code, test cases, and debugging traces that teach models to write production-quality software” source ↗

Sells RL environments“Provides custom RL environments built on top of real APIs, MCP servers, and developer tools” source ↗

Covers coding“Real-world software engineering across integration and observability” source ↗

Sells RL environments“Benchmarks and RL environments built on real professional work (APEX).” source ↗

Covers coding“Software engineeringFor the software development lifecycle.” source ↗

Sells RL environments“300+ RL ENVIRONMENTS” source ↗

Covers coding“spanning RL environments, coding, and the core capabilities every model needs.” source ↗

Sells RL environments“We create rich, complex RL environments that challenge agentic models in novel ways, and design verifiers that reward their behavior.” source ↗

Covers coding“A simulation environment where AI can practice doing work (like writing code or making a spreadsheet)” source ↗

Sells RL environments“We build training gyms for frontier research labs.” source ↗

Covers coding“Company-scale simulations with real codebases and microservices, allowing agents to master the complex, long-horizon workflows required for production.” source ↗

Sells RL environments“Bespoke Labs builds company-scale RL environments and infrastructure to let frontier labs and enterprises train long-horizon agents for production.” source ↗

Covers coding“Building, debugging, and maintaining complex software” source ↗

Sells RL environments“We build simulated applications and workflows where agents can execute complex tasks, delivered to run within your training infrastructure.” source ↗

Covers coding“Work across the software lifecycle, from understanding and debugging an unfamiliar codebase to building features and writing tests.” source ↗

Sells RL environments“Expert-built, containerized tasks for long-horizon agent work, with verifiers, proof of solvability, and pass rates.” source ↗

Covers coding“Our ultra-long-horizon coding benchmark. Frontier models clear only a fraction of its tasks.” source ↗

Sells RL environments“How we run agentic RL across Modal GPUs and our own Kubernetes sandbox fleet.” source ↗

Covers coding“models carry out software engineering work such as building a feature, deploying an application, or debugging an issue in an unfamiliar codebase.” source ↗

Sells RL environments“We build environments and evals for frontier coding agents.” source ↗

Covers coding“Collinear supports cybersecurity and software engineering, computer use, multi-step tool use, long-horizon reasoning, APIs and MCP, and CLI workflows.” source ↗

Sells RL environments“Every simulation lab is a self-contained world where your agent operates, complete with the users, tools, data, and tasks it will face in production.” source ↗

Covers coding“a benchmark of 89 challenging terminal-based software engineering tasks spanning compilation, cryptography, data processing, machine learning, system administration and scientific computing” source ↗

Sells RL environments“An open resource for discovering and experimenting with community RL environments with simple API endpoints and autoscaled infrastructure.” source ↗

Covers coding“Multi-file repos with build, run and test loops.” source ↗

Sells RL environments“We build **RL environments** for frontier AI labs” source ↗

Covers coding“Frontier Coding·Economically Valuable Workflows·Bespoke Evaluations” source ↗

Sells RL environments“Reinforcement learning environments built from real production work” source ↗

Covers coding“Datacurve provides the frontier coding data that powers the world's most advanced models.” source ↗

Sells RL environments“Durable environments for measuring agentic capabilities where real work happens: long horizons, natural instructions, realistic tools, domain-specific judgment, and partial progress.” source ↗

Covers coding“Sharpe designs RL environments for coding, AI training, and ML project management using HUD” source ↗

Sells RL environments“The platform for building RLenvironments” source ↗

Covers coding“helping AI tools (across video, code, image and more) understand quality the way expert creatives do.” source ↗

Sells RL environments“Trajectories & RL” source ↗

Covers coding“including data targeting SWE-Bench (real-world GitHub bug resolution and patch generation) and Terminal-Bench” source ↗

Sells RL environments“We design and maintain sandboxed RL/RLVR environments that let models learn through interaction with realistic, high-stakes systems.” source ↗

Covers coding“Write CUDA kernels for the sliding-window attention layer of Inkling, Thinking Machines Lab's open-weights model, which no public kernel covers.” source ↗

Sells RL environments“That's why we're building RL environments for training capable, better-aligned superintelligences.” source ↗

Covers coding“Private evals, and data for SWE, Tau, MLE, MMMU, and more” source ↗

Covers coding“Software engineering work that targets GDPval capabilities, the economically valuable tasks professionals handle across dozens of occupations.” source ↗

Sells RL environments“Refresh builds simulated worlds for coding and computer use, with verifiable rewards that move models forward.” source ↗

Sells RL environments“BenchFlow builds the environments AI agents learn in” source ↗

Covers coding“doubling frontier-task generation across math, code, and SWE.” source ↗

Sells RL environments“Campaign is our self-play task generation infrastructure. It transforms your proprietary data and evals into new sets of environments” source ↗

Covers coding“Coding, debugging, database exploration, and API testing.” source ↗

Sells RL environments“Cua has 4K+ environments and 400K+ tasks spanning desktop apps, cross-app workflows, browser surfaces, and generated state variations.” source ↗

Covers coding“Deep in domains like coding, physical sciences, RSI, and alignment” source ↗

Sells RL environments“We specialize in creating environments and datasets for RL by leveraging our experience in simulation and model training.” source ↗

Covers coding“Designing, building, and testing complex engineering repositories.” source ↗

Sells RL environments“We turn these worlds into post-training datasets and reinforcement learning environments to train the next frontier of AI.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Covers coding“Pre-built datasets spanning RL environments, coding, and core capabilities.” source ↗

Sells RL environments“Interactive environments where agents take on open-ended tasks and make real decisions, with verifiers built to reward the behavior that matters.” source ↗

Covers coding“Your tasks help AI learn from its mistakes and get better at real-world coding.” source ↗

Sells RL environments“Adapt the commit into a task description, golden patch (the solution), and test patch (verification tests).” source ↗

Covers coding“Plans, edits files, runs commands in a local sandbox.” source ↗

Sells RL environments“Google Cloud provides the infrastructure to scale our environments and agent training.” source ↗

Covers coding“Hard, verified coding tasks. From code no model has seen.” source ↗

Sells RL environments“Long-horizon RL tasks for post-training. Tasks, not repo access: each ships in a runnable environment with a graded verifier” source ↗

Sells RL environments“Purpose-built reinforcement learning environments for long, complex tasks” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“We’re releasing a human-validated subset of SWE-bench that more reliably evaluates AI models’ ability to solve real-world software issues.” source ↗

Tests coding“An agentic benchmark evaluating AI capabilities in terminal environments through software engineering, system administration, and data processing tasks.” source ↗

Tests coding“DeepSWE 1.1 (mini-swe-agent harness run by Datacurve): Fable (max) 70%, GPT 5.5 (xhigh) 67%” source ↗

Tests coding“Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior.” source ↗

Tests coding“Terminal-native evaluation of expert work: tasks that stall a competent practitioner, graded only by execution.” source ↗

Tests coding“a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“SWE-Bench Pro is a benchmark designed to provide a rigorous and realistic evaluation of AI agents for software engineering.” source ↗

Tests coding“SWE-bench is a benchmark for evaluating large language models on real world software issues collected from GitHub.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“DeepSWE is a long-horizon software engineering benchmark that delivers four major advances over today's public benchmarks” source ↗

Tests coding“we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code” source ↗

Tests coding“a new benchmark in the SWE-bench family designed to evaluate the software engineering capabilities of LLMs across a range of programming languages.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at $1 million USD total in real-world payouts.” source ↗

Tests coding“We announce a subset of BigCodeBench, BigCodeBench-Hard, which includes 148 tasks that are more aligned with the real-world programming tasks.” source ↗

Tests coding“Initial set of tasks focused on edit, refactor, and bugfix problems.” source ↗

Tests coding“a new benchmark in the SWE-bench family designed to evaluate the software engineering capabilities of LLMs across a range of programming languages.” source ↗

Tests coding“140 executable tasks across 12 domains, each built around a targeted ML component, a controlled edit surface, and multi-setting evidence for transfer.” source ↗

Tests coding“including understanding paper contributions, developing a codebase, and successfully executing experiments” source ↗

Tests coding“Informatics olympiad programming problems” source ↗

Tests coding“an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching.” source ↗

Tests coding“A benchmark for evaluating LLM-generated CUDA code on **correctness** and **performance**.” source ↗

Tests coding“Initial set of tasks focused on edit, refactor, and bugfix problems.” source ↗

Tests coding“we introduce Rule2DRC, a large-scale benchmark for DRC script coding agents with 1,000 rule-to-script tasks” source ↗

Tests coding“We introduce SWE-fficiency, a benchmark for evaluating repository-level performance optimization on real workloads.” source ↗

Tests coding“even with code execution in the E-Bench-Code extension, reliability (Pass^3) remains below 70%” source ↗

Tests coding“Hy-Backend 2.0 (internal) | **35.2** | 33.8\* | 34.9\*” source ↗

Tests coding“All models are evaluated on our in-house self-built benchmark (300 tasks) using the Claude Code scaffold with a 200-turn budget.” source ↗

Tests coding“demonstrates strong GPU kernel optimization capabilities on Kernel Bench L3 (1.98x median speedup, 96% win rate)” source ↗

Tests coding“we first compile Themis-CodeRewardBench, a benchmark to evaluate code RMs across five preference dimensions (i.e., criteria) and eight programming languages” source ↗

Tests coding“We introduce Vibe Code Bench, a benchmark of 100 web application specifications (50 private validation, 50 held-out test) with 964 browser-based workflows” source ↗

Tests coding“Senior SWE-Bench tasks are sourced from PRs in repos spanning libraries to multi-service applications, authored by engineers with hundreds of commits in their respective repos.” source ↗

Snorkel AI · built it with others“Senior SWE-bench built by Snorkel with Princeton University and UW-Madison” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“IDE-Bench is a comprehensive framework for evaluating AI IDE agents on real-world software engineering tasks through an IDE-native tool interface.” source ↗

Tests coding“CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting)” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“Measures AI performance on real-world software engineering tasks, from bug fixes to feature builds. Built in collaboration with Cognition.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“standardize a CodeForces evaluation benchmark to assess the competition-level coding capabilities of LLMs and reflect their corresponding sophisticated reasoning abilities.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“focusing on real-world applications such as software engineering and open-world information retrieval” source ↗

Tests coding“SWE-bench Lite provides a smaller, carefully selected subset of 300 tasks from the full benchmark” source ↗

Tests coding“Code Migration grades behavior, not code similarity: a migration scores by the share of hidden behavior tests the rebuilt program passes.” source ↗

Tests coding“A command-line interface (OpenHands) for tasks in code, data, and file manipulation.” source ↗

Tests coding“HiL-Bench includes two domains: software engineering and text-to-SQL.” source ↗

Scale AI · built it“We design HiL-Bench to reflect this; tasks in this benchmark require agents to ask the right questions for full clarity” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

BenchFlow · built it“We publish the environments and benchmarks we use to evaluate agents on real work — papers, datasets, and live sites. SkillsBench, ClawsBench, Robo Use” source ↗

Tests coding“FrontierSWE is an ultra long-horizon coding agent benchmark that tests implementation, performance eng and ML research” source ↗

Proximal · built it“FrontierSWE is developed by the Proximal team in collaboration with external collaborators from academia and industry.” source ↗

Tests coding“UI-Bench: Evaluating AI Frontend Generation” source ↗

Tests coding“App-Bench is a benchmark designed to evaluate how well AI-driven tools can automatically build modern web applications.” source ↗

Tests coding“asking them to construct executable backtesters from natural language strategy descriptions and market assumptions” source ↗

Collinear · built it“Evaluations we build to find where models fail” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“Code Index is a live and evolving coding-benchmarks leaderboard composed of novel, contamination-free collections.” source ↗

Tests coding“We task models with writing, from scratch, a Game Boy Advance (GBA) emulator in Rust that compiles to WebAssembly.” source ↗

Mechanize · built it“We’re releasing GBA Eval as an example of what it looks like to invest time in careful grading for a single long-horizon SWE task.” source ↗

Tests coding“we post-trained Qwen3.8-27B on our tasks and measured it on Terminal-Bench 3.0.” source ↗

Tests coding“FrontierSWE is an ultra long-horizon coding agent benchmark that tests implementation, performance eng and ML research” source ↗

Proximal · built it“FrontierSWE is developed by the Proximal team in collaboration with external collaborators from academia and industry.” source ↗

Tests coding“we use continuous supply of fresh tasks collected using SWE-rebench methodology to build a contamination-free benchmark for agentic software engineering.” source ↗

Tests coding“the first benchmark to test LLMs’ ability to generate a complete working web application from only a text specification.” source ↗

Tests coding“AI agents have become surprisingly proficient at software engineering over the past year” source ↗

Tests coding“Codebase QnA consists of 124 tasks that targets the upstream capability of Software Engineering - deep code comprehension that precedes any code change.” source ↗

Tests coding“we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents.” source ↗

Tests coding“Performance results of AI coding agents on React Native code generation tasks, measuring success rate for common task groups, token usage, and cost.” source ↗

Tests coding“ReactBench is an evaluation for coding agents on realistic React work.” source ↗

Tests coding“Surgical GPU kernel benchmark. 6 active CUDA problems, frontier coding agents, roofline-based metric (achieved TFLOPS or GB/s vs hardware peak).” source ↗

Tests coding“In game development, agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations” source ↗

Tests coding“Test Writing consists of 90 high quality tasks that target a coding agent’s Test Writing ability.” source ↗

Tests coding“Unusual machine learning tasks testing adaptability” source ↗

Tests coding“Aider’s polyglot benchmark tests LLMs on 225 challenging Exercism coding exercises across C++, Go, Java, JavaScript, Python, and Rust.” source ↗

Tests coding“Android developers face specific challenges that aren't covered by existing benchmarks, so we created one that focuses on a north star of high quality Android development.” source ↗

Tests coding“Industrial Computer-Aided Design (CAD) code generation requires models to produce executable parametric programs from visual or textual inputs.” source ↗

Tests coding“We first manually construct the first class-level code generation benchmark ClassEval of 100 class-level Python code generation tasks” source ↗

Tests coding“Each eval has a set prompts for coding a Convex backend, a set of human-curated solutions, and a script for evaluating the LLM's output.” source ↗

Tests coding“Introduced problems focused on codebase understanding, bugfinding, planning, and code review.” source ↗

Tests coding“we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code.” source ↗

Tests coding“We’re excited to introduce FrontierCode, a benchmark that measures how well models can truly meet the standards of high-quality production codebases.” source ↗

Tests coding“Both MBPP and MBPP+ referred in our leaderboard use a subset (399 tasks) of hand-verified problems from MBPP-sanitized (427 tasks)” source ↗

Tests coding“McEval is a massively multilingual code benchmark covering 40 programming languages with 16K test samples” source ↗

Tests coding“All bugs included in RepairBench are single-function bugs, meaning that the reference patch changes only one function” source ↗

Tests coding“The task of SWT-Bench is to reproduce a reported issue by adding an appropriate test case to the projects' test suite.” source ↗

Tests coding“we introduce $\emph{AICD Bench}$, the most comprehensive benchmark for AI-generated code detection.” source ↗

Tests coding“We announce a subset of BigCodeBench, BigCodeBench-Hard, which includes 148 tasks that are more aligned with the real-world programming tasks.” source ↗

Tests coding“Defects4J is a collection of reproducible bugs and a supporting infrastructure with the goal of advancing software engineering research.” source ↗

Tests coding“ENAMEL is a rigorous and high-standard benchmark for evaluating the capability of large language models (LLMs) in generating efficient code.” source ↗

Tests coding“we have developed a comprehensive code evaluation dataset FullStack Bench focusing on full-stack programming” source ↗

Tests coding“we have developed a comprehensive code evaluation dataset FullStack Bench focusing on full-stack programming” source ↗

Tests coding“the agent is only given the codebase of the paper and must install all libraries and dependencies, run the code” source ↗

Tests coding“We constructed a dataset, Merge-Bench, of 7938 real-world merge conflict hunks from 1439 GitHub repositories.” source ↗

Tests coding“We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python.” source ↗

Tests coding“we propose NaturalCodeBench (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks.” source ↗

Tests coding“we introduce PerfCodeBench, an executable benchmark for evaluating LLMs on high-performance code optimization.” source ↗

Tests coding“Each task includes a natural language prompt, canonical solution, and deterministic test verification via classical circuit simulation.” source ↗

Tests coding“we initiate the RepoQA benchmark to evaluate LLMs on long-context code understanding.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“Evaluation criteria include syntax correctness, functional accuracy, synthesizability, and post-synthesis quality (PPA).” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“The benchmark that measures what AI coding agents actually do in production” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?” source ↗

Tests coding“The task of issue resolving is to modify a codebase to generate a patch that addresses a given issue.” source ↗

Tests coding“The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models” source ↗

Tests coding“SWE-bench tests AI systems' ability to solve GitHub issues.” source ↗

Tests coding“DrugDiscoveryBench: Can Coding Agents Assist Early-Stage Drug Discovery?” source ↗

Scale AI · built it“Today we're introducing READY (Reliable Enterprise Agent Deployment), a suite of industry-specific benchmarks that measure agents the way enterprises actually use them” source ↗

Tests coding“RSI Bench is an ongoing effort to evaluate whether AI agents can develop the capabilities required to advance AI R&D.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“All tasks are sourced from actual GitHub issues, representing real software engineering problems.” source ↗

Tests coding“We evaluate DeepSeek Coder on various coding-related benchmarks.” source ↗

Tests coding“We further curate LiveCodeBench-Plus, a 91-problem benchmark combining evolved and difficult original LCB-v6 tasks” source ↗

Tests coding“Defensive agents must deliver root-cause fixes that preserve legitimate behavior.” source ↗

Tests coding“Computationally intensive scientific coding across physics, materials science, chemistry, mathematics and biology.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“Long-horizon software-engineering work on private enterprise codebases — not coding puzzles.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“SWE-Bench Pro is a benchmark designed to provide a rigorous and realistic evaluation of AI agents for software engineering.” source ↗

Tests coding“We give frontier AI coding agents 24 hours to write a complete software GBA emulator with WebAssembly support.” source ↗

AfterQuery · contributed tasks or data“end-to-end legal tasks drawn from real cases across 28 practice areas, with a public case co-created with AfterQuery.” source ↗

Tests coding“Coding agents against real vulnerabilities in real codebases, with strict gates and both programmatic and judge verification on every task.” source ↗

Collinear · built it“Evaluations we build to find where models fail” source ↗

Collinear · built it“To apply our TraitBasis method systematically, we developed τ-Trait, a new benchmark designed specifically to measure agent robustness.” source ↗

BenchFlow · built it“We publish the environments and benchmarks we use to evaluate agents on real work — papers, datasets, and live sites. SkillsBench, ClawsBench, Robo Use” source ↗

BenchFlow · built it“We publish the environments and benchmarks we use to evaluate agents on real work — papers, datasets, and live sites. SkillsBench, ClawsBench, Robo Use” source ↗

BenchFlow · built it“We publish the environments and benchmarks we use to evaluate agents on real work — papers, datasets, and live sites. SkillsBench, ClawsBench, Robo Use, PostTrain, and FrontierPhysics.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Snorkel AI · contributed tasks or data“Snorkel AI served as Harvey’s data partner for BigLaw Bench: Research, bringing its expertise in creating complex expert data for frontier AI” source ↗

Tests coding“we ran 153 autonomous runs on the nanoGPT optimizer speedrun across 18 frontier models, testing multiple seeds per model.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“VADER comprises 174 real-world software vulnerabilities, each carefully curated from GitHub repositories and annotated by security experts.” source ↗

Tests coding“The benchmark spans 242 expert-validated programming problems across 15 library-design tasks in four languages.” source ↗

Tests coding“SlopCodeBench: A community benchmark measuring code erosion” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“Each model starts from an empty workspace and constructs a complete application from a product specification.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“We introduce T2J-Bench, a benchmark for codebase conversion that reformulates conversion as transfer under a fixed equivalence contract.” source ↗

Tests coding“users submit diffs, the harness applies each diff inside the task image, runs Lean/SafeVerify checks” source ↗

Tests coding“On the repo-level code generation benchmark VIBE-Pro, M2.7 scored 55.6%, nearly on par with Opus 4.6” source ↗

Tests coding“Kimi-Linear Decode: never wrote a kernel — open run page” source ↗

Tests coding“we introduce ABC-Bench, a benchmark explicitly designed to evaluate agentic backend coding within a realistic, executable workflow.” source ↗

Tests coding“The Agent Security League is an open benchmark tracking the functional and security correctness of popular AI coding agents across real-world tasks.” source ↗

Tests coding“asks the LLM to refactor 89 large methods from large python classes.” source ↗

Tests coding“a benchmark for evaluating temporal visual reasoning via reconstructing executable web animation code from videos.” source ↗

Tests coding“We present ArchBench, the first unified platform for benchmarking LLM capabilities on software architecture tasks.” source ↗

Tests coding“we introduce ArtifactsBench, a new benchmark and paradigm for the automated, multimodal evaluation of visual code generation.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“CAD/technical code evaluation tasks” source ↗

Tests coding“A benchmark for evaluating AI coding agents on real-world tasks within small-scale codebases (<10k LoC) that aren't part of LLM training data.” source ↗

Tests coding“We introduce CUDABEAVER, a benchmark for CUDA debugging from real failing workspaces produced during LLM-based CUDA generation.” source ↗

Tests coding“We introduce Delulu, a verified multi-lingual benchmark of 1,951 FIM samples across 7 languages and 4 hallucination types.” source ↗

Tests coding“DevBench is a comprehensive benchmark designed to evaluate LLMs across various stages of the software development lifecycle” source ↗

Tests coding“we introduce GitTaskBench, a benchmark designed to systematically assess this capability via 54 realistic tasks across 7 modalities and 7 domains.” source ↗

Tests coding“PassBench, 200 curated long-tail fusible tasks (comprising 2,060 subgraphs in total) evaluated under the Error-aware Speedup Score (ES\_t)” source ↗

Tests coding“3DCodeBench evaluates how effectively 12 advanced VLMs can serve as procedural 3D modelers by translating text and image references into procedural code for 3D modeling software.” source ↗

Tests coding“we introduce BIRD-Python, a benchmark designed for cross-paradigm evaluation.” source ↗

Tests coding“CanItEdit is a benchmark for evaluating LLMs on instructional code editing, the task of updating a program given a natural language instruction.” source ↗

Tests coding“CFEval | 2099 | 1929 | 2043 | 2001 | - | 2056 | **2134**” source ↗

Tests coding“a benchmark capable of evaluating Large Language Models (LLMs) concise code generation abilities in 60 programming languages” source ↗

Tests coding“\textbf{\dataname} is an 11-domain, 54-leaf, 1,000-query WebDev benchmark spanning both static-presentation and interactive-application tasks” source ↗

Tests coding“We introduce CUDAHercules, a benchmark that evaluates generated CUDA against end-to-end human-expert SOTA systems.” source ↗

Tests coding“We introduce an evaluation framework of 500 C verification tasks across five property types (memory safety, overflow, termination, reachability, data races)” source ↗

Tests coding“Provide a rigorous, sterile environment to test kernels generated by AI coding agents.” source ↗

Tests coding“The USACO benchmark evaluates AI agents on competitive programming problems from the USA Computing Olympiad.” source ↗

Tests coding“KernelBench_X is a reproducible evaluation harness for **Triton kernel code generation**.” source ↗

Tests coding“we present KOCO-BENCH, a novel benchmark designed for evaluating domain specialization methods in real-world software development.” source ↗

Tests coding“each task in KramaBench asks a system to build a _complete_ data pipeline: load raw files, clean them, transform them, and compute a final answer.” source ↗

Tests coding“introducing Long Code Arena, a suite of six benchmarks for code processing tasks that require project-wide context.” source ↗

Tests coding“We introduce LongWebBench, a benchmark for evaluating long-horizon webpage generation from both structural and functional perspectives.” source ↗

Tests coding“A 12-task scientific-compute benchmark for **Apple Silicon Metal** kernels, paired with a lightweight evolutionary harness for LLM-driven kernel search.” source ↗

Tests coding“MLX Benchmark V2 easy split measuring Apple MLX framework knowledge on lower-difficulty questions.” source ↗

Tests coding“a paradigm where chemists interact with LLMs to generate executable programs for molecular tasks” source ↗

Tests coding“A large part of this project was to find the 'best' prompting strategies for performing NL to MongoDB queries.” source ↗

Tests coding“We introduce NL2Scratch, an executable benchmark for natural-language-to-Scratch generation comprising 311,648 parser-valid NL--program pairs” source ↗

Tests coding“We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries.” source ↗

Tests coding“We introduce PDEAgent-Bench, to the best of our knowledge, the first multi-metric, multi-library benchmark for PDE-to-solver code generation.” source ↗

Tests coding“We present ProcCtrlBench, a benchmark for execution-process evaluation in LLM coding agents.” source ↗

Tests coding“we introduce RepoBench, a new benchmark specifically designed for evaluating repository-level code auto-completion systems.” source ↗

Tests coding“SWE-PRBench is a benchmark of **350 pull requests with human-annotated**” source ↗

Tests coding“Benchmark CLI for comparing AI coding agents on TypeScript workloads. Run an agent, execute tests, get pass/fail — across models and providers.” source ↗

Tests coding“introduce \textsc{VIBench}, a benchmark for measuring VIB in direct and agentic code generation across 20 provider-selectable software-integration scenarios” source ↗

Tests coding“Models generate executable programs, which we render and score against the source image using a VLM rater with dataset-specific rubrics” source ↗

Tests coding“we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation.” source ↗

Tests coding“we introduce WebIGBench, the first benchmark designed to evaluate code generation for interactive webpages with complex interactions.” source ↗

Tests coding“AA LiveCodeBench score of 86.8% and MMLU-Pro at 87% — solid general-knowledge and code-generation breadth at the lower-compute tier, verified on an independent harness.” source ↗

Tests coding“This program exists to run untrusted model-generated code.” source ↗

Tests coding“LiveCodeBench Lite Code Generation” source ↗

Tests coding“a large-scale, continuously updated, contamination-aware benchmark for evaluating LLMs on competitive programming and code reasoning tasks.” source ↗

Tests coding“We introduce LiveCodeBench Pro, a benchmark composed of problems from Codeforces, ICPC, and IOI that are continuously updated to reduce the likelihood of data contamination.” source ↗

Tests coding“Evaluating challenging long-horizon software engineering tasks in public open source repositories” source ↗

Tests coding“Evaluating challenging long-horizon software engineering tasks in public open source repositories” source ↗

Tests coding“FeatureUnion not working when aggregating data and pandas transform output selected” source ↗

Tests coding“First version of a Dockerized SWE-bench evaluation tool” source ↗

Tests coding“SWE-bench JavaScript: Evaluates software-engineering agents on realistic issue resolution, repository navigation, testing, or maintenance workflows.” source ↗

Tests coding“SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution” source ↗

Idler · built it“We created a digital twin, ShelfLife, of a live e-commerce company.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“we introduce APPS, a benchmark for code generation.” source ↗

Tests coding“evaluate three solver-construction paradigms: native algorithmic search (Python), constraint modeling through a Python solver API (Python + OR-Tools)” source ↗

Tests coding“105 real bugs, hidden in two production codebases.” source ↗

Tests coding“DSBench-Hard is DeepSeek's internal hard benchmark for complex coding-agent capabilities.” source ↗

Tests coding“Software integration and production-failure diagnosis with the Terminus-2 harness.” source ↗

Tests coding“Software integration and production-failure diagnosis with the Inspect harness.” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“DeepSWE 1.1 (mini-swe-agent harness run by Datacurve): Fable (max) 70%, GPT 5.5 (xhigh) 67%” source ↗

Tests coding“Each run asks a CLI agent to maximize one base model on one benchmark.” source ↗

Tests coding“Whether Claude Opus 5 belongs in _your_ routing table depends on your languages, your codebases, and your task mix” source ↗

Tests coding“We present OPENHARMONY BENCH, an app-level coding benchmark for evaluating LLM-based coding agents on OpenHarmony ArkTS applications.” source ↗

Tests coding“VulcanBench Coding Intelligence Index v1 evaluates coding agents on post-cutoff open-source engineering tasks using hidden functional tests and regression guards.” source ↗

Tests coding“On FrontierCode 1.1 Extended, our benchmark for real-world engineering tasks that grades mergeability and quality” source ↗

Tests coding“We are excited to introduce **FrontierCode 2.0**, a new generation of our coding benchmark series and our best attempt yet to evaluate user experience.” source ↗

Tests coding“We introduce ScarfBench, a benchmark for behavior-preserving cross-framework refactoring of enterprise Java applications.” source ↗

Tests coding“we propose SWE-bench Multimodal (SWE-bench M), to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software.” source ↗

Tests coding“Improved coding &amp; agent performance over K2.6: +21.8% on Kimi Code Bench v2” source ↗

Tests coding“our internal benchmark that tests coding agents' multi-step execution, tool use, and end-to-end problem solving across diverse engineering domains” source ↗

Tests coding“Measuring how well AI agents can post-train language models” source ↗

Tests coding“QwenReactBench is Qwen's internal benchmark for React application generation, reported as a BT/Elo rating.” source ↗

Tests coding“FullStack 测完整的全栈开发能力，Hard 则专门放高难度 Agent 任务。” source ↗

Tests coding“Agent evaluations for Next.js coding tasks, powered by `@vercel/agent-eval`.” source ↗

Tests coding“Spider 2.0-Lite | Text-to-SQL task | 547 | BigQuery(214), Snowflake(198), SQLite(135)” source ↗

Tests coding“Artificial Analysis Coding Index represents the weighted average of coding benchmarks in the Artificial Analysis Intelligence Index (Terminal-Bench v2.1, SciCode)” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Tests coding“BridgeBench: the official AI benchmark of the vibe coding movement” source ↗

Tests coding“Six 2024 IOI programming problems evaluated with Vals AI's archived v1 C++ agent harness.” source ↗

Tests coding“The model receives a fully working web app. It must handle up to ten “PM-style” requests (in plain English) like adding a new feature, fixing a bug” source ↗

Tests coding“Benchmarks coding agents beyond issue resolution across Codebase Q&A (124 tasks), test writing (90), and refactoring (70)” source ↗

Tests coding“Implementation of a fully functional software requires planning system architecture, organization of storage, and easily extendable modules.” source ↗

Tests coding“A Research Coding Benchmark Curated by Scientists” source ↗

Tests coding“A Research Coding Benchmark Curated by Scientists” source ↗

Tests coding“A Research Coding Benchmark Curated by Scientists” source ↗

Tests coding“This setting isolates core coding and reasoning skills on well-defined, contextualized scientific steps.” source ↗

Tests coding“288 subtasks with background; v1.0.1; 3 runs; pass@1; no model tools; isolated execution; 300 seconds” source ↗

Tests coding“we introduce \textbf{ProCodeBench}, a real-world benchmark for proactive intent prediction.” source ↗

Tests coding“We present ArchBench, the first unified platform for benchmarking LLM capabilities on software architecture tasks.” source ↗

Tests coding“We present ArchBench, the first unified platform for benchmarking LLM capabilities on software architecture tasks.” source ↗

Tests coding“We present ArchBench, the first unified platform for benchmarking LLM capabilities on software architecture tasks.” source ↗

Tests coding“When we launched WebDev Arena, it introduced the first large-scale, human-in-the-loop benchmark for AI coding.” source ↗

Tests coding“Performance, cost, and execution time for leading coding agents on end-to-end software engineering tasks” source ↗

Tests coding“BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies.” source ↗

Tests coding“This repository contains the data files and the utility framework of the BugsJS benchmark.” source ↗

Tests coding“we construct Cangjie-bench, a multi-task benchmark based on the novel statically-typed language Cangjie.” source ↗

Tests coding“CodeContests is a competitive programming dataset for machine-learning.” source ↗

Tests coding“ContextBench consists of 1,136 issue-resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts.” source ↗

Tests coding“We introduce DS-1000, a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas.” source ↗

Tests coding“we present the first instruction-guided code completion benchmark, Controllable Code Completion Benchmark (C3-Bench), comprising 2,195 carefully designed completion tasks.” source ↗

Tests coding“`webdev` (Code Arena)” source ↗

Tests coding“We introduce ProjDevBench, an end-to-end benchmark that provides project requirements to coding agents and evaluates the resulting repositories.” source ↗

Tests coding“Enables real-time comparison of coding capabilities” source ↗

Tests coding“Codegolf v2.2 | pass@1 | 0-shot | 11.0 | 16.8” source ↗

Tests coding“Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation” source ↗

Tests coding“We present RoundTripCodeEval (RTCE), a benchmark of four code execution reasoning tasks that evaluates round-trip consistency through execution-free, exact-match assessment” source ↗

Tests coding“We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories from SWE-Bench” source ↗

Tests coding“It includes 14 datasets for 10 diversified programming language tasks covering code-code (clone detection, defect detection, cloze test, code completion, code refinement, and code-to-code translation)” source ↗

Tests coding“A comprehensive evaluation framework for code generation models across **4 specialized domains**” source ↗

Tests coding“FPBench, a new benchmark of 721 programming tasks across three difficulty levels on three mainstream FP languages: Haskell, Ocaml and Scala.” source ↗

Tests coding“A comprehensive benchmark dataset for VEX (Vector Expression Language) code generation and understanding, extracted from Houdini 21.0.596.” source ↗

Tests coding“existing repository-level code completion benchmarks focus almost exclusively on software code and largely overlook hardware description languages” source ↗

Tests coding“We create the first massively multilingual code generation benchmark by using MultiPL-E to translate two popular Python code generation benchmarks to 18 additional programming languages.” source ↗

Tests coding“none jointly requires repository navigation, hierarchy-aware localization, Electronic Design Automation (EDA) executable verification, and maintenance-style patching” source ↗

Tests coding“This focuses on each agent's ability to produce mergeable code, regardless of whether they iterate publicly (with drafts) or privately.” source ↗

Tests coding“Quixbugs is intended for investigating cross-language performance by _multi-lingual_ program repair tools.” source ↗

Tests coding“SocialBias-Bench, a benchmark of 343 real-world coding tasks spanning seven demographic dimensions.” source ↗

Tests coding“PolyBench is a multi-language benchmark designed to evaluate AI coding assistants across diverse programming tasks and languages.” source ↗

Tests coding“whether Vision-Language Models (VLMs) can process annotated network sketches into correct optimization code remains unexplored” source ↗

No line from its own site or paper places it in Coding yet; the catalog lists it here.

Sells data“We specialize in creating environments and datasets for RL” source ↗

Covers coding“Deep in domains like coding, physical sciences, RSI, and alignment” source ↗

Sells data“Prebuilt datasets, curated for signal, reviewed for quality, and structured to drop into your training stack without translation work.” source ↗

Covers coding“Datacurve provides the frontier coding data that powers the world's most advanced models.” source ↗

Sells data“We provide ready-to-use datasets across software engineering, machine learning engineering, autonomous research, and model post-training.” source ↗

Covers coding“building post-training datasets for software engineering and AI R&D” source ↗

Sells data“A frozen archive of news, SEC filings, arXiv, Wikipedia and the wider web, with point-in-time search and fetch for forecasting and reproducible evals and training.” source ↗

Covers coding“a benchmark of 89 challenging terminal-based software engineering tasks spanning compilation, cryptography, data processing, machine learning, system administration and scientific computing” source ↗

Sells data“We pay top engineers $100-300/hr for coding challenges that can stump our AI model.” source ↗

Covers coding“Your tasks help AI learn from its mistakes and get better at real-world coding.” source ↗

Sells data“Every datapoint is created from scratch by senior practitioners, primarily senior software engineers alongside vetted domain specialists.” source ↗

Covers coding“The most effective approach is to use curated, real-world engineering tasks that represent a reliable ground truth.” source ↗

Sells data“Exported as a tar.gz with its sha256, QC evidence and solver trajectories included.” source ↗

Covers coding“Hard, verified coding tasks. From code no model has seen.” source ↗

Sells data“Vetted experts from our network of 400,000+ work on your platform, managed by your team or our leads.” source ↗

Covers coding“Work across the software lifecycle, from understanding and debugging an unfamiliar codebase to building features and writing tests.” source ↗

Sells data“Draws on 300,000+ verified practitioners across medicine, law, finance, engineering, and more” source ↗

Covers coding“Spans expert-written code, test cases, and debugging traces that teach models to write production-quality software” source ↗

Sells data“Fully managed, large-scale data curation with dedicated expert teams.” source ↗

Covers coding“use case specific (coding, RAG, tool use etc.).” source ↗

Sells data“We turn these worlds into post-training datasets and reinforcement learning environments” source ↗

Covers coding“Designing, building, and testing complex engineering repositories.” source ↗

Sells data“500B+ tokens of agent training data generated” source ↗

Covers coding“Collinear supports cybersecurity and software engineering, computer use, multi-step tool use, long-horizon reasoning, APIs and MCP, and CLI workflows.” source ↗

Sells data“A frontier human data and evaluation lab making AI better for creativity.” source ↗

Covers coding“helping AI tools (across video, code, image and more) understand quality the way expert creatives do.” source ↗

Sells data“We run the rollouts on the same environments and deliver verified trajectory datasets, packaged for your data-ingestion pipeline” source ↗

Covers coding“Coding, debugging, database exploration, and API testing.” source ↗

Sells data“Text and image data from real people playing, captured at the source with clear provenance.” source ↗

Covers coding“long-horizon planningcodespatialmemorynegotiationcollaboration” source ↗

Sells data“Exploring why human judgment, skill, and expertise remain essential to training modern AI and how people shape the future of artificial intelligence.” source ↗

Covers coding“Across several frontier models running DeepSWE-1.1, over 80% of agent rollouts contain reasoning about an imagined grader.” source ↗

Sells data“DataVendorConnect post-training suppliers with labs.” source ↗

Covers coding“Sharpe designs RL environments for coding, AI training, and ML project management using HUD” source ↗

Sells data“Our **AI recruiter** sources vetted specialists for any domain, on demand.” source ↗

Covers coding“Multi-file repos with build, run and test loops.” source ↗

Sells data“Rich computer-use data built around complete workflows, synchronized interaction signals, and verifiable outcomes.” source ↗

Covers coding“Browser useCodingDesignSpreadsheetsProductive workflows” source ↗

Sells data“30k+ experts: physicians, lawyers, engineers, consultants.” source ↗

Covers coding“Real-world software engineering across integration and observability” source ↗

Sells data“We're building the infrastructure for advancing intelligence through expert human data, real-world training environments, and contextual evaluations” source ↗

Covers coding“Source code documentation, engineering workflows, bug tracking, product planning, and technical knowledge bases” source ↗

Sells data“Expert-validated outputs that establish what successful completion looks like” source ↗

Covers coding“Building, debugging, and maintaining complex software” source ↗

Sells data“We produce exclusive and off-the-shelf (OTS) datasets built around multi-step hierarchical reasoning, tool-augmented workflows, and domain-specific problems” source ↗

Covers coding“including data targeting SWE-Bench (real-world GitHub bug resolution and patch generation) and Terminal-Bench” source ↗

Sells data“Email contact@refresh.dev to request environments or discuss a bespoke dataset” source ↗

Covers coding“Software engineering work that targets GDPval capabilities, the economically valuable tasks professionals handle across dozens of occupations.” source ↗

Sells data“We look for the most brilliant minds on the planet in every domain – doctors, lawyers, investment bankers, Fields Medalists, Harvard professors” source ↗

Covers coding“spanning RL environments, coding, and the core capabilities every model needs.” source ↗

Sells data“Private evals, and data for SWE, Tau, MLE, MMMU, and more” source ↗

Covers coding“Private evals, and data for SWE, Tau, MLE, MMMU, and more” source ↗

Covers coding“Review Pull Request” source ↗

Sells data“Our vast network of vetted coding experts offers scalable data production for the languages, coding domains, and programming expertise of your choice.” source ↗

Covers coding“Turn your agent or model into an expert coder with curated custom datasets.” source ↗

Sells data“We connect the world's top red teamers, researchers, and engineers to mission-critical projects at frontier AI labs.” source ↗

Covers coding“How robust are coding agents to attacks they've never seen?” source ↗

Sells data“Request dataset samples or join our global expert network to advance superintelligence.” source ↗

Covers coding“Software engineeringFor the software development lifecycle.” source ↗

Sells data“Start with reviewed reasoning datasets or build targeted data packs around your model's actual errors, tasks, and evaluation results.” source ↗

Covers coding“train and evaluate agents in verifier-backed RL gyms for code, algorithms, science, and mathematics.” source ↗

Sells data“Data grounded in real expertise, from practitioners at the frontier of the sciences, medicine, law, finance, engineering, and the humanities.” source ↗

Covers coding“Pre-built datasets spanning RL environments, coding, and core capabilities.” source ↗