Directory label
Long-horizon agent benchmarks and RL environments
A label other catalogs put on entries, not one of the domains on our map. On the map, each entry here sits in the domain its own site or paper leads with.
Tasks that take many steps or a long session to finish, where a model has to plan, keep track of state and recover from its own mistakes before the work is done.
- RL environments
- 26
- AI benchmarks
- 122
- Shares entries with
- 18 categories
Snapshot 2026-11-05. How categories are built · Where the data comes from
In short
- What are RL environments for long-horizon tasks?
- A simulated workplace where an AI agent practises a task (a code repository, a browser, business software) and an automatic grader scores each attempt. Labs train on thousands of these attempts. Bought to improve models. For long-horizon tasks, that means tasks that take many steps or a long session to finish, where a model has to plan, keep track of state and recover from its own mistakes before the work is done.
- Who builds RL environments for long-horizon tasks?
- Scale AI, Snorkel AI, Prime Intellect, Mercor, Surge AI and Bespoke Labs, among 25 confirmed below.
- What are the main long-horizon agent benchmarks?
- Humanity's Last Exam, Terminal-Bench, OSWorld 2.0, Agents' Last Exam, APEX-Agents and MCP Atlas, among 67 confirmed below.
RL environments 26
-
Scale AI
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Snorkel AI
Sells RL environments“Snorkel helps frontier labs and AI teams develop specialized training data and environments that set their models and agents apart.” source ↗
-
Prime Intellect
Sells RL environments“Turn any task into an RL environment. Init, develop, eval, and push with the Prime CLI.” source ↗
-
Mercor
Sells RL environments“Benchmarks and RL environments built on real professional work (APEX).” source ↗
-
Surge AI
Sells RL environments“We create rich, complex RL environments that challenge agentic models in novel ways, and design verifiers that reward their behavior.” source ↗
-
Bespoke Labs
Sells RL environments“Bespoke Labs builds company-scale RL environments and infrastructure to let frontier labs and enterprises train long-horizon agents for production.” source ↗
-
Parsewave
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Proximal
Sells RL environments“How we run agentic RL across Modal GPUs and our own Kubernetes sandbox fleet.” source ↗
-
Andon Labs
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Collinear
Sells RL environments“Every simulation lab is a self-contained world where your agent operates, complete with the users, tools, data, and tasks it will face in production.” source ↗
-
Good Start Labs
Sells RL environments“Interactive worlds with clean, verifiable rewards, where models learn by doing.” source ↗
-
General Reasoning
Sells RL environments“An open resource for discovering and experimenting with community RL environments with simple API endpoints and autoscaled infrastructure.” source ↗
-
Huzzle Labs
Sells RL environments“We build **RL environments** for frontier AI labs” source ↗
-
Idler
Sells RL environments“Reinforcement learning environments built from real production work” source ↗
-
HUD
Sells RL environments“The platform for building RLenvironments” source ↗
-
ARIMLABS
Sells RL environments“Carefully calibrated training environments for the agents of tomorrow.” source ↗
-
Vmax
Sells RL environments“Campaign is our self-play task generation infrastructure. It transforms your proprietary data and evals into new sets of environments” source ↗
-
Andromede
Sells RL environments“Andromede is an RL data lab generating environments, tasks, and verifiers from real-world data.” source ↗
-
EdotEnv
Sells RL environments“we build RL environments at scale from financial market data that increase in difficulty, so agents can continuously hillclimb.” source ↗
-
Rise Data Labs
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
AIChamp
Sells RL environments“We build the twin, then robot-AI teams, PLC changes, vision and equipment options are tested in it.” source ↗
-
pre.dev
Sells RL environments“Long-horizon RL tasks for post-training. Tasks, not repo access: each ships in a runnable environment with a graded verifier” source ↗
-
Tacit Labs
Sells RL environments“help build Tacit's scientific agent environments: integrated environments where language-model agents can use tools, inspect data, make decisions” source ↗
-
Anthromind
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Diffuse Labs
Sells RL environments“We build reinforcement learning environments that frontier labs use to train their models.” source ↗
1 not yet confirmedListed in the catalog; no line from its own site or paper confirms it yet.
AI benchmarks 122
-
Humanity's Last Exam
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Terminal-Bench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
OSWorld 2.0
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Agents' Last Exam
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
APEX-Agents
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
MCP Atlas
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Senior SWE-bench
Snorkel AI · built it with others“Senior SWE-bench built by Snorkel with Princeton University and UW-Madison” source ↗
-
Continual Learning Bench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
APEX-Agents 1.1
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
APEX-SWE
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
APEX-Accounting
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Remote Labor Index
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
HiL-Bench
Scale AI · built it“We design HiL-Bench to reflect this; tasks in this benchmark require agents to ask the right questions for full clarity” source ↗
-
YC-Bench
Collinear · built it“Evaluations we build to find where models fail” source ↗
-
VERITAS
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Vending-Bench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Long-Horizon Planning
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
TB3 results
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
FrontierSWE (v1)
Proximal · built it“FrontierSWE is developed by the Proximal team in collaboration with external collaborators from academia and industry.” source ↗
-
Terminal-Bench 2.0
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
SWE-Bench Pro v1 (2025-09-19) and v2
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
GDP.xlsx
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
DrugDiscoveryBench with Phylo
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
READY or Not enterprise agent deployment benchmarks
Scale AI · built it“Today we're introducing READY (Reliable Enterprise Agent Deployment), a suite of industry-specific benchmarks that measure agents the way enterprises actually use them” source ↗
-
RSI Bench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
PRBench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
SWE-Bench Pro - Private Dataset
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
AutoResearchExam
Bespoke Labs · built it“To address this gap, we introduce AutoResearchExam, a benchmark of open ended ML research tasks” source ↗
-
data-eng-bench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
CWE-bench v1
Collinear · built it“Evaluations we build to find where models fail” source ↗
-
TraitBasis / Impatient Users Confuse AI Agents
Collinear · built it“To apply our TraitBasis method systematically, we developed τ-Trait, a new benchmark designed specifically to measure agent robustness.” source ↗
-
Vending-Bench 2 / Vending-Bench Arena
Andon Labs · built it“We're releasing Vending-Bench 2, a benchmark for measuring AI model performance on running a business over long time horizons.” source ↗
-
Blueprint-Bench 2, 2026-05
Andon Labs · built it“Blueprint-Bench 2 builds on our original paper with an agent-only evaluation, improved scoring, and a persistent notepad for cross-apartment learning.” source ↗
-
Drone-Bench, with Anthropic Project Pilot, 2026-07
Andon Labs · built it“As part of our continued effort to understand AI’s impact on the physical world, we release Drone-Bench” source ↗
-
Benchmarking Agents in Insurance Underwriting Environments
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
BigLaw Bench: Research
Snorkel AI · contributed tasks or data“Snorkel AI served as Harvey’s data partner for BigLaw Bench: Research, bringing its expertise in creating complex expert data for frontier AI” source ↗
-
Measuring Autonomous AI Research
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
General Agent synthetic environment
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Democratizing Diplomacy harness paper
Good Start Labs · built it“We created the first Diplomacy environment where even small models can play full games.” source ↗
-
Humor / Bad Cards Arena (LOL Arena) leaderboard
Good Start Labs · built it“We run transparent, reproducible evaluations on these game environments, and we can evaluate ahead of a release.” source ↗
-
DAYJOB (Healthcare/Finance) - arXiv:2610.01306
Surge AI · built it“We introduce DAYJOB, 130 long-horizon tasks built by professionals in healthcare and finance that start from a brief request” source ↗
-
ComplexConstraints and Beyond
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
EnterpriseBench CoreCraft - arXiv:2602.16179
Surge AI · built it“We introduce CoreCraft, the first environment in EnterpriseBench, Surge AI's suite of agentic RL environments.” source ↗
-
The Hierarchy of Agentic Capabilities - arXiv:2601.09032
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Antidote and Hemingway-bench leaderboards
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
KellyBench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Blueprint-Bench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Butter-Bench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
LibraryDesignBench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
SlopCodeBench
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
APEX (AI Productivity Index) v1 / APEX-1
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
SWE-Marathon-Ext
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Diplomacy Arena leaderboard
Good Start Labs · built it“We created the first Diplomacy environment where even small models can play full games.” source ↗
-
AI_Diplomacy open-source harness
Good Start Labs · built it“We launched our first project AI Diplomacy.” source ↗
-
Chartography - arXiv:2608.10677
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
HANDBOOK.md - arXiv:2607.25398
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
GDP.pdf - arXiv:2607.11192
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Riemann-Bench - arXiv:2604.06802
Surge AI · built it“We introduce RIEMANN-BENCH, a private benchmark of 25 expert-curated problems designed to evaluate AI systems on research-level mathematics” source ↗
-
InsureBench: insurance underwriting and claims benchmark
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
CorpLaw (measured 2026-08-18; access on request):
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
AdvancedIF - arXiv
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
HANDBOOK.md Agents
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
SWE Atlas: Codebase QnA, Test Writing, Refactoring
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Tuesday Work Index leaderboard
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
Andon FM
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
-
ShelfLife (measured 2026-08-13):
Idler · built it“We created a digital twin, ShelfLife, of a live e-commerce company.” source ↗
-
ShelfLife E-Sim (measured 2026-08-26):
No line from its own site or paper places it in Long-Horizon yet; the catalog lists it here.
55 not yet confirmedListed in the catalog; no line from its own site or paper confirms it yet.
- LifeSciBench
- FrontierSWE v2
- Agent-environment papers
- Firehorse open harness library
- inspect-openreward
- ConjectureBench: 14,865 open math problems
- Dynamic Semantic Tags technical report with Intuit
- The Valley of Code Reasoning
- Cats Confuse Reasoning LLM
- Self-rationalization improves LLM as a fine-grained judge
- AI bosses studies, parts 1 and 2
- FinQA / SnorkelFinance environment and benchmark
- RIFT / Hitchhiker's Guide to Rubric Quality
- Shrinking the Generation-Verification Gap with Weak Verifiers
- Snorkel: Rapid Training Data Creation with Weak Supervision
- Human Baselines for Benchmarks
- INTELLECT-3 Technical Report
- INTELLECT-2
- Systematic Reward Hacking study
- True Agents Model the World / ECHO
- Scaling Agentic RL: 365k+ SWE/terminal/search tasks
- Data-Centric Interpretability for LLM-based Multi-Agent RL
- StarSkirmish Bench
- 1830 railroad game to Finance-Agent transfer study
- Cross-Benchmark Transfer from RL on Agentic Coding Tasks - arXiv
- Post-Training on Office Work Improves Software Engineering - arXiv
- Cross-Benchmark Generalization in Long-Horizon Agents - arXiv
- BackSearch point-in-time search for leak-free forecasting evals
- Galactica, Llama 2, Llama 3
- OpenThoughts datasets
- GEPA
- Bespoke-Nimble-9B model
- Bespoke Curator
- Evalchemy
- Bespoke-MiniCheck
- Project Vend
- Cheating in Drone-Bench
- ApexAgents SkyRL 397B RL training recipe
- COS-PLAY
- Jev verifier cost study
- PopuLoRA
- ~1k JavaScript coding tasks in Harbor format
- OpenEnv Visual Memory and Spreadsheet environments on Hugging Face
- Messier
- Predicting Task Difficulty Without Rollouts
- House Rules
- PROPEL
- Robustly Evaluating Cyber Capabilities
- HuzzleWorld computer-use models
- SheetBench-50 (released 2025-10-01)
- Scale Labs (launched 2026-03-09
- Learning from Less: RLVR in Low Data Regimes
- ACE (AI Consumer Index)
- Universal offline sandbox escape disclosure
- Rubric-based RL papers
Next: pick another domain. Tags come from the RL Research daily scrape, canonicalized nightly.