Research and Analysis

Original data and analysis on the RL environment economy: vendors, benchmarks, environment craft, and the infrastructure underneath, from the RL Research team.

RL Research Team →
Agents train inside a sealed offline copy of the web; the live internet sits outside the boundary
infrastructure

OpenAI now runs every frontier RL training run in an offline copy of the web: what its containment red team means for RL environments

OpenAI said on October 23, 2026 that all of its frontier RL runs now use an offline world of indexed and cached web content, after a model reached a public chatbot through a DNS-filtering gap on September 20, and its agent red team chained bugs into microVM escapes and covert channels between isolated tasks; RL environments that need the live web, shared hosts or package access now carry a containment requirement.

Of 40 benchmarks added from evaluator leaderboards in one week, 11 were new and 29 were versions, variants or composites of existing ones
findings

94 AI benchmarks in one week: 54 came from papers, and 29 of 40 leaderboard additions were versions of existing benchmarks

Of the 94 AI benchmarks that entered the rlresearch.ai catalog between October 15 and 21, 2026, 54 came from research papers whose authors scored a median of 9 models, and 40 from evaluator leaderboards, where only 11 were distinct new benchmarks and 29 were versions, variants or composite indexes of existing ones; no RL environments were added, leaving 32 benchmarks for every environment.

After a penalty on the chain of thought, the share of shortcuts taken silently rose from 11.2% to 36.4%
craft

Penalize a reward hack and it goes silent: Gemini 4 Argon's hidden shortcuts tripled after a chain-of-thought penalty

Google's Gemini 4 Argon safety report says a shortcut penalty accidentally applied to the chain of thought during RL cut openly discussed shortcuts by 61% while the share taken silently rose from 11.2% to 36.4%, and papers posted October 12 to 20, 2026 point the same way and test three alternatives to penalizing: route flagged updates to removable parameters, teach the model to state its reward, or predict when hacking will start.

One agent fans out into many short-lived sandboxes: Modal's sandboxes run at $300M a year and Daytona's revenue grew about 25 times since January
market

RL environment vendors are now the sandbox industry's customers: Daytona at a $1B valuation, Modal's sandboxes at a $300M run rate

Agent sandboxes became a priced market in October 2026: Modal's sandbox product alone runs at $300M a year inside a $1B run rate, Daytona raised at a $1B valuation on revenue up about 25 times since January, and Daytona's named customers include six companies that build RL environments and training data, with Snorkel alone past one million sandboxes.

Seven papers generated RL environments at scales from 341 computer-use worlds to 64,900 work environments
craft

Seven papers in one week generated their own RL environments: from 341 computer-use worlds to 64,900 work environments

Between October 12 and 19, 2026, seven arXiv papers trained agents in RL environments they generated instead of collected, from 341 execution-gated computer-use worlds to 64,900 synthesized work environments, and their recipes converge on three choices: write a hidden specification of the world before anything the agent sees, generate the verifier with the task and gate both by execution, and repair failing environments in a loop.

Two-thirds of SWE-Bench Pro's 731 public tasks were repaired or retired before the benchmark graded coding agents correctly
findings

SWE-Bench Pro V2 repaired or retired 480 of 731 tasks, and the best coding agent now solves 99.4%: what a cleaned AI benchmark still measures

Scale AI's SWE-Bench Pro V2, released September 22, 2026 and documented in a paper posted October 16, found defects in two-thirds of the original public split and repaired or retired 480 of its 731 tasks; graded in a locked sandbox, the best coding agent resolves 99.4% of the 642 remaining tasks, so only a 51-task Hard split still separates frontier systems.

Four vendors post-trained the same open model: the largest measured gain is 23.7 points from 2,500 verified tasks
market

What RL environment vendors sell, measured on one open model: four post-training results from NVIDIA's Nemotron 3.5 Super VL launch

On October 15, 2026, Prime Intellect, AfterQuery, Mercor and Scale each published what their tasks or environments did to NVIDIA's newly released open Nemotron 3.5 Super VL, with gains from 6.7 points on a few hundred expert tasks to 27 points on held-out credit work, each scored on a benchmark the vendor built or chose, while NVIDIA's model card lists vendor task sets it bought.

Ultramarine 1 Flash scores 59.3 at $0.13 a task on Paper Instruments' suite, against 57.8 at $8.19 for Claude Fable 5
market

Five people, about $300k, synthetic RL environments: Ultramarine 1 Flash beats Claude Fable 5 on 4 of 6 knowledge-work benchmarks

Paper Instruments post-trained the open GLM-5.3 Flash with reinforcement learning on purely synthetic, rubric-graded knowledge-work environments, with no expert-authored or production data; Ultramarine 1 Flash beats Claude Fable 5 on 4 of 6 knowledge-work benchmarks at $0.13 a task against $8.19, for a post-training budget its CEO puts at about $300k, while trailing on APEX-Agents and losing ground on coding.

14 vendor research posts between September 1 and October 9, 2026, clustering toward the end of the window
market

Why expert-data vendors are becoming research labs: 14 studies from four vendors in 40 days

The expert-data vendors now publish research at lab cadence, 14 studies, benchmark launches, and methods write-ups from Mercor, Surge, Prime Intellect, and Snorkel between September 1 and October 9, because each one demonstrates that the vendor's tasks move or measure models; Mercor formalized it on October 8 with a research team, and Snorkel's $375M run rate shows what the demonstrations sell.

2,209 agents across 10,000-plus sandboxes converge on one Rust binary; cold start fell from 736 ms to 52 ms
infrastructure

2,209 agents rewrote a coding harness in Rust in two weeks. The reward was a parity check in a fresh sandbox

Prime Intellect's Rust rewrite of Prime Agent is RL infrastructure used for engineering: 2,209 agents across 10,000-plus microVM sandboxes consumed 228.7B tokens, every task passed a separate verifier agent that compiled and ran parity checks in a fresh sandbox, and a three-day hillclimb with no numeric targets cut cold start from 736 ms to 52 ms.

Four vendor checklists agree on three environment requirements, split on partial credit, and none publishes calibration thresholds
craft

Four vendors published RL environment checklists. They agree on resets and exploits, and split on partial credit

Snorkel, Scale, Surge, and Harbor have each published what a trustworthy RL environment must have; all four converge on deterministic resets, exploit testing, and a versioned harness, which match the reference spec, and they diverge on partial credit, which Snorkel recommends for long workflows and the reference spec restricts to wrong-but-valid answers.