What RL environment vendors sell, measured on one open model: four post-training results from NVIDIA's Nemotron 3.5 Super VL launch
By RL Research Team+23.7 points on AutomationBench-AA from 2,500 verified tasks and about 1,450 GPU-hours (Prime Intellect); 47% to 74% of 100 held-out credit tasks fully solved (AfterQuery)
NVIDIA released the open Nemotron 3.5 Super VL on October 15, 2026. The same day, four companies that sell RL environments or expert tasks published what their data did to it:
- Prime Intellect: 2,500 verified enterprise tasks and about 11 hours of training lifted the model’s AutomationBench-AA score from 34.0% to 57.7%.
- AfterQuery with Hebbia: the share of 100 held-out credit tasks fully solved rose from 47% to 74%, three points short of GPT-6 Astra at about a seventh of the cost.
- Mercor: a few hundred tasks per domain moved three of its APEX benchmarks by 6.7 to 8.2 points.
- Scale: reported cost ratios, with accuracy shown only as unlabeled points on its charts.
The model card shows the other side of the trade: NVIDIA lists task sets from Mercor and Turing as purchased datasets. Every vendor scored its own run, on a benchmark it built or picked.
Key Takeaways
- Largest gain: Prime Intellect’s 2,500 tasks, +23.7 points on the public AutomationBench-AA split, for roughly 1,450 GPU-hours.
- Fewest tasks: Mercor’s few hundred per domain moved APEX scores 6.7 to 8.2 points.
- Both sides of the trade: Mercor appears in NVIDIA’s training data and in the launch’s customer results.
What did each vendor deliver?
Each vendor supplied the tasks and the grading; the training method differed:
- Prime Intellect: 2,500 tasks modeled on Zapier’s AutomationBench, business workflows across simulated CRMs, inboxes, help desks and spreadsheets. Each task has a request, a starting world and programmatic checks on the final state. NVIDIA trained on them with GRPO (a reinforcement learning method that scores a group of attempts at the same task against each other), 100 steps on 128 GB200 GPUs.
- AfterQuery with Hebbia: credit tasks built around syndicated loan agreements, seeded by Hebbia’s former credit investors and written by AfterQuery’s expert network, trained with NVIDIA’s NeMo RL library on 32 H200 GPUs. A judge model graded each answer against an expert rubric.
- Mercor with Harvey, Cognition and Ramp: a few hundred tasks per domain from its APEX environments. Legal and software used supervised fine-tuning (SFT), training on attempts its graders had selected; accounting used reinforcement learning.
- Scale: post-trained the model for Skydance’s vendor-credit finance agent and EY’s M&A due-diligence agent, and launched the work as a product, Scale Post-Training.
Prime Intellect @PrimeIntellect A still from Prime Intellect's launch video. The 2,500 tasks cover six business functions, from lead dedup in sales to access reviews in HR, each a request inside simulated apps with checks on the final state.
Chart: NVIDIA, Nemotron 3.5 Super VL launch post, October 15, 2026 · source
What did NVIDIA buy before the launch?
The model card itemizes its post-training data, and vendor names sit in the table next to the synthetic sets:
- Mercor: Terminal-Bench-like tasks curated by Mercor, and a set called Mercor SWE-AgentsV1.
- Turing: a Turing Math Data Pack.
- Patronus: dialog-memory data under an external vendor agreement.
For the Mercor tasks and the Turing pack, the card’s source column reads “Undisclosed - purchased dataset”.
NVIDIA’s launch post gives the totals: about 26 trillion training tokens, then roughly 3.8 million post-training samples and reinforcement learning across dozens of environments, released in NeMo Gym. The Nemotron 3 Ultra recipe used 55.
Closed labs publish no such table. Here a lab names the vendors it pays, and Mercor sits on both sides: selling tasks to the model’s builder, then specializing the finished model for enterprises.
Mercor @mercor Mercor's launch card names the three enterprises it specialized the model for: Harvey in legal, Cognition in software and Ramp in accounting. The same company sold NVIDIA task sets used to train the model in the first place.
What did the results cost?
Training compute is published for one run. Prime Intellect’s took 128 GB200 GPUs, 32 training and 96 generating attempts, for about 11 hours, roughly 1,450 GPU-hours. AfterQuery names its hardware, 32 H200 GPUs, but not the run length. Mercor and Scale publish neither.
The vendors priced inference instead. Scale reports three customer results:
- Skydance: matched the frontier model Skydance runs in production, at about 40 times lower inference cost.
- EY: matched EY’s frontier model at an eighteenth of the cost.
- An insurer: beat its production model by up to 31 points at a quarter of the cost.
Frontier labs buy these vendors’ tasks for a higher score. Here the buyer is an enterprise trading a few points for a far cheaper model it runs itself, the per-workflow economics of selling environments.
Chart: Scale AI, Scale Post-Training launch post, October 15, 2026 · source
What can these numbers not tell you?
Four limits apply to all four results:
- Self-graded: Mercor tested on APEX, which it built. Prime Intellect used AutomationBench-AA’s public split; Artificial Analysis and Zapier hold a private split of 657 tasks that the case study does not report.
- Different metrics: mean rubric scores, full-credit rates and partial credit zeroed by a broken guardrail do not convert into each other.
- Judge errors: AfterQuery excluded tasks with no recorded reward, 8 of 100 for the post-trained model and 5 for the base model.
- Gameable tasks: Prime Intellect reports that roughly 15% of its generated tasks could be gamed before it added dedicated checks, the reward-hacking risk every rubric and checker carries.
What this means
The launch put four vendors’ products on one model in public, and what they sold was specialization: a cheaper open model brought close to a frontier model on one customer’s workflow. A buyer comparing them still needs one held-out test run across all four, which none of the posts provides. Data vendors keep moving up the stack, and the card shows labs still buy from them first.
FAQ
What is Nemotron 3.5 Super VL?
NVIDIA’s open 120-billion-parameter multimodal model for agents, released October 15, 2026. Organizations can post-train it on their own data and run it on their own infrastructure.
What is AutomationBench-AA?
Zapier’s AutomationBench, graded with Artificial Analysis’s method. Agents carry out business workflows across 47 simulated SaaS applications, and deterministic checks inspect the final state of those applications. A task earns partial credit for each objective met and scores zero if the agent breaks any condition it was told to leave alone.
Which vendors does the Nemotron 3.5 Super card name as data sources?
Mercor, for Terminal-Bench-like tasks listed as a purchased dataset and for a set named SWE-AgentsV1; Turing, for a math data pack also listed as purchased; and Patronus, for dialog-memory data under an external vendor agreement. The card does not disclose sizes or prices.