Five people, about $300k, synthetic RL environments: Ultramarine 1 Flash beats Claude Fable 5 on 4 of 6 knowledge-work benchmarks

A team of five post-trained an open model into a knowledge-work agent that beats Claude Fable 5 on four of six knowledge-work benchmarks, using only synthetic RL environments. Paper Instruments’ Ultramarine 1 Flash, released October 14, 2026, applies reinforcement learning (RL) to the open GLM-5.3 Flash and gains 4.1 points on average across six knowledge-work benchmarks while generating 6.5% fewer tokens. On the company’s own suite it costs $0.13 per task, against $8.19 for Claude Fable 5. Every training task is synthetic: 88 generated environment types across finance, law, consulting and operations, each graded by an AI judge against a rubric, with no expert-authored tasks and no production traces. CEO Daanish Khazi puts the post-training bill at about $300k all-in. The gains have three limits: Fable 5 still leads on Mercor’s APEX-Agents, the newer Fable 5.1 scores higher on Paper Instruments’ own suite, and both coding benchmarks went down.

Key Takeaways

• Against the base model: up on all six knowledge-work benchmarks, by 1.4 to 7.3 points; down 1.1 on Terminal-Bench 2.1 and 1.9 on DeepSWE 1.1.
• Against Claude Fable 5: ahead on JobBench, FrontierFinance, AutomationBench and the internal suite; behind on APEX-Agents 1.1 by 6.6 points and on Paper Instruments’ own DiligenceBench by 2.3.

What did Paper Instruments train on?

Published RL post-training for agents mostly uses verifiable tasks, where a test decides the reward, and coding dominates. A diligence memo or a litigation chronology has no unit test, so Paper Instruments writes the reward per task. It weighed three sources of such tasks and kept one:

• Production traces from real users: realistic, but users leave assumptions unstated, and replaying a rollout needs a faithful copy of production state.
• Expert-authored tasks: well defined and gradable, but costly to keep broad, and narrow rubrics still reject defensible answers.
• Synthetic generation: the one it used, for full control over the task mix and cheap enough to filter hard.

The curriculum covers four domains, 19 practice areas and 88 environment types, generated along sampling axes with ten million to ten billion combinations so tasks do not collapse onto a few templates. Environments hold SEC filings, synthetic contracts and emails, and live systems of record such as a CRM or an ERP. Each task takes several dozen agent loops and review rounds to produce, with classifiers screening for answer leakage and reward hacking, and ships in the Harbor format.

How do you grade work that has no unit test?

With a rubric, and Paper Instruments puts the effort into each criterion’s wording. A criterion copied from one golden answer rejects valid alternatives; one that only asks for an appropriate recommendation can reward plausible-sounding errors. So each criterion states one proposition, met or unmet, and names the entities, dates and assumptions needed to check it. Deliverables (Excel, PowerPoint, Word, PDF) are rendered to text and scored by an LLM judge using the company’s open-source rubric package, with a prompt tuned for agreement with human experts.

This is the per-criterion pattern our checklist comparison found vendors adopting for open-ended work, and it carries the judge-reliability risk that comes with any AI grader. Paper Instruments locates the fix in criterion boundaries rather than in a stronger judge.

How does Ultramarine score against Claude, GPT and Gemini?

Pick a benchmark below; the last tab is cost per task on the company’s internal suite.

Ultramarine wins on finance, automation and job-task benchmarks. Counting only the four independent knowledge-work benchmarks, it beats Fable 5 on three. The exception is APEX-Agents, Mercor’s suite of expert-built professional tasks, where Gemini 3.8 Flash and Fable 5 both lead. The launch claims compare against Fable 5. On Paper Instruments’ own suite, four models at maximum reasoning score higher than Ultramarine: GPT-6 Sol at about 10 times Ultramarine’s cost, and Claude Opus 5.5, GPT-6 Astra and Fable 5.1 at 43 to 66 times.

What does a $300k model mean for the expert-data market?

“It was post-trained by a team of 5 for ~$300k all-in.”

A slide from the launch video in the post: output tokens at $0.50 per million against $50 for Claude Fable 5. Per task on Paper Instruments' suite the gap is $0.13 against $8.19.

The vendors that sell expert-authored tasks argue that their data is what moves models. Paper Instruments bets the other way in public: generate every task and spend on compute, filtering and rubric wording. Our read: synthetic tasks win on breadth and cost, and expert data keeps its strongest case on the benchmark built from expert work, where Ultramarine closed less than half the gap to Fable 5. Coding points the same way: Surge’s office-work training improved coding, while this curriculum gave a little back.

The rest of the recipe is published too:

• Harness: Feather, a Python agent loop whose training system prompt is about 50 tokens, against roughly 1,800 to 4,400 in the production harnesses Paper Instruments surveyed. The harness and training integrations are on GitHub as feather-train, MIT-licensed.
• Sandboxes: each rollout runs in an isolated E2B sandbox with the agent loop outside it, so the model cannot touch its harness, credentials or grader, as our microVM post recommends.

What this means

A small team can now buy breadth in knowledge-work RL with compute instead of experts: synthetic environments, single-proposition rubrics and an AI judge took an open Flash model past Fable 5 on four of six knowledge-work benchmarks for about $300k. Buyers should test the two remaining gaps, the expert-built benchmark and coding, before switching.

FAQ

What is Ultramarine 1 Flash?

Paper Instruments’ agentic model for professional knowledge work, released October 14, 2026: GLM-5.3 Flash post-trained with reinforcement learning on synthetic finance, legal, consulting and operations tasks graded by rubrics. It averages 4.1 points above its base model on six knowledge-work benchmarks and costs $0.13 per task on the company’s internal suite.

Is Ultramarine open source?

Not yet in full. Ultramarine is available through Paper Instruments’ API at $0.15 per million input tokens and $0.50 per million output tokens, and CEO Daanish Khazi wrote that the weights will follow two weeks after release. The training harness, the rubric grading package and the Word, Excel and PowerPoint libraries are already open source.

What else has Paper Instruments published?

Its research blog shows the run-up to Ultramarine:

• Kos-1 Lite, a medical model, March 2026
• Kos-1 Experimental, environment-free RL on Kimi K2.5, May 2026
• Notes on Choosing a Rubric Judge, June 2026
• DiligenceBench, its equity-research benchmark, July 2026
• Paper Office, the agent-first Office file libraries, September 2026

What is non-verifiable RL?

Reinforcement learning on tasks no script or unit test can check, such as a diligence memo. The reward comes from a rubric whose criteria an AI judge marks met or unmet, so it is only as reliable as the criteria are precise.