Who defines ground truth for AI agents?

Human experts define ground truth for AI agents: authors create private truth, reviewers adjudicate it, and models never approve it. The bars rlresearch.ai’s reference spec proposes a trusted environment publish are verifier (the program that grades each episode) precision and recall at or above 0.95, hard gates at exactly 1.00, and mean absolute reward error measured against adjudicated expert scores. The pattern in rlresearch.ai’s reference environment specs is three roles with hard access boundaries. Authors create cases and private truth. Performers complete work in a clean VM seeing only public assets. Reviewers adjudicate correctness but cannot change a score without a reason code, a logged justification category. Models draft, generate variants, and red-team, meaning they attempt to break the environment; approval and release promotion stay human. The thresholds are the reference spec’s proposed bars, not measured industry norms. This verification work is what Mercor’s 27% gross margin (its 2025 figure) pays for.

Key Takeaways

• Trusted environments separate three roles architecturally: authors who create private truth, performers who see only public assets, and reviewers who cannot change a score without a reason code.
• Models draft, generate variants, and red-team. They never approve truth, because LLM judges carry systematic biases and score near random on hard reasoning pairs (Zheng et al., 2023; JudgeBench, 2024).
• The published bars in the reference spec: verifier precision and recall at or above 0.95, hard gates at exactly 1.00, and reward error measured against adjudicated expert scores.

What is the separation-of-duties pattern?

The reference environment specs converge on three roles with hard access boundaries:

• The author creates cases, private truth, acceptable alternatives, and reviewer guidance. The author cannot see future performer identities or model trajectories.
• The performer completes independent multi-hour work orders inside an instrumented VM, seeing only the public incident, assets, tools, policies, and empty templates. The performer cannot access private fault state, verifier code, calibration fixtures, or other completions.
• The reviewer adjudicates correctness, alternate valid approaches, ambiguity, and every verifier false positive or false negative. The reviewer cannot silently change a score without a reason code.

The environment’s access controls enforce the separation. Reviewers adjudicate because annotators disagree: Plank (2022) argued against treating a single gold label as truth, and disagreement here is adjudicated with reasons rather than averaged.

What do models do and not do?

Models and agents draft, generate bounded variants, and red-team the environment. They do not approve truth or promote releases. A model can generate candidate cases, propose fault scenarios, or attempt to exploit the verifier. A model cannot define what correct work is, validate the verifier’s calibration, or approve an environment for release.

The paper that established LLM-as-a-judge also cataloged its systematic biases: position, verbosity, self-enhancement (Zheng et al., 2023). JudgeBench then showed strong judges scoring near random guessing on objectively checkable hard reasoning pairs, so judge reliability collapses where expert verification matters most. In RLVR (reinforcement learning with verifiable rewards) training loops specifically, rule-based verifiers produce false negatives while model-based verifiers get hacked into false positives (Huang et al., 2025).

Mercor’s enterprise framework defines the verifier as an isolated program returning a score in [0, 1] with a stated basis. Who calibrates those verifiers is left open there; in the designs reviewed, human experts do.

Which calibration bars does the reference spec propose?

Precision is the share of verifier-passed submissions that are correct on expert review; recall is the share of correct submissions the verifier passes. Reviewer agreement is the share of adjudications two reviewers make the same way, and reported ambiguity is the share of cases a reviewer flags as unclear. Mean absolute reward error is measured against adjudicated expert scores.

Reward models now get their own accuracy benchmarks: RewardBench exists because reward models were being deployed with no published accuracy at all. Environments deserve the same treatment. In the reference spec, an environment passes these bars before benchmark release or RL training, and the thresholds are fixed in the spec before any code.

Why does the Tuesday Index need published criteria?

Surge AI launched the Tuesday Index on August 18, 2026, a composite score (one index number aggregating many task-level scores) for AI at work, built from eight Surge benchmarks. On August 20, Surge reported Alibaba’s Qwen 3.8 Max 8.6 points above Qwen 3.7 Max. When one score moves 8.6 points in a release, its grading definition matters as much as the value. Surge lists the eight component benchmarks; without per-benchmark weights and rubrics, an 8.6-point move cannot be attributed to the model or to the criteria.

The environment market faces the same problem. A verifier that returns a score without publishing its calibration thresholds publishes no calibration and cannot be audited.

“Introducing Tuesday: A Frontier Index for AI at Work”

The index this post is about, as Surge published it. Every bar is one composite score, which is the problem: the definition matters as much as the value.

What this means

In our reading, Mercor’s 27% gross margin (2025) is what calibrated ground truth costs to produce. The quality bar is separation of duties, published thresholds, and human approval of truth, with hard gates that zero reward on any violation.

FAQ

What is a hard gate?

A hard gate is the integrity gate held to precision and recall of exactly 1.00. It rejects fabricated evidence, prohibited actions, and private-state access before any reward is scored. There is no tolerance for a gate that blocks valid submissions or passes invalid ones.

Why must models never approve truth?

A model that approves truth creates a self-referential loop: the model defines what correct work looks like, is evaluated against that definition, and scores well by construction. Human experts, authors and reviewers, define truth. Models draft, generate variants, and red-team. The separation prevents the model from grading itself.

What is the Tuesday Index?

The Tuesday Index is a composite score for AI at work, launched by Surge AI on August 18, 2026. Qwen 3.8 Max moved 8.6 points in one release on August 20, 2026. Composite indices concentrate trust in whoever defines the scoring criteria, the same trust problem the environment market faces with verifier calibration.