AI benchmarks cover only 3.5% of real work

Our study joining 202 O*NET occupational tasks, from the US Department of Labor’s occupational database, against 223 candidate benchmarks with an LLM judge (a language model scoring each task-benchmark pair against a rubric) finds definitive coverage of 3.5%. Only 7 of 202 tasks are definitively covered by an existing benchmark. The study cost $13.79 across 441 judge calls, and a robustness check, a recall audit of 4,935 task-benchmark pairs against every unlinked benchmark, found zero missed matches for $8.82, bringing the combined cost to $22.61. Benchmark success materially overstates workflow competence: existing benchmarks test isolated calculations or document operations rather than the multi-step occupational workflows they claim to represent. Vendor benchmarks show the same gap from the other side: every frontier model scored below 25% strict pass@1 on Surge AI’s HANDBOOK.md (June 25, 2026), and frontier agents completed under 25% of Mercor’s APEX-Agents tasks (January 2026).

Key Takeaways

• Joining 202 O*NET tasks against 223 benchmarks with an LLM judge found only 7 tasks (3.5%) definitively covered; the other 96.5% has no benchmark for environment builders to start from.
• A bounded recall audit (21 tasks, 4,935 task-benchmark pairs) found zero missed matches, which supports reading the gap as coverage rather than search failure. Both studies together cost $22.61.
• Vendor benchmarks corroborate it: frontier agents under 25% on Mercor’s APEX-Agents (January 2026) and every model below 25% strict pass@1 on Surge’s HANDBOOK.md (June 25, 2026).

How was the coverage study conducted?

The study evaluated 7 occupations and 202 tasks from O*NET 30.3 against 223 unique candidate benchmarks across 255 occupation links, with every task receiving its complete candidate pool in a single judge pass. The judge consumed 3.67M input tokens and 646K output tokens.

OpenAI’s GPTs are GPTs established task-level O*NET analysis as the way to measure what LLMs can touch, using human and model raters over the same taxonomy. The study applied the same task-level logic to benchmarks instead of wage exposure: for each occupational task, does any benchmark require performing it? The judge applied a strict rubric: a benchmark counts only when it requires the task’s core action, and sharing a domain does not qualify.

The dominant result is partial coverage. Existing benchmarks often test an isolated calculation, a question-answering skill, a document operation, or a simplified decision rather than the complete occupational workflow. Tasks whose covering verdict rested on a catalog description alone, with no fetched source to verify it, are counted separately and do not count as definitive. Of the 643 covering or partial verdicts retained, 564 used fetched task-level evidence rather than catalog descriptions alone, and 496 contained a quotation verified against the fetched source.

What does the recall audit prove?

A bounded recall audit tested the three highest-value not-yet-covered tasks in each occupation against 235 reputable agentic, multimodal, robotics, and coding benchmarks that no occupation links to. The audit covered 21 tasks and 4,935 task-benchmark pairs in 252 judge calls, costing $8.82.

The result: zero missed covering benchmarks, a 0% task-level recall-failure rate, and 46 partial matches. The partial matches reinforce the central finding. Broad agentic benchmarks exercise a component of the work without requiring the complete workflow:

• finance analysis without examining facilities and producing a client recommendation
• legal research without gathering case evidence or negotiating a settlement
• scientific experimentation without planning the complete validation program

The measurement literature has documented this for years. An Oxford-led review of 445 LLM benchmarks found pervasive construct-validity failures, benchmarks that do not measure what they claim to (Bean et al., 2025), and Raji et al. (2021) warned against reading any narrow benchmark as general capability. Even the covered 3.5% deserves a skeptical read: audits of popular agentic benchmarks show task-setup and reward flaws can distort measured performance by up to 100% relative (Zhu et al., 2025).

Chart: Surge AI, June 25, 2026 · source

What do vendor benchmarks report?

Two vendor results corroborate the coverage gap from different directions. Mercor’s APEX-Agents benchmark, launched in January 2026, reports frontier agents under 25% completion on professional tasks, and only 40% even with 8 tries. The benchmark uses 480 tasks across 33 simulated worlds in investment banking, management consulting, and law (the setup is documented in the APEX-Agents paper).

Surge AI launched HANDBOOK.md on June 25, 2026: 65 tasks across five enterprise domains (finance, medical billing, insurance, logistics, and HR), each governed by a company handbook averaging 43 pages (up to 124), graded deterministically against a per-task rubric of required outputs and prohibited actions. On strict pass@1, which requires every criterion in the rubric to be met, every frontier model scored below 25%. Failure modes include firing employees without authorization, clearing self-submitted expenses, and submitting expired medical records to insurers.

What this means

The 96.5% of tasks without definitive benchmark coverage is the addressable market for environment builders. Every task in that gap is a workflow no existing benchmark tests end to end, and every such workflow is a candidate for a production-grade RL environment with a verifiable reward, a grade computed from the end state rather than from a human opinion.

FAQ

What counts as “definitively covered”?

A task is definitively covered when a benchmark requires performing the task’s core action, with the match verified against the benchmark’s own task material. An isolated calculation, a simplified decision, or a document operation that is one step in the workflow does not count, and neither does a match resting on a catalog description alone.

Why does partial coverage not count?

Partial coverage means a benchmark tests a component of the task without requiring the complete multi-step workflow. A model that passes a partial benchmark has demonstrated competence at one step, not at the occupation-level task the benchmark claims to represent.

What is the combined cost of both studies?

The coverage study cost $13.79 and the recall audit cost $8.82, for a combined $22.61. Publishing study costs alongside findings is part of the methodology: a claim about benchmark coverage should state what it cost to establish.