94 AI benchmarks in one week: 54 came from papers, and 29 of 40 leaderboard additions were versions of existing benchmarks

94 AI benchmarks entered our catalog between October 15 and 21, 2026. Most of the ones that came with third-party scores were new cuts of benchmarks that already existed.

• From research papers (54): 53 of them posted to arXiv in July and August, each scored by its own authors on a median of 9 models.
• From evaluator leaderboards (40): run by evaluators that test models themselves, including Kaggle, Artificial Analysis, Vals AI, Google DeepMind, LlamaIndex and Nous Research. By our classification, only 11 of the 40 were distinct new benchmarks. The other 29 were new versions, archived old versions, tracks, harness variants or composite indexes, such as FACTS Search V2 listed next to an archived V1, and four Kaggle tracks of SciCode.

The same seven days added no RL environments. The catalog holds 3,611 benchmarks against 113 environments, 32 to 1, close to the 31 to 1 our July census found.

Key Takeaways

• Papers: 54 new benchmarks, 34 of them scored on 10 or fewer models chosen by their authors.
• Leaderboards: 11 of 40 additions were distinct benchmarks; 29 re-cut existing ones.
• Environments: none added in the week, while benchmarks grew by 94.

Where did the week’s 94 benchmarks come from?

The two routes deliver different things. A paper benchmark arrives with a new task set and the authors’ choice of models; the agentic ones from this batch include:

• ActBench: behavioural safety of cowork agents, 12 models.
• DDBench: agentic code repair in distributed systems, 10 models.
• TREK: complex trip planning for tool-using agents, 14 models.
• ToolFailBench: diagnosing why tool calls fail, 14 models.
• Agentic Commerce World: an auditable environment for agent-run commerce, 9 models.
• FinReportBench: institution-grade financial reports, 7 models.

An evaluator benchmark arrives with scores on current models, run by a party that did not train them, but usually on a task set that existed before.

What were the 40 leaderboard additions?

Sorted by our reading of each entry:

• Distinct new benchmarks (11): Vals AI’s Robo-bench and ClaimBench, Nous Research’s Hermes Bench, LlamaIndex’s ParseBench and ExtractBench, Kaggle’s Adversarial Customer Service, Human Behavior Bench and Game Arena, Artificial Analysis’s PersonalityBench, MazeBench and STEMBench.
• Tracks, components and harness variants (16): four SciCode tracks run on Kaggle; five components of the Artificial Analysis Decision Index, including runs of HLE, When2Call and WildGuardTest; BrowseComp+ on a fixed corpus; LOCA-Bench at 256K tokens; ARC-AGI-3 through a provider adapter; provider-reported MultiChallenge runs; Kaggle’s Global MMLU Lite and MGSM Bengali; and ExtractBench Pro.
• Versions and archived versions (9): FACTS Search and FACTS Grounding each as V2 with V1 archived, Vals AI’s Finance Agent v2.1 with revised questions and rubrics, its legal-agent benchmark v1 archived from before a grading fix, Code Migration v1.1, EMB v1.1 and the current 101-question set of DeepMind’s 1H-VideoQA.
• Composite indexes (4): the Artificial Analysis Decision Index, the Vals Science Index, the Hermes Index and the FACTS Benchmark Suite.

Kaggle @kaggle

One of the 11 distinct new benchmarks: Human Behavior Bench, launched by Kaggle with Rehearsals. It asks models to predict how real people behave and choose; among the frontier models in Kaggle's chart, the top five sit within 1.4 points.

Why do versions outnumber new benchmarks?

Because benchmarks wear out and get repaired. SWE-Bench Pro V2, whose paper was posted October 16, repaired or retired 480 of 731 tasks, and its fifth lesson for maintainers is to version the benchmark and release all grades.

The evaluators above already do it. When FACTS and the legal-agent benchmark changed, the old version’s results stayed up as a separate, archived entry. Harbor’s task versioning described the same rerun, regrade and reuse discipline at the start of October; the leaderboards that publish frontier scores now practice it.

Vals AI @ValsAI

“minor version updates to four of its benchmarks”

Vals AI's index moved to v2.2 because four benchmarks under it changed: Finance Agent v2.1, Excel Modeling Benchmark v1.1, Code Migration v1.1 and Harvey's LAB v1.1. Three of those revisions, and the archived version of the fourth, are among the week's 29 re-cuts.

The cost lands on anyone reading a score. FACTS Search now means two protocols, SciCode four tracks, and a model’s ARC-AGI-3 result depends on the harness adapter. A score without its version, track and harness is not comparable to the next one, which is why our catalog keeps them as separate entries.

Why did no RL environments arrive?

Our read: benchmarks are published to be cited, while environments are mostly sold to labs under contract, so few become public, and the ones that do, such as Mercor’s APEX family, tend to publish as benchmarks. From October 15 to 21 the count was 94 benchmarks to no environments. In July we counted 3,029 benchmarks against 99 environments, and 3.5% task coverage of real work; the benchmark side has grown by 582 since, the environment side by 14.

What this means

Benchmark counts overstate how much new measurement exists: in the evaluator additions of October 15 to 21, re-cuts outnumbered new benchmarks 29 to 11, and the paper benchmarks mostly lack scores beyond their authors’ handful of models. Anyone citing a benchmark score should name its version, track and harness, and anyone counting the field should count distinct task sets.

FAQ

How many AI benchmarks are there?

The rlresearch.ai catalog listed 3,611 AI benchmarks on October 21, 2026, from research papers, evaluator leaderboards and vendor sites, against 113 RL environments. Counts depend on whether versions, tracks and harness variants are listed separately; in the week of October 15 to 21, those made up 29 of 40 leaderboard additions.

What is the difference between a benchmark version and a variant?

A version changes the benchmark itself: new or repaired tasks, rubrics or grading, as with FACTS Search V2 or Finance Agent v2.1. A variant runs the same benchmark differently: a separate track, a different context length, a fixed document corpus or another agent harness. Both change the score, so neither is comparable to the original without saying which one it is.

Why are paper benchmarks scored on so few models?

The authors run them, usually once, on the models they can access when they write the paper. In this batch the median was 9 models and 34 of 54 papers reported 10 or fewer. Scores on later models arrive only if an evaluator or a model developer adopts the benchmark.