Canary Tools C

AI benchmark

Tier
C why
Snapshot

What is Canary Tools?

Canary Tools measures how LLM agents reason about tool selection by planting diagnostic decoy tools in an MCP tool set and scoring models' susceptibility across six weakness types.

  • General capability mostly

    “We introduce canary tools: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness.”
    arxiv.org/abs/2608.04719v1 ↗

Paper: arxiv.org/abs/2608.04719v1 ↗

Tier C

C

Niche: a handful of models scored, or built by an RL vendor.

How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.

  • 7 models scored
  • in curated RL directories (1 pt)
How we got to tier C
  1. Adoption: 7 models scored. Tier C.
  2. Not yet read on X.
  3. In curated RL directories: 1 point, a floor of none.
  4. Confirmed: a line from its own site or paper places it.

A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.

Where Canary Tools sits

Its domain on the map, live. Hover a logo for its name; click to open it. Open it on the full map

The galaxy could not load. See every vendor on the board.

Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-10-19 snapshot.