IDE-Bench B

AI benchmark · Coding

Domains
Coding Computer Use Enterprise Workflows
Tier
B why
Snapshot

Open benchmark ↗afterquery.com/research

What is IDE-Bench?

IDE-Bench evaluates LLMs acting as IDE agents on 80 real-world software engineering tasks across eight never-published repositories using an IDE-native tool interface.

  • Coding mostly

    “IDE-Bench is a comprehensive framework for evaluating AI IDE agents on real-world software engineering tasks through an IDE-native tool interface.”
    arxiv.org/abs/2601.20886 ↗

Paper: arxiv.org/abs/2601.20886 ↗

Tier B

B

Established: 15 or more models scored, or curated RL directories rate it.

How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.

  • 15 models scored
  • listed with AfterQuery
  • in curated RL directories (4 pts)
How we got to tier B
  1. Adoption: 15 models scored; listed with AfterQuery. Tier B.
  2. On X: unclear, none buzz No move.
  3. In curated RL directories: 4 points, a floor of B.
  4. Confirmed: a line from its own site or paper places it.

A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.

Training methods

Methods IDE-Bench uses, offers or builds for, with public evidence: its own website says so, or two independent sites do.

  • RLVR Reinforcement Learning with Verifiable Rewards On its own website

    “Launching the RLVR stage on Tinker:”

    afterquery.com
  • GRPO Group Relative Policy Optimization On its own website

    “Not all tasks are equally useful for RL. Tasks the model solves consistently or fails every time provide no learning signal. We select tasks where the SFT checkpoint has a 10–80% solve rate. These are the tasks where GRPO can compute meaningful advantages: some attempts succeed, some fail, and the model can learn from the difference.”

    afterquery.com
  • SFT Supervised Fine-Tuning On its own website

    “Our training pipeline had two sequential stages. The first stage teaches the model what good terminal-agent behavior looks like via SFT on gold trajectories. The second stage uses RL to push the model toward solving harder problems it only partially gets right after SFT.”

    afterquery.com

Where IDE-Bench sits

Its domain on the map, live. Hover a logo for its name; click to open it. Open it on the full map

The galaxy could not load. See every vendor on the board.

More Coding AI benchmarks

Coding RL environments

Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-11-05 snapshot.