OctoBench C

AI benchmark · Instruction Following

Domains
Instruction Following
Tier
C why
Snapshot

Full listing name: OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding

What is OctoBench?

Modern coding scaffolds turn LLMs into capable software agents, but their ability to follow scaffold-specified instructions remains under-examined, especially when constraints are heterogeneous and persist across interactions. To fill this gap, we introduce OctoBench, which benchmarks scaffold-aware instruction following in repository-grounded agentic coding. OctoBench includes 34 environments and 217 tasks instantiated under three scaffold types, and is paired with 7,098 objective checklist items. To disentangle solving the task from following the rules, we provide an automated observation-and-scoring toolkit that captures full trajectories and performs fine-grained checks. Experiments on eight representative models reveal a systematic gap between task-solving and scaffold-aware compliance, underscoring the need for training and evaluation that explicitly targets heterogeneous instruction following. We release the benchmark to support reproducible benchmarking and to accelerate the development of more scaffold-aware coding agents.

Tier C

C

Niche: a handful of models scored, or built by an RL vendor.

How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.

  • 8 models scored
  • in curated RL directories (1 pt)
How we got to tier C
  1. Adoption: 8 models scored. Tier C.
  2. Not yet read on X.
  3. In curated RL directories: 1 point, a floor of none.
  4. Not yet confirmed by a line from its own site or paper.

A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.

Not on the map yet: no line from its own site or paper places it in a domain.

More Instruction Following AI benchmarks

Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-10-11 snapshot.