Terminal-Bench A
AI benchmark · Coding
- Tier
- A why
- Snapshot
What is Terminal-Bench?
Terminal-Bench measures how well AI agents complete hard, realistic tasks in computer terminal / command-line environments, verified by tests.
Coding mostly
“a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows”
arxiv.org/abs/2601.11868 ↗
Paper: arxiv.org/abs/2601.11868 ↗
Lineage
Benchmarks this one is built on and built from: versions, subsets, extensions and the indexes that combine them.
Derived benchmarks
Long-Horizon Terminal-Bench extension
“Long-Horizon Terminal-Bench measures how well LLM agents sustain progress over hundreds of steps on 46 containerized terminal tasks across nine categories, graded with hidden verifiers that award partial credit.”
SEAGym extension
“SEAGym is an evaluation environment that measures how self-evolving LLM agents' harness updates affect training, held-out ID/OOD transfer, replay, and cost, instantiated on Terminal-Bench 2.0 and HLE.”
TB3 results later version
“Terminal-Bench 3.0 (TB3) measures how well AI agents complete hard coding tasks in terminal environments, scored by pass@1 over a 74-task snapshot.”
Terminal-Bench 2.0 later version
Terminal-Bench 2.0 (ICLR 2026; multi-institution later version
Terminal-Bench 2.1 later version
Terminal-Bench 2.1 Extended later version
Terminal-Bench 3.0 later version
Terminal-Bench 4.0 later version
Terminal-Bench-Science extension
Terminal-Bench-Science 0.1 extension
WolfBench extension
“WolfBench measures how consistently AI models and coding-agent combinations solve Terminal-Bench 2.0 tasks, reporting five metrics: solid, worst, average, best, and ceiling.”
Included in
“The Artificial Analysis Coding Index scores models on coding as a weighted average of coding benchmarks such as Terminal-Bench v2.1 and SciCode.”
Artificial Analysis Coding Agent Index
“An Artificial Analysis aggregate index scoring coding agents on end-to-end software engineering tasks, combining DeepSWE, Terminal-Bench and SWE-Atlas-QnA.”
“Hermes Index averages model scores across four agentic benchmarks (Hermes Bench, Terminal-Bench 4.0, Terminal-Bench-Science, SkillsBench) run in the Hermes Agent harness, reporting mean score and cost per task.”
Tier A
Widely used: a lab reports it, or 50 or more models are scored on it.
How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.
- 8 models scored
- listed with Snorkel AI
- in curated RL directories (3 pts)
- its family includes Terminal-Bench 2.1 (tier S)
On X · well regarded, moderate buzz, from 11 posts about Terminal-Bench in the last month
Widely used as the reference score for terminal agents and harnesses, with labs and startups citing results, though some note cost and reproducibility caveats.
How we got to tier A
- Adoption: 8 models scored; listed with Snorkel AI. Tier C.
- On X: well regarded, moderate buzz. “Widely used as the reference score for terminal agents and harnesses, with labs and startups citing results, though some note cost and reproducibility caveats.” No move.
- In curated RL directories: 3 points, a floor of B. Raised to B.
- Root of a benchmark family that includes Terminal-Bench 2.1 (tier S): raised to A, one step below.
- Confirmed: a line from its own site or paper places it.
A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.
Training methods
Methods Terminal-Bench uses, offers or builds for, with public evidence: its own website says so, or two independent sites do.
- RLHF Reinforcement Learning from Human Feedback On its own website
“Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
snorkel.ai“…es. Our helper model rated them. We kept the highest-rated response as “accepted” and marked the lowest-rated response as “rejected,” and used these to align the model. We performed self-generated responses to mimic the reinforcement learning with human feedback (RLHF) pipeline, and at the same time, not rely on outputs of larger models like other approaches.”
snorkel.ai - RLVR Reinforcement Learning with Verifiable Rewards On its own website
“Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes”
snorkel.ai - DPO Direct Preference Optimization On its own website
“Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
snorkel.ai“…It beat the next-best competitor by more than seven points and maintained its rank for longer than a month. Recently, I talked with Matt Casey, data science content lead at Snorkel AI, about how we built the model using direct preference optimization, how it changed the AlpacaEval leaderboard, and how it will help us build better, more useful large language models for Snorkel’s customers.”
snorkel.ai
Where Terminal-Bench sits
Its domain on the map, live. Hover a logo for its name; click to open it. Open it on the full map
Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-11-05 snapshot.