Senior SWE-bench B
AI benchmark · Coding
- Built by
- Snorkel AI
- On X
- @snorkelai ↗
- Tier
- B why
- Snapshot
What is Senior SWE-bench?
Senior SWE-bench evaluates coding agents on senior-level software engineering work: under-specified feature requests, bugs requiring runtime investigation, and code quality (tasteful solves).
Coding mostly
“Senior SWE-Bench tasks are sourced from PRs in repos spanning libraries to multi-service applications, authored by engineers with hundreds of commits in their respective repos.”
senior-swe-bench.snorkel.ai/ ↗
Who built Senior SWE-bench
Snorkel AI built it with others
“Senior SWE-bench built by Snorkel with Princeton University and UW-Madison”
snorkel.ai/research/ ↗
Lineage
Benchmarks this one is built on and built from: versions, subsets, extensions and the indexes that combine them.
Built on
SWE-bench original
“Senior SWE-bench evaluates coding agents on senior-level software engineering work: under-specified feature requests, bugs requiring runtime investigation, and code quality (tasteful solves).”
Tier B
Established: 15 or more models scored, or curated RL directories rate it.
How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.
- 19 models scored
- built by Snorkel, Snorkel AI
- in curated RL directories (7 pts)
On X · unclear, low buzz, from 3 posts about Senior SWE-bench in the last month
Mostly Snorkel-affiliated people promoting Senior SWE-bench's inclusion in Fireworks AI's Specialized Intelligence Index; little independent discussion or reported results.
How we got to tier B
- Adoption: 19 models scored; built by Snorkel, Snorkel AI. Tier B.
- On X: unclear, low buzz. “Mostly Snorkel-affiliated people promoting Senior SWE-bench's inclusion in Fireworks AI's Specialized Intelligence Index; little independent discussion or reported results.” No move.
- In curated RL directories: 7 points, a floor of B.
- Confirmed: a line from its own site or paper places it.
A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.
Training methods
Methods Senior SWE-bench uses, offers or builds for, with public evidence: its own website says so, or two independent sites do.
- RLHF Reinforcement Learning from Human Feedback On its own website
“Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
snorkel.ai“…es. Our helper model rated them. We kept the highest-rated response as “accepted” and marked the lowest-rated response as “rejected,” and used these to align the model. We performed self-generated responses to mimic the reinforcement learning with human feedback (RLHF) pipeline, and at the same time, not rely on outputs of larger models like other approaches.”
snorkel.ai - RLVR Reinforcement Learning with Verifiable Rewards On its own website
“Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes”
snorkel.ai - DPO Direct Preference Optimization On its own website
“Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
snorkel.ai“…It beat the next-best competitor by more than seven points and maintained its rank for longer than a month. Recently, I talked with Matt Casey, data science content lead at Snorkel AI, about how we built the model using direct preference optimization, how it changed the AlpacaEval leaderboard, and how it will help us build better, more useful large language models for Snorkel’s customers.”
snorkel.ai
Where Senior SWE-bench sits
Its domain on the map, live. Hover a logo for its name; click to open it. Open it on the full map
Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-11-05 snapshot.