An Anthropic researcher resigned over self-improving AI: human grading is the one loop stage that does not scale

Human grading is the one stage of the AI training loop that does not scale with compute, and the race Jacob Coxon resigned over on September 8, 2026 is a race to automate the rest of it. The race is visible in rlresearch.ai model score tracking: after a September 9 refresh that changed 172 model scores, 0.25 points separate the top two models, 84.36 against 84.11. Coxon quit Anthropic and wrote that his employer and OpenAI, where he says he spent a combined three years, are racing to build self-improving superintelligence and taking a reckless gamble with human lives; wire services picked it up within hours. Today’s models improve through a simple loop: give the model real work with a checkable answer, score the attempt, adjust the model toward what scored well, repeat millions of times. Self-improvement means the model starts running parts of its own loop. One stage still needs people: deciding what counts as a correct answer, and it does not get cheaper as compute grows.

Key Takeaways

• The warning is about a loop, not a moment: models train on scored tasks, and stronger models increasingly help build the next round of tasks and scoring tools.
• The loop already works at the largest scale. A vendor published results on September 1 showing a 397B-parameter model improving 70 percent relative on expert tasks through this kind of training.
• People still write the answer key and check the graders; that stage is both the bottleneck and the point where oversight can happen.

“racing straight to self-improving superintelligence and gambling with our lives”

What is the loop, in plain terms?

Think of training an AI agent like coaching with an exam that grades itself:

1. Set the exam. Build an environment: a realistic task with a hidden correct answer, like a bug to fix or a ledger to reconcile.
2. Let the model try. The model attempts the task, sometimes for hours, using tools like a real worker would.
3. Score the attempt. A verifier, which is a program that checks the work, returns a pass or a score.
4. Update the model. Its weights are adjusted toward whatever scored well. Go to step 1.

Researchers call this reinforcement learning with verifiable rewards, documented as a recipe by Lambert et al. (2024). The self-improving part arrives when the model’s own output feeds the loop: stronger models draft new exam tasks, probe the graders for loopholes, and grade work where no simple checker exists. Each of those substitutions removes a person, and speeding the loop up is why they are made. The fear Coxon describes, per coverage of his full thread, is that the loop speeds up faster than anyone can audit what the graders are rewarding.

Is the loop working today?

Yes, and in public. On September 1, Mercor published a training guide showing a 397B-parameter model trained with this loop on 1,928 expert-written work tasks. Its first-try success rate on a held-out exam rose 70 percent relative to where it started. No web-crawled data was involved: the capability came from purchased, expert-built tasks. There is now a cataloged industry of benchmarks and RL environments selling those tasks, with vendor margins our economics analysis has priced.

The race Coxon describes also shows up in our score tracking. The two leading models are separated by roughly the size of the drift a single routine refresh produces, as our leaderboard-stability analysis measured in late August. No lab can pause and keep its lead, which is his stated explanation for why his employer kept racing: stop, and someone less careful pulls ahead.

Where do humans still sit in the loop?

At one stage: deciding what counts as correct. Our research on who defines ground truth found that in environments labs trust, models may draft tasks and stress-test graders, but people write the answer key and check the graders against expert judgment. The checking cannot be loosened, because agents exploit any grader with a loophole. Unlike compute, expert grading gets harder as models get stronger, because the loopholes get subtler.

That is why the resignation’s two policy requests (coordination between US labs, and possibly a temporary halt on capability gains) point at the same place our data does. The answer key is the one place oversight can attach, because whoever calibrates the graders decides what the loop optimizes. Today that calibration is done by vendors under commercial contracts rather than under any regulation.

What this means

Self-improvement runs on a supply chain with public prices, public results, and one stage that still requires human experts. In our reading, the resignation is evidence that demand for that human stage will persist, and it shows where oversight can attach: the grading.

FAQ

Who is Jacob Coxon and what did he claim?

Jacob Coxon is a researcher who describes three years of pretraining work across OpenAI and Anthropic. In his September 8 resignation post he argued both labs are racing toward self-improving systems without adequate safeguards, and that many insiders privately consider the risk existential. Those claims about internal beliefs are his; neither company had publicly responded as of September 9, 2026.

What is an RL environment?

An RL environment is a packaged, repeatable work task with a hidden correct answer and a grader. It is the exam in the loop: the model practices on it, the grader scores it, and the score trains the model. Building good ones is expert labor, which is why a vendor market exists.

Do models train themselves today?

Models do not train themselves end to end today. They already write task drafts and test graders for loopholes, but in trusted setups a person still approves what counts as correct. The trend is toward automating more stages, which is why the one human gate carries so much weight.