HumanEval D

AI benchmark · Coding

Domains
Coding
Tier
D why
Snapshot

Open benchmark ↗github.com/openai/human-eval

What is HumanEval?

HumanEval measures the functional correctness of model-generated Python code by running completions against unit tests and reporting pass@k.

Lineage

Benchmarks this one is built on and built from: versions, subsets, extensions and the indexes that combine them.

Derived benchmarks

  • EvalPlus extension

    “EvalPlus measures the functional correctness of LLM-generated code using test-augmented benchmarks HumanEval+ and MBPP+, ranked by pass@1.”

  • HumanEval+ extension

  • HumanEval-Mul extension

  • MultiPL-E extension

    “MultiPL-E measures multi-language code generation by translating HumanEval and MBPP unit-test-driven benchmarks into 18 additional programming languages.”

Tier D

D

Long tail: little or no usage data yet.

How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.

  • no usage data
  • in curated RL directories (1 pt)

On X · mixed, moderate buzz, from 12 posts about HumanEval in the last month

Still routinely cited in model releases and quantization tests, but increasingly viewed as saturated and contamination-prone, with people preferring private or harder evals.

@OrcaRouter ↗ · @net_termina ↗ · @ViC305 ↗

How we got to tier D
  1. Adoption: no usage data. Tier D.
  2. On X: mixed, moderate buzz. “Still routinely cited in model releases and quantization tests, but increasingly viewed as saturated and contamination-prone, with people preferring private or harder evals.” No move.
  3. In curated RL directories: 1 point, a floor of none.
  4. Confirmed: a line from its own site or paper places it.

A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.

Where HumanEval sits

Its domain on the map, live. Hover a logo for its name; click to open it. Open it on the full map

The galaxy could not load. See every vendor on the board.

More Coding AI benchmarks

Coding RL environments

Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-11-05 snapshot.