Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing C

Name: Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing
Creator: Gray Swan AI

AI benchmark

Publisher: Gray Swan AI
On X: @GraySwanAI ↗
Tier: C why
Snapshot: October 11, 2026

Open benchmark ↗grayswan.ai/research

Full listing name: Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing / ARTEMIS

Latest from Gray Swan AI on X

@GraySwanAI ↗

Ever convinced an AI to do something it wasn’t supposed to? Test your creativity in 6 scenarios and get a chance to win $100 every time you trick the AI. Free, no coding skills needed, just typing. Get started at
— GraySwanAI (@GraySwanAI) Oct 10, 2026
You built an AI. Your old boss stole it, raised a billion dollars, and launches it in 7 days. It runs his whole life now. His email, his travel, his vault. He thinks he built it. He didn't. You did. You know how it breaks... Breaking Ties is live.
— GraySwanAI (@GraySwanAI) Oct 6, 2026
A car dealership has an AI salesperson. It's been told: never sell below sticker price. Your job is to talk it into selling you a sportscar for $1. Pull it off and you've earned a raffle ticket toward $100. There are four more like it. No code, just typing.
— GraySwanAI (@GraySwanAI) Sep 29, 2026

What is Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing?

Evaluates AI agents against human cybersecurity professionals on penetration testing of a live university network of ~8,000 hosts, scoring valid vulnerabilities found and submission quality.

Security mostly
“We present the first comprehensive evaluation of AI agents against human cybersecurity professionals in a live enterprise environment.”
arxiv.org/abs/2512.09882 ↗

Who built Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing

Led by Stanford University.

Paper: arxiv.org/abs/2512.09882 ↗

Tier C

Niche: a handful of models scored, or built by an RL vendor.

How much stands behind a tier: usage, who backs it, a confirmed line from its own site or paper, and a clear read on X. High with three of the four, medium with two, low otherwise.

built by Gray Swan AI
no proper name (a paper title or description)
in curated RL directories (2 pts)

How we got to tier C

Adoption: built by Gray Swan AI; no proper name (a paper title or description). Tier D.
Not yet read on X.
In curated RL directories: 2 points, a floor of C. Raised to C.
Confirmed: a line from its own site or paper places it.

A clear view on X moves a tier one step up or down; most entries don't move. How tiers work.

Where Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing sits

Its realm in the universe, live. Hover a star for its name; click to open it. Open it in the universe

The galaxy could not load. See every vendor on the board.

Next: browse all benchmarks. Entry from the RL Research daily scrape of public sources, 2026-10-11 snapshot.