Autonomous pentest benchmark
We run frontier and open LLMs as autonomous penetration testers against live, private targets and score every result against a code-verified answer key. We measure two different skills: coverage — how much of a target they find — and exploitation — how far they can chain a foothold into a real objective.
Leaderboard
Two labs, two skills. Halcyon measures coverage — a broad, multi-service app seeded with a wide surface of independent vulnerabilities, where the question is how many of them a model finds. Meridian measures exploitation — fewer findings, but each is a step in a chain that has to be reasoned through to an objective (get a shell, become admin). Each lab is scored out of 500; Overall is the two added together, out of 1000. Each score is the mean of 3 independent runs — one autonomous single-shot session each, no memory between them; we average three because a single run is noisy.
| # | Model | Overall/1000 | CoverageHalcyon · /500 | ExploitationMeridian · /500 |
|---|---|---|---|---|
| 01 | deepseek-v4-flash deepinfra | 224 /1000 | 75 | 149 |
| 02 | minimax-m3 opencode-go | 184 /1000 | 49 | 135 |
| 03 | hy3 litellm | 183 /1000 | 69 | 114 |
| 04 | minimax-m2.7 opencode-go | 134 /1000 | 65 | 69 |
| 05 | gpt-5.6-luna litellm | 83 /1000 | 40 | 43 |
Each score is the mean of 3 independent runs — one autonomous session each, no memory carried between them. We average three because a single run is noisy; the per-model page shows every run.
Methodology
A run is one full engagement: the model gets a target and a scope, does its own recon, then hunts and exploits — a single autonomous pass, no memory of any other run. We run each model three independent times per lab and score the mean, because one run is noisy and a stable number needs a few. Coverage (Halcyon) scores breadth — the share of planted vulnerabilities a model finds; Exploitation (Meridian) scores how far it chains a foothold toward an objective, verified by secret markers it can only recover by completing each step. Each lab is out of 500; the two add to an Overall /1000.
Each lab scores out of 500 — the mean of 3 independent runs. A model runs the target three times, each a fresh single-shot session with no memory of the others; we average the three because a single run is noisy. A model's Overall /1000 is its two lab scores added together — cost and time are reported alongside, never scored.
The engine
Every scan runs in a locked, disposable sandbox: all Linux capabilities dropped except NET_RAW(so its scanner works), under a deny-by-default firewall whose only scan-reachable host is the in-scope target — plus the model's own API endpoint and DNS. Inside, an autonomous agent maps the surface, then hunts: it runs its own tools, chases leads, and writes up each vulnerability it can prove.
The results cross the boundary, the methods don't. Commands, payloads, prompts and the agent's reasoning stay inside the sandbox — only proven findings are published.
The benchmark scores the Hunter alone — no reviewer, no gate. That isolates the model's raw hunting ability, which is what a model swap actually changes. The reviewer is what the shipped product adds.
In the shipped product a second, skeptical agent re-tests every claim before anything reaches a client. The benchmark scores the raw hunter alone — so the number reflects the model, not our filtering.
The labs
Halcyon and Meridian are live, deliberately-vulnerable applications — one seeded with independent vulnerabilities across the OWASP spectrum (coverage), one a multi-service app whose weaknesses chain toward objectives (exploitation). Each is deployed on its own isolated host.
We keep the exact targets and their answer keys private, so no model can be tuned to them — what we publish is how we score and what we find. And we hold ourselves to it: we audited our own answer keys at the source-code level and proved every lab is fully solvable. A benchmark is only as honest as its ground truth.
What we're learning
Too early to say anything real from a handful of models on two labs — any “finding” here would just be noise. We're holding this section until enough models are benchmarked to draw an honest conclusion. Check back as the board fills out.
Limitations
A benchmark that hides its limits is the thing we're trying not to build. So, plainly:
A model strong here may be weak on a different stack. Read the scores as capability on controlled targets — not production readiness. More labs are on the roadmap.
Agents that ace labs still drop sharply on real, unstructured engagements. This measures hunting and exploitation on known-vulnerable targets, nothing more.
Results move run to run; we average the valid runs, report the reliable floor, discard and re-run degraded ones, and mark a result provisional until it has enough valid runs. Small gaps between models are likely noise, not signal.
Full limitations & reproducibility →·How we keep the harness honest →
We add models and labs and publish findings as runs complete. Get an email when the leaderboard moves — no spam.