The lab

The target and the ground truth

A benchmark is only as good as its answer key. Ours is a private, multi-service application whose every vulnerability we re-verified in source code — because the documentation that shipped with it was wrong.

Why not a public CTF

Most AI-security benchmarks reuse public CTF challenges or historical CVEs. That creates a contamination problem: today's frontier models were very likely trained on those exact write-ups. A model that “solves” a well-known challenge may be recalling a published solution, not hunting for the bug.

We score against a target that has never been public and appears in no training set. The number therefore reflects hunting ability, not memory. This is the single question every serious reader asks first — and a private target is the cleanest answer to it.

What's inside

Four services on one host: a Node/Express API, a Python cloud panel, a legacy PHP box, and a public web portal front-end. The three backend services are seeded with 25 planted vulnerabilities spanning the OWASP spectrum — SQL injection, remote code execution, server-side template injection, IDOR, authentication bypass, unrestricted file upload, secrets exposure and account takeover.

Multiple services is a deliberate choice. A real engagement is never one bug on one endpoint; it's deciding which of several systems to attack and how deep to go. Because each planted vulnerability is scored individually, one target behaves like 25 graded tasks, not a single pass/fail.

The ground truth is built from code, not docs

The target ships with a white-box list of its own vulnerabilities. We did not trust it. We audited all four services at the source level, and the shipped list was wrong in ways that would have silently broken the benchmark:

• One documented “XXE” vulnerability does not exist — the code uses an XML parser that cannot resolve external entities, so the classic file:///etc/passwdexploit is impossible. Scoring against a bug that can't be found would penalise every honest hunter and reward the ones that hallucinate it.
• Endpoints were wrong. A login the docs place at /login actually lives at /. A one-character mismatch means a perfectly good finding is recorded as a miss.
• Several exploit mechanisms were mis-described— an injection that “bypasses auth” actually needs a different technique; a data leak returns different fields than claimed.

Siblings and decoys

Two of the services expose the same path — a login endpoint on each. A finding therefore has to be attributed to the right service, or a model gets credit for a vulnerability it never touched. Our scorer disambiguates by service and port, so a report is matched to the exact planted vuln or to none.

We also planted decoys: a backup file full of stale, fake credentials, and dead code that looks exploitable but is never served. A model that reports a trap as a real finding is penalised on precision. Finding real bugs is the point; inventing them is the failure mode we measure.

Frozen, versioned, private

The target is frozen and versioned. Every leaderboard row records the target and engine version it ran against, so scores stay comparable as we make changes. Models never see the target before evaluation.

We keep the exact target and its answer key private — but we publish everything else: the methodology, the vulnerability taxonomy, the scorer, and redacted per-run results. You can audit exactly how we score without being able to tune a model to what we score.

What we don't claim

One target is not a population. A model that's strong here could be weak elsewhere, and any target ages as frameworks move on. We treat this honestly: results are reported per vulnerability and per service so the picture reads as many data points rather than one verdict, and the roadmap is more targets. We say this plainly, because a benchmark that hides its limits is exactly the thing we're trying not to build.

Next → How the engine works