The engine

How the engine works

architectureengine 98c06f3+bp215 min read

The engine is not a scanner running a checklist. It is an autonomous agent that does its own reconnaissance and exploitation inside a locked, disposable sandbox. This page is the detailed version: what actually happens on a run, why the sandbox is built the way it is, and what makes the benchmark a fair measure of the model rather than of our plumbing.

A run, end to end

A run is one complete penetration test. The engine orchestrates the phases — it does not script the pentest. In order:

Provision. A fresh, single-use sandbox is created (container + private network + firewall sidecar). No state is carried over from any previous run.
Brief. The agent receives the target and the scope (rules of engagement) — and nothing else on a black-box lab: no hints, no vulnerability list. On an authenticated (grey-box) lab it also receives a low-privilege foothold (see below).
Recon. The agent discovers the reachable services and maps the surface itself — ports, endpoints, parameters, technologies. We provide the phase; the agent provides the work.
Hunt. The open-ended loop: the agent runs its own tooling, chases leads across many turns, forms and tests hypotheses, and tries to prove each candidate vulnerability.
Collect. Only findings the agent can demonstrate are written up and pulled from the sandbox. A claim with no working proof never becomes a finding.
Teardown.The entire sandbox is destroyed. Telemetry (tokens, cost, wall-clock, the agent's terminal turn) is recorded on the way out.

The results cross the boundary, the methods don't. Commands, payloads, prompts and the agent's reasoning stay inside the sandbox — only proven findings are published.

The sandbox is the security boundary

The agent runs real exploits — real remote code execution, real injection — so containment is not optional; it is the whole point. Every run gets a fresh container with:

All Linux capabilities dropped except NET_RAW — the single capability a SYN scanner needs. In particular the container holds no NET_ADMIN, so it cannot alter its own firewall from the inside.
A read-only root filesystem, a non-root user, and no-new-privileges — a popped shell inside the sandbox cannot escalate to tamper with the host or the controls.
A deny-by-default network.The only hosts the agent can reach are the in-scope target, the model's own API endpoint, and DNS. A firewall sidecar enforces egress at the packet level; because the agent container has no NET_ADMIN, those rules are immutable from within. Private (RFC-1918) ranges are dropped by default — the in-scope target is the sole allowed host — and cloud-metadata (169.254.169.254) is always blocked, so an SSRF found in the lab can only reach what the lab intends — never our infrastructure.
Ephemerality. Container, network, and volumes are single-use and torn down at the end of the run. Nothing the agent writes survives.

The one host that is notthe target — the model's API endpoint — is the only way data leaves the box. That is an accepted, disclosed part of the design: to be driven by a hosted model, the target's HTTP responses necessarily pass through that model's provider. Self-hosted models close even that gap.

Black-box vs grey-box entry

Labs come in two tiers, matching how real engagements are scoped. On a black-box (surface) lab the agent gets only the URL and must find everything unauthenticated. On a grey-box (authenticated) lab — the exploitation lab is one — the client hands over a genuine low-privilegeaccount: credentials, the login route, and the scope. That foothold is staged into the agent's working notes at the start of the run, exactly as a client would email a tester a viewer account. Higher-privilege roles (operator, admin) exist but their passwords are never provided — reaching them is the point of the engagement, and only exploitation gets you there, never logging in.

Recon, then the hunt

Recon is a mapping phase: the agent enumerates the reachable services, routes, parameters and technologies and builds its own picture of the attack surface. Then the hunt begins, and it is entirely the agent's: it runs standard offensive tooling and its own scripts inside the sandbox, probes for the vulnerability classes it suspects, follows one finding into the next, and iterates over many turns. Nothing about which bug to look for, or how, comes from us — the engine only holds the ring. This is the difference from a scanner: a scanner emits every template that matches; the agent reasons about this target and pursues a line of attack.

How a candidate becomes a finding

A model doesn't get credit for suspecting a bug — it gets credit for proving one. To count, the agent must demonstrate the issue and write a structured finding: the vulnerability type, where it lives, and the proof it used to establish it. The collector that pulls findings out of the sandbox is deliberately tolerant of how the model formats its output (fenced JSON, a bare array, concatenated objects) — a genuinely-hunted finding must never be lost to a formatting quirk, because on the benchmark there is no reviewer downstream to recover it. What it will not do is invent proof: no demonstration, no finding.

This is our answer to “self-graded homework.”An eval where success is one model's unverified opinion is worthless. In the benchmark, a finding counts only if it matches a vulnerability we verified in the target's source code. The verdict is mechanical, not editorial.

The benchmark measures the hunter alone

For the leaderboard we deliberately run the engine hunt-only: any post-hunt filtering is switched off, and the engine's own deterministic hygiene lane (TLS/header auto-findings) is neutralised, so the score reflects the model's ownfindings and nothing else. This isolates the single thing a model swap actually changes — the model's ability to find and prove vulnerabilities — instead of measuring our filtering.

Three runs per lab, averaged

A single pass is a noisy sample — the same model on the same target can find more on one attempt and less on the next. So a benchmark result is not one run but the mean of three runs (R1 · R2 · R3), each a fresh, self-contained engagement — one run gets lucky or unlucky, three average that out.

Each run is a fresh sandbox, a fresh agent, a fresh database record. The model is dropped on the target cold and works the whole engagement — recon, hunt, exploit — start to finish, and then the box is destroyed.
The three runs are independent: no overview, no notes, nothing is carried from one into the next. Run 2 doesn't know what run 1 found; it re-hunts the surface from scratch. We average across them precisely because a lone run isn't reliable — the mean, reported with a floor, is the honest summary of how a model does on this target.
A run that ends without the agent ever concluding is a measurement artifact — flagged degraded, discarded, and re-run until there are three valid ones. Nothing is shared between models either: every model faces the target fresh.

Same engine, swappable brain

Every model on the leaderboard drives the exact same engine — the same agent driver, the same sandbox, the same phases — pinned to one version and stamped on every result. We are not building a bespoke agent around each model; we hold the harness constant and swap only the brain. That is the only way a ranking of models means anything: if the scaffold changed per model, you would be ranking our engineering, not their hunting.

One clarification, because it looks like a confound and isn't: the provider label on a row (e.g. deepinfra, opencode.ai/zen) is only where the model is served — the inference endpoint. The agent driver and the sandbox are byte-for-byte identical across all of them; only the model behind the API changes. And when we improve the engine, the version string changes and we re-run — old and new scores never mix in the same column.

What leaves the sandbox, and what doesn't

The public site shows what each model found— the vulnerability class and where it lives, per run, on the model's page — and, for exploitation, how far up each chain it got. It never shows the agent's commands, payloads, prompts, or reasoning: those stay inside the sandbox. The results cross the boundary; the methods do not. This is a benchmark, not a how-to — and we keep the labs and their answer keys private.

Next → How we score models, out of 1000