Technical blog
No marketing. Five posts on the methodology, the targets, the engine, the scoring, and the harness we audit to keep it honest — with the diagrams, the honest numbers, and the limits. Everything you'd need to decide whether to trust the leaderboard.
Two skills, measured separately: coverage (how much a model finds) and exploitation (how far it chains a foothold into an objective). How a test runs, and how depth is verified with planted markers.
Why we score against a private, code-verified target instead of public CTFs — and why building the answer key from source code (not the README) is the whole game.
One autonomous agent in a locked sandbox, a skeptical reviewer that re-tests every claim, and a hard rule: nothing is scored unless it can be proven.
Recall, consistency and precision — what each measures, why cost is excluded, how a private answer key defeats contamination, and where this benchmark's limits are.
The scoring bugs we found in our own harness and fixed: excluding truncated scans, refusing to score consistency from one run, measuring cost honestly, and never penalising a model for how it formats its findings.