The harness

Keeping the benchmark honest

integrityscorer v18 min read

The easiest way to build a dishonest benchmark is to trust your own harness. A model's score has to reflect its hunting — never a quirk of how we run it or count it. So we audit the harness the same way we audit the target: adversarially, from two directions, and we publish what we find. This post is the bugs we found in our own scoring, and how we fixed them.

We ran two independent audits — one over the raw run telemetry, one over the scoring code — and had them check each other. Where they disagreed, we dug until they agreed. Everything below survived that.

A truncated run is not a bad result

Every run is one autonomous attempt. Occasionally an attempt ends early for reasons that have nothing to do with the model's skill — an inference endpoint hiccups, a stream drops, and the agent's loop stops before it ever reaches a natural conclusion. The run “completes,” but it completed at a third of the depth of a healthy run, having barely started hunting.

If you score that as an ordinary low result, you punish the model for your infrastructure. We caught exactly this in the wild: one run of a model found nine vulnerabilities; the very next run of the same model, cut short, found two — and it was quietly dragging that model's reliability score down by two-thirds.

So a run that ends without the agent ever concluding is flagged as degraded, thrown out, and re-run— the harness keeps going until it has the three valid runs a result is supposed to have. A degraded attempt never enters a score and is never published: a truncated sample is not a measurement, so it isn't one of the three runs. We record that it happened — how many attempts a run took — but its findings stay private.

Reliability needs at least two valid runs

Because a model is non-deterministic, one run is noise — the same model can vary sharply from run to run. So we run several and report what the model does reliably: the floor and ceiling across runs, and how many vulns it found in every run. This is a reported reliability band, nota scored pillar — an earlier version of this benchmark weighted “consistency” directly; we removed it, because a per-run average already prices variance in (a bug found once in three runs already earns a third of the credit).

It is also meaningless from a single run: with one run, “found every time” and “found once” are the same sentence. So the reliability band requires at least two valid (non-degraded) runs. Below that threshold a result is marked provisional— we report its recall and precision but not a reliability it hasn't earned, and we say so plainly on the leaderboard. A provisional row is a promise to finish measuring.

We don't penalise a model for its handwriting

The hunter writes up each vulnerability it proves. Different models write differently — one emits a clean object per finding, another wraps everything in a code fence, another hands back a single list of all its findings at once. None of that is a statement about hunting ability; it's handwriting.

An earlier version of our collector only understood one of those shapes and silently dropped the rest. That is a benchmark bug, not a model signal — and with no reviewer in the loop on a raw-hunter run, nothing would have recovered the dropped findings. It now reads every common shape identically, for every model. A finding is only ever lost when the model genuinely didn't prove it — never because of how it chose to write it down.

Cost is reported honestly, or not at all

Cost sits next to the score, never inside it — but reporting it carries its own small honesty trap: when a provider doesn't return usage for a run, the naive number is $0.00, which reads as “free.” It wasn't free; it was unmeasured. We show unmeasured cost as “—”, and we average cost only over the runs we could actually measure, so a metering gap can never quietly halve a model's reported price.

Pinned and versioned

None of this would matter if the harness drifted between models. It doesn't — every model drives the same engine, pinned to one version (see how the engine works). What matters for honesty is that the version is stamped on every row: the leaderboard only ever compares models measured on the same version, and when we change how scoring works, that stamp changes with it — old and new numbers never mix in a column.

Why publish our own bugs?Because “trust us” is exactly what a serious benchmark can't say. The target is private so it can't be gamed — which means the one thing we can open is the harness. A benchmark is only as honest as the machinery that scores it, and the only way to earn that word is to show the machinery, including the parts we had to fix.

← Back to the blog