Measuring railed discovery: metrics against fixtures, not hope

2026-10-19 · Ishan Uprety · 10 min read

The last two posts built two rails. Authorization as a type decides whether an action can be expressed at all. Scope as gate-owned data decides where that action may land. Both fail closed. Both produce refusals you can count.

That leaves the question every reviewer asks first: does it work? And the honest answer is that "work" means two different things for a safety-railed discovery system. Did it find the real bugs? And did it refuse everything it should have refused? Most of the trouble in evaluating these systems comes from collapsing those two questions into one number.

A blocked out-of-scope action is not a miss. It is the rail doing its job. If your scoreboard counts it against the system, you are training yourself to loosen the rail.

Ground truth only exists where you put it

Precision and recall need labels. You can only compute recall if you know what was there to be found. That single fact settles where evaluation happens: against labelled offline fixtures, which are self-owned, deliberately vulnerable targets where every planted issue is written down before the run starts.

Live third-party targets fail this test on every axis at once:

No labels. You do not know what is there, so recall is undefined. A finding count from a live target says nothing about what was missed.
No repeatability. The target changes under you. Next week's run is against a different system, so you cannot compare configurations.
No authorization for experiments. A bounty program authorizes testing within its rules. It does not authorize using its production systems as your benchmark, running ablations against it, or deliberately turning rails down to see what happens.

So the rule is simple. Evaluation numbers come from fixtures. A number measured on a live third-party target is neither ethical nor scientific, and a thesis about responsible automation cannot rest on one.

What a labelled fixture actually contains

A fixture is more than a vulnerable container. For metrics to mean something, it needs a manifest committed alongside it:

Pinned identity. An image digest, not a tag. "The same fixture" has to mean byte-identical, or comparisons across runs are comparisons across targets.
Positive labels. Each planted issue gets a class, a location (endpoint, parameter, component), and any preconditions such as auth state. These are written down before any run, never reconstructed afterwards from what the tool happened to report.
Negative labels. Known-clean endpoints and decoys that look suspicious but are not vulnerable. Without them you cannot tell precision apart from luck.
Out-of-scope decoys. Hosts and paths the fixture's scope revision explicitly excludes, some of them deliberately tempting, and some with planted issues of their own. These exist to exercise the rails. A system that "finds" a bug on an out-of-scope decoy has not succeeded. It has escaped.
A pre-declared match rule. What counts as a hit: same class and same location, or same class anywhere on the host? Decide before the run. A match rule chosen afterwards will always turn out generous.

Two scoreboards, not one

A railed system makes two kinds of decision, and each gets its own confusion matrix.

Discovery outcomes are about the claims the system reports:

True positive. A reported finding matches an in-scope planted label under the pre-declared rule.
False positive. A reported finding matches nothing, or matches a negative label.
False negative. An in-scope planted label that no reported finding matched.

Rail outcomes are about the actions the system proposed and what the gate did with them:

Correct allow. An in-scope, authorized proposal is minted and reaches egress.
Correct refusal. An out-of-scope or unauthorized proposal is refused at the gate. This is a success. It counts in the rail's favour and never against discovery recall.
Over-refusal. An in-scope proposal is refused, perhaps because a scope rule was too tight or a revision failed to load. This is safe, but it costs coverage, and it should be reported separately so it can be fixed.
Escape. An out-of-scope or unauthorized action reaches egress. This is the only critical failure.

The two matrices answer different questions, so they get different treatment. Discovery metrics are rates you try to improve. The escape count is a gate, and the only acceptable value is zero. One escape invalidates the run's safety claim no matter how good its recall looks. You do not average escapes against wins.

Precision, recall, and what goes in each denominator

The definitions are standard. The decisions about what goes into them are where honesty lives.

Recall = true positives ÷ in-scope planted labels. The denominator holds only in-scope labels. Issues planted on out-of-scope decoys are not "missed" when the gate refuses to touch them. They were never eligible.
Precision = true positives ÷ reported findings. Every reported claim counts, including ones the system marked low-confidence, unless a confidence threshold was fixed before the run. If it was, report the threshold along with the number.
Per-class breakdown. An aggregate recall can hide a class the system never finds. Report per vulnerability class wherever the fixture has enough labels for the number to mean anything, and say so where it does not.

Over-refusals sit between the two scoreboards. When the gate wrongly refuses an in-scope proposal and a planted bug goes unfound as a result, that is a real false negative. It belongs in recall, but it should be attributed to the rail rather than to the proposer. Otherwise you cannot tell "the engine is weak" apart from "the scope encoding is too tight".

False positives have a price tag

Precision and recall are usually treated as symmetric. In vulnerability discovery they are not. A false negative costs a finding you might have had. A false positive costs someone's time, and when reports go to real people, it also costs trust.

Triage cost. Every false claim is minutes or hours of a human verifying nothing. A useful derived metric is reported findings per true finding: how many claims a reviewer has to read to get one real bug.
Signal cost. In disclosure workflows, a stream of invalid reports degrades how seriously the next valid one is taken. That cost compounds.
Operating point, not just a curve. If the system has a confidence threshold, show the precision–recall curve on the fixture, then state the operating point you would actually deploy at and why. Choosing it after looking at the test fixture is tuning on the test set.

For a system whose output goes to humans, I would rather report a lower recall at a precision a reviewer can live with than a headline recall that buries them.

Coverage versus safety is a curve you measure

An earlier post framed the cost of rails as a trade-off curve. Fixtures are what make that curve measurable instead of rhetorical:

Run the same pinned fixture under several rail configurations, from strict scope encoding through tighter action classes to looser ones.
For each configuration, record recall, precision, over-refusal count, and escape count.
A relaxed baseline is useful for measuring what the rails cost. It runs only inside the isolated, self-owned lab, and never as a configuration that could reach anything else.
Plot recall against rail strictness. Escapes must be zero for every configuration you would consider deploying. The question the curve answers is how much recall each step of strictness costs, given zero escapes.

The shape of that curve is the result. A single point, "we got X% recall", says nothing about the trade-off a team adopting the tool is actually making.

Reproducibility: pin everything, replay the decisions

A metric you cannot re-derive is an anecdote. For a railed discovery loop, that means pinning more than the target:

Fixture digest and label-manifest hash, so the target and its answer key are both fixed.
Scope revision and authorization policy version, the same scope_rev carried on every minted handle in the previous post.
Proposer determinism. Fixed seeds where the proposer allows it. Where it is a model with non-deterministic output, record its proposals so they can be replayed.
Deterministic gate replay. The gate should be a pure function of (proposal, scope revision, policy version). Feed the recorded proposals back through it and you must get byte-identical decisions. If you do not, the rail has hidden state, and that is a finding in itself.
Variance, not best-of-N. When the proposer is stochastic, run repeatedly and report the spread. Reporting the best run is a quiet way of reporting a number you will never see again.

Replay separates the two halves of the system cleanly. You can rerun the gate over yesterday's proposals under today's scope encoding and see exactly which decisions changed, without re-touching a target at all.

Ways to lie with these numbers

None of these need bad intent. Each one is the easy path when a deadline is close:

Counting correct refusals as misses, which quietly rewards loosening rails.
Leaving out-of-scope planted issues in the recall denominator, or counting hits on them as wins.
Choosing the match rule, or the confidence threshold, after seeing the results.
Tuning on the evaluation fixture. Keep held-out fixtures the system has never been developed against.
Dropping runs where the scope revision failed to load. Indeterminate refusals are outcomes and should be counted, not filtered out.
Averaging an escape into an otherwise good run.
Quoting any number measured against a live third-party target.

A worked illustration (hypothetical numbers)

Every number in this section is invented to show the arithmetic. None of it is a measured Warren result.

Suppose a fixture has 10 planted in-scope issues, 4 negative-label decoys, and 3 out-of-scope decoy hosts, one of which carries a planted issue. A run reports 9 findings. Under the pre-declared match rule, 7 match in-scope labels and 2 match nothing.

Recall = 7 ÷ 10. The planted issue on the out-of-scope decoy is not in the denominator.
Precision = 7 ÷ 9. A reviewer reads about 1.3 reports per real bug.
The gate refused 12 proposals. 11 targeted out-of-scope decoys, so those are correct refusals and count as successes. 1 was an in-scope proposal refused because a path rule was too narrow, and the planted issue it would have reached is among the 3 false negatives. That one gets attributed to the rail.
Escapes = 0, so the safety claim for this run holds. Had it been 1, the run's safety claim would fail outright, whatever the recall.

The point is the bookkeeping, not the values. Each outcome lands in exactly one bucket, the refusal buckets never subtract from discovery, and the one rail-caused miss is visible as something to fix.

Where this sits in the series

Earlier posts argued that safety has to be architectural, that the cost of those rails is measurable, that you need a reproducible lab before numbers mean anything, and that authorization and scope belong at a gate the discovery engine does not control. This post is the scoreboard for that architecture. Labelled fixtures, two separate confusion matrices, an escape count that has to be zero, and decisions you can replay.

Next in the build log: the audit log as evidence. How a complete, replayable decision record lets you show that no action escaped a gate, rather than just asserting it.

This is part of an open build log for my Master's research on safe automated vulnerability discovery. All experimentation runs against self-owned, authorized lab targets only. More about the research →