A benchmark result is not a fitness claim

This is commentary, and the view in it is mine rather than the organisation’s. It is here because the competence it argues for is in the framework and the reasoning behind it is not.

A benchmark establishes performance on a fixed set of items, under stated conditions, at a point in time. It is a useful instrument for comparing systems on that set. It establishes nothing about a system’s behaviour on your task, with your data, under your consequence of error — and yet a benchmark figure is routinely the only evidence offered before a system goes into use.

Three gaps that a number cannot close

The distribution gap: benchmark items are curated, and production inputs are not. The consequence gap: a benchmark scores every item alike, while in your use one failure mode may be trivial and another catastrophic. The freshness gap: a system that scored well six months ago may have been changed since, and a score is not monitoring.

None of this is an argument against benchmarks. It is an argument that a benchmark result is an input to an evaluation rather than a substitute for one, which is why D6 asks at L1 for the recognition that a published result does not establish fitness, and at L3 for the ability to say precisely what a given claim does and does not establish.

What I would ask for instead

Fitness criteria derived from the intended use, written before development; an evaluation set sampled from realistic conditions with cases held back; and a defined threshold at which the system may not be used. If a vendor cannot say what its system must achieve to be acceptable for your task, the number it is quoting you is not about your task.

The uncomfortable part, for anyone selling assurance: doing this properly is slower and less quotable than a leaderboard position, and it will sometimes conclude that a system should not be deployed. That is the point of it.