Model releases arrive with a column of numbers, and the numbers are usually accurate. The difficulty is that they answer a narrower question than the reader is asking. A benchmark says how a model performed on a fixed set of problems, once, under conditions the publisher chose. A deployment decision needs to know how it will behave on your problems, repeatedly, under conditions you did not choose.
What is actually in a published score?
Take a recent frontier release we covered: a new state of the art on a long-horizon software engineering benchmark at 77.9%, first place on an end-to-end business automation benchmark at 51.3%, 91.7% on long video understanding, and a tie for first at 68% on remediating security flaws. Every one of those is a real measurement. None of them tells you the spread across repeated runs, and none of them was produced on your workload.
Some index construction is more honest than others. The same release cites an index weighting sectors by their contribution to national output, which at least states its weighting rather than leaving it implicit. A benchmark that does not say what it is averaging over is averaging over something.
Why is contamination the base rate?
Public evaluation sets leak into training corpora within months of release. That is not an occasional accident; it is the normal lifecycle of a popular benchmark. A team choosing a model on public scores is therefore increasingly measuring how much of the test set a model has already seen, which is a real property and not the one being shopped for.
The implication is uncomfortable but simple: the more widely cited a benchmark is, the less it discriminates. The scores converge, the gaps fall inside noise, and the ranking stops carrying information long before anybody stops quoting it.
Why the mean is the wrong statistic
On single-pass reasoning benchmarks, the distance between the best open-weight models and the best closed ones is now within noise. Run the same prompt twenty times and the picture separates: closed models cluster, open ones spread. We ran a consistency evaluation across nine models and found that leaderboard rank predicted mean accuracy well and predicted worst-case behaviour badly.
For a chat product that spread is tolerable. For anything that writes to a database, calls a tool or spends money, the worst case is the whole problem — a system is only as reliable as its bad runs, and the mean conceals exactly those.
Why two numbers are rarely comparable
Conditions move results more than model quality does, and they are often unstated. One hardware analysis we covered put an inference chip ahead of every other part on throughput per megawatt without using speculative decoding, while the chips it was compared against were using it. The comparison is still informative, but only because the analyst said so; a table that does not mention it is comparing two different things in the same column.
| Question | Why it changes the answer |
|---|---|
| How many runs, and what was the spread? | A mean over one pass hides the failure rate that will define your incident log |
| When was the evaluation set published? | Anything widely available for a year may be in the training data |
| What inference techniques were enabled? | Speculation, caching and batching move throughput several-fold and are often unstated |
| On what hardware, at what concurrency? | Figures quoted at low concurrency do not survive contact with production traffic |
| Who constructed the benchmark, and do they sell something? | Not disqualifying, but it decides which numbers get published and which do not |
| Does the task resemble yours at all? | The only question whose answer you can establish yourself |
What a usable evaluation looks like
The teams doing this well maintain a private, versioned set drawn from their own traffic, refresh a portion of it each quarter, and never publish it. That is considerably more work than downloading a benchmark, and it is the only approach we have seen hold up. The versioning matters as much as the privacy: without it you cannot tell a model regression from a change in your own test set.
- Draw cases from real traffic, including the ones that went wrong.
- Score variance alongside the mean, and set the acceptance threshold on the worst case where the system acts on its output.
- Refresh part of the set on a schedule, and keep the old versions so that comparisons across time remain possible.
- Re-run it on the model you already use, not only on candidates — a provider's silent update is indistinguishable from a bad day otherwise.
- Keep it unpublished. The moment it is public, it starts rotting.
None of this makes public benchmarks useless. They are a reasonable filter for deciding what to test and a poor basis for deciding what to deploy. The distinction costs an afternoon to maintain and is the difference between a number that describes the field and a number that describes your system.
Questions
- Why do leaderboard scores converge?
- Partly because models improve and partly because popular evaluation sets leak into training corpora within months of release. As that happens the benchmark measures memorisation as much as capability, and the gaps between models fall inside noise.
- Should I pick a model by its benchmark rank?
- Use the rank to choose what to test, not what to deploy. In our comparison across nine models, rank predicted mean accuracy well and worst-case behaviour badly — and worst case is what matters anywhere the model writes, calls a tool or spends money.
- How many times should an evaluation be run?
- Enough to see the spread rather than the mean. Repeating the same prompt twenty times is what separated clustered models from spread ones in the comparison we ran; a single pass cannot show that difference at all.
- Why should a private evaluation set never be published?
- Because publication starts the contamination clock. A set drawn from your own traffic, versioned and partly refreshed each quarter, keeps discriminating precisely because nobody can train on it.