AI Benchmarking: Turning Scores Into Audit Evidence

Key takeaways

• AI benchmarking measures how a model performs on a fixed set of tasks under a known scoring rule. It answers which model is stronger on that test, not whether your system is fit for its intended purpose.
• Under the EU AI Act, providers of high-risk systems must declare their accuracy metrics in the instructions for use. The duty is to state and substantiate your own metric, not to cite someone else’s leaderboard.
• Public scores are weakening as evidence: the most-cited tests are saturated, and their published question sets leak into training corpora.
• A benchmark result becomes admissible evidence only when you can show what was run, on which data, against which model version, by whom, and when.
• The evidence that survives an audit almost always comes from an internal test set derived from your intended purpose, not from a public ranking.

What AI benchmarking actually measures

An AI benchmark is a standardised test: a fixed dataset of inputs, a set of expected outputs, and a scoring rule that turns model behaviour into a comparable number. AI benchmarking is the practice of running models against those tests and interpreting what comes back. Because every model faces the same inputs under the same rule, a benchmark converts an impression into a figure that two people can disagree about with the same facts in front of them. That is genuinely useful, and it is also where the trouble starts. A benchmark is a finite sample drawn from one distribution. The work you deploy an AI model into is an open-ended set of situations drawn from a much messier one. The score is always a proxy, and the distance between the proxy and the thing you care about is never printed on the leaderboard.

The three parts of a benchmark

Every benchmark has three components, and each one fails independently.

• The dataset: the inputs, and usually the ground-truth outputs to score against.
• The evaluation method: the rule that converts outputs into a score. Answer matching for multiple choice, unit-test execution for code, or a model-based judge for open-ended text.
• The leaderboard: the ranking that makes results comparable and, incidentally, makes them worth gaming.

Ground truth deserves particular attention because it silently caps everything above it. If the answer key is wrong, a correct model response is scored as an error, and the entire result inherits that noise.

Capability families, and what each one is blind to

No single test captures what a model can do, so the field splits evaluation by capability. Reading across families is the only honest way to see a model’s shape.

• Knowledge (MMLU, MMLU-Pro): breadth of factual recall. The original MMLU is largely saturated, so small differences between frontier models carry no signal.
• Reasoning (GPQA Diamond, Humanity’s Last Exam): multi-step problem solving on deliberately hard questions. Hard today, a contamination target tomorrow.
• Coding (SWE-bench Verified, LiveCodeBench): fixing real repository issues rather than writing isolated functions. Function-level tests have lost most of their discriminating power.
• Agents (GAIA): planning and tool use across multiple steps. Results swing on the scaffolding around the model as much as the model itself.
• Multimodal (MMMU): reasoning across text, images and diagrams. A strong text model can post a weak score here.
• Human preference (Arena Elo): which answer people prefer in blind comparisons. This measures preference and style, not correctness, and the two diverge more often than the ranking suggests.

For teams working specifically with language models, the family-by-family detail sits in our guide to LLM benchmarks. What follows applies to any model class.

Where benchmark scores become a regulatory obligation

Most writing on AI benchmarking treats it as an engineering habit. For anyone placing a system on the European market, it is closer to a documentation duty, and the framing changes what a good benchmark looks like.

EU AI Act Article 15: declare your accuracy metric

Article 15 of the EU AI Act requires high-risk AI systems to achieve an appropriate level of accuracy, robustness and cybersecurity, and to perform consistently in those respects throughout their lifecycle. The operative sentence for governance teams is the next one: the levels of accuracy and the relevant accuracy metrics must be declared in the accompanying instructions for use. Read that carefully, because it inverts the usual reflex. The obligation is not to score well on a recognised public test. It is to name the metric you consider relevant to your system, state the level you achieve, and stand behind both. A leaderboard position does not discharge this duty, and citing one in place of your own measurement is a gap an assessor will find quickly. The Act does anticipate shared measurement standards. It directs the Commission, working with stakeholders and organisations such as metrology and benchmarking authorities, to encourage the development of benchmarks and measurement methodologies. That work is in progress rather than settled, so for now the burden of choosing and justifying a metric sits with the provider. Article 15 applies from 2 August 2026, which is inside the planning horizon of most programmes described in our EU AI Act operator’s guide.

General-purpose models: Article 55 and the Code of Practice

Providers of general-purpose models carry a separate and more explicit evaluation duty. The GPAI Code of Practice published by the EU AI Office in July 2025 requires providers of models with systemic risk, currently pegged to training compute above 10^25 FLOPs, to maintain a Safety and Security Framework covering model evaluations, red-teaming, post-market monitoring, cybersecurity and incident reporting. Here the evaluation is not a comparison exercise at all. It is a safety argument, and it has to be maintained after release rather than produced once at launch. Deployers building on top of these models inherit the consequences, which is why our note on general-purpose AI treats provider evaluation evidence as a procurement input rather than a technical curiosity.

ISO 42001 and NIST AI RMF: the management-system view

Two voluntary frameworks turn benchmarking from an activity into a governed process. ISO/IEC 42001 places performance evaluation inside a certifiable management system. Once a benchmark run lives in clause 9, it stops being a slide and becomes a record with an owner, a cadence, and an audit trail. The NIST AI Risk Management Framework is more direct. Its MEASURE function applies metrics, benchmarks, tests and analytics to trustworthiness characteristics, and calls for regular benchmarking of both the system and the risk-management practice around it. NIST’s work on test, evaluation, validation and verification sets the bar plainly: TEVV processes must be objective, repeatable or scalable, in place, followed and documented, and they should adhere to scientific, legal and ethical norms in an open and transparent process. That single sentence is the most useful specification of a defensible benchmark anyone has written. Note what it does not say. It says nothing about which test you run, and everything about whether someone else could run it again and get the same answer.

Why a public benchmark score is weak evidence

The case against relying on published scores is not a contrarian opinion. It comes from the benchmarking literature itself.

Saturation and contamination

The most-cited tests have saturated. Frontier models cluster within a point or two of each other, and the remaining spread often reflects ambiguous or mislabelled questions rather than any capability difference. A test where everyone scores an A cannot do the one job a benchmark exists to do. Contamination is the sharper problem. Benchmarks are published openly so the field can reproduce them, which means their questions and answers end up in the web-scale corpora used to train the next generation of models. A model that has effectively seen the test during training will ace it without generalising. Surveys of contamination in language-model evaluation describe the effect bluntly: the score measures recall, not capability, and you cannot tell the difference from the leaderboard.

Goodhart’s Law and construct validity

When a measure becomes a target, it stops being a good measure. Once a benchmark is the number labs advertise and buyers screen on, the incentive shifts from building better models to posting better scores. This is now well documented. An interdisciplinary meta-review associated with the European Commission’s Joint Research Centre, Can We Trust AI Benchmarks?, identified nine systemic issues in AI benchmarking, including misaligned incentives, construct-validity failures and gaming risks. Construct validity is the quiet one: it asks whether a benchmark measures the thing it claims to measure at all, and a surprising share of published benchmarks cannot demonstrate that they do. Aspen Digital’s policy primer Benchmarks 101 gives the failure taxonomy in three parts: low-quality data, flawed measurement or design, and misinterpreted results. Its worked example is worth keeping in mind the next time a headline score lands. Researchers found that roughly 30 percent of the biology and chemistry questions in Humanity’s Last Exam, a benchmark built specifically to be hard, had wrong or unsupportable answers.

The lab-versus-deployment gap

Benchmarks measure in the lab, not in the wild. They are also used earlier in the lifecycle than most people assume, at goal definition and model building, whereas impact assessment and red-teaming come later, once a system exists to be probed. The consequences are not academic. Aspen Digital documents dermatology models benchmarked on the International Skin Imaging Collaboration dataset that scored highly overall, then degraded significantly on darker skin once performance was split by skin tone. The aggregate number was accurate and the conclusion drawn from it was wrong. That example draws the boundary this article is about. A capability score is not a fundamental-rights assurance, and no amount of it substitutes for the disaggregated testing described in our guide to AI bias. Fairness testing is a separate obligation with a separate method, not a subsection of performance measurement.

What a benchmark can and cannot evidence

Governance teams need a boundary they can apply without re-litigating the science each time. This is ours.

The read-out is short. A benchmark is admissible as evidence only when you can show what was run, on which data, against which model version, by whom, and when. Everything in the left column depends on that record existing. Nothing in the right column becomes true because a score was high. Where a benchmark cannot carry the weight, the obligation falls back to the controls described in our AI risk management guide.

Building an internal benchmark that survives an audit

The practical conclusion of everything above is that the benchmark worth investing in is usually one you build.

Start from the intended purpose, not the leaderboard

Intended purpose is the anchor concept in the EU AI Act, and it is the right starting point for a test set. Write down what the system is for, who it affects, and what a wrong answer costs in each case. Then assemble real inputs it will encounter, including the awkward ones: the ambiguous requests, the edge cases, the inputs that arrive malformed. This is more work than reading a ranking, and the resulting evaluation will not generalise beyond your use case. That is the point. A model that leads a public leaderboard and performs poorly on your task is the wrong model for your deployment, whatever its scores.

Data provenance and held-out design

Three properties separate a test set you can rely on from one that quietly misleads you.

1. The model has not seen the data. Use held-out data, freshly generated data, or data time-windowed to postdate the model’s training cutoff. Otherwise you are measuring memorisation.
2. The data reflects your real domain. Generic web text tells you nothing about how a model handles clinical notes, loan files or internal tickets, where the vocabulary, structure and ambiguity are nothing like a clean encyclopedia paragraph.
3. The ground truth is genuinely correct. Every score is measured against an answer key, and a key with errors caps a good model while rewarding the wrong behaviour.

These pull against each other, and it is worth saying so plainly. The most realistic data is usually the most sensitive, so it cannot be published, and publishing it would contaminate future models anyway. Hand-labelling at scale introduces exactly the ground-truth errors the third property warns about. Teams resolve this with synthetic generation, careful de-identification, or narrow but well-curated sets. There is no option that avoids the trade-off entirely.

Versioning, reproducibility and retention

Repeatability and reproducibility are not engineering niceties here. The AI Verify Testing Framework treats them as a governance principle in their own right, and structures every check as an Outcome, a Process, and the Evidence that validates it. That pattern maps directly onto what an assessor will ask for. A benchmark run should leave behind a record containing, at minimum:

• the test protocol and the scoring rule
• the dataset version and a content hash
• the model identifier, version and API endpoint
• the harness, parameters, and whether the run was zero-shot or few-shot
• the date, the operator, and the raw outputs alongside the computed score
• the retention period and where the artefacts live

That list is what converts a number into an artefact. It is also the difference between a system that is auditable and one that merely performed well once, a distinction we develop further in auditability in AI. These records belong with the rest of the technical documentation the EU AI Act requires, not in a separate folder owned by the data-science team.

How to interrogate a vendor’s benchmark claims

Benchmark tables are the first evidence any vendor puts forward, and they are rarely challenged. Seven questions do most of the work.

1. Which scoring rule was used, and does it match our comparison? Accuracy, pass@k and Elo are not interchangeable. A pass@1 and a pass@10 for the same model measure different things, and an Elo gap describes head-to-head win rate, not a percentage of capability.
2. Zero-shot or few-shot? Few-shot numbers are systematically higher.
3. Full test set or a verified subset? Human-reviewed subsets with broken items removed report cleaner and higher figures.
4. Who ran the evaluation? A score a vendor produced on its own harness is not comparable to an independent measurement.
5. Is the test set public, and could it have leaked into training? If the vendor has not considered contamination, the score cannot be relied on.
6. When was it measured, relative to the model version behind the API today? Providers update models silently behind stable API names.
7. Will you re-run this on our held-out set, under NDA? The answer to this one is the most informative of all.

A vendor who cannot answer these is asking you to accept a marketing number as an assurance artefact. That is a third-party risk finding, not a technical footnote, and it belongs in the same assessment as the platform questions covered in our comparison of AI governance tools.

FAQ

What is AI benchmarking? AI benchmarking is the practice of running AI models against standardised tests, each made of a fixed dataset and a scoring rule, to produce comparable performance numbers. It is used to compare candidate models, track progress between versions, and catch regressions. It measures performance on the benchmark’s tasks, which may or may not resemble the tasks you care about. Does the EU AI Act require benchmarking? Not in those words. Article 15 requires high-risk systems to achieve appropriate accuracy, robustness and cybersecurity, to perform consistently across the lifecycle, and to declare the levels of accuracy and the relevant accuracy metrics in the instructions for use. That effectively requires a defensible measurement practice, but it does not prescribe a named public benchmark. The Act separately directs the Commission to encourage the development of benchmarks and measurement methodologies with metrology and benchmarking authorities, work that is still under way. Providers of general-purpose models with systemic risk face a more explicit model-evaluation duty under Article 55 and the GPAI Code of Practice. Are public AI benchmarks still useful? Yes, as a first filter rather than a final verdict. Public benchmarks narrow a field of candidates quickly and expose obvious weaknesses. They cannot tell you how a model behaves on your data, in your domain, under your constraints, and saturation and contamination have widened that gap. Treat a leaderboard position as a hypothesis to test, not a result to cite. What is benchmark data contamination? Contamination happens when a benchmark’s test questions and answers appear in a model’s training data. Because benchmarks are published openly for reproducibility, they are routinely crawled into training corpora. A contaminated model reproduces answers it has memorised, posting a high score that collapses on fresh problems of the same type. It is difficult to detect, rarely disclosed, and the main reason public scores drift away from real-world performance. How is AI benchmarking different from an AI audit? A benchmark measures how a model performs on a defined task. An AI audit examines the system and the organisation around it: governance, risk management, data provenance, human oversight, documentation and the evidence behind each claim. Benchmark results are one input to an audit. On their own they answer a narrow technical question, and auditors will ask how the run was designed, versioned and retained before they accept the number at all. How many benchmarks should we run on a high-risk system? Fewer than most teams expect, chosen more carefully. One internal test set derived from the intended purpose is worth more than five public ones, because it is the only test whose results speak to fitness for purpose. Add a fairness evaluation with results disaggregated across affected groups, and a resilience test covering adversarial and out-of-distribution inputs. Those three, versioned and repeatable, produce a stronger file than any number of leaderboard citations.

Conclusion

AI benchmarking is not broken, but it is widely misfiled. Treated as a ranking exercise it produces procurement theatre: a table of numbers nobody can reproduce, measuring tasks nobody in the room has examined, on model versions that have since been replaced. Treated as a measurement practice it produces something an assessor can actually use. The shift is small and mostly organisational. Derive the test from the intended purpose. Hold the data out. Record the protocol, the versions, the operator and the date. Keep the artefacts where the rest of the technical file lives. Say clearly what the result does not prove. That last discipline matters most. The strongest position in front of a regulator is not a high score. It is a documented measurement, an honest statement of its limits, and controls covering everything the measurement leaves open. Wiring those records to the controls they support, rather than scattering them across slide decks and notebooks, is the practical work of an AI governance framework.