Key takeaways
- AI bias is a systematic, repeatable skew in an AI system’s outputs that produces unfair or inaccurate results for particular people or groups.
- The most cited taxonomy, from NIST SP 1270, sorts AI bias into three categories: systemic, statistical, and human.
- Under Article 10 of the EU AI Act, examining and mitigating bias in high-risk training data is a legal duty from 2 August 2026, not a nice-to-have.
- Marquee failures such as COMPAS, Amazon’s recruiting tool, and the Optum health algorithm were all foreseeable and preventable with documented bias examination.
- Bias is governable: with the right records, roles, metrics, and continuous monitoring, it becomes an auditable control instead of a reputational surprise.
What is AI bias?
AI bias is a systematic error in how an AI system represents the world, one that consistently advantages or disadvantages certain individuals or groups. It is not a single glitch or a one-off wrong answer. It is a pattern: the same kind of person gets the same kind of unfair result, again and again, because the pattern is baked into the data, the model, or the way people use its output.
It helps to separate two ideas that share the word “bias”. In statistics, bias is a neutral quantity: the difference between a model’s average prediction and the true value. In society, bias means an unjust outcome for someone because of who they are. AI bias sits exactly where these two meet: a measurable statistical skew that lands as real-world harm on real people.
That is why NIST frames AI bias as socio-technical rather than purely computational. You cannot debug your way out of it with a cleaner loss function alone, because the causes reach back into how data was collected, which problem was chosen, and how humans read the results. For a governance team, that framing matters: it means bias controls have to cover data, models, and people, not just code.
The three types of AI bias
The reference taxonomy comes from NIST SP 1270, which groups AI bias into three categories: systemic, statistical, and human. Most real incidents are a blend of all three, but naming them separately is what lets a team assign owners and controls to each.
Systemic bias
Systemic bias comes from society and institutions, then gets encoded into data. If a hiring process favoured one group for a decade, a model trained on that history learns the same preference as if it were a rule. The model is not inventing the unfairness; it is inheriting and scaling it.
Statistical and computational bias
Statistical bias comes from the data and the maths. Common mechanisms include statistical imbalance (some groups are underrepresented in the training set), proxy variables (a feature such as postcode or healthcare cost that stands in for a protected attribute), and contextual mismatch (a model trained in one setting used in another). These are the failures most amenable to technical detection and correction.
Human and cognitive bias
Human bias enters through the people who design, label, and use the system. Automation bias, our tendency to trust a machine’s output over our own judgement, is especially relevant to governance: a well-calibrated model can still cause harm if operators defer to it without question. Confirmation bias in how results are interpreted compounds the problem.
Where bias enters the AI lifecycle
Bias is not injected at a single point. It accumulates across the lifecycle, which is why one-off checks miss it. ISO/IEC TR 24027, the technical report on bias in AI systems, sets out techniques to assess bias across every phase, from data collection through to use.
The main entry points are worth listing because each needs its own control:
- Data collection and sampling: who is over- or under-represented in the training set.
- Labelling: subjective or inconsistent labels encode the labeller’s assumptions.
- Feature selection: including a proxy for a protected attribute reintroduces bias you thought you removed.
- Training and design choices: the objective you optimise for defines what “good” means, and can quietly trade fairness for accuracy.
- Testing and evaluation: if you only measure aggregate accuracy, group-level harm stays invisible.
- Deployment and feedback loops: a biased model shapes the data it later trains on, so the skew compounds over time.
A governance program maps a control to each of these stages rather than treating bias as a final-step audit.
Real-world AI bias examples
The clearest way to understand AI bias is through cases where it caused documented harm. Each of the examples below is famous for the failure, but the more useful lesson is what a governed process would have caught first.
The COMPAS recidivism tool, used in US courts to score a defendant’s risk of reoffending, was analysed by ProPublica. Black defendants who did not go on to reoffend were labelled high-risk at nearly twice the rate of comparable white defendants, roughly 44.9% against 23.5%. The governance lesson: group-level error rates, not just overall accuracy, belong in every evaluation report.
Amazon built an experimental recruiting tool that learned from a decade of mostly male resumes and began downgrading applications that mentioned women. The company scrapped it in 2018. The lesson: historical data carries historical discrimination, and a proxy as innocent as a club name can encode gender.
A health risk-prediction algorithm used across US hospitals, studied by Obermeyer and colleagues in Science, affected roughly 200 million people. It used healthcare cost as a proxy for healthcare need. Because less was historically spent on Black patients with the same conditions, the model understated their needs. Working with the vendor, the researchers cut the bias by about 80%. The lesson: proxy variables need explicit review, and a fix is often possible once the proxy is named.
Facial recognition systems have repeatedly shown higher error rates for demographic groups underrepresented in their training images, turning a data gap into unequal accuracy. The lesson: representativeness is a measurable property you can test for before deployment.
AI bias and the law: EU AI Act, NIST, ISO
This is where AI bias stops being an ethics discussion and becomes a compliance obligation. Article 10 of the EU AI Act requires providers of high-risk AI systems to examine training, validation, and test datasets for possible biases that could affect health, safety, or fundamental rights, and to take measures to detect, prevent, and mitigate them. Data must be relevant, sufficiently representative, and appropriate for the people the system will affect.
One clause deserves special attention. Article 10(5) creates a narrow, conditional exception allowing providers to process special-category (sensitive) data strictly to detect and correct bias, with safeguards. In plain terms: you may sometimes need protected-attribute data to prove your system is fair, and the law makes room for that under strict conditions.
Article 15 adds requirements for accuracy, robustness, and cybersecurity, and Article 27 introduces a Fundamental Rights Impact Assessment for certain deployers. These high-risk obligations apply from 2 August 2026, which turns bias examination into a dated deadline rather than an aspiration.
A subtle but important point from the research literature: fairness metrics and legal non-discrimination are not the same thing. As one analysis of the Act notes, there is no single fairness number to optimise. The obligation is to examine for bias and mitigate it where feasible, documenting your choices, not to declare victory on one metric while failing others.
Two frameworks translate these duties into practice. The NIST AI Risk Management Framework puts fairness under its Measure and Manage functions, and ISO/IEC 42001, the AI management system standard, gives you the governance structure to run bias examination as a repeatable, certifiable process. Together with ISO/IEC TR 24027, they form a practical crosswalk from regulation to daily operations.
How to detect and measure AI bias
You cannot manage what you do not measure, and bias measurement means testing outcomes across groups, not just checking overall accuracy. A few metrics recur in practice:
- Statistical parity: does the positive outcome rate match across groups?
- Equal opportunity: among people who qualify, are they selected at similar rates regardless of group?
- Disparate impact, often expressed as the adverse-impact ratio or the four-fifths rule: is one group selected at less than 80% the rate of the most-selected group?
No single metric is complete, and some are mathematically impossible to satisfy at once, which is why the choice of metric is itself a governance decision that should be recorded. Alongside the metrics, teams should actively hunt for proxy variables and document the representativeness of their datasets. The through-line is repeatability: a bias test that runs once before launch is evidence of intent; a bias test that runs on a schedule is evidence of control.
How to mitigate AI bias
When a test surfaces bias, mitigation techniques fall into three stages of the pipeline. Pre-processing acts on the data: re-sampling under-represented groups or re-weighting examples so the training set better reflects reality. In-processing acts on the model: adversarial debiasing or fairness constraints added to the training objective. Post-processing acts on the output: adjusting decision thresholds per group to equalise error rates.
None of these is a silver bullet, and each involves trade-offs that a governance process should surface rather than hide. Feature review removes or neutralises proxies. Data gap remediation fills the holes that caused statistical imbalance. Above all, human oversight keeps a person accountable for the decision, which is both good practice and, for high-risk systems, a legal expectation. The point is not to chase a perfect model but to show a defensible, documented process for reducing harm.
Governing AI bias: an operational program
The difference between an organisation that gets surprised by a biased model and one that does not is rarely the algorithm. It is the program around it. Bias governance turns scattered, one-off fixes into a standing control with four ingredients.
First, records: a bias examination log for every high-risk system, capturing what data was checked, which metrics were used, what was found, and what was done. This is exactly the evidence Article 10 expects you to produce on request. Second, roles: clarity on who owns bias for a given system, and how responsibility splits between the provider that builds it and the deployer that uses it. Third, continuous monitoring: bias drifts as data and behaviour change, so the examination has to repeat on a cadence, not sit frozen at launch. Fourth, an audit trail: sign-offs, versions, and evidence linked to each AI system so an assessor, or a regulator, can follow the decision.
This is where a dedicated AI governance platform earns its place over a folder of spreadsheets. When every bias test, mitigation, and sign-off lives in one system of record, tied to the specific AI system and the specific obligation, the program becomes auditable by design. For a deeper library of governance practice, our Industry Insights hub covers the adjacent controls that bias examination connects to, from data governance to human oversight.
FAQ
What are the three types of AI bias?
The NIST SP 1270 taxonomy names three categories: systemic bias, which society and institutions encode into data; statistical or computational bias, which comes from the data and the model, including imbalance and proxy variables; and human bias, which enters through the people who design, label, and use the system. Most real incidents combine all three, so a good governance program assigns a control to each rather than treating bias as one problem.
What is an example of AI bias?
A well-documented case is the Optum health risk algorithm studied in Science, which affected roughly 200 million patients. It used healthcare cost as a stand-in for healthcare need, and because less had historically been spent on Black patients with the same conditions, it understated their needs. Once the proxy was identified, researchers reduced the bias by about 80%, which shows both the harm and the fixability of proxy-driven bias.
Can AI bias be eliminated completely?
No, and claiming otherwise is a red flag. Bias can be measured, reduced, and controlled, but some residual skew almost always remains, and different fairness metrics can conflict mathematically. The realistic goal, and the one the EU AI Act sets, is to examine for bias, mitigate it where feasible, and document the trade-offs, rather than to certify a perfectly fair system.
Is AI bias illegal under the EU AI Act?
Bias itself is not criminalised, but for high-risk systems the failure to examine and mitigate it is a breach of Article 10 from 2 August 2026. Providers must check datasets for bias, ensure representativeness, and keep evidence. Article 10(5) even permits limited use of sensitive data to detect and correct bias, under safeguards, so the law treats bias examination as a required, documented activity.
How do you test an AI system for bias?
You measure outcomes across protected groups using metrics such as statistical parity, equal opportunity, and disparate impact (the four-fifths rule), rather than relying on aggregate accuracy. You also search for proxy variables and document dataset representativeness. Crucially, the test should run on a schedule so it produces ongoing evidence of control, not a single pre-launch snapshot.
What is the difference between bias and fairness in AI?
Bias is the systematic skew in a system’s behaviour; fairness is the goal of outcomes that do not unjustly depend on who someone is. They are related but not opposite: reducing bias is how you pursue fairness, but because fairness has several competing definitions, removing one form of bias can trade off against another. That tension is why fairness is a governance decision, not just a modelling metric.
Conclusion
AI bias is not an abstract ethics debate to be settled someday. It is a concrete, measurable property of a system, and under the EU AI Act it is now a documented, auditable duty with a deadline. The organisations that stay out of trouble treat bias the way they treat any other risk: they name its types, test for it across the lifecycle, mitigate it with known techniques, and keep the evidence in one place. Do that, and bias stops being a headline waiting to happen and becomes just another control you can prove. See how AI Sigil helps teams govern bias and fairness systematically.