Key takeaways
- Algorithmic bias is a systematic, repeatable error in an automated system that produces unfair or discriminatory outcomes for certain people or groups, not a random mistake.
- Bias rarely comes from one line of code. It enters through data, model design, and the human and institutional context around the system.
- The EU AI Act treats bias as a legal obligation: Article 10 requires providers of high-risk systems to examine and mitigate it, with high-risk duties enforceable from 2 August 2026.
- Measuring bias means comparing outcomes across protected groups using fairness metrics such as demographic parity, equal opportunity, and equalized odds.
- Lasting control comes from governance, not a one-off fix: documented bias testing, human oversight, and monitoring recorded inside an AI management system.
What is algorithmic bias?
Algorithmic bias describes systematic and repeatable errors in a computer system that create unfair outcomes, such as privileging one group of users over another without a justified reason. The decisive word is systematic. A model that is wrong at random is simply inaccurate. A model that is consistently wrong against the same group is biased, and that consistency is what turns a technical flaw into a legal and reputational problem. Three terms are often blurred together, so it helps to separate them. Data bias refers to skew in a dataset, for example a training set that under-represents older applicants. AI bias is the broad umbrella for any unfair behavior of an AI system. Algorithmic bias sits between the two: it is the unfair pattern an algorithm produces once data, design choices, and deployment combine. Data bias is one cause; algorithmic bias is the observable result. The most useful framing for governance comes from the US National Institute of Standards and Technology. In NIST SP 1270, bias in AI is treated as socio-technical: it cannot be removed by adjusting code alone, because it reflects choices about data, objectives, and the social setting the system runs in. That framing has a direct consequence for compliance. A credible response has to cover process, people, and governance, not just a model metric. For an organization building an AI management system, this is the line between a defensible control and a box-ticking exercise.
Where algorithmic bias comes from
NIST SP 1270 sorts the origins of bias into three categories. Each one calls for a different kind of control, which is why naming the category is the first practical step.
Computational and statistical bias
This is bias that comes from the data and the model itself. Training data that under-represents a group, labels that encode past prejudice, or a proxy variable that quietly stands in for a protected characteristic all push predictions in a skewed direction. A frequent culprit is the objective function. Optimizing for a single target, such as cost or engagement, can disadvantage groups that the target does not describe well. The model is doing exactly what it was told to do, and that is the problem.
Systemic bias
Systemic bias is inherited from institutions and history rather than from a specific dataset error. If a recruitment process favored one demographic for years, a model trained on that record learns the same preference even when no protected attribute appears in the features. The algorithm becomes faithful to the past and unfair to the present. Removing the sensitive field does not help, because other variables act as stand-ins for it.
Human bias
Human bias enters through the people who design, build, and use the system. Cognitive shortcuts shape which features are selected, how outputs are read, and when a reviewer defers to the machine. Automation bias, the tendency to trust an automated output even when other evidence contradicts it, is one of the most common and most dangerous, because it can turn a flawed recommendation into a final decision while a human is nominally in the loop. These categories line up with the machine learning lifecycle. The EDPB AI Auditing Checklist, prepared for the European Data Protection Board by Dr. Gemma Galdon Clavell, locates bias at three moments: pre-processing (data collection and preparation), in-processing (model training and inference), and post-processing (how a prediction becomes a decision in the world). Naming the moment makes the fix concrete, because a labeling problem cannot be solved with a deployment rule, and a deployment problem cannot be solved by cleaning data.
Algorithmic bias in the real world
Definitions land better against documented cases, and there are several that regulators and researchers return to. In criminal justice, the COMPAS recidivism tool became the reference example after a 2016 ProPublica investigation found that Black defendants were almost twice as likely to be wrongly flagged as high-risk (about 45 percent) as white defendants (about 23 percent), while white defendants were more often mislabeled as low-risk despite going on to reoffend. In healthcare, an algorithm studied by Obermeyer and colleagues in Science (2019) influenced care decisions for roughly 200 million people in the United States. It used healthcare cost as a proxy for healthcare need. Because less money had historically been spent on Black patients with the same conditions, the model rated them as lower-need. Working with the vendor, the researchers cut the measured bias by about 80 percent once the proxy was corrected, a reminder that bias is often fixable when it is actually looked for. In hiring, Amazon stopped using an experimental recruiting tool after finding it downgraded resumes containing the word women’s and favored male applicants, because it had been trained on a decade of mostly male resumes. In consumer finance, the Apple Card drew regulatory scrutiny in 2019 after reports that women were offered lower credit limits than men with comparable profiles. The pattern is consistent across every sector. No team set out to discriminate. Bias arrived through proxies, historical data, and unexamined assumptions, which is precisely why it has to be tested for rather than assumed away.
Who is responsible: providers and deployers
A recurring source of confusion is who owns the bias problem. The EU AI Act answers this by splitting duties between providers, who develop or place a system on the market, and deployers, who put it to use. Providers carry the data and model duties. They build the training pipeline, choose the objective, and therefore sit closest to computational and statistical bias. Deployers carry the context duty. They decide where and on whom a system is used, which is where systemic and automation bias often surface. A bias-free model can still produce unfair outcomes if it is deployed on a population it was never validated for. For regulated organizations, this split has a practical implication: bias controls have to live on both sides of a vendor relationship. Procurement is the natural checkpoint. A deployer that asks a provider for its bias-testing evidence, its representativeness analysis, and its intended-use limits before signing turns a contractual moment into a control. AI Sigil’s AI governance platform is built around exactly this handoff, so that provider evidence and deployer assessments sit in one record.
Why algorithmic bias is now a compliance obligation
For most of the past decade, bias was discussed as an ethics question. Under the EU AI Act it is a legal one, with concrete duties and deadlines. Article 10 of the AI Act requires that training, validation, and testing data for high-risk systems be relevant, sufficiently representative, and to the best extent possible free of errors and complete for the intended purpose. It explicitly requires data governance to include examination in view of possible biases that are likely to affect health and safety, negatively affect fundamental rights, or lead to discrimination prohibited under Union law, along with measures to detect, prevent, and mitigate those biases. Article 10(5) goes further, permitting providers to process special categories of personal data when strictly necessary to detect and correct bias, subject to safeguards. The regulation treats bias testing as a duty, not an optional good practice. Obligations for the Annex III high-risk categories become enforceable on 2 August 2026. Deployers have their own duty. Article 27 requires certain deployers of high-risk systems, including many public bodies, to carry out a Fundamental Rights Impact Assessment before first use, describing the affected groups, the specific risks of harm, the human oversight in place, and the complaint mechanisms available. Bias is central to that assessment, because a single biased system in education can affect non-discrimination, the right to education, and child rights at the same time. It helps to separate two layers of law. The AI Act imposes an ex-ante, technical duty to examine and mitigate bias during development. EU non-discrimination law operates ex-post, on outcomes, when a person is harmed. Recent legal scholarship stresses that the two are complementary rather than identical, so meeting one does not automatically satisfy the other. A defensible program plans for both, and documents the reasoning. Standards turn the obligation into an operating model. The NIST AI Risk Management Framework places bias work inside its MEASURE function, giving teams a shared vocabulary for testing and tracking it. ISO/IEC 42001, the AI management system standard, provides the documented home for bias controls, owners, and evidence. Together they move bias from a research topic to a managed control with an audit trail.
How to detect and measure algorithmic bias
Detection begins before any metric is calculated: you have to decide which groups to protect. These usually track legally protected characteristics such as race, gender, age, religion, or disability, along with their close proxies, and the choice should be recorded. With the groups defined, you compare outcomes across them. The fairness metrics most audit methodologies rely on, including the EDPB checklist, are straightforward to compute and to evidence:
- Demographic parity: each group receives positive outcomes at a similar rate.
- Equal opportunity: qualified people receive positive outcomes at a similar rate across groups.
- Equalized odds: error rates, both false positives and false negatives, are similar across groups.
- Risk difference (p1 minus p2) and risk ratio (p1 divided by p2): simple, auditable measures of the gap between two groups.
No single metric is correct for every situation, and several of them cannot all be satisfied at once for mathematical reasons. The choice of metric is therefore a governance decision in its own right, one that should be justified in writing, then tested against ground truth and against the representativeness of the training data. A fairness number with no recorded rationale is weak evidence in an audit.
How to mitigate and govern algorithmic bias
Mitigation that survives scrutiny is repeatable. A practical workflow, drawn from the EDPB audit methodology, runs in four steps that any governance team can document.
- Model card: record the system’s purpose, owner, data sources, and intended use.
- System map: capture how the model, the wider technical system, and the decision process connect, and who is accountable at each stage.
- Moments and sources of bias: identify where bias can enter across pre-processing, in-processing, and post-processing.
- Bias testing: run the fairness metrics above on the defined protected groups, then record the results and the actions taken.
Each step maps onto the three lifecycle stages NIST SP 1270 recommends for managing bias: pre-design, design and development, and deployment. Controls placed early, such as representative data and a well-chosen objective, are cheaper and more effective than corrections bolted on after release. For higher-risk or unsupervised systems, an adversarial audit, which probes the system with crafted inputs, can surface bias that standard testing misses. Technical fixes are only half the answer. Because bias is socio-technical, durable control depends on governance: a named owner for every model, meaningful human oversight that resists automation bias, a complaint mechanism for affected people, and periodic re-testing whenever the system or its population changes. Under the AI Act the evidence is part of the obligation, so this work has to be written down. Recording bias tests, mitigation decisions, and monitoring inside one AI inventory and management system turns scattered good intentions into an auditable control that maps directly to Article 10 and to ISO/IEC 42001.
Frequently asked questions
What is algorithmic bias? Algorithmic bias is a systematic, repeatable error in an automated system that produces unfair or discriminatory outcomes for certain individuals or groups without a justified reason. It differs from random inaccuracy because it consistently disadvantages the same people, which is what makes it both measurable and legally relevant. What is an example of algorithmic bias? A well-documented case is a healthcare algorithm that used past spending as a proxy for medical need. Because less had historically been spent on Black patients with the same conditions, the system rated them as lower-need, affecting care for around 200 million people until the proxy was corrected and the bias fell by roughly 80 percent. What is the difference between algorithmic bias and AI bias? AI bias is the broad umbrella for any unfair behavior of an AI system. Algorithmic bias is the specific unfair pattern an algorithm produces once data, design, and deployment interact. Data bias, meaning skew in the underlying dataset, is one of its main causes rather than a synonym. What is automation bias? Automation bias is the human tendency to over-trust an automated output, even when other evidence suggests it is wrong. It matters for AI oversight because it can quietly turn a flawed recommendation into a final decision, defeating the purpose of having a human in the loop. Is algorithmic bias illegal under the EU AI Act? The AI Act does not ban bias outright, but it makes examining and mitigating bias a legal obligation for high-risk systems under Article 10, with those duties enforceable from 2 August 2026. Separately, EU non-discrimination law can apply when biased outputs cause harm to a person. How do you measure algorithmic bias? You define protected groups, then compare outcomes across them using fairness metrics such as demographic parity, equal opportunity, equalized odds, and the risk difference or risk ratio between groups, testing the results against ground truth and the representativeness of the training data.
Conclusion
Algorithmic bias is no longer a question of whether a system is ethical in the abstract. It is a question of whether an organization can show, with evidence, that it looked for unfair outcomes, measured them, and acted. The EU AI Act, NIST SP 1270, and ISO/IEC 42001 now point to the same expectation: bias is a managed, documented, auditable control rather than a one-time clean-up. Teams that treat it that way, with clear ownership, fairness testing, and monitoring kept in one place, will meet the 2026 deadlines and build systems that customers and regulators can trust. AI Sigil helps regulated organizations put exactly these controls into practice inside a single AI governance platform.