Key takeaways
- Explainability is the ability to describe, in terms a person can understand, why an AI system produced a given output. It is not the same as interpretability, which concerns how the model works inside.
- Under the EU AI Act, explainability has shifted from good practice to legal duty: Article 13 requires high-risk systems to be transparent enough for deployers to interpret their output, and Article 86 grants affected people a right to explanation from 2 August 2026.
- The NIST AI Risk Management Framework lists explainable and interpretable among its seven characteristics of trustworthy AI, and ISO/IEC 42001 expects explainability to be managed and evidenced.
- A governance program should produce several kinds of explanation, covering rationale, responsibility, data, fairness, safety and impact, not a single technical readout.
- Explainability is auditable. The real work is documenting which explanations you give, to whom, and how you assure them across the AI lifecycle.
What explainability means in AI (and what it is not)
Explainability is the capacity to describe the behaviour of an AI system in language a person can follow. The clearest way to hold it in mind comes from a comparison the search results keep circling back to. As Splunk puts it, explainability answers the question of why a model produced a result, while interpretability answers how the model works internally. The two ideas are related, and they are routinely confused, but they are not interchangeable. A second distinction sharpens the picture. The Alan Turing Institute separates transparency, being open about how and why a system is used, from explainability, being able to give clear and accessible reasons for a specific output. Transparency is a posture. Explainability is a deliverable: an account that a named audience can actually use. It also helps to separate global from local explanations. A global explanation describes how a model behaves in general, across its whole input space. A local explanation accounts for one specific decision, the loan that was declined or the transaction that was flagged. Compliance questions are almost always local, because the person affected wants to know about their case, not the model in the abstract. An AI governance platform has to support both, and keep them apart.
Explainability vs interpretability
If you take one distinction from this guide, take this one, because the search demand for it is relentless and the regulatory stakes depend on it. Interpretability is a property of the model. A linear regression or a short decision tree is interpretable because a person can trace the path from input to output by reading the model itself. Explainability is broader. It is the ability to produce a human-understandable account of a result even when the model is a deep network whose internal weights no one can read directly. The NIST AI Risk Management Framework frames the pair in a way governance teams can reuse: explainability concerns the mechanisms underlying an AI system’s operation, while interpretability concerns the meaning of its output in the context of its designed purpose. In practice, interpretability is often treated as a subset of explainability. You can have an explainable account of an un-interpretable model, and that is exactly the situation most enterprises face.
Why explainability is now a compliance requirement
For years explainability lived in research papers and responsible-AI manifestos. It now lives in law and in certifiable standards, which changes who has to care and why. The NIST framework names explainable and interpretable as one of seven characteristics of trustworthy AI, alongside valid and reliable, safe, secure and resilient, accountable and transparent, privacy-enhanced, and fair with harmful bias managed. NIST is voluntary, but it has become the common operating language for AI risk, and most enterprise programs map their controls to it. ISO/IEC 42001, the first certifiable AI management system standard, turns that language into something auditable. An organization that seeks certification has to show that explainability is governed: that it is considered when models are selected, that explanations are produced where stakeholders need them, and that the evidence exists to prove it. NIST tells you what to address; ISO 42001 gives you a management system to demonstrate that you addressed it. The two are complementary, and many programs run the framework inside the standard. The decisive shift is legal. The EU AI Act makes a sufficient degree of explanation a binding obligation for high-risk systems, and it gives individuals a route to demand one. That is a different world from a best-practice checklist, and it is the reason explainability now belongs on a compliance roadmap rather than only on a data-science backlog.
What the EU AI Act requires, by role and risk tier
The Act does not impose one flat duty. It assigns different obligations to providers and deployers, and it scales them by risk tier. Getting the mapping right is the difference between a defensible program and a paperwork exercise.
Providers of high-risk systems (Article 13)
The core obligation sits with whoever places a high-risk system on the market. Article 13(1) states that “high-risk AI systems shall be designed and developed in such a way as to ensure that their operation is sufficiently transparent to enable deployers to interpret a system’s output and use it appropriately.” The word interpret is deliberate: the deployer has to be able to make sense of what the system says. That duty is delivered mainly through instructions for use. Article 13 requires them to be concise, complete, correct and clear, and Article 13(3) lists what they must contain, including the system’s characteristics, its level of accuracy, its known limitations, and its technical capabilities to explain its output. Explanation is therefore not an add-on; it is part of the documentation a provider ships. It also underpins Article 14 on human oversight, because a person cannot meaningfully oversee a system whose output they cannot interpret.
Deployers and the right to explanation (Article 86)
The Act also reaches the organization that uses the system. Article 86 provides that any affected person subject to a decision taken by a deployer on the basis of a high-risk system listed in Annex III, where that decision produces legal effects or similarly significantly affects them in a way they consider adverse to their health, safety or fundamental rights, “shall have the right to obtain from the deployer clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken.” This is a right held by individuals and owed by deployers, and it becomes applicable on 2 August 2026. A bank, an insurer or an employer that runs a high-risk system has to be ready to explain a specific adverse decision to the person on the receiving end, in plain terms, on request. That is a local explanation, on demand, with a deadline.
Transparency disclosure is not explainability (Article 50)
One trap is worth naming. Article 50 imposes transparency duties on certain limited-risk systems, such as telling people they are interacting with a chatbot or labelling synthetic media. Those are disclosure obligations, and they are not the same as explaining a decision. A system can satisfy Article 50 disclosure and still owe nothing under Articles 13 or 86, or the reverse. Treat the two as separate lines in your compliance program so a disclosure banner is never mistaken for an explanation.
The six types of explanation a governance program should produce
Most competitor content stops at a single technical explanation, a feature-importance chart, and calls the job done. A governance program needs more, because different audiences need different accounts of the same decision. The Alan Turing Institute sets out six types of explanation that, taken together, give a rounded picture, and they map neatly onto what regulators and affected people actually ask for.
- Rationale: the reasons behind an outcome, expressed in accessible terms rather than raw model internals.
- Responsibility: who is accountable for the system and how a person can contest or seek review of a decision.
- Data: what data was used, where it came from, and how it shaped the result.
- Fairness: the steps taken to ensure the decision is non-discriminatory, which ties explainability directly to bias management.
- Safety: the accuracy, reliability, security and robustness behind the output.
- Impact: the effect the decision has on the individual and on society.
The same source frames the discipline around four maxims: be transparent, be accountable, consider context, and reflect on impacts. These are operating commitments rather than slogans. They tell a team to tailor each explanation to its audience, to assign clear ownership, and to think about the consequences of giving, or withholding, an explanation. A regulator examining an adverse decision will rarely be satisfied by a rationale alone; they will want the responsibility, data and fairness accounts as well.
Explainability techniques and their governance trade-offs
Explanations come from somewhere, and the method you choose carries governance consequences. There are two broad families. Intrinsic explainability comes from using a model that is interpretable by design, such as a linear model, a generalized additive model or a shallow decision tree. The explanation is the model. The trade-off is that these models can underperform on complex problems. Post-hoc explainability sits on top of an opaque model. Techniques such as SHAP and LIME approximate which features drove a particular prediction. Counterfactual explanations describe the smallest change that would have altered the outcome, which is often the most intuitive account for an affected person, because it answers the question of what they would have needed to be different. Surrogate models and feature-importance methods give a global view of behaviour. The headline trade-off is between predictive performance and ease of explanation. The instinct to reach for the most accurate model and bolt an explainer on afterwards is understandable, but post-hoc explanations are approximations and can mislead if treated as ground truth. The governance choice is not which technique is best in the abstract, it is which technique gives the audience in front of you, a regulator, an auditor, or an affected individual, an account they can rely on for that decision. Record the method, its limitations, and why it was chosen.
Operationalizing explainability as an auditable control
The gap between a responsible-AI principle and a passed audit is documentation. Explainability becomes a control when you can show, for a given system, which explanations you provide, to whom, by what method, and who signed off. The Alan Turing Institute pairs its explainability principle with an Explainability Assurance Management template, an iterative tool for planning, implementing and evidencing explanation activities across the lifecycle, so that the clarification of system outputs to affected stakeholders is documented rather than improvised. The structure matters more than the specific template. A workable control has four parts: a record of which explanation types each system owes and to which audiences; the technique used to produce each one; the evidence that the explanation was actually available when needed; and a named owner who reviews it as the model changes. This is where explainability stops being a data-science concern and becomes a program. Mapping each high-risk system to its Article 13 and Article 86 obligations, attaching the explanation artifacts, and keeping the evidence current as models are retrained is exactly the kind of work an AI governance platform is built to carry. Done once, by hand, it is a project. Done continuously, across a portfolio, it needs a system of record.
FAQ
What is the difference between explainability and interpretability? Interpretability is a property of the model: a simple model whose logic a person can read directly is interpretable. Explainability is the broader ability to give a human-understandable account of a result, even for a complex model no one can read directly. A common shorthand is that interpretability answers how the model works, while explainability answers why a particular output occurred. In governance terms, you can owe an explanation for a model that is not itself interpretable, which is the situation most enterprises are in. Is the EU AI Act’s right to explanation already in force? Not yet. Article 86, the right of affected people to obtain clear and meaningful explanations of a high-risk decision from the deployer, becomes applicable on 2 August 2026. Article 13 transparency obligations for providers of high-risk systems apply on the schedule set for high-risk requirements. Organizations running high-risk systems should prepare the explanation capability now rather than waiting for the date. Does explainability apply to every AI system? No. The binding obligations in the EU AI Act focus on high-risk systems. Limited-risk systems carry transparency disclosure duties under Article 50, such as disclosing that content is AI-generated, which is different from explaining a decision. Minimal-risk systems carry no specific explainability duty under the Act. That said, NIST and ISO 42001 treat explainability as a general characteristic of trustworthy AI, so most mature programs apply a proportionate version across the portfolio. What counts as a good explanation for a regulator? A single feature-importance chart rarely does. A regulator examining an adverse decision typically wants several accounts together: the rationale for the outcome, who is responsible and how it can be contested, what data was used, the fairness steps taken, and the safety basis of the output. Tailoring the explanation to the audience and documenting it are as important as the technical method behind it. Can we just use SHAP or LIME and call it explainability? Those techniques are useful, but they are post-hoc approximations of an opaque model, not the whole of explainability. They can mislead if treated as exact truth, and they answer a technical question, not the governance ones about responsibility, data and impact. Use them as one input, record their limitations, and pair them with the documentation and ownership that make explainability auditable. How does explainability relate to human oversight? They are tightly linked. Article 14 of the EU AI Act requires effective human oversight of high-risk systems, and a person cannot meaningfully oversee a system whose output they cannot interpret. Explainability is the precondition for oversight: without an understandable account of why the system produced a result, the human in the loop is approving outputs blind.
Conclusion
Explainability has crossed from principle to obligation. It is no longer enough to say a model is accurate; under the EU AI Act, NIST and ISO 42001, an organization has to be able to say why a system reached a decision, in terms the right audience can use, and to prove it did. The teams that treat this as a documentation and governance problem, rather than a one-off technical trick, will be ready for Article 86 in 2026 and for the audits that follow. The practical first step is modest: list your high-risk systems, decide which explanations each one owes and to whom, and start keeping the evidence. See how the AI Sigil platform turns that into a repeatable control.