One Major Risk of Generative AI Models, Explained

Key takeaways

• The single most material risk of generative AI models is unfaithful output, what NIST formally calls confabulation and what most teams know as hallucination.
• Hallucination dominates because it has the highest production incidence in real deployments and because it amplifies every other risk: an unfaithful output makes bias, privacy leaks, and intellectual property infringements harder to detect.
• The defensible reference taxonomy is NIST AI 600-1, which catalogues 12 risks unique to or exacerbated by generative AI and maps each one to the four functions of the NIST AI Risk Management Framework.
• Under the EU AI Act, the same risk surfaces in Article 9 (risk management system), Article 13 (transparency), Article 15 (accuracy, robustness, cybersecurity), Article 50 (synthetic content labelling), and Articles 53 and 55 (general-purpose and systemic-risk obligations).
• A workable governance pattern has four layers: pre-deployment evaluation, retrieval grounding with confidence gating, output mediation with human review on high-stakes flows, and post-deployment monitoring with incident reporting.

The one major risk: unfaithful output (hallucination)

If you must pick a single answer to the question, pick hallucination, the tendency of a generative model to produce content that sounds confident but is factually wrong, fabricated, or unsupported by any source the model was given. NIST uses the term confabulation in its Generative AI Profile, partly to underline that the failure is structural rather than incidental: the model is not lying, it is sampling from a probability distribution that happens to assign mass to false statements (NIST AI 600-1).

Three reasons make this the dominant risk in 2026.

First, production incidence. Academic work mapping real-world generative AI incidents finds that unfaithful output is the most-reported failure mode across deployed systems, ahead of bias, privacy leakage, or prompt injection (arXiv 2505.22073). Three high-profile precedents are now case law in the way risk officers reason about generative AI: the Air Canada chatbot decision, in which a Canadian tribunal held the airline liable for a refund policy invented by its assistant; the Mata v. Avianca sanctions, in which a New York federal judge sanctioned attorneys who filed a brief citing six entirely fictional cases produced by ChatGPT; and the Australian filings in 2024 and 2025 in which lawyers were referred to professional bodies for similar hallucinated citations.

Second, compounding effect. A bias incident in a deterministic system surfaces as a measurable disparity in outcomes. A bias incident in a hallucinating model can hide inside a fluent paragraph that sounds authoritative. The same is true of privacy: an unfaithful summary of a medical record may invent a diagnosis, mixing a real privacy exposure with a fabricated one. Hallucination is the failure mode that makes every other failure mode harder to audit.

Third, regulatory weight. The EU AI Act does not name hallucination as such, but it does require providers of high-risk AI systems to design them so that they reach appropriate levels of accuracy and robustness throughout their lifecycle (Article 15), and to provide instructions for use that disclose performance characteristics and known limitations (Article 13). For general-purpose AI models, the obligations escalate under Article 53, and for models posing systemic risk, the EU AI Office’s GPAI Code of Practice imposes a full Safety and Security Framework including pre-deployment evaluations and post-market monitoring (EU GPAI Code of Practice, July 2025).

One risk is the headline. Twelve are the structure underneath it.

The full risk landscape: NIST’s 12-category taxonomy

Why NIST AI 600-1 is the reference

Most competing articles list eight, ten, or twelve risks without a shared backbone, which makes the lists hard to compare and harder to operationalize. NIST AI 600-1 fixes that. Published on 26 July 2024, the Generative AI Profile was developed by a public working group of more than 2,500 contributors and identifies 12 risks that are either unique to generative AI or significantly exacerbated by it. Each risk is mapped to the four functions of the underlying NIST AI Risk Management Framework 1.0 (Govern, Map, Measure, Manage), with more than 200 recommended actions distributed across them.

The 12 risks, mapped to EU AI Act articles and OWASP LLM Top 10

The table does two jobs at once. For a US-anchored team, it preserves the NIST vocabulary already in use. For an EU-anchored team, it shows which regulator obligation attaches to each risk. The OWASP column gives a working bridge to the security architects, who tend to use the LLM Top 10 v2025 as their shared language.

Using the table for prioritization

The taxonomy is not a checklist. Prioritization is the work. For each risk, ask two questions: how likely is this failure given the way your system is built and deployed, and how severe is the consequence if it fires. A clinical decision-support assistant prioritizes confabulation, harmful bias, and human-AI configuration. A code-generation assistant prioritizes confabulation, information security, and intellectual property. A consumer chatbot prioritizes dangerous content, information integrity, and data privacy. The taxonomy lets every team work from the same vocabulary while landing on different priority orderings.

What makes generative AI’s risk profile different

Three properties of generative AI break the assumptions classical software risk management relies on.

Scale and speed. A single prompt produces content at internet scale. A misconfigured customer assistant can publish thousands of incorrect refund commitments before anyone notices, as Air Canada discovered. The blast radius of a single bad release is no longer bounded by user volume; it is bounded by generation volume.

Stochastic outputs. Classical software has a deterministic test oracle: given an input, the correct output is fixed and testable. Generative models sample from a distribution. The same prompt produces different outputs across runs, and the same model behaves differently after a routine fine-tune. This breaks unit testing, regression testing, and most acceptance criteria written for deterministic software. Evaluation has to shift from “does the output equal X” to “is the output within an acceptable distribution”, which is a harder question with weaker tooling.

Emergent capabilities and value-chain opacity. Behaviors that were absent from the training data can appear at scale, sometimes unannounced between checkpoints. At the same time, responsibility is layered: a foundation model provider trains the model, an integrator fine-tunes and wraps it, a deployer puts it in front of users. The EU AI Act addresses this with the value chain provisions in Article 25 and the GPAI provider duties in Article 53, but in practice the deployer still owns the user-visible failure. The EU GPAI Code of Practice draws an additional line at the systemic-risk threshold of 10^25 floating-point operations for training, above which providers must run a Safety and Security Framework, including model evaluations and red-team testing.

Governing the dominant risk: a four-layer pattern

A defensible governance posture for the single dominant risk maps neatly onto the NIST AI RMF Measure and Manage functions, and onto the EU AI Act’s Articles 9, 14, 15, 17, and 72.

Layer 1: Pre-deployment evaluation

Before a generative system reaches users, it should pass a documented evaluation suite covering its expected failure modes. For confabulation, that means hallucination benchmarks (TruthfulQA, HaluEval, domain-specific evals built from your own ground-truth set), red-team prompts designed to elicit fabricated citations, and adversarial tests drawn from MITRE ATLAS techniques. The NIST AI RMF Playbook describes the Measure function in operational terms; the EU AI Act formalizes the duty in Article 15, which requires high-risk systems to be designed and developed in such a way that they achieve an appropriate level of accuracy, robustness, and cybersecurity, and perform consistently in those respects throughout their lifecycle (Article 15).

Layer 2: Retrieval grounding and confidence gating

The architectural pattern that most reliably reduces hallucination at runtime is retrieval-augmented generation with strict grounding. The model is forced to answer from retrieved documents, with explicit citation, and to abstain when retrieval confidence is below a configured threshold. This shifts the failure mode from “answer wrong” to “refuse to answer”, which is dramatically cheaper to operate. Confidence gating is also one of the few patterns that satisfies the transparency duty of Article 13, which requires the design to enable users to interpret the system’s output appropriately.

Layer 3: Output mediation

For high-stakes flows, retrieval is not enough. Output mediation adds a validation layer between the model and the user: a second model checks the first model’s output, a rule-based validator enforces structural constraints, or a human reviews the output before it is acted on. The decision of where to mediate is governed by impact. Clinical, legal, and financial decisions need human-in-the-loop. Informational outputs may need only automated checks. The choice is the substance of Article 14 (human oversight): providers must design high-risk AI systems so that natural persons can effectively oversee them and override their outputs.

Layer 4: Post-deployment monitoring and incident reporting

Nothing in Layers 1 to 3 catches the drift that emerges only after deployment, when prompts diverge from the eval set and users find edge cases the team did not anticipate. Article 72 of the EU AI Act formalizes post-market monitoring, and Article 73 sets reporting duties for serious incidents. The OECD’s working definition of an AI incident provides a shared vocabulary (OECD AI Paper No. 16), and the NIST AI RMF Manage function lists the operational practices: live evaluation against rolling benchmarks, user feedback loops, anomaly detection on input distributions, and a mature incident response process owned by a named function in the organization.

FAQ

What are the four types of AI risk? The most commonly cited four-part split comes from the NIST AI Risk Management Framework’s trustworthy AI characteristics. The functions Govern, Map, Measure, and Manage describe the lifecycle, while the trustworthy AI characteristics group risks into four practical buckets: safety and security risks, fairness and bias risks, transparency and accountability risks, and privacy and data governance risks. Other taxonomies (OECD, EU AI Act risk categories, ISO 23894) use different splits. For decision-making, the more granular NIST AI 600-1 12-risk taxonomy is more useful than any four-bucket model.

What is a major concern in the use of generative AI? The major concern is unfaithful output: the model returns a confident answer that is wrong, fabricated, or unsupported by evidence. Concrete consequences include legal liability when an assistant misrepresents company policy, professional sanctions when fabricated citations end up in court filings, reputational damage when synthetic content is mistaken for genuine reporting, and clinical or financial harm when a hallucinated recommendation is acted on. The concern is not theoretical: documented incidents now include the Air Canada chatbot precedent and the Mata v. Avianca sanctions.

What is one concern associated with generative AI models in software development? The most acute concern in software development is the generation of insecure code. Coding assistants happily produce snippets that import deprecated libraries, hardcode credentials, miss input validation, or reproduce vulnerable patterns memorized from training data. OWASP captures the surrounding family of risks in LLM05 Improper Output Handling and LLM01 Prompt Injection, and NIST SP 800-218A extends the Secure Software Development Framework to AI-assisted development. Concrete controls include mandatory code review of AI-generated code, secret scanning, dependency vetting, and refusal patterns when the assistant is asked to produce security-sensitive code.

How does the EU AI Act regulate generative AI specifically? The EU AI Act treats generative AI under three layers. Article 50 sets transparency duties for synthetic content (deepfake labelling, machine-readable marks). Article 53 sets baseline obligations for providers of general-purpose AI models: technical documentation, copyright policy, training-data summary, and assistance to downstream providers. Article 55 adds duties for general-purpose AI models with systemic risk, including model evaluations, adversarial testing, serious-incident reporting to the AI Office, and cybersecurity. The EU GPAI Code of Practice operationalizes Articles 53 and 55 for providers above the 10^25 FLOPs training compute threshold.

What is the difference between bias and hallucination? Bias is systematic skew in outputs across protected attributes or groups: the model is more likely to recommend a male candidate, more likely to misrecognize a darker-skinned face, more likely to produce a stereotype. Hallucination is unfaithful generation: the model invents a citation, a refund policy, a person, a quote, a case. Both are NIST AI 600-1 risks but they are distinct categories. They also call for different mitigations: bias is addressed through dataset curation, fairness evaluations, and outcome audits; hallucination is addressed through retrieval grounding, confidence gating, and output mediation.

Are there governance frameworks built specifically for generative AI? Yes. The four most actionable today are NIST AI 600-1 (US, July 2024), the EU GPAI Code of Practice (EU, July 2025), the OWASP Top 10 for LLM Applications (industry, November 2024), and MITRE ATLAS (industry, evolving). NIST AI 600-1 is the canonical risk taxonomy with action mappings. The EU GPAI Code of Practice operationalizes the AI Act for general-purpose providers. OWASP supplies the developer-facing vulnerability list. MITRE ATLAS catalogues adversarial techniques. Used together, they cover taxonomy, regulatory operationalization, application security, and threat modelling.

Will hallucination be solved by larger models? Probably not by scale alone. Larger models reduce certain hallucination types and amplify others, particularly confident misstatements in domains under-represented at training time. The serious academic position in 2025 is that hallucination is intrinsic to autoregressive generation and must be governed at the system level, with grounding, mediation, and monitoring, not waited out at the model level (arXiv 2504.08526).

Conclusion

If you need one answer to take into an exam, a board meeting, or a vendor review, it is this: the major risk of generative AI models is unfaithful output, also called hallucination or confabulation. If you need to defend that answer or operationalize it, the underlying landscape has twelve risks in the NIST AI 600-1 taxonomy, each one anchored to specific EU AI Act articles, and each one governed by a small number of well-understood architectural patterns. The mistake most teams make is treating risks one at a time. The opportunity is to adopt a taxonomy, map it to a control library, and run the four-layer governance pattern through a continuous improvement loop.

At AI Sigil we ship a risk register pre-mapped to NIST AI 600-1 and to the EU AI Act’s high-risk and GPAI obligations, with controls per category and audit-ready evidence collection. Risk is one part of the work. The other part is keeping it managed.