Your AI Is Showing — Confident, wrong, and your users believed it

Your AI Is Showing — Confident, wrong, and your users believed it

LLM09: Misinformation — Confident, wrong, and your users believed it

Part 9 of 10 — OWASP Top 10 for LLMs

In 2024, Air Canada's AI chatbot told a grieving passenger that he could book a full-price bereavement ticket and apply for a discounted refund afterward — a policy that didn't exist. The passenger relied on that information, bought the ticket, and applied for the refund. Air Canada denied it, saying the chatbot was wrong and they weren't responsible for what it said.

A tribunal disagreed. Air Canada's chatbot provided misinformation to travelers, leading to operational disruptions and legal complications. The airline was successfully sued as a result.

The model wasn't hacked. Nobody manipulated it. It just — confidently, fluently, helpfully — made something up. And a court decided that deploying a system that makes things up, in a context where users will rely on those things, makes you liable for what it invents.

This is LLM09.

What is it, actually?

LLMs don't retrieve facts from a database. They predict the next token based on patterns learned during training. Most of the time those predictions align with reality. Sometimes they don't — and the model has no reliable way to know the difference.

This is called hallucination: the model generates content that is plausible, fluent, and confidently stated — but factually wrong. It's not lying. It doesn't know it's wrong. It's pattern-matching its way to an answer that sounds right.

While hallucinations are a major source of misinformation, they are not the only cause. Biases introduced by training data and incomplete information also contribute. And overreliance compounds the risk: when users place excessive trust in LLM outputs and fail to verify independently, misinformation gets integrated into critical decisions — amplifying harm across healthcare, legal, and business contexts.

There are three forms this takes in practice:

Factual hallucination — the model states something false as if it were true. A date, a name, a statistic, a policy, a legal citation. A large-scale audit of 111 million references across 2.5 million academic papers found a sharp rise in non-existent citations following widespread LLM adoption — with a conservative estimate of 146,932 hallucinated citations in 2025 alone.

Source fabrication — the model invents a source that doesn't exist. A study, a regulation, a court case, a scientific paper — complete with author names, publication dates, journal titles. It looks like a citation. It goes nowhere.

Package hallucination (the actively exploited one) — adversaries identify commonly hallucinated code library names, publish malicious packages under those names, and wait for developers to unknowingly install them via AI coding assistants. The model recommends a package. The developer installs it. The package is malware.

The developer's view — what went wrong and how to fix it

The uncomfortable engineering reality: you cannot eliminate hallucinations. They are an inherent property of how language models work. Every model, including the best ones available today, will hallucinate under the right conditions. Your job isn't to make hallucinations impossible — it's to make their consequences bounded.

1. Ground responses in verified sources — use RAG for factual queries

The most effective mitigation for factual accuracy is retrieval. Instead of asking the model to generate facts from memory, give it the facts and ask it to reason over them.

# ❌ Asking the model to recall a policy from training
response = llm.generate("What is our refund policy?")
# Model may hallucinate a policy that doesn't exist
# ✅ Ground it — retrieve the actual policy, then have the model summarize it
policy_doc = knowledge_base.retrieve("refund policy")
response = llm.generate(
    f"Based ONLY on the following policy document, answer the question.\n"
    f"If the answer is not in the document, say so explicitly.\n\n"
    f"Policy: {policy_doc}\n\nQuestion: What is our refund policy?"
)

# ❌ Asking the model to recall a policy from training
response = llm.generate("What is our refund policy?")
# Model may hallucinate a policy that doesn't exist
# ✅ Ground it — retrieve the actual policy, then have the model summarize it
policy_doc = knowledge_base.retrieve("refund policy")
response = llm.generate(
    f"Based ONLY on the following policy document, answer the question.\n"
    f"If the answer is not in the document, say so explicitly.\n\n"
    f"Policy: {policy_doc}\n\nQuestion: What is our refund policy?"
)

The key addition: "If the answer is not in the document, say so explicitly." This shifts the model from pattern-matching to retrieval — and forces it to admit uncertainty rather than fill the gap with invention.

2. Never let the model generate code package names unsupervised

Package hallucination is actively exploited. When your AI coding assistant suggests a library, verify it exists before installing.

# Before installing ANY AI-recommended package:
pip index versions <package-name>  # Check it exists on PyPI
npm view <package-name>             # Check it exists on npm
# Better: maintain an allowlist of approved packages
# and have the AI choose from that list only

# Before installing ANY AI-recommended package:
pip index versions <package-name>  # Check it exists on PyPI
npm view <package-name>             # Check it exists on npm
# Better: maintain an allowlist of approved packages
# and have the AI choose from that list only

In agentic coding systems, build in a verification step before any package install. A hallucinated package name that doesn't hit your allowlist gets flagged for human review.

3. Surface uncertainty explicitly — don't let the model fake confidence

LLMs produce fluent, confident responses — misinformation can easily be mistaken for verified fact. The default behavior of most LLMs is to answer as if they're certain, even when they're not. Change that default.

SYSTEM_PROMPT = """
You are a helpful assistant. Follow these rules strictly:
- If you are not certain of a fact, say "I'm not certain, but..." before stating it
- If asked for a specific statistic, date, or citation you cannot verify, say so
- Never invent sources, citations, or references — if you don't have one, say you don't
- Prefer "I don't know" over a confident wrong answer
"""

SYSTEM_PROMPT = """
You are a helpful assistant. Follow these rules strictly:
- If you are not certain of a fact, say "I'm not certain, but..." before stating it
- If asked for a specific statistic, date, or citation you cannot verify, say so
- Never invent sources, citations, or references — if you don't have one, say you don't
- Prefer "I don't know" over a confident wrong answer
"""

4. Validate AI-generated citations before they go anywhere

If your application generates citations, references, or links — validate them before showing users.

def validate_citation(citation):
    # Check DOI exists
    if citation.doi:
        response = requests.get(f"https://doi.org/{citation.doi}")
        if response.status_code != 200:
            return CitationStatus.HALLUCINATED
    # Check author + title against known databases
    if not crossref_lookup(citation.author, citation.title):
        return CitationStatus.UNVERIFIED
    return CitationStatus.VERIFIED

def validate_citation(citation):
    # Check DOI exists
    if citation.doi:
        response = requests.get(f"https://doi.org/{citation.doi}")
        if response.status_code != 200:
            return CitationStatus.HALLUCINATED
    # Check author + title against known databases
    if not crossref_lookup(citation.author, citation.title):
        return CitationStatus.UNVERIFIED
    return CitationStatus.VERIFIED

This won't catch everything — but it catches the most obvious fabrications before they reach users who might rely on them.

5. Add human review gates for high-stakes outputs

In healthcare, legal, and financial contexts — don't let AI outputs go directly to end users for consequential decisions. Put a professional in the loop. A lawyer using an AI legal assistant asked the agent to draft a case motion. The agent generated hallucinated citations. The lawyer didn't verify before signing. He violated Rule 11 — fined $3,000 and his license was revoked. The AI was a tool. The professional was responsible. The workflow didn't include a verification step. That's the gap.

The business and client view — what this actually costs you

The Air Canada case established a precedent that's worth understanding clearly: deploying an AI system that gives incorrect information to users, in a context where users will rely on that information to make decisions, creates liability — even if the error was the AI's and not a human's.

"Our AI made a mistake" is not a defense. It's a description of the problem.

LLM-generated misinformation can be harder to detect for humans compared to human-written misinformation with the same semantics — it can have more deceptive styles and potentially cause more harm. That's the part that makes this uniquely dangerous at scale. The model writes confidently. It uses the right vocabulary. It structures its answers well. The error is invisible until someone checks — and most people don't check.

Hallucinations in court filings by prominent law firms as recent as April 2026 indicate that the hallucination problem is far from solved, even when sophisticated parties are using AI. If law firms are still submitting hallucinated citations in 2026, it's not a matter of technical sophistication. It's a matter of workflow — nobody built in the verification step.

For any business deploying AI in customer-facing or decision-support roles, the risk framework is straightforward:

High stakes + no verification = liability. If your AI gives advice that users act on — travel, medical, legal, financial — and that advice is wrong, you are responsible for the downstream decision.

Scale amplifies everything. A human advisor gives wrong advice to one person. An AI chatbot gives wrong advice to 10,000 simultaneously, all in the same confident tone, all without a disclaimer.

"The AI said so" is not due diligence. For professionals — lawyers, doctors, financial advisors — relying on AI output without independent verification isn't a defense. It's negligence.

Three takeaways

🔧 For developers: You cannot eliminate hallucinations — but you can bound them. Ground factual responses in retrieved documents, validate citations before surfacing them, and build explicit uncertainty into your system prompts. Never let the model fake confidence it doesn't have.

💼 For businesses: Air Canada lost a legal case because their chatbot made something up. The legal standard isn't whether the AI was responsible — it's whether your product gave users incorrect information they relied on. Review your AI's outputs in customer-facing contexts with that lens.

🔍 For everyone: 146,932 hallucinated citations published in academic papers in 2025 alone. The model sounds certain because it's designed to sound certain — not because it is. Verification is not optional. It's the gap between a useful tool and a liability.

Last up — LLM10: Unbounded Consumption. Your AI bill just hit $40,000 and it's Tuesday. Denial-of-wallet is real.

#AI Security · #LLM · #OWASP · #AI Ethics · #Machine Learning