LLM07: Misinformation (Hallucination, Ungrounded Claims)

Description

Misinformation is output that is incorrect, incomplete, unsupported or misleading, and credible enough that something acts on it. The something is often not a person. A model’s claim becomes a package name in a lockfile, an argument to a payment tool, an inferred state passed between agents (“customer is verified”, “nightly backup completed”), a citation rendered as a link, or a price restated to a customer in writing. The causes are mundane: hallucination, thin or stale retrieval, unvalidated tool output, and an interface that gives a guess and a grounded fact the same typography.

In deployed integrations this is a coding assistant recommending a dependency that does not exist, a RAG answer whose citations do not support the sentences beside them, a support bot confirming a refund term nobody wrote, or an agent reporting a job finished that never ran. The 2026 edition moved this entry up two places from LLM09:2025, pulled there by an incident corpus that ranked it far higher than the practitioner vote did, and reframed it: the model will be wrong, so the containing system must be built to survive that. The 2023 list called this Overreliance, now a contributing factor usually embedded in system design rather than in user behaviour. Adjacent categories own the mechanisms: claiming a hallucinated name on a registry is LLM04, executing bad generated code is LLM10, deliberately seeded false content is LLM05 and LLM01.

Keywords: llm misinformation, hallucination, slopsquatting, rag citation integrity, groundedness, sycophancy, false commitments

Examples/Proof

• Repeatable hallucinated dependency
  • Run a fixed coding prompt set N times and collect every import and install target. A nonexistent name recurring across most runs is predictable, and so pre-registrable.
• Citation that does not support the claim
  • Ask a question with no answer in the corpus. An answer arriving with resolvable-looking citations, instead of an abstention, proves the citation layer is decoration.
• Commitment invented under pushback
  • Push a support bot through authority claims and repeated disagreement until it confirms a discount or exception in writing. The transcript is the finding.
• Fabricated state handed downstream
  • Make a tool return an empty result and see whether the agent reports success. A completion claim with no matching tool output is a false state later steps will trust.

Detection and Monitoring

• Groundedness and abstention per answer
  • Log retrieved chunk IDs and the sentence each citation supports; alert when a citation resolves to nothing or its quote is absent. An abstention rate near zero on out-of-corpus queries means the no-answer threshold is not wired up.
• First-seen dependency gate
  • Diff lockfile changes against the registry; alert on packages with no download history, no source repository, or a publish date later than the suggestion.
• Commitment-language detection
  • Scan outbound support transcripts for confirmation phrasing about price, refund and policy exception, and reconcile against the system of record.

Remediation

1. Ground claims before anything acts on them
   • Resolve names, prices, policy terms and identifiers against an authoritative system before they enter a file, a tool argument or a customer-facing message.
2. Separate generation from execution
   • Have the model propose a structured claim, verify it in code, then act. Prose is not a transport for a fact a later step depends on.
3. Make abstention a first-class output
   • Enforce a retrieval or reranker score floor server-side and return an explicit no-answer instead of an unsourced guess.
4. Verify state independently of the model’s report
   • Confirm completions and approvals from the system that would have performed them, not from the agent’s summary.
5. Limit blast radius
   • Least privilege on install, spend and write paths; stage irreversible actions; keep a rollback ready.

Prevention Checklist

• Every dependency an assistant suggests is resolved against a registry allowlist before install
• Citations are emitted by the retriever and post-verified against the sentence they support
• The answer path can return “no answer” and does so on out-of-corpus queries
• Prices, refund terms and policy exceptions come from a system of record, not generated text
• Multi-turn adversarial suites run per release and report rates, not single screenshots