AI hallucination
A hallucination is output from an AI model that is fluent and confident but false or unsupported: an invented fact, a made-up citation, a policy that does not exist, or a number that appears nowhere in the source. It happens because language models generate plausible text rather than looking facts up, so it can be reduced but not fully eliminated.
01why it matters for a business
Hallucinations are the main reason AI output cannot be trusted blindly, and they have already had real consequences. In the widely reported Mata v. Avianca case in 2023, lawyers were sanctioned by a US federal court after filing a brief containing case citations ChatGPT had invented. In 2024, a Canadian tribunal held Air Canada responsible for incorrect bereavement fare information its website chatbot gave a customer. In both cases the organization owned the error, not the model.
For a business, the practical response is design, not hope. Ground answers in your own data through retrieval, require citations, instruct the model to say when it does not know, check outputs against source systems wherever numbers or commitments are involved, test with AI evaluations, and keep people reviewing anything high stakes.
02what it looks like in practice
A distributor's AI assistant drafts quotes from customer emails. In testing, it occasionally fills in a volume discount the customer asked about but the contract does not include. The fix is structural: prices and discounts now come only from a tool that queries the pricing system, the model is instructed never to state a price it did not receive from that tool, and a validation step compares every number in the draft with the system's values before a rep sees it.
03common mistakes
- Assuming a newer or bigger model has solved the problem. Rates vary by task, and no model is immune.
- Letting a model answer from general knowledge where your specific policy or data is what matters.
- No way for users to verify answers. Citations and links to sources make errors visible.
- Measuring hallucinations once at launch and never again.
04related terms
- Retrieval-augmented generation (RAG)Retrieval-augmented generation (RAG) is a technique in which an AI system first searches your own documents or data for passages relevant to a question, then gives those passages to a language model to write its answer.
- AI guardrailsGuardrails are the controls around an AI system that keep its inputs, outputs, and actions within acceptable limits: input filtering, output checks for policy, format, and sensitive data, limits on which tools and data it can reach, spending caps, and rules that route risky cases to a person.
- AI evaluations (evals)Evals are structured tests that measure how well an AI system performs on a defined task, using a set of real or realistic inputs with known good outcomes.
- Human in the loop (HITL)Human in the loop is a design pattern in which a person reviews, approves, or corrects an AI system's output at defined points before it takes effect.
- Generative engine optimization (GEO)Generative engine optimization (GEO) is the practice of improving how often and how accurately a brand, product, or source appears in answers produced by generative AI systems such as ChatGPT, Claude, Gemini, Perplexity, and Google's AI features.
05where insomnia club fits
Insomnia Club's AI implementation work grounds answers in your own data and adds evaluation and guardrails, so the system is proven to behave before it reaches customers.
see AI implementation →tell us what keeps you up at night.
Scoped by the people who ship it. Priced before we start.
book a call drop your number