Why does an AI make things up rather than say it does not know?
A language model has no built-in step where it checks a fact. It predicts the next piece of text, and when the true answer was rare in its training data, a confident guess is often the most likely-looking continuation. Researchers at OpenAI argued in a September 2025 paper, "Why language models hallucinate", that the way models are trained and tested makes it worse: the scoring rewards a lucky guess over "I don't know", much as a multiple-choice exam with no penalty for wrong answers rewards guessing.
Hallucinations gather where facts are thin: exact numbers, recent changes, small local businesses, niche rules. That is a fair description of most things an owner wants to ask.
Has a made-up answer ever cost a business money?
Yes, and the business paid. In February 2024, British Columbia's Civil Resolution Tribunal ruled against Air Canada after its website chatbot told a grieving customer they could claim a bereavement discount after travelling, which was not the airline's policy. The airline argued the chatbot was responsible for its own words. The tribunal disagreed and ordered Air Canada to pay the difference.
Closer to home, in October 2025 Deloitte agreed to refund part of its fee to the federal Department of Employment and Workplace Relations after a report it wrote was found to contain references that did not exist. The corrected version disclosed that a generative AI model had been used in the drafting.
The rule for an owner is short. Whatever goes out under your name is yours, whoever drafted it. The Australian Consumer Law's ban on misleading claims applies to an ad a chatbot wrote exactly as it applies to one you wrote (ACCC, advertising and selling guidance).
What does an invented click price do to a builder's budget? (example figures)
A granny flat builder in Bathurst asks a chatbot what a click costs for "granny flat builders Bathurst". The answer comes back without hesitation: about $2.50. The owner sets a Google Ads budget of $750 a month expecting 300 clicks and, at a 5 percent enquiry rate, 15 enquiries. Then Google's Keyword Planner, which estimates from Google's own auction data, shows top-of-page bids of $6 to $11.
| What the chatbot implied | What Keyword Planner suggests | |
|---|---|---|
| Cost per click | $2.50 | About $8 |
| Clicks for $750 | 300 | 94 |
| Enquiries at 5 percent | 15 | About 5 |
| Builds won at 1 in 5 | 3 | 1 |
The $2.50 had no source, because there was none to have. The model produced a number shaped like a click price. An owner who plans on it decides after a month that Google Ads does not work, when the plan was wrong on the first day.
How do you catch a hallucination before it costs you?
Treat every number, name, date, price, law and quote in an AI answer as unchecked until you have seen it somewhere else. Ask for the source and open it, because invented references often look perfect and lead nowhere. Ask the same question in a fresh chat: if the figure changes, it was never known.
For anything about your own business, give the model the facts instead of asking it to remember them, or use a tool that answers from your documents, which is retrieval-augmented generation. Keep a person between the AI and the customer for anything that goes out.
How does Cora keep advice tied to your numbers?
CoraVision ties Cora's figures to your own accounts: once connected, it reads Google Ads, Meta and Google Analytics 4 each night, so the cost per click in the conversation comes from your own campaigns. The dashboard's Trust Score says how far those numbers can be trusted, which is the honest answer when tracking is only half set up. And because you make every change yourself, a suggestion that looks wrong stops with you.
Questions
Is an AI hallucination the same as lying?
No intent is involved. The model has no sense of true and false, only of which words are likely to come next, so it gives a wrong answer in the same confident tone as a right one. For a customer who acts on it, the effect is the same.
Will newer models stop making things up?
They make fewer errors in their makers' own tests, but no model maker claims zero. OpenAI's 2025 paper on the subject argues accuracy can never reach 100 percent, because some questions have no answer a model could know. Checking stays part of the job.
Does asking for sources stop an AI inventing things?
It helps you check, but it does not stop the problem. Models can invent sources with real-sounding titles, authors and links. Open each one and find the claim in it, or use a tool that searches the web or your documents before it answers, so the links point at pages that exist.