Generative AI cannot guarantee that what it says is true, cannot reliably explain how it reached an answer, and cannot be held responsible for its output. The U.S. National Institute of Standards and Technology (NIST) calls the first problem confabulation: confidently stated content that is false. Courts and tribunals in both the United States and Canada have since placed responsibility for such errors on the people and companies that use the tools.
Key takeaways
- NIST describes confabulation as a natural result of how generative models are designed, so it cannot be fully removed by better wording of a request.
- A model can give a convincing explanation of its reasoning even when its answer is wrong.
- In Mata v. Avianca (June 22, 2023), a federal court fined lawyers USD 5,000 for filing opinions that ChatGPT had invented.
- A February 14, 2024 ruling in Moffatt v. Air Canada made the airline answerable for wrong information from its own chatbot.
- The practical rule: use AI to draft and sort, and have a person verify every fact, number, and citation before it is relied on.
Why a fluent answer can still be false
NIST's Generative AI Profile (NIST AI 600-1), published in July 2024, explains the mechanism. Confabulation, it says, is a natural result of the way generative models are designed: they produce outputs that approximate the statistical distribution of their training data. A large language model, for example, predicts the next word in a sentence. That process can produce accurate text, and it can also produce text that is wrong or contradicts itself. The profile notes that the problem is especially relevant for open-ended prompts that call for long answers, and in fields that need deep domain knowledge.
The same document points out why the errors are hard to catch. Users tend to believe false content because the response sounds confident. The model may also add invented reasoning or citations that appear to support its answer, which can lead people to trust it more than they should. In NIST's words, a model can offer logical steps for how it arrived at an answer even when the answer itself is incorrect.
Two court cases about responsibility
The legal record shows where the burden lands. In Mata v. Avianca, a federal court in New York issued an opinion and order on sanctions on June 22, 2023. Two lawyers and their firm had submitted non-existent judicial opinions, with fake quotes and citations, created by ChatGPT, and then stood by them after the court questioned them. Judge P. Kevin Castel wrote that there is "nothing inherently improper about using a reliable artificial intelligence tool for assistance," but that lawyers keep a gatekeeping role over the accuracy of what they file. The respondents were ordered to pay a USD 5,000 penalty jointly and severally.
In Canada, the British Columbia Civil Resolution Tribunal decided Moffatt v. Air Canada (2024 BCCRT 149) on February 14, 2024. A chatbot on the airline's site had told a traveler that he could apply for a bereavement fare after buying his ticket, which another page on the site contradicted. Law-firm summaries of the ruling report that the tribunal found negligent misrepresentation and rejected the argument that the chatbot was a separate entity, holding the airline answerable for information on its own website, whether it came from a static page or a chatbot. The decision is from a small-claims tribunal and is not a binding precedent for other courts, but it shows how the question is likely to be framed.
A task-by-task limits table
The table lists recurring limits and a matching check. It is a simplified guide based on the NIST profile and the two cases, not a complete list.
| Task | Limit | Check before relying on it |
|---|---|---|
| Citing cases, studies, or statistics | The model can produce citations that look real and do not exist | Find each source in an official database or the publisher's page |
| Summarizing records, such as patient notes | NIST warns that an invented summary of patient information could lead to a wrong diagnosis or treatment | A qualified person compares the summary with the original record |
| Explaining its own reasoning | Stated steps may not match why the answer is right or wrong | Test the conclusion independently |
| Describing what it is | A model can falsely claim to be human or to have human traits | Treat such claims as output, not evidence |
| Answering for a business or a filing | The tool carries no legal responsibility | A named person reviews and signs off |
What these limits do not mean
None of this says AI tools are useless. The court in Mata did not condemn the use of a reliable tool, and drafting, sorting, and translating text are tasks where a reviewed draft saves time. The limits describe where the checking has to happen. A model that is right nine times out of ten still makes its tenth error with the same confident tone as the other nine, so the reader cannot tell them apart without a second source.
A workable habit has three steps: ask the tool for a draft, mark every factual claim, number, and quotation in it, and confirm each against a primary source. High-stakes uses, such as law, medicine, and money, call for a licensed professional to review the output. This article gives general information and is not legal or medical advice.
FAQ
Why does AI invent facts?
According to NIST, generative models produce output that approximates the statistical patterns of their training data. They do not look up a fact and check it, so a plausible but false statement can come out as easily as a true one.
Can better prompts eliminate the errors?
Clear prompts can reduce some mistakes, but NIST describes confabulation as a natural result of how these models are built, so a human check is still needed for anything that matters.
Who is responsible when an AI tool gives wrong information?
In the two cases above, the lawyers who filed the output and the airline that deployed the chatbot were held responsible. These are individual rulings, not general law, so the answer in another situation depends on its facts.
Is it safe to ask an AI tool for medical or legal guidance?
It can help a person prepare questions or understand terms, but it should not replace a doctor or lawyer. NIST specifically names healthcare as an area where false generated content creates risk.
Related reading
- Limits of AI in education
- Why AI misreads human emotions
- AI and the need for coding
- A beginner's learning path for artificial intelligence
More in AI Basics.
Last verified: October 2026
Sources
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, National Institute of Standards and Technology, July 2024
- Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y.), order imposing sanctions, June 22, 2023
- British Columbia Civil Resolution Tribunal, decision in Moffatt v. Air Canada, 2024 BCCRT 149, dated February 14, 2024
- Moffatt v. Air Canada: misrepresentation by an AI chatbot, McCarthy Tetrault, February 2024