AI is not broken as a whole, but current systems fail in documented and fairly predictable ways: they state invented facts, perform unevenly across groups, and degrade when conditions differ from their training data. The practical rule is to match the amount of checking to the cost of an error, and to verify any claim that could affect money, health, legal standing, or safety.
Key takeaways
- Hallucination is partly a scoring problem: a 2025 paper argues that training and benchmarks reward confident guessing over admitting uncertainty.
- Accuracy can differ sharply by group; a 2018 audit of commercial face analysis found error rates up to 34.7% for darker-skinned women against 0.8% for lighter-skinned men.
- Models trained on one setting often do worse in another, a problem called distribution shift.
- Organizations that deploy AI are held responsible for its output, as a 2024 British Columbia tribunal ruling on an airline chatbot shows.
Why chatbots state things that are false
Language models generate the most plausible next words, and plausible is not the same as true. A paper by Kalai, Nachum, Vempala, and Zhang (arXiv, September 4, 2025) argues that hallucinations persist because most benchmarks give no credit for admitting uncertainty, so a model that guesses scores better than one that abstains. The authors suggest changing how existing leaderboards are scored rather than only adding new tests. A model can therefore sound certain while being wrong, and fluent wording says nothing about accuracy.
The consequences are not confined to trivia. In Moffatt v. Air Canada (2024 BCCRT 149, reported in February 2024), the airline's website chatbot told a customer that a bereavement fare could be claimed after travel, which contradicted the airline's own policy page. The British Columbia Civil Resolution Tribunal rejected the argument that the chatbot was separately responsible, treating it as part of the airline's website, and ordered CAD 812.02 in total (CAD 650.88 in damages, CAD 36.14 in interest, and CAD 125 in fees). It is one small claims decision, not a general rule of law.
Unequal accuracy and the data behind it
An AI system is only as even as its training and test data. The Gender Shades study (Buolamwini and Gebru, 2018) tested three commercial gender classification systems and found error rates of up to 34.7% for darker-skinned women, while the lowest error, for lighter-skinned men, reached at most 0.8%. The authors noted that two common benchmark datasets were about 79.6% and 86.2% lighter-skinned. The lesson is not that every system repeats this result today, but that a single overall accuracy figure can hide large gaps between groups. Reported results should be broken down by the groups the system will actually serve.
Brittleness and distribution shift
Image and language models can be thrown off by small changes. In a 2014 paper on adversarial examples, Goodfellow, Shlens, and Szegedy argued that neural networks' vulnerability to small, deliberately chosen input changes comes mainly from their near-linear behavior, and showed a fast way to generate such inputs. A broader problem is distribution shift. The WILDS benchmark (Koh and colleagues, 2020) collects ten datasets built from real deployment shifts, such as images from different hospitals or camera traps, and reports that standard training gives substantially lower accuracy on out-of-distribution data than on in-distribution data. A model validated in one hospital, season, or region should not be assumed to work in another without testing.
Uneven capability and measurement gaps
The Stanford 2026 AI Index describes a "jagged frontier": one model earned an International Mathematical Olympiad gold medal while the top model read analog clocks correctly only 50.1% of the time. The report also counts 362 documented AI incidents in 2025, up from 233 in 2024, though rising counts partly reflect more deployment and more reporting. It describes responsible AI benchmark reporting as spotty, and notes that improving one dimension, such as safety, can reduce another, such as accuracy. Strong scores on a benchmark therefore say little about performance on a specific task.
How organizations manage failure
The US National Institute of Standards and Technology published the AI Risk Management Framework 1.0 on January 26, 2023, organized around four functions: Govern, Map, Measure, and Manage. A Generative AI Profile followed on July 26, 2024. Both are voluntary. NIST's page states that AI RMF 1.0 is being revised as part of the White House AI Action Plan, so the current text should be checked before citing it. Typical measures include testing on realistic data, monitoring after release, documenting limits, and keeping a human responsible for consequential decisions.
A checklist for verifying AI output
The table below is practical guidance derived from the failure modes above, not a standard.
| What the output contains | Check to run |
|---|---|
| Names, quotes, and citations | Search for the exact source; confirm the author, title, and year exist and say what is claimed. |
| Numbers and statistics | Find the original table or report; confirm the year, unit, and population. |
| Laws, prices, and policies | Open the regulator's or company's own page and check the effective date. |
| Recent events | Confirm with a dated news source; a model may lack current information. |
| Medical, legal, or financial points | Treat as background only and confirm with a qualified professional. |
| Step-by-step reasoning or code | Run the code or redo one step independently; ask the tool for a different method and compare. |
FAQ
Can hallucinations be eliminated?
Current research describes them as reducible rather than gone. The 2025 paper by Kalai and colleagues attributes them to statistical pressures in training and to benchmarks that penalize uncertain answers, and it proposes scoring changes to reduce them.
Is a confident answer more likely to be correct?
Not reliably. Models often produce fluent, assured wording whether or not the content is accurate, so confidence of tone should not replace checking a source.
Who is responsible when an AI tool gives wrong information?
It depends on the jurisdiction and the facts. In the Moffatt decision, the tribunal held the airline responsible for what its chatbot said. Readers facing a real dispute should consult a lawyer rather than rely on one ruling.
Does AI failure mean it should not be used?
No. It means use should match the stakes. Drafting, summarizing, and brainstorming tolerate occasional errors if a person reviews the result, while decisions about health, money, or legal rights need verified sources.
Related reading
- Weighing the benefits and risks of AI
- Whether AI could surpass human intelligence
- A review of AI and human reasoning
- Can AI evolve and adapt?
More in AI Policy and Ethics.
Last verified: October 2026
Sources
- Kalai, A. T., Nachum, O., Vempala, S. S., Zhang, E., Why Language Models Hallucinate, arXiv:2509.04664, 2025
- National Institute of Standards and Technology, AI Risk Management Framework, 2023, with Generative AI Profile (NIST AI 600-1), 2024
- Buolamwini, J., Gebru, T., Gender Shades, Proceedings of Machine Learning Research 81, 2018
- Goodfellow, Shlens, and Szegedy, Explaining and Harnessing Adversarial Examples, arXiv:1412.6572, 2014
- Koh, P. W., et al., WILDS: A Benchmark of in-the-Wild Distribution Shifts, arXiv:2012.07421, 2020
- Stanford Institute for Human-Centered AI, 2026 AI Index Report, 2026
- Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, 2024 (award amounts as reported by Mondaq, February 2024)