A model is calibrated when its stated confidence matches how often it is right: among answers given with 80 percent confidence, about 80 percent should be correct. A 2017 ICML study by Guo, Pleiss, Sun, and Weinberger found that modern deep neural networks were poorly calibrated, and a 2022 study of language models found that larger models can be well calibrated on some question formats and can estimate whether they know an answer. Uncertainty in AI is therefore measurable, and the measurement shows where outputs can and cannot be trusted.
Key takeaways
- Confidence scores from a model are only useful if they have been checked against real outcomes, a property called calibration.
- Guo et al. (2017) reported that modern neural networks are poorly calibrated, and that one-parameter temperature scaling fixed much of the gap on their tests.
- Kadavath et al. (2022) reported that larger language models are well calibrated on multiple-choice and true/false questions in the right format, with weaker results on new tasks.
- NIST's AI Risk Management Framework, released January 26, 2023, is a voluntary U.S. framework for managing AI risk, and measuring model behavior is one of its four functions.
- A confidence number never replaces human review in high-stakes settings.
Two kinds of uncertainty that get mixed up
Uncertainty in a prediction has at least two sources. One comes from the world: a coin flip, a patient's unpredictable response, or tomorrow's weather cannot be known exactly no matter how good the model is. The other comes from the model: too little training data, an input unlike anything seen before, or a poor fit. The first cannot be reduced by building a better system, while the second sometimes can.
Weather forecasting offers a familiar example of the first kind stated openly. The U.S. National Weather Service defines probability of precipitation as the likelihood of a measurable amount of liquid precipitation, 0.01 inch or more, at a given point in the forecast area during a specified period, normally 12 hours. A forecast of 30 percent is a statement about uncertainty, not a failure to decide. Most machine learning products hide this layer and show a single answer, which makes the output look more certain than the underlying math supports.
What calibration means and how it is checked
A classifier usually produces a score between 0 and 1 for each prediction. Calibration asks whether that score behaves like a probability. The common test groups predictions into equal-width confidence bins, compares the average confidence in each bin with the share of correct answers, and takes a weighted average of the gaps. Guo et al. called this summary expected calibration error (ECE) and used 15 bins.
The table below is an illustrative example with invented numbers, not measured data. It shows a model that is overconfident, which is the pattern Guo et al. described for modern networks.
| Confidence bin | Share of predictions | Average confidence | Actual accuracy | Gap |
|---|---|---|---|---|
| 0.5 to 0.7 | 10% | 0.62 | 0.55 | 0.07 |
| 0.7 to 0.9 | 30% | 0.82 | 0.70 | 0.12 |
| 0.9 to 1.0 | 60% | 0.97 | 0.85 | 0.12 |
Weighting each gap by its share of predictions gives an error of about 0.12. In plain terms, when this imaginary model says it is 97 percent sure, it is right 85 percent of the time. A user who reads the score at face value would trust it too much.
What the published research found
Guo and colleagues tested image and document classifiers and reported that, unlike networks from about a decade earlier, modern ones are poorly calibrated. They found that depth, width, weight decay, and batch normalization all influence calibration. Their simplest fix, temperature scaling, divides a network's raw outputs by one number fitted on held-out data. For a ResNet-110 on CIFAR-100, ECE fell from 16.53 percent to 1.26 percent in their results. Because the fix does not change which class is predicted, accuracy stays the same while the confidence becomes more honest.
Kadavath and colleagues, in "Language Models (Mostly) Know What They Know" (July 2022), examined text models. They reported that larger models are well calibrated on multiple-choice and true/false questions when the question is posed in a suitable format. Models could also propose an answer and then estimate the probability that it is correct, a measure the authors call P(True), and could be trained to predict whether they know an answer before attempting it, called P(IK). The authors noted that calibration was weaker on tasks different from the training tasks. Both results come from specific benchmarks and model sizes, and neither guarantees that a deployed chatbot's stated confidence is reliable.
Where uncertainty reaches people
Users meet model uncertainty in several forms: a fraud alert with a risk score, a transcription flagged as low confidence, a chatbot that answers fluently even when wrong, or a medical image tool that returns a probability. The risk is greatest when fluency is mistaken for knowledge. A text generator can produce a polished, wrong sentence with the same tone as a correct one.
NIST's AI Risk Management Framework, released on January 26, 2023, names four functions: govern, map, measure, and manage. NIST published a companion Generative AI Profile on July 26, 2024. The framework is voluntary, and it does not prescribe a single uncertainty metric.
Questions to ask about any confidence number
- Was the score checked against real outcomes on data resembling the intended use?
- Does the tool report calibration, such as ECE, or only accuracy?
- What happens on inputs unlike the test data, such as a new population or a different language?
- Is the score shown as a probability, a category, or an unlabeled bar?
- Who reviews low-confidence cases, and what is the process when the model is confidently wrong?
- Has the model been updated since the last calibration check?
This article offers general information about how AI systems express confidence. It is not professional advice for any medical, legal, or financial decision, and it does not recommend a product.
FAQ
Does a high confidence score mean an AI answer is correct?
Not by itself. The score is meaningful only if the model has been checked for calibration on similar data. Guo et al. showed that accuracy and calibration can diverge in modern networks.
Can a language model tell when it does not know something?
Partly. Kadavath et al. reported that larger models can estimate the chance that their own answers are correct and can be trained to predict whether they know an answer, but results were weaker on unfamiliar tasks.
Is calibration the same as accuracy?
No. Accuracy counts how often predictions are right. Calibration compares stated confidence with that rate. A model can be accurate and overconfident at the same time.
Is there a legal requirement to report AI uncertainty?
No single rule covers it. The NIST framework is voluntary, and requirements differ by sector and jurisdiction. Rules relevant to a specific product should be checked with the regulator or a qualified professional.
Related reading
- Predictive analytics and data-driven decisions
- How AI affects modern society
- The components of an AI system
- Well-known AI technologies
More in AI Basics.
Last verified: October 2026
Sources
- On Calibration of Modern Neural Networks, Guo, Pleiss, Sun, and Weinberger, ICML, 2017
- Language Models (Mostly) Know What They Know, Kadavath et al., arXiv, 2022
- Forecast terms, probability of precipitation, National Weather Service, NOAA
- AI Risk Management Framework, National Institute of Standards and Technology, 2023