Most clinical AI tools are statistical prediction models: they take patient data and output an estimated risk, and statistics supplies the tests for whether that estimate can be trusted. Two checks matter most. Discrimination asks whether the model ranks sicker patients above healthier ones, and calibration asks whether a predicted risk of 20 percent matches about 20 events per 100 patients. The BMJ's TRIPOD+AI statement, published on April 16, 2024, set a 27-item reporting checklist for studies that use either regression or machine learning.
Key takeaways
- A model with a good ranking score can still mislead: two heart-risk models differed by 0.005 in AUC yet selected 110 and 206 men per 1,000 for treatment at the same threshold.
- Calibration has four levels, from average risk to every patient subgroup, and the strongest level is described by its authors as utopian.
- Low disease prevalence can make even an accurate test mostly wrong when it flags someone (illustrative calculation below).
- TRIPOD+AI replaced the 2015 checklist and asks for performance with confidence intervals, including for key subgroups.
Where statistics ends and AI begins
Logistic regression and a neural network answer the same kind of question: given age, lab values, and images, how likely is a diagnosis or outcome? The difference lies mostly in how flexibly the model fits the data, and flexibility raises the risk of fitting noise. For that reason the reporting guidance does not treat the two families separately. The TRIPOD+AI authors, led by Gary Collins and Karel Moons, wrote a single checklist, described in a BMJ paper, for prediction model studies that use regression or machine learning. It covers studies that develop a model, evaluate one, or do both, and it supersedes the 2015 TRIPOD checklist.
The checklist has 27 main items, which break into 52 subitems, plus a separate 13-item abstract checklist. It was built through literature reviews, a two-round Delphi survey, and a consensus meeting that included patient and public representatives. Fairness runs through it, with items on known health inequalities and on model performance across sociodemographic groups.
Discrimination is not calibration
A review in BMC Medicine by Ben Van Calster and colleagues (2019) argues that calibration receives too little attention and warns that poorly calibrated algorithms can be misleading and potentially harmful for clinical decisions. Its example compares two cardiovascular risk models. QRISK2-2011 had an AUC of 0.771 and was well calibrated. The NICE version of the Framingham model scored slightly higher on AUC, at 0.776, but overestimated risk. At a 20 percent treatment threshold, QRISK2 selected 110 men per 1,000 and the Framingham version selected 206. Same ranking quality, almost twice the number of men flagged.
| Level | What it checks | Typical tool |
|---|---|---|
| Mean calibration | Average predicted risk against the overall event rate | Calibration-in-the-large |
| Weak calibration | Whether risks are too extreme or too moderate | Intercept (target 0) and slope (target 1) |
| Moderate calibration | Predicted risk matches observed proportions across the range | Flexible calibration curve |
| Strong calibration | Match for every combination of predictors | Called a utopian goal by the authors |
A model that shrank when it moved
The same review walks through a model for obstructive coronary artery disease. It was developed on 5,677 patients, 31 percent of whom had the disease, and validated on 4,888 patients, 44 percent of whom had it. The validation AUC was 0.69, the calibration intercept was -1.04, and the slope was 0.63. An intercept below zero signals overestimated risk, and a slope below one signals predictions that are too extreme, a typical sign of overfitting. Refitting the coefficients improved calibration. The case shows why a model validated in one population cannot be assumed to carry over to another with a different disease rate.
The review also gives rough sizing guidance: a precise flexible calibration curve needs about 200 patients with the event and 200 without. With less data, only the intercept and slope can be assessed, and in some situations the authors say no model should be developed at all.
Why a flagged patient is often not sick
The next calculation is an illustrative example, not data from a study. Suppose a condition affects one in 100 patients, and a test has 90 percent sensitivity and 90 percent specificity. Among 10,000 patients, 100 have the condition and the test flags 90 of them. Of the 9,900 without it, 10 percent, or 990, are flagged wrongly. A flagged patient therefore has 90 chances in 1,080, or about 8.3 percent, of truly having the condition. Sensitivity and specificity describe the test; the share of flagged patients who are truly ill also depends on prevalence. This is the reason published evaluations should state the population and the disease rate.
Checklist for reading a clinical AI paper
- Is the outcome defined and was the model tested on patients it never saw?
- Are confidence intervals given, and for subgroups too?
- Does the paper show a calibration curve, intercept, and slope, or only AUC?
- Does the disease rate in the test data match the setting where the tool would be used?
- Was the study reported against TRIPOD+AI?
This article explains statistical concepts used in published research. It is not medical advice and does not assess any specific tool. Decisions about testing or treatment belong with a patient and a clinician.
FAQ
Is AI just statistics?
Prediction models in medicine, whether regression or machine learning, estimate probabilities from data, so statistical tests apply to both. Machine learning methods are more flexible and need larger samples and stricter validation.
What does an AUC of 0.77 mean?
It means the model ranks a randomly chosen patient with the outcome above a randomly chosen patient without it about 77 percent of the time. It says nothing about whether the predicted risk values are accurate.
What is overfitting?
Overfitting means a model has learned noise in its development data. According to the BMC Medicine review, overfitted models tend to give risks that are too extreme, overestimating high risks and underestimating low ones.
Does TRIPOD+AI make a model safe?
No. It is a reporting guideline. It helps readers judge a study by requiring the details, and it does not certify performance.
Related reading
- how Canada regulates AI in health care
- AI in drug delivery
- AI in organ transplantation
- AI and dietary supplement selection
More in AI in health.
Last verified: October 2026
Sources
- TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods, Collins GS, Moons KGM et al., BMJ, 2024
- Calibration: the Achilles heel of predictive analytics, Van Calster B et al., BMC Medicine, 2019