Health AI is fair to the extent that it performs comparably across the groups it is used on, and published studies show that this often fails without anyone intending it. A widely used U.S. risk algorithm underestimated the needs of Black patients because it predicted spending instead of illness, and chest X-ray models underdiagnosed several under-served groups. Regulators in Canada, the United States, and the United Kingdom, and the World Health Organization (WHO), now ask developers to test for these gaps before and after deployment.
Key takeaways
- A 2019 Science study found that a commercial care-management algorithm used health care cost as a stand-in for need, which understated illness in Black patients.
- A 2021 Nature Medicine study found that chest X-ray classifiers more often labeled sick patients from several under-served groups as healthy.
- WHO lists inclusiveness and equity as one of six principles for health AI, and warns that systems trained mainly on data from high-income countries may not work elsewhere.
- Health Canada, the FDA, and the U.K. regulator jointly listed representative data and post-deployment monitoring among 10 good practice principles in 2021.
When the label is the problem: a cost-based risk score
Health systems use prediction tools to decide which patients with complex needs get extra support. In a study published in Science on October 25, 2019, Ziad Obermeyer and colleagues examined one such commercial algorithm. At the same risk score, Black patients were considerably sicker than White patients, judged by signs of uncontrolled illness. The authors estimated that fixing the disparity would raise the share of Black patients receiving additional help from 17.7 percent to 46.5 percent.
The cause was the prediction target. The algorithm predicted future health care costs, not illness. Because unequal access to care means less money is spent on Black patients than on White patients with the same needs, cost looked like a good proxy by some accuracy measures while carrying a large racial bias. The authors described the algorithm as typical of an industry-wide approach affecting millions of patients, and they concluded that convenient proxies for the true outcome can be an important source of algorithmic bias. The study does not claim that every cost-based tool behaves this way.
When the data are the problem: chest X-ray models
A second case concerns imaging. Seyyed-Kalantari and colleagues, writing in Nature Medicine on December 10, 2021, tested chest X-ray classifiers on three large public datasets: MIMIC-CXR with 371,858 images, CheXpert with 223,648, and ChestX-ray14 with 112,120. The models consistently underdiagnosed, meaning they labeled sick patients as healthy, at higher rates for female patients, Black patients, Hispanic patients, patients aged 0 to 20, and patients on Medicaid insurance, which the authors used as a proxy for lower socioeconomic status. Rates were higher still for intersectional groups, such as Hispanic female patients.
The result was not uniform. In one dataset, the largest underdiagnosis rates appeared in male patients and in patients over 80, and the authors suggested small samples or dataset selection as possible reasons. These were public research datasets and research models, so the study shows what can happen, not how often it happens in clinical products.
Where bias can enter and what is asked
| Stage | Documented example | Check named by regulators or WHO |
|---|---|---|
| Choosing what to predict | Cost used in place of illness | Define the intended use and test performance for the groups affected |
| Building the dataset | Different underdiagnosis rates by sex, race, age, and insurance | Make clinical study participants and datasets representative of the intended patient population |
| Moving to a new setting | Models trained mostly on high-income country data may not suit other settings | Reflect the diversity of settings in design and assess the system in real use |
| After launch | Performance can change as data and practice change | Monitor deployed models and manage the risks of retraining |
What the guidance says
In its June 28, 2021 guidance on AI ethics and governance in health, WHO sets out six principles. One is ensuring inclusiveness and equity, which means designing and sharing AI so it can be used as widely and equitably as possible regardless of age, sex, income, ethnicity, or other characteristics. The report also warns of biases encoded in algorithms.
On October 27, 2021, Health Canada, the FDA, and the U.K. Medicines and Healthcare products Regulatory Agency published ten guiding principles for good machine learning practice in medical devices. The third asks that clinical study participants and datasets be representative of the intended patient population, with attention to characteristics such as age, sex, race, and ethnicity, so that results generalize, bias is managed, and underperformance can be seen. The tenth asks that deployed models be monitored. The International Medical Device Regulators Forum built on this work in a final document released in January 2025. These are principles for developers, not a guarantee about any specific product.
Questions to ask about a health AI tool
- What exactly does the model predict, and is that a direct measure of need or a proxy such as cost?
- Were performance results reported separately for sex, age, race, ethnicity, and insurance status?
- Does the patient population in the evidence match the population where the tool is used?
- Who monitors results after launch, and how are problems reported?
The studies and documents summarized here are background reading, not guidance for any individual patient, and no product is assessed. Anyone with a personal health question should take it to a clinician.
FAQ
Does the Science study mean all health algorithms are biased?
No. It examined one commercial algorithm and showed how one design choice, predicting cost, produced a large gap. It also offers a general lesson about proxies, which can be checked tool by tool.
Is a model that scores well on accuracy automatically fair?
No. In the Science study, cost looked like an effective proxy for health by some measures of predictive accuracy, yet large racial biases arose. Accuracy overall and accuracy within each group are different questions.
Which groups did the chest X-ray study find were underdiagnosed?
Female patients, Black patients, Hispanic patients, patients aged 0 to 20, and patients on Medicaid insurance, with higher rates for intersectional groups. One of the three datasets showed a different pattern.
Do these principles carry legal force?
The principles from WHO and the three device regulators are guidance. Legal duties depend on the country, the product, and whether it is regulated as a medical device, so a specific case needs a qualified professional.
Related reading
- How Canada regulates health AI
- AI and patient care
- AI in nursing education
- AI and equity in special education
More in AI in health.
Last verified: October 2026
Sources
- Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366(6464), 2019
- Seyyed-Kalantari L, et al. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine 27(12), 2021
- WHO guidance on ethics and governance of AI in health, World Health Organization, June 2021
- Good Machine Learning Practice for Medical Device Development: Guiding Principles, Health Canada, U.S. FDA, and MHRA, October 2021
- Good machine learning practice for medical device development, International Medical Device Regulators Forum, January 2025