An artificial neural network is a stack of layers of adjustable numbers, called weights, that is trained on labeled examples until its outputs match the labels. In medicine the examples are usually images. A 2017 Nature study trained one on 129,450 clinical images covering 2,032 diseases and reported performance on par with 21 board-certified dermatologists on two biopsy-confirmed tasks, and a 2022 Science Advances study showed that such models can fail on skin tones that training sets under-represent.
Key takeaways
- A network has no medical knowledge of its own: it adjusts weights until its predictions agree with the labels it is given, so the labels and images set the limit.
- The 2017 skin cancer network was trained directly from pixels and disease labels and was tested against 21 dermatologists on two binary tasks.
- A 2021 review of 70 dermatology AI studies found skin tone information reported in only 7 of them (10.0 percent).
- On the 656-image Diverse Dermatology Images set, tested models lost 27 to 36 percent of their ROC-AUC compared with their original results, and fine-tuning on that set closed the gap between light and dark skin.
- This article is general information about how the technology works. It is not medical advice, and it does not describe how any condition is diagnosed or treated.
How a network learns from labeled images
A medical image enters a neural network as a grid of numbers. Each layer combines the numbers from the layer before using its weights and passes the result on, and the last layer outputs a score for each label, such as "malignant" or "benign". At first the weights are arbitrary and the scores are poor. Training compares the scores with the labels attached to each image, measures the error, and nudges every weight in the direction that reduces it, repeated over the whole image set many times.
Two points follow. The network is only ever told what the label is, never why, so it can use any pattern in the pixels that predicts the label, including patterns unrelated to disease. And its score is a statement about images like those it was trained on. How well it does on images from a different clinic, camera, or patient group is a separate question that has to be tested.
A landmark test: skin lesions in 2017
Andre Esteva and colleagues published Dermatologist-level classification of skin cancer with deep neural networks in Nature in 2017. They trained a single convolutional neural network, a type that scans images with small sliding filters, end-to-end from images, using only pixels and disease labels as inputs. The training set held about 129,450 clinical images spanning 2,032 diseases, which the authors describe as roughly 100 times larger than earlier sets.
For testing, the network was compared with 21 board-certified dermatologists on biopsy-confirmed images in two binary tasks: keratinocyte carcinomas versus benign seborrheic keratoses, and malignant melanomas versus benign nevi. The network performed on par with all tested experts in both. The authors suggest that mobile devices could extend dermatologists' reach beyond the clinic, which is a hope and not a result of the study. A binary comparison on selected, biopsy-confirmed images is narrower than examining a patient.
What the training data decides
A 2021 scoping review in JAMA Dermatology by Roxana Daneshjou and colleagues examined 70 studies published from January 1, 2015 to November 1, 2020. Together the studies used 1,065,291 images to develop or test algorithms, and only 257,372 of them (24.2 percent) were publicly available. Patient race or ethnicity was described for at least one data set in 14 studies (20.0 percent), and skin tone information appeared in 7 (10.0 percent). Of 56 studies that developed new algorithms for skin cancers, 36 (64.3 percent) met the review's gold-standard labeling criteria.
The same group then tested what such gaps do. In Disparities in dermatology AI performance on a diverse, curated clinical image set, published in Science Advances in 2022, they built the Diverse Dermatology Images set of 656 pathologically confirmed images. It holds 208 images of Fitzpatrick skin types one and two, 241 of types three and four, and 207 of types five and six. Three published algorithms scored lower on it than in their original tests, with ROC-AUC (a score where 0.5 is chance and 1.0 is perfect) down 27 to 36 percent. Dermatologists also performed worse on the darker-skin and rare-disease images. Fine-tuning the models on the new images closed the gap between light and dark skin.
ROC-AUC on the diverse image set, by skin type
The table gives the values reported for the three algorithms in the 2022 paper. The algorithms are shown as numbers here, and their names are in the paper.
| Algorithm | All 656 images | Skin types one and two | Skin types five and six |
|---|---|---|---|
| 1 | 0.65 | 0.64 | 0.55 |
| 2 | 0.56 | 0.61 | 0.50 |
| 3 | 0.67 | 0.72 | 0.57 |
A score of 0.50 is no better than chance, so one algorithm was at chance on skin types five and six in this test. These figures describe a research test of three specific algorithms in 2022 and say nothing about any current product.
Questions to ask about a medical-image network
- Which labels were used, and were they confirmed by biopsy or by a clinician's opinion?
- How many images, from how many sites, devices, and patient groups?
- Was the test set kept apart from training, and was it collected somewhere else?
- Are results reported for each skin tone, age group, and sex, or only overall?
- Does the comparison with clinicians use the same images and the same two choices the network had?
FAQ
Does a neural network understand medicine?
No. It finds statistical patterns that link pixels to labels. The 2017 network used only pixels and disease labels, with no medical rules supplied.
Why would a model do worse on some skin tones?
The 2021 review found that most data sets did not report skin tone, and the 2022 study found that the tested models lost accuracy on a balanced set. Fine-tuning on that set closed the gap, which points to training data as a main cause, though the papers do not claim it is the only one.
Does matching dermatologists mean the network can replace them?
The 2017 study compared the network with experts on two binary tasks using selected images. Replacing a clinician involves the whole patient visit, which that comparison did not test.
Does a published result mean a tool is approved for patients?
No. A paper reports a research test. Whether a tool is authorized for patient care is a separate regulatory decision, and the papers cited here make no such claim.
Related reading
- How AI relates to machine learning
- Applications of artificial neural networks
- A practical guide with worked examples
- Alternatives to neural networks
More in AI in health.
Last verified: October 2026
Sources
- Dermatologist-level classification of skin cancer with deep neural networks, Esteva A et al., Nature 542, 2017
- Lack of Transparency and Potential Bias in Artificial Intelligence Data Sets and Algorithms: A Scoping Review, Daneshjou R et al., JAMA Dermatology 157(11), 2021
- Disparities in dermatology AI performance on a diverse, curated clinical image set, Daneshjou R et al., Science Advances 8, 2022