Problems in AI research are hard to find because a model can score well for the wrong reasons: test data leaking into training, mislabeled test examples, or results that nobody can repeat. A 2023 review in the journal Patterns found data leakage errors in 17 scientific fields, affecting at least 294 papers. Common fixes are separating test data from the start, auditing benchmark labels, and publishing code and checklists.
Key takeaways
- Data leakage means information from outside the training set reaches the model, which inflates test scores.
- Kapoor and Narayanan counted at least 294 affected papers across 17 fields, calling the number a lower bound.
- A 2021 study of 10 widely used datasets estimated at least 3.3 percent label errors in their test sets.
- NeurIPS 2019 introduced a code submission policy, a reproducibility challenge, and a checklist for authors.
- A strong headline score is a claim to check, not a result to trust.
Why a good score can mislead
Most machine learning results are reported as one number on a test set: accuracy, error rate, or a similar measure. The number means something only if the test set is truly unseen, correctly labeled, and similar to the situations the model will face. Each of those conditions can fail quietly. Unlike a broken bridge, a flawed experiment still produces a clean table of results.
Data leakage
Leakage happens when the model gets access to information it would not have in real use. The authors of the Patterns review, Sayash Kapoor and Arvind Narayanan of Princeton University, published August 4, 2023, sort leakage into eight types. These range from textbook errors, such as failing to separate training and test data, to open problems, such as a test set that does not match the population a claim is about.
An illustrative example: a team scales every number in a dataset using the overall average, and only afterward splits the data into training and test sets. The test rows have already influenced the scaling, so the model has seen a faint trace of the answers. The effect can be small or large, and nothing in the final code looks wrong.
The same paper includes a case study of civil war prediction. Of 12 papers with available code and data, four claimed that complex machine learning models beat baseline models, and all four had leakage. After correction, the complex models generally did no better than decades-old logistic regression. The authors propose model info sheets, a 21-question template in which researchers argue that their models are free of leakage.
Errors in the test sets themselves
Benchmarks carry their own risk. Curtis Northcutt, Anish Athalye, and Jonas Mueller, in a paper at NeurIPS 2021 (Datasets and Benchmarks track), examined 10 popular datasets and estimated at least 3.3 percent label errors in their test sets, with the ImageNet validation set at 6 percent or more. When the authors corrected for the errors, model rankings shifted: ResNet-18 overtook ResNet-50 on ImageNet once the share of mislabeled test examples rose by 6 percent. A leaderboard difference of a fraction of a point may therefore reflect noise in the labels more than a better model. The point connects to computer vision systems, where such benchmarks are standard.
Results nobody can repeat
Reproducibility asks a plain question: can another group get the same result from the same materials? A report on the NeurIPS 2019 reproducibility program, by Joelle Pineau and seven co-authors, describes three parts of the initiative: a code submission policy, a community-wide reproducibility challenge, and a Machine Learning Reproducibility checklist included in paper submission. The report was submitted to arXiv on March 27, 2020 and was listed as forthcoming in the Journal of Machine Learning Research.
Where each problem tends to appear
| Problem | Where it hides | Check that catches it |
|---|---|---|
| Leakage | Data preparation steps run before the train-test split | Split first, then fit every step on training data only |
| Label errors | Benchmark test sets used for years | Audit a sample of test labels; report results on cleaned subsets |
| Non-representative test data | A test set that differs from real users | State the target population and test on it |
| Missing materials | Papers without code, data, or settings | Publish code, seeds, and configuration; use a checklist |
| Weak baselines | Comparisons to untuned simple models | Tune simple models as carefully as new ones |
Questions to ask of any AI research claim
- Was the test set separated before any processing?
- Does the test data resemble the people or situations the claim covers?
- Are code and data available, and has anyone else reproduced the result?
- Is the improvement larger than known label noise in the benchmark?
- Was a simple baseline given the same tuning effort?
FAQ
What is data leakage in machine learning?
It is the use of information during training that would not be available at prediction time, such as test rows shaping a preprocessing step. It makes a model look better than it will perform in practice.
Does leakage mean the researchers cheated?
Not necessarily. Many leakage types are honest mistakes in data handling, which is why the Patterns authors recommend checklists rather than assuming bad faith.
Are benchmark scores still useful?
Yes, with limits. They allow comparison, but label errors and test sets that differ from real use mean small gaps between models should be read cautiously.
What is a reproducibility checklist?
It is a list of items authors confirm when submitting a paper, such as whether code, data, and training settings are provided. NeurIPS used one in 2019.
Related reading
- Characteristics of hard AI problems
- Limits of toy problems in research
- Why the problem domain matters
- Computer vision applications and challenges
More in AI Basics.
Last verified: October 2026
Sources
- Kapoor S., Narayanan A., Leakage and reproducibility crisis in machine-learning-based science, Patterns, August 4, 2023
- Northcutt C., Athalye A., Mueller J., Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, NeurIPS Datasets and Benchmarks, 2021
- Pineau J. et al., Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program), arXiv:2003.12206, 2020