An AI program differs from traditional software in where its behavior comes from: a conventional program does what its code specifies, while a machine learning system behaves according to patterns learned from data, so the data and the trained model have to be tested alongside the code. A 2017 paper by Google engineers proposes 28 specific tests and monitoring needs for that reason, grouped into four areas: data, model, infrastructure, and monitoring.
Key takeaways
- In conventional software, behavior is written down as rules; in machine learning, it is inferred from training data and cannot be fully specified in advance.
- The ML Test Score paper (Breck and colleagues, 2017) lists 28 tests across four areas and scores a system by its weakest area.
- In the paper's survey of several dozen Google teams, none of the tests was implemented by more than 80 percent of teams.
- An overall accuracy figure can hide failures for a subgroup, so quality is checked on slices of data.
- Ongoing maintenance is a defining cost: a 2015 paper describes hidden technical debt in ML systems.
Where the behavior comes from
A payroll calculator follows rules that a developer wrote: if the input is X, the output is Y. A reviewer can read the code, list the cases, and confirm each one. A spam filter or image classifier works differently. Its developers supply examples and a learning algorithm, and the program derives its own decision rules from them. The result can be accurate on cases it has never seen, but no one wrote down what it should do for every input.
The ML Test Score paper, published at the IEEE Big Data conference in 2017, draws an analogy to compilation. Training is like compiling, except that the source is both code and training data. By that analogy, data deserves testing like code, and a trained model needs the practices a compiled binary needs: debuggability, rollbacks, and monitoring. The authors say a model's prediction behavior is difficult to specify a priori, which is why they build the rubric around checks on data and behavior rather than a list of expected outputs.
What the 28 tests cover
The paper groups its tests into four areas of seven each. Generic engineering practice, such as unit test coverage and a clear release process, is deliberately left out, because the rubric targets what is specific to ML. The table gives three examples per area, quoted in short form from the paper's own lists.
| Area | What it checks | Example tests |
|---|---|---|
| Features and data | The inputs the model learns from | Feature expectations are captured in a schema; the data pipeline has appropriate privacy controls; all input feature code is tested |
| Model development | The model itself | Offline and online metrics correlate; a simpler model is not better; quality is sufficient on important data slices |
| ML infrastructure | Training and serving machinery | Training is reproducible; models are canaried before serving; serving models can be rolled back |
| Monitoring | Behavior after release | Training and serving are not skewed; models are not too stale; prediction quality has not regressed |
A traditional program has analogues of some of these, such as reproducible builds and rollback. The data tests and the slice and staleness tests have no close equivalent, because a conventional program does not age when the world around it changes.
Why one accuracy number is not enough
The paper gives an illustration of the slicing problem. Global accuracy could improve by 1 percent while accuracy for one country drops by 50 percent, and a single summary metric would hide the drop. Slices are meant to separate subsets that might behave differently, for example users by country or movies by genre. This is an illustrative example from the paper, not a measured result.
The same logic matters outside engineering teams. A vendor statement that a product is "95 percent accurate" says little until the question of which data, which groups, and which time period has an answer.
Scoring and how common these practices were
The rubric awards points per test and takes the minimum score across the four areas as the overall result, so a strong pipeline cannot offset an untested data stage. Scores between zero and one suggest serious holes in reliability, and scores above three suggest strong automated testing and monitoring suitable for mission-critical systems. The paper is candid that the practices are not universal: in a survey of several dozen Google teams, none of the tests was implemented by more than 80 percent of them, and the authors interviewed 36 teams while building the rubric. The findings describe one company's systems in 2017, so they say nothing about current industry-wide practice.
Maintenance as a separate cost
An earlier paper, Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015, D. Sculley and nine co-authors), argues that fast ML gains carry ongoing maintenance costs. It names risk factors specific to ML, including boundary erosion, entanglement, hidden feedback loops, undeclared consumers, and data dependencies. In plain terms, a model can quietly start feeding on its own outputs, or other systems can come to depend on its predictions without anyone recording the link. Both papers come from a large company's experience with continuously retrained systems, and smaller or one-time models may face fewer of these problems.
The ML Test Score publication page lists the 2017 authors as Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley.
FAQ
Is AI software just software with more math?
It is software, but the behavior depends on training data as well as code, which changes how it is tested, released, and monitored.
Can an AI program be fully tested like a calculator?
Not in the same way. The set of possible inputs is too large to list, so tests rely on representative data, data slices, and monitoring after release.
Does a trained model stay correct once released?
Not necessarily. The ML Test Score includes a test on model staleness, since the world that produced the training data can change.
Do these papers apply to every AI product?
They describe supervised systems at Google, trained continuously and serving predictions. The authors say most recommendations also apply to other ML systems, but they do not claim coverage of every kind of AI.
Related reading
- Older AI methods compared with machine learning
- Limited memory AI and its constraints
- How AI programs handle large data sets
- Whether AI can exceed human capabilities
More in AI Tools and Industry.
Last verified: October 2026
Sources
- Breck E, Cai S, Nielsen E, Salib M, Sculley D, The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction, IEEE Big Data, 2017
- Sculley D et al., Hidden Technical Debt in Machine Learning Systems, Advances in Neural Information Processing Systems 28, 2015