Training a machine learning model means adjusting it on example data until its errors shrink, then measuring it on examples it has never seen. The standard way to run that measurement is to split the data into three sets: a training set the model learns from, a validation set used to compare choices, and a test set held back for a final check. Google's machine learning course illustrates a split of roughly 70 percent, 15 percent, and 15 percent.
Key takeaways
- Testing a model on the data it trained on is, in the scikit-learn documentation's words, a methodological mistake, because a model that repeats its training labels scores perfectly and still fails on new cases.
- Overfitting means the model memorizes the training set so closely that it predicts poorly on new data.
- Two causes recur: a training set that does not represent real data, and a model that is too complex.
- A test set that is used again and again "wears out," so fresh data is needed to restore an honest check.
- Duplicates between training and test data spoil the result, since the only fair test is on new examples.
The three data sets and their jobs
The training set is where the model learns. The validation set gives early feedback while the model is being built, and engineers use it to compare settings and design changes. The test set is touched last, as a double-check once the validation results look good. Google's course calls a two-way split of training and test data a decent idea, but recommends three sets because the validation set absorbs the repeated trial and error that would otherwise contaminate the test set.
The same course gives three conditions for a good validation or test set: it is large enough to give statistically meaningful results, it represents the dataset and the real-world data the model will meet, and it contains no examples that also appear in the training set.
Overfitting and how it shows up
Google's course defines overfitting as creating a model that matches the training set so closely that it fails to make good predictions on new data. The opposite goal is generalization, a model that predicts well on data it has not seen. A loss curve plots the model's error against training iterations. When the training curve keeps falling while the validation curve starts to rise, the two curves diverge, which signals overfitting. The course names two broad causes: a training set that does not adequately represent real-life data, and a model that is too complex.
The scikit-learn documentation adds a subtler problem, leakage. Tuning settings repeatedly against the test set lets knowledge of that set "leak" into the model, so the reported score no longer reflects generalization. Preprocessing steps such as standardizing values should also be learned from the training set only and then applied to held-out data.
A worked example with made-up numbers
The table below is an illustrative example. It supposes a class has 200 labeled photos of leaves from two tree species and wants to train a simple classifier. The counts follow the 70/15/15 pattern shown in Google's course; no real dataset or result is implied.
| Set | Photos | Used for |
|---|---|---|
| Training | 140 | Fitting the model |
| Validation | 30 | Comparing settings, such as how complex the model should be |
| Test | 30 | One final score, reported once |
If the same tree was photographed from three angles, all three photos belong in the same set. Otherwise near-duplicates sit on both sides of the split and the test score looks better than it should.
When a small dataset cannot spare a third set
Scikit-learn notes that partitioning into three sets sharply cuts the data available for learning, and that results can depend on one random choice of split. Its alternative is k-fold cross-validation: the training data is divided into k parts, the model is trained on k minus one parts and checked on the remaining one, and the process repeats so that every part serves once as the check. The reported performance is the average. The documentation says this can be computationally expensive but does not waste much data, which matters for small samples. A separate test set is still held out for the final evaluation.
A checklist for a school project
- Decide the split before looking at results, and write it down.
- Keep the test set sealed until the final report.
- Remove duplicates and near-duplicates across sets.
- Compare the training score with the validation score; a wide gap points to overfitting.
- Use only data that students and families have agreed to share, and leave personal details out of datasets.
- Report the size of each set next to the score, so readers can judge how much to trust it.
FAQ
What is the difference between a validation set and a test set?
The validation set is used repeatedly during building to compare choices. The test set is used once at the end. Using the test set for tuning defeats its purpose.
Is a higher training score always better?
No. A very high training score with a much lower validation score is the typical sign of overfitting. The score on unseen data is the one that counts.
How much data is needed to split into three sets?
No fixed number applies. The sets must be large enough to give statistically meaningful results. When data is scarce, cross-validation makes better use of what exists.
Why not train on all the data and skip the test set?
A model that simply repeats the labels it has seen would earn a perfect score and still fail on new cases, so an unseen test set is the only way to learn how it behaves in use.
Related reading
- Where AI learning datasets come from
- Whether AI really learns
- Traditional and modern AI methods
- Pattern recognition in machine learning
More in AI in education.
Last verified: October 2026
Sources
- Datasets: Dividing the original dataset, Machine Learning Crash Course, Google for Developers, 2026
- Overfitting, Machine Learning Crash Course, Google for Developers, 2026
- Cross-validation: evaluating estimator performance, scikit-learn documentation, 2026