- Foundation
- About 15 min
- 5 practice questions
Model training and testing: training, validation and test sets and the workflow
Split data into training, validation and test sets, explain why a model is never judged on its training data, and follow the workflow from collecting data to monitoring a deployed model for data drift.
If something looks wrong, please tell us.
By the end of this lesson you can
- Explain the job of the training, validation and test sets and work out a split.
- Explain why a score on training data is falsely high.
- List the six workflow steps and explain why a deployed model is monitored for data drift.
If a teacher tests a student with the exact questions the student practised, a high score proves little: the student may simply have memorised the answers. Machine learning has the same problem, and the solution is the same: keep some questions back. This lesson explains how labelled data is divided, why the division matters, and the full journey of a model from raw data to daily use.
Three sets, three jobs
Before training, the labelled data is shuffled and divided so that no example sits in more than one part:
| Set | Used for | A typical share |
|---|---|---|
| Training set | Learning: the model adjusts its learned numbers (its parameters, such as the slope of a line) to fit these examples | About 70% |
| Validation set | Comparing models and choosing settings fixed before training (hyperparameters), such as k in k-NN or the depth of a tree | About 15% |
| Test set | One final, honest check on examples the model has never seen | About 15% |
Worked example 1: making a split
A courier company has 2,000 deliveries labelled 'on time' or 'late'. With a 70/15/15 split: training = 70% of 2,000 = 1,400; validation = 15% of 2,000 = 300; test = 15% of 2,000 = 300. Check: 1,400 + 300 + 300 = 2,000. After training and tuning, the final model gets 255 of the 300 test deliveries right. Test accuracy = 255 ÷ 300 × 100 = 85%. This is an honest estimate of how it may perform on future deliveries, because those 300 played no part in building it.
2,000 labelled deliveries (70/15/15 split)
-
Training set: 1,400
- Learning: the model adjusts its parameters
-
Validation set: 300
- Comparing models and choosing hyperparameters
-
Test set: 300
- One final, honest check
- 255 of 300 right: 85% test accuracy
Why you never test on training data
A model can memorise its training examples. The extreme case is k-NN with k = 1: when asked about a training example, its nearest neighbour is that very example, so it simply repeats the stored label and can score close to 100% on the training data while telling you nothing about new cases. A training score measures memory; a test score measures learning. Two related traps:
- Data leakage: test examples, or copies of them, slip into the training set, so the test score is inflated. Removing duplicates before splitting prevents one common cause.
- Tuning on the test set: if you keep changing the model until the test score looks good, the test set has quietly become part of training. That is the validation set's job; the test set is used once, at the end.
Worked example 2: choosing with the validation set
Two models are built for the courier data. Model A scores 97% on training data and 80% on validation data. Model B scores 88% on training and 86% on validation. Choose by validation score, not training score: model B is better at examples it has not seen. Only model B is then run once on the test set, giving the final figure to report.
Train on the training set, tune on the validation set, judge once on the test set, and keep monitoring after deployment.
The machine learning workflow
The machine learning workflow
- Collect Data that matches the real task and people.
- Clean Fix errors, missing values, duplicates and labels.
- Train Train models; compare them on the validation set.
- Test Once, on the untouched test set.
- Deploy Put it into use in an app, website or office process.
- Monitor Keep checking results; retrain when they slip.
- Collect data that matches the real task and the people the model will serve.
- Clean it: fix errors, handle missing values, remove duplicates and check labels.
- Train one or more models on the training set, comparing them on the validation set.
- Test the chosen model once on the untouched test set, using measures that fit the problem.
- Deploy it: put it into use in an app, a website or an office process.
- Monitor it: keep checking its results, and retrain when they slip.
Data drift: why monitoring never stops
A model learns the world as it was when its data was collected, but the world changes. Spending habits shift, new pest varieties appear on crops, and spammers invent new wording. When the data a deployed model receives slowly moves away from its training data, its accuracy falls. This is data drift. Monitoring means comparing live results with the test results and checking whether incoming data still looks like the training data. Suppose a crop-disease app scored 90% on its test set, but checks on recent farmer photos show only 78% correct: the drop signals drift, and the model should be retrained on newer, labelled photos. (The figures are illustrative.)
Key takeaways: the training set teaches, the validation set chooses and the test set judges, once. A score on training data shows memory, not learning, and leaked or reused test data inflates scores. The workflow is collect, clean, train, test, deploy, monitor, and monitoring continues for as long as the model is in use, because data drifts.
Check your understanding
-
1. A school has 500 labelled answer sheets and keeps 20% aside as a test set, with no separate validation set. How many sheets are used for training?
- 100
- 250
- 480
- 400
Show answer
Answer: D. Test set = 20% of 500 = 100 sheets. Training set = 500 − 100 = 400 sheets.
-
2. Why is a model checked on a test set instead of on its training data?
- To see how it performs on examples it has never seen
- Because the training data is always wrong
- Because the test set is always larger
- To make the model train faster
Show answer
Answer: A. A model can memorise its training examples, so a score on them looks better than its real performance. The test set is kept out of training, so it gives an honest estimate of how the model handles new data.
-
3. A team wants to choose the best value of k for k-NN by comparing several values. Which data should it use to compare them?
- The training set
- The test set
- The validation set
- Any rows chosen at random from all the data
Show answer
Answer: C. Comparing settings (hyperparameters) such as k is the validation set's job. Using the test set would make it part of model building, so it could no longer give an honest final score; the training set would favour models that memorise.
-
4. In the machine learning workflow, which step comes right after a model is deployed?
- Monitoring its performance over time
- Collecting the first data
- Cleaning the data
- Training the first model
Show answer
Answer: A. The workflow is collect, clean, train, test, deploy, monitor. After deployment the model must be monitored because real-world data can drift and reduce its accuracy.
-
5. A model is run once on a test set of 400 examples and gets 360 right. What is its test accuracy?
- 85%
- 90%
- 80%
- 36%
Show answer
Answer: B. Accuracy = correct ÷ total × 100 = 360 ÷ 400 × 100 = 90%.
Sign in to keep track of the lessons you finish.