- Foundation
- About 16 min
- 5 practice questions
Overfitting and evaluation: accuracy, the confusion matrix, precision and recall
Spot overfitting and underfitting from training and test scores, see why accuracy misleads when one class is rare, and fill in a confusion matrix to calculate accuracy, precision and recall correctly.
If something looks wrong, please tell us.
By the end of this lesson you can
- Recognise overfitting and underfitting from training and test scores, and name a fix for each.
- Explain why accuracy misleads when one class is rare.
- Fill in a confusion matrix and calculate accuracy, precision and recall from it.
Building a model that scores well on paper is easy; building one that works on new cases is harder. This lesson covers the two classic ways a model fails to generalise and the measures used to judge a model honestly. These ideas also help you question any claim that 'the AI is 99% accurate'.
Overfitting and underfitting
Overfitting happens when a model learns the training data too closely, including its noise and accidents, so it does very well on training data but poorly on new data. It is like a student who memorises last year's answers word for word and fails when the questions change. Underfitting is the opposite: the model is too simple to capture the real pattern, so it does poorly even on the training data, like a student who learned only the chapter titles.
| Situation | Training score | Test score | Typical fixes |
|---|---|---|---|
| Underfitting | Low | Low | A more flexible model, more useful features, longer training |
| Good fit | High | Close to training | Keep it; monitor it on new data |
| Overfitting | Very high | Much lower | More data, a simpler model, earlier stopping, limiting tree depth |
Worked example 1. Model X scores 99% on training data and 72% on test data: the large gap points to overfitting. Model Y scores 61% on training and 60% on test: both are low, which points to underfitting. Model Z scores 88% on training and 86% on test: a small gap and a good score suggest a good fit.
Why accuracy misleads on unbalanced data
Accuracy is the share of all predictions that are correct. It misleads when one class is rare, which is common: frauds, rare diseases and defective parts are all rare. Out of 1,000 card transactions, suppose 10 are fraud. A lazy model that always says 'not fraud' is right on 990, so its accuracy is 990 ÷ 1,000 = 99%, yet it catches none of the frauds. We need measures that look at the rare class directly.
The confusion matrix
A confusion matrix counts the four possible outcomes of a yes/no prediction. Call the rare class we are hunting for 'positive' (here, fraud):
The four outcomes in a confusion matrix
-
Predicted fraud
- True positive (TP): really fraud
- False positive (FP): really genuine, a false alarm
-
Predicted genuine
- False negative (FN): really fraud, a missed fraud
- True negative (TN): really genuine
- True positive (TP): predicted fraud, really fraud.
- False positive (FP): predicted fraud, really genuine (a false alarm).
- False negative (FN): predicted genuine, really fraud (a missed fraud).
- True negative (TN): predicted genuine, really genuine.
Accuracy = (TP + TN) ÷ total. Precision = TP ÷ (TP + FP): of those flagged, how many were right? Recall = TP ÷ (TP + FN): of the real positives, how many were caught?
Worked example 2: a better fraud model
A better model flags 20 of the 1,000 transactions, and 8 of those 20 are real frauds. Fill the matrix step by step: TP = 8. FP = 20 − 8 = 12. There were 10 frauds, so FN = 10 − 8 = 2. TN = 1,000 − 8 − 12 − 2 = 978.
| Item | Predicted fraud | Predicted genuine | Total |
|---|---|---|---|
| Actually fraud | TP = 8 | FN = 2 | 10 |
| Actually genuine | FP = 12 | TN = 978 | 990 |
| Total | 20 | 980 | 1,000 |
Accuracy = (8 + 978) ÷ 1,000 = 986 ÷ 1,000 = 98.6%. Precision = 8 ÷ (8 + 12) = 8 ÷ 20 = 40%. Recall = 8 ÷ (8 + 2) = 8 ÷ 10 = 80%. Compare the lazy model: TP = 0, FN = 10, so its recall is 0 ÷ 10 = 0%, and its precision cannot even be calculated because it flags nothing. The better model has slightly lower accuracy (98.6% against 99%) but catches 8 of the 10 frauds. Accuracy alone would have picked the useless model.
Precision or recall: which matters more?
It depends on which mistake costs more. In screening for a serious disease, a missed patient (a false negative) is costly, so recall matters most; a false alarm only leads to a further test by a doctor. In a spam filter, hiding a genuine email such as an interview call letter (a false positive) is costly, so precision matters most. Raising one usually lowers the other, so the choice is a decision for people, not the model. A single score that balances the two, the F1 score, is also widely used.
Precision or recall?
-
Precision
TP ÷ (TP + FP): of those flagged, how many were right?
- Matters most when a false positive is costly
- Spam filter: hiding a genuine email such as an interview call letter
-
Recall
TP ÷ (TP + FN): of the real positives, how many were caught?
- Matters most when a false negative is costly
- Disease screening: a missed patient
Key takeaways: a big gap between training and test scores signals overfitting; low scores on both signal underfitting. Accuracy can look excellent while a model misses every rare case. Build the confusion matrix first, then calculate precision = TP ÷ (TP + FP) and recall = TP ÷ (TP + FN), and choose the measure that matches the costlier mistake.
Check your understanding
-
1. A model scores 98% on its training data but only 65% on new test data. What is the most likely problem?
- Underfitting
- Overfitting
- The test set is too small to matter
- The model has no parameters
Show answer
Answer: B. A very high training score with a much lower test score means the model memorised the training data instead of learning a general pattern. That is overfitting. Underfitting would give low scores on both.
-
2. Out of 500 patients, 5 have a rare disease. A model always predicts 'healthy'. What is its accuracy?
- 1%
- 95%
- 100%
- 99%
Show answer
Answer: D. It is right on every healthy patient: 500 − 5 = 495. Accuracy = 495 ÷ 500 = 99%, even though it detects none of the 5 sick patients, so its recall is 0%. This is why accuracy misleads when one class is rare.
-
3. A fraud model flags 25 transactions; 10 of them are real frauds. There were 20 frauds in total. What is its recall?
- 50%
- 40%
- 25%
- 80%
Show answer
Answer: A. Recall = frauds caught ÷ all real frauds = TP ÷ (TP + FN) = 10 ÷ 20 = 50%. Precision would be 10 ÷ 25 = 40%, a different measure.
-
4. A confusion matrix shows TP = 30, FP = 10, FN = 20 and TN = 940. What is the precision?
- 60%
- 75%
- 97%
- 40%
Show answer
Answer: B. Precision = TP ÷ (TP + FP) = 30 ÷ (30 + 10) = 30 ÷ 40 = 75%. Recall would be 30 ÷ (30 + 20) = 60%, and accuracy (30 + 940) ÷ 1,000 = 97%; neither is precision.
-
5. For a screening test for a serious disease, where every positive result is checked again by a doctor, which measure matters most?
- Accuracy, because the disease is rare
- Precision, because false alarms are the worst mistake
- Recall, because missing a sick person is the costliest mistake
- Training score, because it is the highest
Show answer
Answer: C. A missed patient (false negative) may go untreated, while a false alarm only leads to a further check. Recall measures how many real cases are caught, so it matters most here. Accuracy misleads for rare diseases, and a training score says nothing about new patients.
Sign in to keep track of the lessons you finish.