• Foundation
  • About 16 min
  • 5 practice questions

Overfitting and evaluation: accuracy, the confusion matrix, precision and recall

Spot overfitting and underfitting from training and test scores, see why accuracy misleads when one class is rare, and fill in a confusion matrix to calculate accuracy, precision and recall correctly.

If something looks wrong, please tell us.

By the end of this lesson you can

  • Recognise overfitting and underfitting from training and test scores, and name a fix for each.
  • Explain why accuracy misleads when one class is rare.
  • Fill in a confusion matrix and calculate accuracy, precision and recall from it.

Building a model that scores well on paper is easy; building one that works on new cases is harder. This lesson covers the two classic ways a model fails to generalise and the measures used to judge a model honestly. These ideas also help you question any claim that 'the AI is 99% accurate'.

Overfitting and underfitting

Overfitting happens when a model learns the training data too closely, including its noise and accidents, so it does very well on training data but poorly on new data. It is like a student who memorises last year's answers word for word and fails when the questions change. Underfitting is the opposite: the model is too simple to capture the real pattern, so it does poorly even on the training data, like a student who learned only the chapter titles.

Reading training and test scores together
SituationTraining scoreTest scoreTypical fixes
UnderfittingLowLowA more flexible model, more useful features, longer training
Good fitHighClose to trainingKeep it; monitor it on new data
OverfittingVery highMuch lowerMore data, a simpler model, earlier stopping, limiting tree depth

Worked example 1. Model X scores 99% on training data and 72% on test data: the large gap points to overfitting. Model Y scores 61% on training and 60% on test: both are low, which points to underfitting. Model Z scores 88% on training and 86% on test: a small gap and a good score suggest a good fit.

Why accuracy misleads on unbalanced data

Accuracy is the share of all predictions that are correct. It misleads when one class is rare, which is common: frauds, rare diseases and defective parts are all rare. Out of 1,000 card transactions, suppose 10 are fraud. A lazy model that always says 'not fraud' is right on 990, so its accuracy is 990 ÷ 1,000 = 99%, yet it catches none of the frauds. We need measures that look at the rare class directly.

The confusion matrix

A confusion matrix counts the four possible outcomes of a yes/no prediction. Call the rare class we are hunting for 'positive' (here, fraud):

The four outcomes in a confusion matrix

  • Predicted fraud
    • True positive (TP): really fraud
    • False positive (FP): really genuine, a false alarm
  • Predicted genuine
    • False negative (FN): really fraud, a missed fraud
    • True negative (TN): really genuine
  • True positive (TP): predicted fraud, really fraud.
  • False positive (FP): predicted fraud, really genuine (a false alarm).
  • False negative (FN): predicted genuine, really fraud (a missed fraud).
  • True negative (TN): predicted genuine, really genuine.

Accuracy = (TP + TN) ÷ total. Precision = TP ÷ (TP + FP): of those flagged, how many were right? Recall = TP ÷ (TP + FN): of the real positives, how many were caught?

Worked example 2: a better fraud model

A better model flags 20 of the 1,000 transactions, and 8 of those 20 are real frauds. Fill the matrix step by step: TP = 8. FP = 20 − 8 = 12. There were 10 frauds, so FN = 10 − 8 = 2. TN = 1,000 − 8 − 12 − 2 = 978.

Confusion matrix for the better model
ItemPredicted fraudPredicted genuineTotal
Actually fraudTP = 8FN = 210
Actually genuineFP = 12TN = 978990
Total209801,000

Accuracy = (8 + 978) ÷ 1,000 = 986 ÷ 1,000 = 98.6%. Precision = 8 ÷ (8 + 12) = 8 ÷ 20 = 40%. Recall = 8 ÷ (8 + 2) = 8 ÷ 10 = 80%. Compare the lazy model: TP = 0, FN = 10, so its recall is 0 ÷ 10 = 0%, and its precision cannot even be calculated because it flags nothing. The better model has slightly lower accuracy (98.6% against 99%) but catches 8 of the 10 frauds. Accuracy alone would have picked the useless model.

Precision or recall: which matters more?

It depends on which mistake costs more. In screening for a serious disease, a missed patient (a false negative) is costly, so recall matters most; a false alarm only leads to a further test by a doctor. In a spam filter, hiding a genuine email such as an interview call letter (a false positive) is costly, so precision matters most. Raising one usually lowers the other, so the choice is a decision for people, not the model. A single score that balances the two, the F1 score, is also widely used.

Precision or recall?

  • Precision TP ÷ (TP + FP): of those flagged, how many were right?
    • Matters most when a false positive is costly
    • Spam filter: hiding a genuine email such as an interview call letter
  • Recall TP ÷ (TP + FN): of the real positives, how many were caught?
    • Matters most when a false negative is costly
    • Disease screening: a missed patient
Raising one usually lowers the other, so the choice is a decision for people.

Key takeaways: a big gap between training and test scores signals overfitting; low scores on both signal underfitting. Accuracy can look excellent while a model misses every rare case. Build the confusion matrix first, then calculate precision = TP ÷ (TP + FP) and recall = TP ÷ (TP + FN), and choose the measure that matches the costlier mistake.

Check your understanding

  1. 1. A model scores 98% on its training data but only 65% on new test data. What is the most likely problem?

    1. Underfitting
    2. Overfitting
    3. The test set is too small to matter
    4. The model has no parameters
    Show answer

    Answer: B. A very high training score with a much lower test score means the model memorised the training data instead of learning a general pattern. That is overfitting. Underfitting would give low scores on both.

  2. 2. Out of 500 patients, 5 have a rare disease. A model always predicts 'healthy'. What is its accuracy?

    1. 1%
    2. 95%
    3. 100%
    4. 99%
    Show answer

    Answer: D. It is right on every healthy patient: 500 − 5 = 495. Accuracy = 495 ÷ 500 = 99%, even though it detects none of the 5 sick patients, so its recall is 0%. This is why accuracy misleads when one class is rare.

  3. 3. A fraud model flags 25 transactions; 10 of them are real frauds. There were 20 frauds in total. What is its recall?

    1. 50%
    2. 40%
    3. 25%
    4. 80%
    Show answer

    Answer: A. Recall = frauds caught ÷ all real frauds = TP ÷ (TP + FN) = 10 ÷ 20 = 50%. Precision would be 10 ÷ 25 = 40%, a different measure.

  4. 4. A confusion matrix shows TP = 30, FP = 10, FN = 20 and TN = 940. What is the precision?

    1. 60%
    2. 75%
    3. 97%
    4. 40%
    Show answer

    Answer: B. Precision = TP ÷ (TP + FP) = 30 ÷ (30 + 10) = 30 ÷ 40 = 75%. Recall would be 30 ÷ (30 + 20) = 60%, and accuracy (30 + 940) ÷ 1,000 = 97%; neither is precision.

  5. 5. For a screening test for a serious disease, where every positive result is checked again by a doctor, which measure matters most?

    1. Accuracy, because the disease is rare
    2. Precision, because false alarms are the worst mistake
    3. Recall, because missing a sick person is the costliest mistake
    4. Training score, because it is the highest
    Show answer

    Answer: C. A missed patient (false negative) may go untreated, while a false alarm only leads to a further check. Recall measures how many real cases are caught, so it matters most here. Accuracy misleads for rare diseases, and a training score says nothing about new patients.

Privacy · Terms · Refunds · Delivery