• Foundation
  • About 16 min
  • 5 practice questions

Doing machine learning well: overfitting, bias, accuracy and the workflow

Recognise overfitting and underfitting, see how biased data and misleading accuracy cause harm, follow the six-step ML workflow and learn the key terms.

If something looks wrong, please tell us.

Building a model that scores well on paper is easy; building one that works fairly on real people is harder. This lesson covers the common ways ML goes wrong and the habits that prevent them. These ideas matter for exams and for judging any claim that 'the AI decided'.

Overfitting and underfitting

Overfitting happens when a model learns the training data too closely, including its noise and accidents, so it does very well on training data but poorly on new data. It is like a student who memorises last year's answers word for word and fails when the questions change. Underfitting is the opposite: the model is too simple to capture the real pattern, so it does poorly even on the training data, like a student who learned only the chapter titles.

Comparing training and test scores
SituationTraining scoreTest scoreTypical fixes
UnderfittingLowLowUse a more flexible model, add useful features, train longer
Good fitHighClose to trainingKeep it; check it on new data regularly
OverfittingVery highMuch lowerMore data, a simpler model, stop training earlier, limit tree depth

Worked example 1. Model X scores 99% on training data and 72% on test data: the large gap points to overfitting. Model Y scores 61% on training data and 60% on test data: both are low, which points to underfitting. Model Z scores 88% on training and 86% on test: a small gap and a good score suggest a good fit.

Bias in data

A model learns whatever patterns its data contains, including unfair ones. If a hiring model is trained on past decisions that favoured one group, it can learn to repeat that favour. If a voice system is trained mostly on speakers of one language or accent, it may work badly for others. If a face dataset has few darker-skinned faces, recognition can be less accurate for them. This is data bias. Remedies include collecting data that represents everyone the model will serve, checking results separately for each group, and keeping a person responsible for important decisions.

Why accuracy can mislead

Accuracy is the share of predictions that are correct. It is misleading when one class is rare.

Worked example 2. Out of 1,000 card transactions, 10 are fraud. A lazy model that always says 'not fraud' is right on 990 of them, so its accuracy is 990 ÷ 1,000 = 99%, yet it catches none of the frauds. Two better measures help. Precision asks: of the transactions the model flagged, how many were really fraud? Recall asks: of all the real frauds, how many did the model catch? Suppose a better model flags 20 transactions, of which 8 are real frauds. Precision = 8 ÷ 20 = 40%. Recall = 8 ÷ 10 = 80%. Its accuracy is lower than the lazy model's, but it is far more useful.

Three ways to score a fraud model

  • Accuracy
    • Share of predictions that are correct
    • Misleading when one class is rare
    • Lazy model: 990 ÷ 1,000 = 99%, but catches no frauds
  • Precision
    • Of the transactions flagged, how many were really fraud?
    • Better model: 8 ÷ 20 = 40%
  • Recall
    • Of all the real frauds, how many were caught?
    • Better model: 8 ÷ 10 = 80%

High training score + low test score = overfitting. Low on both = underfitting. For rare events, look at precision and recall, not accuracy alone.

The ML workflow

The ML workflow

  1. Collect Data that matches the real task and people.
  2. Clean Fix errors, missing values, duplicates and labels.
  3. Train Train models; tune settings on a validation set.
  4. Test Once, on the untouched test set.
  5. Deploy Put it into use in an app, website or office process.
  6. Monitor Keep checking results; retrain when needed.
  1. Collect data that matches the real task and the people it will serve.
  2. Clean it: fix errors, handle missing values, remove duplicates and check the labels.
  3. Train one or more models on the training set, tuning settings on a separate validation set.
  4. Test the chosen model once on the untouched test set, using measures that fit the problem.
  5. Deploy it: put it into use in an app, website or office process.
  6. Monitor it: keep checking its results, because the world changes and a model's accuracy can drift; retrain when needed.

Key terms glossary

TermMeaning
ParameterA number the model learns from data, such as a weight
HyperparameterA setting chosen by people before training, such as k in k-NN or the learning rate
Validation setData held back to compare models and tune hyperparameters, separate from the test set
EpochOne full pass through the training data
InferenceUsing a trained model to make predictions
Data driftThe real-world data slowly changing so the model becomes less accurate
OverfittingMemorising training data, doing badly on new data
UnderfittingToo simple to learn the pattern, doing badly everywhere

Key takeaways

  • Compare training and test scores to spot overfitting and underfitting.
  • Biased data produces biased models; check results for every group.
  • Choose measures that fit the problem, and keep monitoring a model after it goes live.

Check your understanding

  1. 1. A model scores 98% on its training data but only 65% on new test data. What is the most likely problem?

    1. Underfitting
    2. Overfitting
    3. The test set is too small to matter
    4. The model has no parameters
    Show answer

    Answer: B. A very high training score with a much lower test score means the model memorised the training data instead of learning a general pattern. That is overfitting. Underfitting would give low scores on both.

  2. 2. Out of 500 patients, 5 have a rare disease. A model always predicts 'healthy'. What is its accuracy?

    1. 1%
    2. 95%
    3. 100%
    4. 99%
    Show answer

    Answer: D. It is right on every healthy patient: 500 − 5 = 495. Accuracy = 495 ÷ 500 = 99%, even though it detects none of the 5 sick patients. This is why accuracy can mislead when one class is rare.

  3. 3. A fraud model flags 25 transactions; 10 of them are real frauds. There were 20 frauds in total. What is its recall?

    1. 50%
    2. 40%
    3. 25%
    4. 80%
    Show answer

    Answer: A. Recall = frauds caught ÷ all real frauds = 10 ÷ 20 = 50%. (Precision would be 10 ÷ 25 = 40%, a different measure.)

  4. 4. A loan model was trained only on applicants from cities. It performs badly for rural applicants. What is the main cause?

    1. Too many epochs
    2. The learning rate was zero
    3. The training data did not represent rural applicants
    4. The model was deployed too early in the day
    Show answer

    Answer: C. A model learns only from the examples it sees. With no rural applicants in the training data, it has not learned their patterns. This is data bias; the fix is representative data and checking results for each group.

  5. 5. In the ML workflow, which step comes right after a model is deployed?

    1. Monitoring its performance over time
    2. Collecting the first data
    3. Cleaning the data
    4. Training the first model
    Show answer

    Answer: A. The workflow is collect, clean, train, test, deploy, monitor. After deployment, the model must be monitored because real-world data can drift and reduce its accuracy.

Privacy · Terms · Refunds · Delivery