• Foundation
  • About 15 min
  • 5 practice questions

Bias and fairness: how unfair outcomes enter AI systems and how to reduce them

Identify how bias enters AI through data and design, recognise it in hiring, lending and face recognition, measure it with simple rates, and list ways to reduce it.

If something looks wrong, please tell us.

People often think a computer is neutral because it follows numbers, not feelings. But an AI system learns from data collected in the real world, and it is designed by people who make choices. If the data or the choices are unfair, the AI can repeat the unfairness - and repeat it at very large scale. This is called algorithmic bias. Understanding where it comes from is the first step to preventing it.

Bias in, bias out

  1. Unfair data or design choices Data from the real world and choices made by people.
  2. The AI learns the pattern
  3. Unfairness repeated at scale The system can repeat it at very large scale.

Where bias comes from

Common sources of bias in AI
Source of biasWhat happensSimple illustration
Historical biasThe data faithfully records an unfair pastPast hiring favoured one group, so the model learns to favour it
Representation (sampling) biasSome groups are missing or too few in the dataA face dataset with few photos of darker-skinned people
Measurement or label biasWhat is recorded is a poor stand-in for what mattersUsing 'number of complaints' as a measure of an employee's quality
Proxy variablesA harmless-looking feature closely tracks a protected traitResidential pin code standing in for community or income
Feedback loopsThe model's own outputs shape the next round of dataMore checks in one area find more cases there, which leads to even more checks
Design choicesThe goal the builders set ignores fairnessOptimising only for overall accuracy, ignoring errors for small groups

Three areas where bias matters

Hiring. A resume-screening model trained on past selections learns what past selectors liked. If those selections were skewed, for example towards men or towards graduates of a few colleges, the model may rank equally able candidates from other groups lower - even if gender is never an input, because words, hobbies or college names can act as proxies.

Loans and credit. A lending model may use location, type of phone or spending patterns. These can track income, caste or community indirectly. People with thin credit histories, such as many first-time borrowers or small farmers, may be scored poorly simply because the data has little about them.

Face recognition. Research has found that face recognition systems can be less accurate for groups that were under-represented in the photos used to train them. When such a system is used for identity checks or policing, a higher error rate for one group means more wrong matches and more wrongful suspicion for that group.

Worked example 1: comparing selection rates

Suppose a screening tool shortlists 60 of 200 applicants from Group A and 15 of 100 applicants from Group B. Working: Group A's selection rate = 60 ÷ 200 = 30%. Group B's rate = 15 ÷ 100 = 15%. The ratio of B's rate to A's = 15 ÷ 30 = 0.5, so Group B is shortlisted at half the rate of Group A. A gap this large does not by itself prove discrimination - the groups may differ in qualifications - but it is a clear warning sign that the tool must be investigated.

Worked example 2: comparing error rates

Suppose a face-matching system is tested on 1,000 photos from each of two groups. It makes 10 wrong matches for Group P and 50 for Group Q. Working: error rate for P = 10 ÷ 1,000 = 1%; for Q = 50 ÷ 1,000 = 5%. The overall error rate is 60 ÷ 2,000 = 3%, which sounds acceptable, but Group Q faces five times the error rate of Group P. This is why systems must be tested group by group, not only on average.

An average can hide an unfair gap

  • Group P
    • 10 wrong matches in 1,000 photos
    • Error rate 1%
  • Group Q
    • 50 wrong matches in 1,000 photos
    • Error rate 5% - five times that of Group P
  • Both groups together
    • 60 wrong matches in 2,000 photos
    • Error rate 3%, which sounds acceptable

Bias in, bias out. A system that is accurate on average can still be unfair to a group - always measure results separately for each group.

How to reduce bias

  1. Collect more balanced and representative data, with consent.
  2. Check features for proxies of protected traits; simply deleting a 'gender' or 'caste' column is not enough.
  3. Measure outcomes and error rates separately for each group before and after launch.
  4. Include people from different backgrounds in the team that designs and tests the system.
  5. Keep human review for serious decisions and give affected people a way to appeal.
  6. Audit the system regularly, because bias can appear later as data changes.

Fairness is not purely technical. Different definitions of fairness - equal selection rates, or equal error rates - can conflict, so people must decide which matters most for a given use and be open about that choice.

Key takeaways: bias enters through unfair historical data, missing groups, poor labels, proxy variables, feedback loops and design goals. It shows up in hiring, lending and face recognition. Detect it by comparing rates across groups; reduce it with better data, proxy checks, diverse teams, human review and regular audits.

Check your understanding

  1. 1. A hiring model is trained on ten years of past selections in which most selected candidates were men. It now ranks women lower. What is the main source of this bias?

    1. Too much computing power
    2. A slow internet connection
    3. Historical bias in the training data
    4. The colour scheme of the software
    Show answer

    Answer: C. The model learned from past decisions that were skewed towards men, so it repeats that pattern. This is historical bias in the data. Computing power, internet speed and screen colours do not decide whom the model favours.

  2. 2. A loan model does not use religion or caste, but it uses residential pin code, which in one city closely follows community lines. This is an example of:

    1. A proxy variable that can carry bias indirectly
    2. Data minimisation
    3. Encryption
    4. Explainability
    Show answer

    Answer: A. A pin code that closely tracks community acts as a stand-in (proxy) for a protected trait, so bias can enter even though caste and religion are not inputs. Data minimisation, encryption and explainability are protections, not sources of bias.

  3. 3. A face recognition system was trained mostly on photos of one group of people. What is it most likely to do?

    1. Be equally accurate for every group
    2. Work only in daylight
    3. Need no testing at all
    4. Make more errors for groups that were under-represented in the training photos
    Show answer

    Answer: D. A model learns best what it has seen most. Groups with few training photos are recognised less accurately, so they suffer more wrong matches. That is why balanced data and group-wise testing are needed.

  4. 4. What is the best first step to find out whether a deployed AI system is biased?

    1. Make the model bigger
    2. Measure and compare its outcomes and error rates for different groups
    3. Remove all human reviewers
    4. Hide the system's decisions from users
    Show answer

    Answer: B. Bias is detected by measuring results group by group, for example selection rates or error rates. A bigger model does not reveal bias, removing reviewers weakens oversight, and hiding decisions works against transparency.

  5. 5. A team deletes the 'gender' column from its data and declares the model fair. Why is this not enough?

    1. Deleting a column always makes a model illegal
    2. Deleting a column reduces accuracy to zero
    3. Other features such as hobbies, names or colleges can still act as proxies for gender
    4. Gender can never affect any decision
    Show answer

    Answer: C. Removing the column does not remove the pattern: other features correlated with gender can carry the same bias. Fairness must be checked by measuring outcomes. Deleting a column is not illegal and does not destroy accuracy.

Privacy · Terms · Refunds · Delivery