• Foundation
  • About 22 min
  • 5 practice questions

Bayes' theorem

Reverse a conditional probability with Bayes' theorem to find the probability of a cause given an observed result, interpret prior and posterior probabilities, and meet the idea of a random variable.

If something looks wrong, please tell us.

The theorem of total probability goes forwards: from causes (which bag, which machine) to the chance of a result (a red ball, a defective item). Often we need to go backwards: we have seen the result and ask which cause is responsible. A red ball was drawn; how likely is it that it came from bag I? The answer is given by Bayes' theorem, named after Thomas Bayes.

Statement and proof

Let E₁, E₂, …, Eₙ be a partition of S, and A an event with P(A) ≠ 0. By the definition of conditional probability and the multiplication theorem, P(Eᵢ|A) = P(Eᵢ ∩ A)/P(A) = P(Eᵢ) P(A|Eᵢ)/P(A). Replacing P(A) by the theorem of total probability gives:

Bayes' theorem: P(Eᵢ|A) = P(Eᵢ) P(A|Eᵢ) / [P(E₁) P(A|E₁) + P(E₂) P(A|E₂) + … + P(Eₙ) P(A|Eₙ)], for i = 1, 2, …, n.

The events Eᵢ are called hypotheses. P(Eᵢ) is the prior (a priori) probability of Eᵢ, before the result is known; P(Eᵢ|A) is the posterior (a posteriori) probability, after A has been observed. Bayes' theorem is also called the formula for the probability of causes. In words: posterior = (that route's product) ÷ (sum of all routes' products).

Applying Bayes' theorem

  1. List the hypotheses E₁, …, Eₙ (a partition) and their priors P(Eᵢ)
  2. Write the likelihoods P(A|Eᵢ) of the observed result
  3. Multiply: P(Eᵢ) P(A|Eᵢ) for each i
  4. Add the products to get P(A)
  5. Posterior P(Eᵢ|A) = its product ÷ P(A)

Worked examples

Worked example 1: which bag?

Using the bags of lesson 4 (bag I: 3 red, 2 black; bag II: 2 red, 4 black; each chosen with probability 1/2), a red ball is drawn. Find the probability that it came from bag I. Products: bag I, (1/2)(3/5) = 3/10; bag II, (1/2)(1/3) = 1/6. Sum P(red) = 7/15. P(bag I | red) = (3/10) ÷ (7/15) = (3/10)(15/7) = 9/14. The red ball has raised the chance of bag I from 1/2 to 9/14, because bag I has a larger share of red balls.

Worked example 2: which machine?

In the factory of lesson 4 (machines A, B, C making 50%, 30%, 20%, with 2%, 3%, 4% defective), an item is found defective. Find the probability that machine C made it. P(C | D) = 0.008/0.027 = 8/27 ≈ 0.30. Although C makes only 20% of the items, it accounts for about 30% of the defective ones. Similarly P(A | D) = 10/27 and P(B | D) = 9/27; the three posteriors add to 1.

Worked example 3: a screening test (illustrative numbers)

Suppose 1% of a population has a certain condition. A test gives a positive result for 95% of people who have it, and also, wrongly, for 5% of people who do not. A person tests positive. What is the probability that the person has the condition? Hypotheses: D (has it), P(D) = 0.01; H (does not), P(H) = 0.99. Products: (0.01)(0.95) = 0.0095 and (0.99)(0.05) = 0.0495. Sum: 0.059. P(D | positive) = 0.0095/0.059 = 95/590 = 19/118 ≈ 0.16. Most positive results come from the large healthy group. This is why a low prior matters so much, and why a positive screening result is usually followed by further tests.

Prior and posterior in worked example 3
HypothesisPriorLikelihood of a positive testProductPosterior
Has the condition0.010.950.009519/118 ≈ 0.16
Does not have it0.990.050.049599/118 ≈ 0.84
Total10.0591

Worked example 4: which coin?

A box holds one fair coin and one coin with heads on both sides. A coin is picked at random and tossed, and it shows a head. Find the probability that it is the two-headed coin. Products: two-headed, (1/2)(1) = 1/2; fair, (1/2)(1/2) = 1/4. Sum 3/4. P(two-headed | head) = (1/2)/(3/4) = 2/3. If the toss had shown a tail, the coin would certainly be the fair one, since the two-headed coin cannot show a tail.

A note on random variables

A random variable is a real-valued function whose domain is the sample space of a random experiment. For example, toss a coin twice, with S = {HH, HT, TH, TT}. If X is the number of tails, then X(HH) = 0, X(HT) = 1, X(TH) = 1, X(TT) = 2. Several random variables can be defined on one sample space: Y = (number of heads) − (number of tails) gives Y(HH) = 2, Y(HT) = Y(TH) = 0, Y(TT) = −2.

Key takeaways

  • Bayes' theorem finds the probability of a cause Eᵢ given an observed result A.
  • Posterior = P(Eᵢ) P(A|Eᵢ) ÷ Σ P(Eⱼ) P(A|Eⱼ); the denominator is the total probability of A.
  • Priors matter: a rare cause stays fairly unlikely even after favourable evidence.
  • The posteriors over a partition always add to 1, which is a good check.
  • A random variable assigns a real number to each outcome of an experiment.

Check your understanding

  1. 1. Bag I has 2 red and 3 black balls; bag II has 4 red and 1 black ball. A bag is chosen at random and a red ball is drawn. What is the probability that it came from bag II?

    1. 1/2
    2. 2/5
    3. 2/3
    4. 4/5
    Show answer

    Answer: C. Products: bag I (1/2)(2/5) = 1/5; bag II (1/2)(4/5) = 2/5. P(red) = 3/5. P(bag II | red) = (2/5)/(3/5) = 2/3.

  2. 2. Machines P and Q make 60% and 40% of the output, with 1% and 2% defective. A defective item is found. What is the probability that Q made it?

    1. 2/5
    2. 4/7
    3. 3/7
    4. 1/2
    Show answer

    Answer: B. Products: P, (0.6)(0.01) = 0.006; Q, (0.4)(0.02) = 0.008. Total 0.014. P(Q | defective) = 0.008/0.014 = 4/7.

  3. 3. In Bayes' theorem, the probabilities P(E₁), P(E₂), …, P(Eₙ) of the hypotheses before the result is known are called

    1. posterior probabilities
    2. conditional probabilities of A
    3. likelihoods
    4. prior probabilities
    Show answer

    Answer: D. P(Eᵢ) are the prior (a priori) probabilities. P(Eᵢ|A), found after observing A, are the posterior probabilities, and P(A|Eᵢ) are the likelihoods.

  4. 4. A box holds one fair coin and one two-headed coin. A coin is picked at random and tossed, and it shows a tail. What is the probability that it is the fair coin?

    1. 1/2
    2. 1/3
    3. 2/3
    4. 1
    Show answer

    Answer: D. Products: fair (1/2)(1/2) = 1/4; two-headed (1/2)(0) = 0. P(fair | tail) = (1/4)/(1/4 + 0) = 1. A two-headed coin cannot show a tail.

  5. 5. Assertion (A): If 1% of a population has a condition, and a test is positive for 95% of those who have it and for 5% of those who do not, then a person who tests positive has the condition with probability 0.95. Reason (R): In Bayes' theorem the posterior probability depends on the prior probability as well as on the likelihood.

    1. Both A and R are true, and R is the correct explanation of A
    2. Both A and R are true, but R is not the correct explanation of A
    3. A is true, but R is false
    4. A is false, but R is true
    Show answer

    Answer: D. R is true. Because of the low prior of 0.01, the posterior is (0.01)(0.95)/[(0.01)(0.95) + (0.99)(0.05)] = 0.0095/0.059 ≈ 0.16, not 0.95. So A is false; 0.95 is P(positive | condition), not P(condition | positive).

Sign in to keep track of the lessons you finish.

Next lesson → Final self-assessment: Probability

Privacy · Terms · Refunds · Delivery