- Foundation
- About 22 min
- 5 practice questions
Bayes' theorem
Reverse a conditional probability with Bayes' theorem to find the probability of a cause given an observed result, interpret prior and posterior probabilities, and meet the idea of a random variable.
If something looks wrong, please tell us.
The theorem of total probability goes forwards: from causes (which bag, which machine) to the chance of a result (a red ball, a defective item). Often we need to go backwards: we have seen the result and ask which cause is responsible. A red ball was drawn; how likely is it that it came from bag I? The answer is given by Bayes' theorem, named after Thomas Bayes.
Statement and proof
Let E₁, E₂, …, Eₙ be a partition of S, and A an event with P(A) ≠ 0. By the definition of conditional probability and the multiplication theorem, P(Eᵢ|A) = P(Eᵢ ∩ A)/P(A) = P(Eᵢ) P(A|Eᵢ)/P(A). Replacing P(A) by the theorem of total probability gives:
Bayes' theorem: P(Eᵢ|A) = P(Eᵢ) P(A|Eᵢ) / [P(E₁) P(A|E₁) + P(E₂) P(A|E₂) + … + P(Eₙ) P(A|Eₙ)], for i = 1, 2, …, n.
The events Eᵢ are called hypotheses. P(Eᵢ) is the prior (a priori) probability of Eᵢ, before the result is known; P(Eᵢ|A) is the posterior (a posteriori) probability, after A has been observed. Bayes' theorem is also called the formula for the probability of causes. In words: posterior = (that route's product) ÷ (sum of all routes' products).
Applying Bayes' theorem
- List the hypotheses E₁, …, Eₙ (a partition) and their priors P(Eᵢ)
- Write the likelihoods P(A|Eᵢ) of the observed result
- Multiply: P(Eᵢ) P(A|Eᵢ) for each i
- Add the products to get P(A)
- Posterior P(Eᵢ|A) = its product ÷ P(A)
Worked examples
Worked example 1: which bag?
Using the bags of lesson 4 (bag I: 3 red, 2 black; bag II: 2 red, 4 black; each chosen with probability 1/2), a red ball is drawn. Find the probability that it came from bag I. Products: bag I, (1/2)(3/5) = 3/10; bag II, (1/2)(1/3) = 1/6. Sum P(red) = 7/15. P(bag I | red) = (3/10) ÷ (7/15) = (3/10)(15/7) = 9/14. The red ball has raised the chance of bag I from 1/2 to 9/14, because bag I has a larger share of red balls.
Worked example 2: which machine?
In the factory of lesson 4 (machines A, B, C making 50%, 30%, 20%, with 2%, 3%, 4% defective), an item is found defective. Find the probability that machine C made it. P(C | D) = 0.008/0.027 = 8/27 ≈ 0.30. Although C makes only 20% of the items, it accounts for about 30% of the defective ones. Similarly P(A | D) = 10/27 and P(B | D) = 9/27; the three posteriors add to 1.
Worked example 3: a screening test (illustrative numbers)
Suppose 1% of a population has a certain condition. A test gives a positive result for 95% of people who have it, and also, wrongly, for 5% of people who do not. A person tests positive. What is the probability that the person has the condition? Hypotheses: D (has it), P(D) = 0.01; H (does not), P(H) = 0.99. Products: (0.01)(0.95) = 0.0095 and (0.99)(0.05) = 0.0495. Sum: 0.059. P(D | positive) = 0.0095/0.059 = 95/590 = 19/118 ≈ 0.16. Most positive results come from the large healthy group. This is why a low prior matters so much, and why a positive screening result is usually followed by further tests.
| Hypothesis | Prior | Likelihood of a positive test | Product | Posterior |
|---|---|---|---|---|
| Has the condition | 0.01 | 0.95 | 0.0095 | 19/118 ≈ 0.16 |
| Does not have it | 0.99 | 0.05 | 0.0495 | 99/118 ≈ 0.84 |
| Total | 1 | 0.059 | 1 |
Worked example 4: which coin?
A box holds one fair coin and one coin with heads on both sides. A coin is picked at random and tossed, and it shows a head. Find the probability that it is the two-headed coin. Products: two-headed, (1/2)(1) = 1/2; fair, (1/2)(1/2) = 1/4. Sum 3/4. P(two-headed | head) = (1/2)/(3/4) = 2/3. If the toss had shown a tail, the coin would certainly be the fair one, since the two-headed coin cannot show a tail.
A note on random variables
A random variable is a real-valued function whose domain is the sample space of a random experiment. For example, toss a coin twice, with S = {HH, HT, TH, TT}. If X is the number of tails, then X(HH) = 0, X(HT) = 1, X(TH) = 1, X(TT) = 2. Several random variables can be defined on one sample space: Y = (number of heads) − (number of tails) gives Y(HH) = 2, Y(HT) = Y(TH) = 0, Y(TT) = −2.
Key takeaways
- Bayes' theorem finds the probability of a cause Eᵢ given an observed result A.
- Posterior = P(Eᵢ) P(A|Eᵢ) ÷ Σ P(Eⱼ) P(A|Eⱼ); the denominator is the total probability of A.
- Priors matter: a rare cause stays fairly unlikely even after favourable evidence.
- The posteriors over a partition always add to 1, which is a good check.
- A random variable assigns a real number to each outcome of an experiment.
Check your understanding
-
1. Bag I has 2 red and 3 black balls; bag II has 4 red and 1 black ball. A bag is chosen at random and a red ball is drawn. What is the probability that it came from bag II?
- 1/2
- 2/5
- 2/3
- 4/5
Show answer
Answer: C. Products: bag I (1/2)(2/5) = 1/5; bag II (1/2)(4/5) = 2/5. P(red) = 3/5. P(bag II | red) = (2/5)/(3/5) = 2/3.
-
2. Machines P and Q make 60% and 40% of the output, with 1% and 2% defective. A defective item is found. What is the probability that Q made it?
- 2/5
- 4/7
- 3/7
- 1/2
Show answer
Answer: B. Products: P, (0.6)(0.01) = 0.006; Q, (0.4)(0.02) = 0.008. Total 0.014. P(Q | defective) = 0.008/0.014 = 4/7.
-
3. In Bayes' theorem, the probabilities P(E₁), P(E₂), …, P(Eₙ) of the hypotheses before the result is known are called
- posterior probabilities
- conditional probabilities of A
- likelihoods
- prior probabilities
Show answer
Answer: D. P(Eᵢ) are the prior (a priori) probabilities. P(Eᵢ|A), found after observing A, are the posterior probabilities, and P(A|Eᵢ) are the likelihoods.
-
4. A box holds one fair coin and one two-headed coin. A coin is picked at random and tossed, and it shows a tail. What is the probability that it is the fair coin?
- 1/2
- 1/3
- 2/3
- 1
Show answer
Answer: D. Products: fair (1/2)(1/2) = 1/4; two-headed (1/2)(0) = 0. P(fair | tail) = (1/4)/(1/4 + 0) = 1. A two-headed coin cannot show a tail.
-
5. Assertion (A): If 1% of a population has a condition, and a test is positive for 95% of those who have it and for 5% of those who do not, then a person who tests positive has the condition with probability 0.95. Reason (R): In Bayes' theorem the posterior probability depends on the prior probability as well as on the likelihood.
- Both A and R are true, and R is the correct explanation of A
- Both A and R are true, but R is not the correct explanation of A
- A is true, but R is false
- A is false, but R is true
Show answer
Answer: D. R is true. Because of the low prior of 0.01, the posterior is (0.01)(0.95)/[(0.01)(0.95) + (0.99)(0.05)] = 0.0095/0.059 ≈ 0.16, not 0.95. So A is false; 0.95 is P(positive | condition), not P(condition | positive).
Sign in to keep track of the lessons you finish.