- Practice
- About 20 min
- 5 practice questions
Total probability and Bayes' theorem
Find the total probability of an event that can happen through several mutually exclusive routes, and reverse the condition with Bayes' theorem to find the probability of a cause given the result.
If something looks wrong, please tell us.
By the end of this lesson you can
- State and prove the theorem of total probability.
- Draw a probability tree for a two-stage experiment.
- State Bayes' theorem and use it to find P(Aᵢ/B) from P(B/Aᵢ).
- Interpret prior and posterior probabilities in everyday situations such as machines and tests.
Many situations happen in two stages: first one of several causes (a box, a machine, a condition) is selected, then a result is observed. Total probability goes forward, from causes to result. Bayes' theorem, named after the English statistician Thomas Bayes, goes backward: having seen the result, how likely is each cause?
Theorem of total probability
Let A₁, A₂, ..., Aₙ be mutually exclusive and exhaustive events (they split S into pieces) and let B be any event. Then B = (A₁ ∩ B) ∪ (A₂ ∩ B) ∪ ... ∪ (Aₙ ∩ B), and these pieces are mutually exclusive. Adding their probabilities and using the multiplication theorem on each:
P(B) = P(A₁) P(B/A₁) + P(A₂) P(B/A₂) + ... + P(Aₙ) P(B/Aₙ)
A two-stage experiment as a tree
-
Cause A₁, probability P(A₁)
- B happens: P(B/A₁)
- Route probability P(A₁) P(B/A₁)
-
Cause A₂, probability P(A₂)
- B happens: P(B/A₂)
- Route probability P(A₂) P(B/A₂)
-
Cause Aₙ, probability P(Aₙ)
- B happens: P(B/Aₙ)
- Route probability P(Aₙ) P(B/Aₙ)
Multiply along each branch, then add the branches that end in B. That sum is P(B).
Worked example 1: two boxes
Box I has 3 white and 2 black balls; Box II has 2 white and 6 black balls. A box is chosen at random and a ball is drawn from it. Find the probability that the ball is white. Let A₁, A₂ = choosing Box I, Box II, so P(A₁) = P(A₂) = 1/2, and let W = white. P(W/A₁) = 3/5 and P(W/A₂) = 2/8 = 1/4. So P(W) = (1/2)(3/5) + (1/2)(1/4) = 3/10 + 1/8 = 12/40 + 5/40 = 17/40.
Bayes' theorem
With the same A₁, ..., Aₙ (mutually exclusive, exhaustive, each P(Aᵢ) > 0) and P(B) > 0, the definition gives P(Aᵢ/B) = P(Aᵢ ∩ B)/P(B). The multiplication theorem gives the numerator, P(Aᵢ ∩ B) = P(Aᵢ) P(B/Aᵢ), and total probability gives the denominator:
P(Aᵢ/B) = P(Aᵢ) P(B/Aᵢ) / [P(A₁) P(B/A₁) + P(A₂) P(B/A₂) + ... + P(Aₙ) P(B/Aₙ)]
In words: the probability of a cause given the result is its own branch divided by the total of all branches leading to that result. The P(Aᵢ) are called prior probabilities (known before the experiment) and the P(Aᵢ/B) posterior probabilities (revised after seeing B). The posterior probabilities of all the causes add up to 1.
Worked example 2: back to the boxes
In Worked example 1 the ball drawn is white. Find the probability that it came from Box I. P(A₁/W) = (1/2)(3/5) / (17/40) = (3/10) × (40/17) = 12/17. Before the draw each box had probability 1/2; seeing a white ball makes Box I, which has more white balls, much more likely. Then P(A₂/W) = 5/17, and 12/17 + 5/17 = 1.
Worked example 3: three machines
In a factory (made-up figures), machines A, B and C make 50%, 30% and 20% of the output, and 2%, 3% and 5% of their respective items are defective. (i) Total probability that an item picked at random is defective: P(D) = (0.5)(0.02) + (0.3)(0.03) + (0.2)(0.05) = 0.010 + 0.009 + 0.010 = 0.029. (ii) An item is found defective. P(A/D) = 0.010/0.029 = 10/29, P(B/D) = 9/29, P(C/D) = 10/29. Machine C makes only a fifth of the output, but its high defect rate makes it as likely a source as machine A.
| Machine | Prior P(Aᵢ) | P(D/Aᵢ) | Branch P(Aᵢ)P(D/Aᵢ) | Posterior P(Aᵢ/D) |
|---|---|---|---|---|
| A | 0.5 | 0.02 | 0.010 | 10/29 |
| B | 0.3 | 0.03 | 0.009 | 9/29 |
| C | 0.2 | 0.05 | 0.010 | 10/29 |
| Total | 1 | 0.029 | 1 |
Worked example 4: a screening test
Suppose 1% of a group have a certain condition. A test is positive for 95% of those who have it, and also (wrongly) positive for 10% of those who do not (made-up figures). If a person tests positive, what is the probability that they have the condition? Branches ending in positive: has the condition, 0.01 × 0.95 = 0.0095; does not, 0.99 × 0.10 = 0.099. Total P(positive) = 0.1085. So P(condition/positive) = 0.0095/0.1085 = 95/1085 = 19/217 ≈ 0.088. Less than 9%: because the condition is rare, most positive results come from the large group without it. Bayes' theorem makes this surprising fact clear.
Key takeaways: the causes must be mutually exclusive and exhaustive; total probability adds the branch products P(Aᵢ)P(B/Aᵢ); Bayes' theorem divides one branch by that total to reverse the condition; a table of priors, branch products and posteriors keeps the working tidy and lets you check that the posteriors add to 1.
Check your understanding
-
1. A₁ and A₂ are mutually exclusive and exhaustive with P(A₁) = 0.6, P(A₂) = 0.4, P(B/A₁) = 0.1 and P(B/A₂) = 0.3. What is P(B)?
- 0.4
- 0.18
- 0.2
- 0.12
Show answer
Answer: B. P(B) = (0.6)(0.1) + (0.4)(0.3) = 0.06 + 0.12 = 0.18. Adding 0.1 + 0.3 = 0.4 ignores the weights P(A₁) and P(A₂).
-
2. With the same data, what is P(A₂/B)?
- 2/3
- 1/3
- 0.3
- 0.12
Show answer
Answer: A. Bayes: P(A₂/B) = (0.4)(0.3)/0.18 = 0.12/0.18 = 2/3. Note that 0.3 is P(B/A₂), the condition the other way round.
-
3. Urn I has 2 red and 3 blue balls; urn II has 4 red and 1 blue. A fair coin decides the urn, and one ball is drawn. What is the probability that it is red?
- 7/10
- 2/5
- 4/5
- 3/5
Show answer
Answer: D. P(red) = (1/2)(2/5) + (1/2)(4/5) = 1/5 + 2/5 = 3/5. The values 2/5 and 4/5 are the chances within one urn only, P(red/urn I) and P(red/urn II).
-
4. Which condition on the events A₁, ..., Aₙ is needed for the theorem of total probability and Bayes' theorem?
- They are independent.
- They are mutually exclusive and exhaustive.
- They are equally likely.
- Each has probability 1/n.
Show answer
Answer: B. The proofs need A₁, ..., Aₙ to split the sample space with no overlaps and nothing left out: mutually exclusive and exhaustive. They need not be equally likely or independent.
-
5. In the urn question above, the ball drawn is red. What is the probability that it came from urn I?
- 2/5
- 2/3
- 1/3
- 1/5
Show answer
Answer: C. P(urn I/red) = (1/2)(2/5)/(3/5) = (1/5)/(3/5) = 1/3. Then P(urn II/red) = 2/3, and the two add to 1.
Sign in to keep track of the lessons you finish.