• Practice
  • About 14 min
  • 5 practice questions

Luck and variance: judging your answering policy over many papers

See how much a score can vary by chance alone, why a lucky or unlucky paper tells you little, and how to judge a policy over several papers.

If something looks wrong, please tell us.

Suppose you try a new policy in one mock and your score goes up by 3 marks. Is the policy better? Maybe, but maybe you were lucky. Marks from guessing are partly chance, and chance can move a single paper by several marks either way. This lesson measures that spread, so you know how big a change must be before it means something. All figures use an invented scheme: +1 for a right answer, -1/3 for a wrong one.

Blind guessing as a coin with four sides

Take 20 pure guesses, each with four options and a one-in-four chance of being right. The expected number right is 20 x 1/4 = 5. In this scheme the expected net score is 5 - 15 x 1/3 = 0. But the actual number right varies from paper to paper.

Net score of 20 blind guesses (chances rounded)
Right out of 20Net scoreChance
2-4.007%
3-2.6713%
4-1.3319%
5+0.0020%
6+1.3317%
7+2.6711%
8+4.006%

The net is -2.67 to +2.67 (between 3 and 7 right) in about 81% of cases, and outside that range in the rest. So 20 blind guesses can easily give a net of plus or minus 3 marks or more, with an average of zero.

Worked example 1: a lucky and an unlucky paper

  1. Step 1: Nitin guesses 20 questions blind and gets 8 right, 12 wrong.
  2. Step 2: his net from them is 8 - 12 x 1/3 = +4. The expected is 0, so he gained 4 marks by luck.
  3. Step 3: Pooja also guesses 20 blind and gets 2 right, 18 wrong.
  4. Step 4: her net is 2 - 18 x 1/3 = -4. She lost 4 marks by chance.
  5. Result: the same policy produced +4 and -4. Neither result says anything about whether guessing is a good policy.

How many papers do you need?

For 20 blind guesses the typical size of the spread (the standard deviation) is about 2.6 marks. If you average over several papers, the spread of the average shrinks: dividing by the square root of the number of papers. With 4 papers it is 2.6 / 2 = 1.3 marks. So a change in your average of about 3 marks over four papers is a stronger signal than a 3-mark jump in one.

Worked example 2: Qasim compares two policies

  1. Step 1: in one mock, policy A (attempt everything) beat policy B (skip blind ones) by 1.5 marks.
  2. Step 2: random spread from guessing alone can be about 2.6 marks, which is larger than 1.5.
  3. Step 3: so one mock cannot tell the policies apart.
  4. Step 4: he decides to use each policy on every second mock for six mocks, recording the net scores and the reasons for each answer.
  5. Result: he will compare averages, and trust the result only if the gap is clearly larger than the spread.

A single paper is a small sample. Judge a policy by its reasons and its averages over several papers, not by one score.

Key takeaways

  • Guessing results vary by chance from paper to paper.
  • The typical spread of results from a few guesses can be several marks.
  • Averages over several papers are steadier than one paper.
  • Judge a policy by the reasons for each answer as well as by the score.

Check your understanding

  1. 1. How many right answers are expected from 20 blind guesses, each with four options?

    1. 5
    2. 4
    3. 10
    4. 15
    Show answer

    Answer: A. 20 x 1/4 = 5.

  2. 2. Scheme +1, -1/3. Someone gets 6 right out of 20 blind guesses. What is the net?

    1. 0
    2. -1.33
    3. +1.33
    4. +2.67
    Show answer

    Answer: C. 6 - 14 x 1/3 = 6 - 4.67 = +1.33.

  3. 3. One mock shows a 2-mark gain from a new policy. What should you conclude?

    1. The new policy is now proven
    2. It may be luck; test again
    3. The new policy is plainly useless
    4. Luck cannot affect a mock score
    Show answer

    Answer: B. A 2-mark change is small compared with the spread chance can produce. More papers are needed.

  4. 4. Averaging over 4 papers reduces the typical spread of the average to

    1. The same
    2. Double
    3. One-quarter
    4. Half
    Show answer

    Answer: D. The spread falls by the square root of the number of papers. The square root of 4 is 2, so it halves.

  5. 5. The best way to judge an answering policy is

    1. By the best single paper you had
    2. By reasons and averages over papers
    3. By a friend's score on a mock
    4. By how confident you felt then
    Show answer

    Answer: B. One paper can mislead. Averages over several papers, and sound reasons for each choice, are better evidence.

Privacy · Terms · Refunds · Delivery