• Practice
  • About 14 min
  • 5 practice questions

Confidence and calibration: using your honest chance of being right

Turn your confidence on each answer into an honest probability, correct it using your own record, and use the break-even rate to decide whether an attempt is worth making.

If something looks wrong, please tell us.

Expected-value arithmetic needs a probability: your chance of being right. But students are often not good at estimating it. Feeling 80% sure may mean you are right 65% of the time. The skill is calibration: making your stated confidence match your results. This lesson shows how to measure your calibration and how to apply it to the attempt decision. All schemes are invented.

The decision rule

For a question with chance p of being right, mark M and penalty P, the expected gain from attempting is p x M - (1 - p) x P. It is positive when p is above the break-even rate P / (M + P).

The same chance, different schemes
Invented schemeBreak-even pEV at p = 0.3EV at p = 0.5
+1, -0.2520%0.1250.375
+3, -125%0.2001.000
+2, -133%-0.1000.500
+1, -150%-0.4000.000

The same belief can mean attempt in one scheme and leave in another. At p = 0.3, the first two schemes gain on average and the last two lose.

Measuring calibration

Tag each practice answer with the confidence you felt: for example 'about 80%', 'about 60%', 'about 40%'. Afterwards group the answers by tag and work out the real hit rate in each group. A tag with a lower real rate than its label means you are overconfident at that level.

Worked example 1: Anil in a +1, -1 scheme

  1. Step 1: Anil gives 'about 60%' to 20 answers in practice and 9 of them are right.
  2. Step 2: his real hit rate is 9 / 20 = 45%, below his label of 60%.
  3. Step 3: at the label, the expected gain per answer is 0.6 x 1 - 0.4 x 1 = 0.2, which looks worth taking.
  4. Step 4: at the real rate, the gain is 0.45 x 1 - 0.55 x 1 = -0.10, which is a loss.
  5. Step 5: over 20 such answers the loss is about 20 x 0.10 = 2 marks.
  6. Result: feeling 60% sure was not enough to justify attempting in this scheme. His real break-even is 50%, and his 'about 60%' answers fall short of it. He now treats that tag as 'leave, unless I can narrow it further'.

Worked example 2: Zoya in a +2, -0.5 scheme

  1. Step 1: Zoya says 'about 80%' on 20 answers and 13 are right: 13 / 20 = 65%.
  2. Step 2: the break-even rate is 0.5 / (2 + 0.5) = 20%.
  3. Step 3: at the real rate of 65%, the gain per answer is 0.65 x 2 - 0.35 x 0.5 = 1.125, which is still well positive.
  4. Step 4: she keeps attempting these answers. Her overconfidence does not change the decision here, but it would matter in a harsher scheme.
  5. Result: calibration matters most when your honest rate is near the break-even rate.

Habits

  • Use a few levels only (for example sure, fairly sure, doubtful), so each group has enough answers to count.
  • Wait for at least 20 answers in a group before trusting its rate.
  • Update your rates every few papers.

Do not trust the feeling; trust the record. Attempt when your real hit rate at that confidence is above the break-even rate.

Key takeaways

  • Break-even p = penalty / (mark + penalty).
  • Compare your real hit rate for a confidence level with the break-even rate.
  • Overconfidence is common; the log corrects it.
  • Small groups give unreliable rates.

Check your understanding

  1. 1. Scheme +3, -1. What is the break-even chance of being right?

    1. 33%
    2. 20%
    3. 25%
    4. 50%
    Show answer

    Answer: C. Break-even = 1 / (3 + 1) = 0.25, which is 25%.

  2. 2. In an invented scheme (+1, -1), a student with a 45% real hit rate should, on average,

    1. Lose marks by attempting
    2. Gain marks by attempting
    3. Break even exactly
    4. Always get full marks
    Show answer

    Answer: A. The break-even rate is 50%, and 45% is below it. Each answer loses 0.10 on average.

  3. 3. A student tags 25 answers 'about 80%', and 15 are right. What is the real hit rate?

    1. 80%
    2. 75%
    3. 40%
    4. 60%
    Show answer

    Answer: D. 15 divided by 25 is 0.6, which is 60%.

  4. 4. Scheme +2, -0.5, chance of being right 0.4. What is the expected gain of attempting?

    1. 0.8
    2. 0.5
    3. 0.3
    4. 1.1
    Show answer

    Answer: B. 0.4 x 2 - 0.6 x 0.5 = 0.8 - 0.3 = 0.5.

  5. 5. Why wait for about 20 answers in a group before trusting its hit rate?

    1. Small groups can be thrown by luck
    2. Twenty answers is a rule of every exam
    3. Small groups never matter in practice
    4. Hit rates change from day to day
    Show answer

    Answer: A. With few answers a lucky or unlucky run can change the rate a great deal. More answers make the rate more stable.

Privacy · Terms · Refunds · Delivery