19 The Final Word on Exam Questions

The statistics of exam questions are unlikely to tax the brains of eminent statisticians for very long. Let's say we want to choose 150 questions randomly from a set of, say, 900 if that's how many we have. Now take one question, the classic query about, among other things, how many angels can realistically be persuaded to dance on the head of a pin. Let's call this, for convenience, question P (P for pin). Every so often, when choosing the question set for our exam, question P will no doubt appear; but how often?

Take the first exam cycle; if we choose 150 questions truly randomly from one large set of 900, then the chance of our question P appearing in the exam is precisely 150/900 or 1 in 6 (or 16.67 % if you like). Put another way, over time it will appear once in every six exams. This means, of course, that if you keep on taking the exam again and again, five times out of every six any effort you have put into remembering the answer to question P (the answer is 14 incidentally) will have been totally wasted.

But hold on; the situation has changed. You now know the answer, so we want it to appear next time. At the next exam cycle, the chances of P appearing are once again 16.67 (but the cumulative probability of the two successive appearances, given that it already had only a 16.67 % of appearing last time) are much less … let's say 1 in 36. Extending this out, the probability of P turning up in three successive exam cycles are getting pretty thin and in four successive cycles, miniscule at best. The odds against you are awful … it's looking like your valuable knowledge of the P answer will most of the time be wasted.