A positive test does not mean what you think it means

Opening

Here is a positive medical test.

The test is pretty good at detecting the disease.

So what is the chance that the person who tested positive actually has it?

Most of us want to answer from the quality of the test. Ninety per cent accurate? Then perhaps something like 90 per cent.

That can be spectacularly wrong.

The idea

Psychologists and statisticians have used versions of this problem for decades because it exposes a small trap in the way probability is usually presented.

Take a classic mammography exercise used in research on statistical reasoning. Its numbers are deliberately simplified.

Imagine 1,000 women of the specified age taking part in routine screening.

Ten have breast cancer.

Of those ten, eight receive a positive mammogram.

The remaining 990 do not have breast cancer. But 95 of them also receive a positive result.

Now forget percentages for a moment and count people.

There are 103 positive results in total.

Eight are women who have cancer.

Ninety-five are women who do not.

So, using those particular numbers, a woman drawn from the positive-result group has a probability of about 8 in 103, roughly 8 per cent, of actually having the disease.

Nothing has gone wrong with the arithmetic. The thing that disappeared in the intuitive answer was the starting population.

The disease was uncommon in the example. Even a relatively small false-positive rate applied to the much larger group without the disease creates a lot of positive results.

This is the base rate, and ignoring it can make a number sound as if it means something it does not.

What changes when you look at it this way

There is another nice twist.

Researchers including Gerd Gigerenzer found that people often reason much better when probabilities are turned into what they call natural frequencies.

Compare these two ways of hearing the same sort of information.

One version gives you prevalence, sensitivity and a false-positive rate as percentages. You have to keep several conditional probabilities in your head and work out how they fit together.

The other says: start with 1,000 people. Ten have the disease. Eight of those test positive. Of the 990 who do not have it, 95 also test positive.

Suddenly you can almost see the answer.

The mathematics did not become easier because someone removed the difficult bits. The same information was rearranged into a form our brains seem to handle more naturally.

That is worth remembering beyond medical tests.

A percentage can be perfectly accurate and still leave you with a terrible picture of what is happening.

Sometimes the quickest way through a wall of impressive-looking numbers is embarrassingly simple.

Ask how many actual people we are talking about.