Risky business: doctors’ understanding of statistics

My criticism of the way test usefulness is portrayed is its tortuous complexity, which is illustrated by the ‘problem’ posed here to apparently test doctors’ understanding of risk. Instead it is a vivid example of how current teaching is causing widespread confusion and putting students and trainee doctors off the subject. This is partly due to historical reasons because the terms were ‘borrowed’ from WW2 radar. Adjusting a dial could change the sensitivity and specificity of radar signals to maximise the chance of spotting aircraft and to minimise seeing birds.

If the ability of a simple (positive/negative) test to predict a diagnosis is being assessed, all we need to know how often a diagnosis was found when the test was positive (e.g. 80/200 patients) and how often the test was positive when the diagnosis was present (e.g. 80/100 patients). If we are also told how many people were studied (e.g. 1000), then this tells us the basic facts about that study. However, similar information will be needed about all other symptoms, signs, test results and diagnoses. Sadly, this is not available and we have to guess to do our job.

Given the above values of 80/200, 80/100 and 1000, the enthusiastic can say that the sensitivity is 0.8, the positive predictive value is 0.4, the false discovery rate is 0.6, the diagnostic prevalence is 0.1, the positive test result prevalence is 0.2, the false negative rate is also 0.2, the false positive rate is 0.133, the specificity is 0.867, the prevalence of the absence of a diagnosis in the study population is 0.9, the negative predictive value is 0.975, etc. This information is of little if any value clinically but is often used to torment people by asking them to think backwards by calculating the positive predictive value from various combinations of the above values. But the answer is obvious from the main study observation of 80/200. It is 80/200 = 0.4.

We need more clarity and less obfuscation. In order to do sensible research, we also need to understand the set theory of differential diagnostic reasoning, diagnostic confirmation and treatment selection [1].

Reference:
1. Llewelyn H, Ang AH, Lewis K, Abdullah A. The Oxford Handbook of Clinical Diagnosis, 3rd edition. Oxford University Press, Oxford, 2014, pages 625 to 627.

It seems ironic in the rapid response by Dr Llewelyn, who is listed as Hon Fellow in mathematics and consultant physician, when it was asserted that:

"Doctors use symptoms and other findings by considering their differential diagnoses, often ranking the diagnoses in order of the frequency in which they occur in the finding. We then look for other findings that occur commonly in one of more diagnoses and rarely or never by definition in others. By doing this we try to assemble a combination of findings within which the frequency of patients with one of the differential diagnoses is high"

and that:

"The specificity, false positive and likelihood ratio have never been a part of differential diagnostic reasoning. "

While it is true that no one (I know of anyway) carries around a set of numbers relating to statistical values of medical tests, the fact is that understanding sensitivity, specificity, true and false postiives/negatives and the like is precisely the basis of why we can still steer our diagnosis towards a set of differentials, since the formulation of diagnosis is based on accepting or rejecting signs and symptoms (if looking for the presence of a specific symptom is a test) found on examination.

For example if one accepts the absence of a pathognomonic symptom or sign does not rule out the possibility a diagnosis, then one accepts the idea of a false negative.

Similarly doctors need to understand the impact and basis of ordering tests for investigations, in which how it would (or should) change the final diagnosis, particularly when the tests are ordered as a screening/'fishing' basis or for confirmation of diagnosis.

The classic example would have been choosing between V/Q scan and CTPA in investigating patients assessed as having high risk or low risk pulmonary embolism. Barring issues with radiation and renal dysfunction (in the use of intravenous contrast), understanding the results of PIOPED and PIOPED II (with its associated pre/post test probability and likelihood ratio) should guide physicians in selecting the test which will provide the result that the physician is willing to accept (without wasting time and money as well as unnecessary risks in performing test which is not going to be helpful).

Perhaps we are too used to pressing a button and getting a (perceived definitive) answer without associated statistical data and provision attached to the test result. This is a common illusion we all are guilty of making.

Understanding the role of medical statistics and accepting no single test is definitive without clauses/provision is an important step back on the path to practising good medicine.

Actually, I agree with John Lowe, as I realised the minute after I fired off my last response. Probability and uncertainty is difficult, especially when predicting the future !!

But in fact that does mean we are both on the same page in the real world - real doctors have serious matters to deal with, and will need a whole new way of thinking to grasp the real relevance of risk !

Gerd Grigerenzer thought natural frequencies would be the solution, but I think pictures, fuzzy thinking, and not precision maths, are probably the way to go.

Thanks to Sam Lewis for his wry observation and valiant attempt to boil down the problem to a manageable clinical compote, but . . . I have to disagree. The true probability does NOT lie between 1 in 50 and 1 in 51, as he says, but rather between 0 in 50 and 1 in 51. To make things more concrete, if we take the former case, ie when the test in question has zero sensitivity - implausible in practice but perfectly conceivable in theory - then none of our positive test results will be true ie include the one unlucky person in a thousand who has the disease, no matter how many times we repeat the test; indeed, under these conditions a negative result would be more predictive than a positive outcome, since if we exclude all the positives (all false) then 1 in 1000 minus 50 = 950 negatives will have the disease, a reverse 'sensitivity' of 1 in 950, which is slightly better than chance, at 1 in 1000.