Observations Statistics in Medicine

Risky business: doctors’ understanding of statistics

BMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g5619 (Published 17 September 2014) Cite this as: BMJ 2014;349:g5619

Re: Risky business: doctors’ understanding of statistics

Christopher Martyn highlights the lamentable state of knowledge of doctors with regard diagnostic tests[1]. Since we can assume that doctors are intelligent beings, and that intelligent beings only learn what is useful and don’t recall what they don’t use regularly, we might surmise that, despite the protestations of the medical statisticians, knowledge of the characteristics of diagnostic tests is not a prime requirement of a practising doctor. Spurred by Dr Martyn’s article we recently performed an informal email survey of 18 local GPs of whom 16 responded. We asked them about two diseases that managed by GPs, often without the input of hospital specialists: asthma and type II diabetes.

We asked the GPs how they diagnosed the disease. For asthma, most followed the BTS/SIGN guidelines but there was no single test to make a clinical decision, and clinical history and examination are just as important as investigations such as spirometry. For type II diabetes some used tests based on glucose and some the more recent test based on HbA1c%. Only two GPs, one for asthma and one for diabetes knew the sensitivity and specificity of the tests they were using. However, the general message was that these diseases are diagnosed by a combination of tests, and it would be difficult to know what the sensitivity and specificity of the combination would be, which may help explain the GPs’ ignorance and lack of interest in the values.

Dr Martyn’s article was based on a survey of doctors in Boston[2]. We wondered how useful the paradigm used to test the doctors in Boston would be for GPs in the UK. Essentially given the sensitivity, specificity and prevalence, one should be able to work out the positive predictive value. We have shown that most GPs in our (albeit small) survey do not this in practice. The problem as highlighted by Dr Martyn is particularly acute when the prevalence is low, since even highly sensitive and specific tests have low positive predictive value, which is commonly overestimated. However, it is unclear what the relevant prevalence or pre-test probability is in these circumstances.

Thus GPs are likely to fail to the same test as the Harvard doctors for a number of reasons:

1) Diagnosis is based on a combination of tests and clinical examination and there is little research based on the sensitivity and specificity of the combination of different examinations as opposed to a one-off test, which is why GPs are unlikely to know the values.

2) It is unclear what is meant by the prevalence of asthma or diabetes for these GPs. It is not the proportion of people in the population with the disease, but rather the proportion of people who come to consult who have the disease (perhaps with similar age and clinical history). This proportion is likely to be quite high and so the issue of overestimating the positive predictive value is less important.

3) The prevalence of the disease will also depend on the severity of the disease being tested for and so this also muddles the calculations.

In conclusion we feel that testing doctors on the interpretation of diagnostic tests, whilst presenting the doctors with an intellectual challenge which one feels they should be capable of answering, does not reflect the day-to-day reality of many doctors’ lives, and is the reason why many doctors do it poorly. Given the lack of data on pre-test probability (indeed even simple prevalence data is difficult to get hold of) in practice, and of the sensitivity and specificity of the test, it is not possible to do the calculation.

One is left with the unsatisfactory combination of scientific principles and intuitive guess work, what David Sackett referred to as “the science of the art of the clinical examination “[3]

MJ Campbell, Professor of Medical Statistics
I Rotherham, Medical Student
C Sefton, Medical Student
J Dickson, GP and NIHR Clinical Lecturer

References
1. Martyn C. Risky business: doctors’ understanding of statistics. BMJ 2014; 349:g5619 doi: 10.1136/bmj.g5619
2. Manrai AK, Bhatia G, Strymish J, Kohane IS, Jain SH. Medicine’s uncomfortable relationship with math: calculating positive predictive value. JAMA Intern Med 2014; 174:991-3.
3. Sackett DL, Rennie DL. The science of the art of the clinical examination. JAMA 1992;262: 2650-2

Competing interests: No competing interests

05 November 2014
Michael J. Campbell
Professor of Medical Statistics
I Rotherham, C Sefton, J Dickson
University of Sheffield
ScHARR, Regent Court, 30 Regent St Sheffield S1 4DA