# The real problem is the biomedical ignorance of statisticians

BMJ 2011; 342 doi: https://doi.org/10.1136/bmj.d2579 (Published 21 April 2011) Cite this as: BMJ 2011;342:d2579## All rapid responses

This is a modified version of my response posted on 27 May

http://www.bmj.com/content/342/bmj.d3026/reply#bmj_el_261219

---------------------------------------------------------------------

---------------------------------------

Statistics, the weed in the biomedical garden.

The insistence that doctors spend more of their hard-pressed time

learning even more statistics, and the weakness of the case for it, makes

a return to the solidity of unadulterated biomedicine, where statistics is

a simple ancillary1, all the more urgent.

I will be apparent in replying to arguments presented2-7 as criticisms of

my suggestion that medical statistics should be put back in its bottle.

My plea for "better tests not better statistics" - tests that give a

yes/no answer, instead of statistical maybe's - is dismissed, because

"medical science just has not delivered" them2. Really? : tests for

pregnancy, whether or not followed by DNA proof of paternity, let alone

identity; measurements of hormones and essential body constituents; ECG

definition of arrhythmias; immuno-definition, from the very sophisticated

to the simple patch tests that confirm sensitivity with a molecular

specificity almost as absolute as the patient's existence; and although

I'll make a full stop with microbiology, I could continue through a forest

of yes/no, or near enough yes/no tests. It isn't hard to imagine what the

inventors of those tests, and those striving to discover and design more

of them, would say to the typically negative belief of the statistically-

minded, that search for such tests "is probably futile"3. And whilst

nobody can disagree that clinicians must be able to interpret "...current

imperfect tests" 3, that's mostly a matter of remembering a few

percentages, not the drudge of learning more statistics.

The biomedical poverty of the statistical case is further illustrated

by belief that: "If we all stuck to...straight yes no answers...modern

medicine would not exist [because] medicine is all about balancing

probabilities" 4 Nonsense; medical advance would not have happened with

that relativist, statistical mindset: think vitamins - scurvy and C,

rickets and D; think hormones - insulin and diabetes, thyroxin and

myxoedema, and think pharmacology and magic bullets, drugs and their

receptors; and so on, for each biomedical advance. Balance of

probabilities be damned; real medicine is about certainty; it is medical

statistics that has to live in a web of uncertainty.

The statistical brainwashing of medical practice is alarming:

Rathbone5, for example, seems to believe my argument can be made

disappear, like a vampire exposed to a clove of garlic, by waving the

label of a cognitive bias of the " base rate fallacy" variety; likewise,

that a test with a "low predictive value" can be rescued by an "..ability

to calculate the probability of ....[its] relative utility". It can't,

which is why we need better tests not a better statistical ability to

understand their defects. Wilson6 belief (despite an unquestionable

concern for hurt feelings, the need to work together for a common goal,

and defence of statistics as a true science) "that there isn't ".. any

possible way to create better diagnostic tests without collaboration

between clinicians and analysts [statisticians]" has little relationship

to biomedical reality: yes/no tests are invented and developed by medical

biologists alone, often on just a few experimental animals, before

toxicity and human studies; it's only then, that poor tests need

statistical collaboration to make them appear to work.

Dressing clinical problems with statistical rags from the wardrobe of

the Emperor's Clothes, has become a contemporary medical fashion. And, as

is the way with any fashion, medical statistics has convinced us

irrationally, making clinicians believe it will make them better

practitioners. But even when "the symptoms, signs, and tests that doctors

rely on have poor performance characteristics" 7 , that state of affairs

won't be improved by more statistical characterisation. Clinical practice

is a professional skill, honed by individual professionals, as it is for

professional musicians; and no amount of statistics will teach a fiddler

how to bow his fiddle. As clinicians, we have to learn how good we are at

eliciting a history and physical signs, and understanding their

significance; and likewise with diagnostic tests; the belief that

statistics can help us improve these skills is based on fashion, not on

reason or common sense.

The malign influence of statistics in medicine operates well beyond

the realm of "tests" and clinical practice, taken up by much of this

correspondence (which started with a response to Heath's review which

cited tests as an example8). Far more important is the way that worship of

this false god has contributed to the failure of the once exciting field

of clinical research, a field which has become effete and without purpose,

producing statistical trivia, from the spurious disasters of the

associative statistics, and their subsequent denial. that feed the media

daily; the epidemiological statistics that can only reveal trivial or

dubious associations9; the horror stories and nonsense "break-throughs";

the encouragement of screening programs with their spurious disease

creation10; and finally, and above all, the choking out of solid clinical

research by scientifically blind trials and studies, where statistical

significance replaces biological relevance.

The death of clinical research is in urgent need of new thought, and

clearing our biomedical garden of the lethal weed of statistics is one

small part of it.*

Sam Shuster

* In response to my original letter, James Penston kindly sent me a copy

of his recent book (Stats.Con. How we've been fooled by statistics-based

research in medicine; The London Press, 2010). It is a wonderful analysis

of the emptiness of medical statistics; a perfect weed-killer.

1 Shuster S. The real problem is the biomedical ignorance of

statisticians

BMJ 2011;342:d2579 (21 April)

2 Hemming K. Let's work together. BMJ 2011;342:d3030 ( 21 May)

3 McNulty SJ, Williams P. Skill of interpreting imperfect

investigations. BMJ 2011; 342:d3026 ( 21 May)

4 Pharoah PD. Balancing probabilities. BMJ 2011;342:d2579 ( 21 May)

5 Rathbone P. Role of cognitive bias. BMJ 2011; 342: d3047 ( 21

May)

6 Wilson A. All inflammation aside. BMJ April 27; 2011

7 Barraclough K . Come clean if you don't know. BMJ 2011; 342:

d3044 ( 21 May)

8 . Heath I. Dare to know: risk illiteracy and shared decision

making. BMJ2011;342:d2075. (6 April.)

9 Levell NJ, Beattie CC, Shuster S, Greenberg DC. Melanoma epidemic:

a midsummer night's dream? Brit J Dermatol. 2009; 161: 630-634.

10 Shuster S. Malignant melanoma: how error amplification by

screening creates spurious disease. Brit J Dermatol 2009; 161: 977-979.

**Competing interests: **
No competing interests

Dr. Shuster's letter would be a good conversation starter on an

important topic, if it weren't so intentionally inflammatory. He frames

the problem as a battle of ignorance, but that is exactly why it shouldn't

be the starting point of a healthy discussion. His letter may be a

reaction to recent studies aiming to demonstrate physicians' ignorance of

statistics [1, 2]. Let me apologize for any hurt feelings on behalf of

statisticians everywhere.

Let's leave alone who knows less about the other's field and consider

the common ground. Both physicians and statisticians have a common goal of

improving patient care. In my experience, the two groups work well

together to bring clinical knowledge together with analytical expertise.

Shuster's letter focused primarily on diagnostic tests and their

statistics. I may lack imagination, but I can't picture any possible way

to create better diagnostic tests without collaboration between clinicians

and analysts. In fact, I can't imagine medicine progressing if we throw

statistics under the bus.

I appreciate Dr. Shuster's challenge that no study should rely

entirely on statistics. But the over-reliance or abuse of statistics is a

study design problems, not an impeachment of statistics. Frankly,

statistics, as a science, is unimpeachable. Statistics have helped to

modernize biomedical science. And in this regard I reject the (only action

item in the letter) to return statistics to an ancillary aid.

References:

1. Heath I. Dare to know: risk illiteracy and shared decision making.

BMJ2011; 342:d2075. (6 April.)

2. Gigerenzer, G. (2002). Calculated risks: How to know when numbers

deceive you. New York: Simon & Schuster.

**Competing interests: **
No competing interests

The insistence that doctors spend more of their hard-pressed time

learning even more statistics, and the weakness of the case for it, makes

all the more urgent the need for a return to the solidity of unadulterated

biomedicine, where statistics is a simple ancillary.

Hemming dismisses my plea for "better tests and not better

statistics" - tests that give a yes/no answer, instead of statistical

maybe's - because "medical science just has not delivered" them. Really?

: tests for pregnancy, whether or not followed by DNA proof of paternity;

measurements of hormones and essential body constituents; ECG definition

of arrhythmias; immuno-definition, from the very sophisticated to the

simple patch tests that confirm sensitivity with a molecular specificity

almost as absolute as the patient's existence; and although I'll make a

full stop with microbiology, I could continue through a forest of near

yes/no tests.

The biomedical poverty of the statistical case is further illustrated

by Pharoah's belief that: "If we all stuck to...straight yes no

answers...modern medicine would not exist [it] is all about balance of

probabilities". Nonsense; medical advance would not have happened with

that relativist, statistical mindset: think vitamins - scurvy and C,

rickets and D; think hormones - insulin and diabetes, thyroxin and

myxoedema, and think pharmacology and magic bullets, drugs and their

receptors; and so on, for each biomedical advance. Balance of

probabilities be damned; real medicine is about certainty; medical

statistics can only live in a web of uncertainty.

The statistical brainwashing of medical practice is alarming:

Rathbone, for example, seems to believe that a test with a "low predictive

value" can be rescued by an "..ability to calculate the probability of

....[its] relative utility". It can't, which is why need better tests not

a better statistical ability.

The malign influence of statistics in medicine operates well beyond

the more simple realm of "tests": worship of this false god is one of the

reasons for the failure of the once exciting field of clinical research,

which has become effete and without purpose. The death of clinical

research is in urgent need of new thought, and clearing our biomedical

garden of the lethal weed of statistics is one small part of it.*

Sam Shuster

* In response to my original letter, James Penston sent me a copy of

his recent book (Stats.Con. How we've been fooled by statistics-based

research in medicine; The London Press, 2010). It is a wonderful analysis

of the emptiness of medical statistics; a perfect weed-killer.

**Competing interests: **
No competing interests

Sam Shuster argues that we need better tests and not better

statistics. Unfortunately medical science just has not delivered on these

near perfect tests (Shuster states he wants tests that give almost yes or

no answers - i.e. perfect screening tests), we would all love tests like

these but they simply don't exist. The review article which is quoted as

revealing the statistical illiteracy of doctors, cites evidence in which

doctors were asked how they would communicate a positive screen for breast

cancer to patients, finding that 60% would miss-communicate this as a 80%

or 90% risk when actually it was about 10%. If this sample is

representative, it means that thousands of women will be given miss-

information and the worry caused by this is terrible.

Doctors need to communicate these risks to their patients, not

statisticians. Statisticians have a duty to ensure they communicate well

with doctors. But to suggest that statistics should become a simple

auxiliary aid and that statisticians are ignorant about biomedical issues

is simply wrong. Many applied statisticians stride to become experts in

particular clinical areas, perhaps in a similar way to how public health

doctors often excel in applied statistics. It is only with this sort of

collaboration can we move forward to better evidence based medicine, by

sharing knowledge between disciplines.

If statisticians worm their way back to become disinterested

technicians, and provide our clinical colleagues with numerical output

only, then doctors will go on miss-communicating these issues to their

patients indefinitely. Furthermore if there is miss-understanding around

these issues in screening then this also means that doctors will

underestimate the harms associated with any consequent invasive diagnostic

procedures (by underestimating false positive screens) and so in turn

counselling patients on overestimated benefits of screening.

Statisticians don't "juggle" results to make tests look better - they

stride to find the least biased most accurate estimate of accuracy.

**Competing interests: **
No competing interests

In his letter Shuster [1] suggests we should direct our efforts away

from encouraging a deeper understanding of statistics towards a search for

better investigations, indeed one which will give a definitive 'yes or

no'. Whilst it may be admirable to strive for a test which is 100%

sensitive and specific, with a 100% positive and negative predictive

value, it is unlikely to be achievable and therefore maybe our energies

should be directed at training ourselves to be better equipped to

interpret our current imperfect tests.

When we first encounter a patient we begin to formulate the likelihood of

a condition being present, based on history and examination, which is then

further modified through investigations.

If we understand what the indications for, and limitations of, any

investigation which we use, we can strengthen our interpretation of the

results and thereby improve their utility. If we calculate a pre-test

probability of a condition to be high, then a positive test has a higher

positive predictive value; if the pre-test probability of a condition is

low, then a negative investigation has a better negative predictive value.

For example, a left bundle branch block on an ECG in the context of 30

minutes of central crushing chest pain, with sweating, nausea etc can be

diagnostic of an acute coronary syndrome; yet the exact same changes on a

routine, asymptomatic, pre-operative ECG would not warrant the patient to

be rushed to the nearest centre for a primary PCI. The same investigation

has a different utility, which is dependent on the skill and knowledge of

the interpreter. This does not require a complex knowledge of statistics,

merely knowledge of the place of investigations in the diagnostic pathway.

All investigations have their limitations and a good doctor should

understand these limitations and only ask for an investigation if they are

able to interpret the result. Even a "perfect" test can become a bad one

when it is used in the wrong patient or the wrong clinical scenario.

Dr S J McNulty MB ChB FRCP

Consultant Endocrinologist

Dr P Williams MB ChB MRCP MRCS Ed (A&E) FCEM

Consultant Physician

Ref

1.BMJ 2011;342:d2579

**Competing interests: **
No competing interests

The use of 'specificity' and 'false positive rate' is confusing and

is not applicable when reasoning with clinical risk or diagnoses. I shall

use the values that confused the group of gynaecologists [1] to illustrate

this.

A doctor (and patient participating in decisions) may wish to know

what proportion of women in a study turned out to have breast cancer. It

was about 1% [1]. When mammograms were done, about 10% were positive (and

so 90% were negative). When the mammogram result was negative, only about

0.1% had breast cancer. When the mammogram result was positive about 9.2%

had cancer [1].

If a mammogram is positive, a surgeon will consider the differential

diagnosis (e.g. breast cancer, benign adenoma, etc) and the proportion of

those with a positive result who have each of these differential diagnoses

(e.g. 9.2% have breast cancer). The surgeon will then consider one of the

possibilities (e.g. breast cancer) and look for a finding (e.g. lymph node

enlargement) that occurs commonly in that that possibility and rarely in

at least one other diagnosis. The specificity and false positive rate are

not used in differential diagnostic reasoning. It is ratios of

'sensitivities' that are used [2].

The proportion of those with a positive screening test result who

have cancer can be observed directly in the study population. It was

about 9.2% (or 0.092). There is no need to 'calculate' it. However it

can be calculated in a roundabout way from other directly observed

proportions by using a version of Bayes theorem as follows:

1/(1+((0.99/0.01)x(0.09/0.9))) = 0.092

Professor Shuster is right about questioning the need for doctors to

do this calculation [3]. Asking them to do so is not a test of 'risk

literacy' but merely a test of unnecessarily applying the arithmetic of

Bayes theorem.

References

1. Heath I. Dare to know: risk illiteracy and shared decision making.

BMJ2011; 342: d2075. (6 April.)

2. Llewelyn H, Ang HA, Lewis K, Al-Abdullah A. The Oxford Handbook of

Clinical Diagnosis, 2nd ed., Oxford University Press, Oxford, 2009

3. Shuster S. The real problem is the biomedical ignorance of

statisticians BMJ 2011; 342:2579 doi:10.1136/bmj.d2579

**Competing interests: **
No competing interests

Shuster criticises the real problem of the devastating biomedical

ignorance of statisticians and goes on to ask, "Why should a doctor need to

know how to calculate the chance of breast cancer in a patient with a

positive screening mammogram". The answer is simple. It is a doctor who

offers a woman a mammogram, not a biomedical statistician, and therefore

it is the doctor who should understand how to interpret the test result.

If we all stuck to the use of tests and interventions with "straight

yes or no" results that even doctors can understand modern medicine would

not exist. Modern medicine is all about the balance of probabilities.

**Competing interests: **
No competing interests

Shuster is trying to have his cake and eat it in his criticism of

statistics in clinical practice (1). He highlights that breast cancer

screening is a "bad" test (by which I think he means it has a low positive

predictive value) but it is precisely because of our ability to calculate

this probability that we can know the relative utility of the test!

Unfortunately this information has been neglected in information given to

women who have had breast cancer screening until relatively recently (2,3)

but it is vital that such information is provided to allow women to make

an informed decision about screening.

The example given by Heath (4) and Shuster's comments highlights the

error known as the base rate fallacy or neglect which is one a number of

cognitive biases that affect our ability the think clearly and correctly.

Contrary to Shuster's view it is only through a greater understanding and

use of statistics - or more correctly probability - that we can counter

such biases and therefore offer our patients better care.

1. Shuster S The real problem is the biomedical ignorance of

statisticians

BMJ 2011;342:2579 doi:10.1136/bmj.d2579

2. McPherson K. Should we screen for breast cancer?

BMJ2010;340:c3106.

3. Gotzshe PC, Hartling OJ, Nielsen M, Brodersen J, J?rgensen KJ.

Breast screening: the facts--or maybe not. BMJ2009;338:446-8

4. Heath I. Dare to know: risk illiteracy and shared decision making.

BMJ2011;342:d2075. (6 April.)

**Competing interests: **
No competing interests

## Re: Dr McNulty & Williams- Intrepreting investigations

In the example quoted , i.e Acute Chest pain with LBBB, the question

for the clinician is 'Does the patient need to be admitted urgently? yes

or no?.

The answer is yes. (one does not need to calculate the precise predictive

values to interpret this imperfect test and make that clinical decision!)

Competing interests:No competing interests18 July 2011