Assessing the quality of research
BMJ 2004; 328 doi: https://doi.org/10.1136/bmj.328.7430.39 (Published 01 January 2004) Cite this as: BMJ 2004;328:39
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
I have been interested in the idea of hierarchy of information and
where it comes from. Poincare is credited with the idea from the turn of
the last century. He used the term hierarchy of fact(1). Tracking down the
quote is a harmless pastime and is like identifying William Occam’s phrase
“entes non multiplicanda sunt praeter necessitatem”. Occam wrote some
years before computers, and in Latin and even I cannot bring myself to
wade through his Quodlibeta, which could be called the razor’s last hiding
place. Putting poincare and hierarchy through a search engine will score
many hits, but not the bit where he may say: “I have had this fantastic
idea…..hierarchies, hierarchies are going to really important in about a
century.” Really good ideas may not be known to be the really good ideas
at the time or perhaps he did not think it was that important. The
earliest I came across ‘hierarchies’ in a law report(2) is 1982. The
report concerned charitable status of a religious institution.
I have always thought that this concept of hierarchy when it is used
in the context of medicine, may be a busted flush. That is, the concept as
good as it is in pure maths, is not exportable. Suppose there is a
Scottish study that shows drug X to beneficial but there is then a larger
meta-analysis that shows X is not so good. Suppose further a Scot comes to
you with a diagnosis for which drug X could be used. Does one give the
drug (according to the Scottish study) or withhold it (according to the
meta-analysis)? The problem with ignoring the meta-analysis and treating
the Scot with drug X is that the hierarchy is inverted – and if one does
that, it is not a hierarchy.
This problem is less theoretical than may appear at first. In the albumin
study, although there was no overall benefit described in a much trumpeted
meta-analysis, there was a study of its use in children with meningococcal
septicaemia which showed a benefit(3). Readers were very keen to have the
opinion of the authors and anyone else who was in the news, such as the
editor(4) of the BMJ saying it was harmful, whether they would forbid its
use in their own children if the diagnosis were meningitis. The journals
did not ring with their cries of: “…and not even in my children!” In the
event, the silence of the great men was audible. The shakers shook not.
The leaders - didn’t.
A thought experiment such as the one above is called a gedanken
experiment and has a long history. Einstein used this a lot. Marie Curie
during a Solvay conference, records him lumbering around a foreshore
saying his most important task was to decide what happened when a ray of
light went through a free falling elevator. The earliest example of
thought experiment that springs to mind is Newton and the apple (a maid of
kent, I think) in the late seventeenth century.
In short, hierarchies may have a place in ordering information. The
issue is that a hierarchy is linear. If the information has more than one
dimension, then there will be more than one way to place them in a
hierarchy and not only that, the orderings may be inconsistent with each
other. The same evidence cannot simultaneously validate both of them. If
one is going to favour one way to order and reject the other why not
reject both and conclude the information cannot be validly ordered?
Priests never had this problem.
Hierarchies have a place in theory surrounding medical practice. They
may have a place near the centre, but I do not think that they are so
important that they should be placed anywhere near the top.
Oliver R Dearlove FRCA
Refs
1.D L Hurd JJ Kipling Origins and Growth of Physical Science, Penguin
Books 1964. Introduction
2. Valley Forge v Americans United 454 US 464 (page 15)
3 Levin M, Galassini R, De Munter C, Nadel S, Habibi P, Britto J, et
al. Improved survival in children admitted to intensive care with
meningococcal disease. Proceedings of the 2nd Annual Meeting of the Royal
College of Paediatrics and Child Health. York: Royal College of
Paediatrics and Child Health , 1998.
4. Dearlove O R. A shaft in the Editor’s eye. British Medical
Journal 1999 317 23 also at http://www.bmj.com/cgi/eletters/317/7154/291
Conflict of interest: The author is a Council member of the Royal
College of Anaesthetists. These views are his own and do not represent
those of the Council (or his employer).
Competing interests:
as script
Competing interests: No competing interests
While I would agree with the broad thrust of the paper by Paul
Glasziou et al, I have to take issue with the statement that none of the
current hierarchies include all important dimensions of study quality.
The hierarchy used by the Scottish Intercollegiate Guidelines Network
(SIGN) is based on both the type and quality of the studies cited as
evidence. This may not be immediately apparent from the summary hierarchy
published in the guidelines, but examination of the full guideline
development process set out in our methodology manual (1) will demonstrate
the extent to which study quality is taken into account. All the aspects
of RCT design mentioned by the authors, for example, are covered.
Most other organisations working in the guideline field follow
similar processes that ensure study quality is adequately taken into
account when developing their evidence base.
1. SIGN50: a guideline developers' handbook. Edinburgh; Scottish
Intercollegiate Guidelines Network; 2002. Available from:
http://www.sign.ac.uk/guidelines/fulltext/50/index.html
Competing interests:
None declared
Competing interests: No competing interests
Paul Glasziou and colleagues draw attention to the inadequacies of
relying solely on epidemiological research, which is a blunt tool
containing many flaws.1 Research into the effects of oral contraceptives
(OCs) and hormone replacement therapy (HRT) is an excellent example.
Epidemiology has been given pride of place, and separated from
accompanying biochemical and pathological investigations, since the end of
the 1960s. By then it had become obvious that no safe combinations of
oestrogens and progestogens were possible.2 Although the dose of
oestrogen was limited, even more powerful progestogens were introduced as
“safer lower dose contraceptives”, regardless of the fact that
progestogens were known to be more likely to cause breast cancer in animal
studies than oestrogens.
Since then numerous flawed epidemiological studies have given so many
conflicting results that has it taken decades to confirm that progestogens
and oestrogens are highly thrombogenic and carcinogenic for women. At
last, older women are being warned against using hormone replacement
therapy, but more very young women than ever are being encouraged to take
progestogenic hormones for contraception. False claims of safety are often
based on including HRT takers in never takers of OCs reference groups and
vice versa. Breast cancer was the commonest cause of death in the
Oxford/FPA contraceptive study in women up to age 74, but the number of
women who had never used these hormones for any reason before they died
from breast cancer was not available from the authors. 3 There is clear
evidence that current users of progestogens and oestrogens have more risk
of breast cancer at any age. Latent periods for cancers and residual
increased risks after hormone exposures may be as long as 20 to 40 years.
Risks increase with further exposures.4
The RCGP OC study found an increased risk of miscarriage among women
stopping for side effects, and post pill amenorrhoea took up to two years
to resolve. In my experience recovery of unexplained infertility can be
hastened by repletion of OC-induced nutritional deficiencies.5
Use of progestogens and oestrogens cause widespread immune dysfunction and
we should once again prioritise inclusion of basic research investigations
which were so revealing in the 1960s.
1. Glasziou P, Vandenbrouke J, Chalmers I. Assessing the quality of
research. BMJ 2004;328:39-41.
2. Grant ECG. Changing oral contraceptives. BMJ 1969;4:789-91 &
Today's Drugs
3. Grant ECG, Price E, Steel CM. Oral contraceptive and smoking
mortality. Lancet 2003;362:1241.
4. Brinton LA, Brogan DR, Coates RJ, et al. Breast cancer risk
among women under 55 years of age by joint effects of usage of oral
contraceptives and hormone replacement therapy. Menopause 1998;5:145-51.
5. Grant ECG. The pill, hormone replacement therapy, vascular and mood
over-reactivity, and mineral imbalance. J Nutr Environ Med 1998;8:105-116.
Competing interests:
None declared
Competing interests: No competing interests
Eminence based medicine
In a article on assessment of the quality of research the Authors are
discussing the need to broaden the scope by which the evidence is assessed
so that the principles of other types of research, addressing questions on
aetiology, diagnosis, prognosis, and unexpected events of treatment will
become equally widely understood. They also suggest the need to redefine
the hierarchies of evidence and of separate grading systems
The Authors
also argue that the main difference between the evidence from randomised
clinical trials and observational studies is that the first kind of
studies address therapeutic interventions while the latter non-therapeutic
questions. This argument is not easily to accept since the real difference
appears rather to be related to the judgement of the value of evidence on
causality. And why would criteria of evidence on causality what regards
the therapeutic interventions differ between the various kinds of studies
that all are in fact observational although experimental and non-
experimental.
The value of evidence on causality of diseases which is the
subject of aetiological investigations should in principle not differ
between interventions and non-interventions since both are of interest for
therapeutic interventions. Thus, the argument on interventions as
discriminating factor is not the real source to the differences between
the locations of the various types of studies in the hierarchies of
evidence. In fact, aetiological studies try to identify causes of a
disease to respond to by means of therapeutic interventions. The
requirements on the strength of evidence should thus be even stronger and
the aetiological investigation appears even more difficult as the
complexity of causal factors in various environments are more difficult to
separate or isolate as compared to conscious, planned interventions that
researchers are aware of and able to control. The argument about the main
difference between RCTs and other kinds of studies is thus not convincing.
The therapeutic interventions mentioned by the Authors are exclusively
biological in nature. The real difference between the various kinds of
study design is rather related to the strength of causal explanations and
predictions they generate. Since observational studies are often
investigating more complex situations and factors that are not possible to
randomise or isolate as easy as in a RCT, the causal explanations provided
by observational research must be seen as equally valid as the evidence
from RCTs. Thinking in terms of vertical hierarchies was probably a
feature of the earlier decades. Today it would be more appropriate to
think in system theoretical terms such as complexities and mutually
including causal layers (e.g. to include side effects due to synergies or
interactions). The RCTs per definition reduce the reality to single causes
and one-dimensionally operating factors by exclusion of the context. Is
there rather a need of separate hierarchies of evidence for various
disciplines after the degree of reductionism? Still today, the criteria of
evidence from quantitative studies formulated by Sir Hill appear to be of
larger scientific value and generalisability than “eminence based
evidence” (Mr Ian Chalmers´' opinion?).
Reference
Hill AB. The environment and disease: association or causation? Proc R Soc
med 1965;58:295-300.
Competing interests:
None declared
Competing interests: No competing interests