Rapid Responses to:

EDUCATION AND DEBATE: Kevork Hopayian The need for caution in interpreting high quality systematic reviews BMJ 2001; 323: 681-684 [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Interpreting Systematic Reviews - More than caution is required

James B Connelly   (10 October 2001)

Interpreting Systematic Reviews - More than caution is required 10 October 2001
James B Connelly, Senior Lecturer in Public Health Nuffield Institute for Health Send response to journal: Re: Interpreting Systematic Reviews - More than caution is required	Sir, Despite problems it is clear that Hopayian(1) still believes systematic reviews can fulfil their promise of providing 'a new era for the objective appraisal of evidence available on a given topic'(1) But things are worse than this. First, it is surely possible to construct plausible, if not ingenious, reasons why one or another trial might be excluded both on a priori basis and, if the overall results displease, on a 'post hoc' basis. Such inclusions or exclusions are, as Hopayian points out, not determinable through using any number of quality scores which not only differ from each other when applied to the same trial(2) but, as Hopayian shows, lack reliability between assessors. (1) Hopayian cites the trial by Cuckler et al as flawed. However the therapeutic mechanism of action of epidural steroid might reasonably be expected to occur within 24 hours and the different rate of cross-overs to 'active treatment' after this period would tend to diminish the observed effect of the steroid in the subsequent intention-to-treat analysis, providing exactly the reason for such a conservative analysis. That a genuine clinical and methodologically based disagreement is possible over such issues, can, and does mean that systematic reviewers can usually justify their inclusions and exclusions, resulting in discordant reported pooled effects(2) and important clinical questions remaining unresolved.(3) The academic debate about systematic reviews, and especially meta- analysis, has displayed a relentless optimism about the possibility of clarity. Contributors to this debate publish better and more refined recommendations for procedural issues such as publication bias, trial heterogeneity and inclusion criteria, but efforts cannot be said to have resulted in clarity regarding important clinical problems. (3) It is always possible through using one or more reasonable argument regarding appropriate procedures to criticise any given systematic review. The situation suggests the truth of Quine's remarks concerning empiricism's vulnerability to such positive or negative attitudes. "Any statement can be held to be true come what may, if we make drastic enough adjustments elsewhere in the system."(4) The necessity now is to look disinterestedly at systematic reviews and especially meta-analysis and come clean that they may not ever fulfil their promise in some important areas. As a final illustration Hopayian would judge as valid a 'large trial' over a 'non-credible meta-analysis'; if this were our choice we all would concur but, as the producers of meta-analyses repeatedly demonstrate, 'credibility' can be contested and then we are thrown back on an uncertain choice between discordant large trials and meta-analysis.(5) Where now are the promises of clarity and objective evidence? References 1. Hopyian K. The need for caution in interpreting high quality systematic reviews. BMJ 2001; 232: 681-4 2. Moher D, Ba P, Jones A, Cook D J, Jaddad A R, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analysis? Lancet 1998; 352: 609- 13 3. Cates C. BMJ 2001; 323: 235 (letter) 4. Quine W V O. Two Dogmas of Empiricism. In From a Logical Point of View. (2nd ed). London: Harvard University Press, 1999 5. LeLorier G, Gregoire A, Benhaddaf J, Laierre J, Dererian F. Discrepancies between meta-analysis and subsequent large randomised, controlled trials. N. Eng. J. Med. 1997; 337: 536-42 Competing interests: none