Letters

The quality of systematic reviews

BMJ 2000; 321 doi: https://doi.org/10.1136/bmj.321.7256.297 (Published 29 July 2000) Cite this as: BMJ 2000;321:297

Review is biased

  1. Stephen Senn, professor of pharmaceutical and health statistics (stephens@public-health.ucl.ac.uk)
  1. University College London, London WC1E 6BT
  2. Centre for Statistics in Medicine, Institute of Health Sciences, Headington, Oxford OX3 7LF
  3. UK Cochrane Centre, Oxford OX2 7LG
  4. Bushey, Hertfordshire WD2 2NN
  5. The London Chest Hospital, London E2 9JX
  6. Fast Cycle Sciences Limited, PO Box 221, Epsom KT17 2WF
  7. GlaxoWellcome, Uxbridge, Middlesex UB11 1BT
  8. Department of Sociology, Social Policy and Social Work Studies, University of Liverpool, Liverpool L69 7ZA
  9. International Health Division, Liverpool School of Tropical Medicine, Liverpool L3 5QA
  10. Department of Clinical Epidemiology and Biostatistics, McMaster University, 1200 Main Street West, Hamilton, Canada L8N 3Z5
  11. Institute of Health Sciences, University of Oxford, Old Road, Headington, Oxford OX3 7LF
  12. Department of Clinical Epidemiology and Biostatistics
  13. Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Canada, L8N 3Z5
  14. Foresight Consultants, Dundas, Ontario, Canada L9H 2R5

    EDITOR—In discussing meta-analyses in the treatment of asthma, Jadad et al state that most reviews published in peer reviewed journals or funded by industry have serious methodological flaws.1 This summary is misleading and could have been put more succinctly as, “most reviews published in peer reviewed journals have serious methodological flaws,” since the industry reviews in their paper were a (similar) subset of the published papers.

    Jadad et al are, however, right in drawing attention to the inherent bias in their Cochrane approved quality index. Cochrane reviews are excellent in many respects but grossly deficient in at least one—namely, the reliance on software, RevMan, that is incapable of satisfying an essential and elementary requirement placed by drug regulators on sponsors, that “the particular model chosen should reflect the state of medical knowledge about the variables to be analysed as well as the statistical design of the trial.”2 RevMan cannot deal appropriately with covariates nor with multicentre, cluster randomised, minimised, or crossover trials. It is thus a suitable tool for single centre, randomised, parallel group trials in which no covariates are measured: a type of trial that is rather rare in my experience. On the other hand, the pharmaceutical industry has long employed doctors and statisticians capable of dealing with the complications of real clinical trials. For a good illustration, see the paper by Richardson and Bablok,3 which Jadad et al did not include.

    The biased and one sided review by Jadad et al cannot be taken as showing the superiority of Cochrane reviews to pharmaceutical industry reviews.

    Footnotes

    • Competing interests Professor Senn is a consultant to the pharmaceutical industry.

    References

    1. 1.
    2. 2.
    3. 3.

    High quality reporting of both randomised trials and systematic reviews should be priority

    1. Douglas G Altman, professor of statistics in medicine (altman@icrf.icnet.uk),
    2. Jonathan J Deeks, senior medical statistician,
    3. Mike Clarke, associate director (research),
    4. Christopher Cates, general practitioner
    1. University College London, London WC1E 6BT
    2. Centre for Statistics in Medicine, Institute of Health Sciences, Headington, Oxford OX3 7LF
    3. UK Cochrane Centre, Oxford OX2 7LG
    4. Bushey, Hertfordshire WD2 2NN
    5. The London Chest Hospital, London E2 9JX
    6. Fast Cycle Sciences Limited, PO Box 221, Epsom KT17 2WF
    7. GlaxoWellcome, Uxbridge, Middlesex UB11 1BT
    8. Department of Sociology, Social Policy and Social Work Studies, University of Liverpool, Liverpool L69 7ZA
    9. International Health Division, Liverpool School of Tropical Medicine, Liverpool L3 5QA
    10. Department of Clinical Epidemiology and Biostatistics, McMaster University, 1200 Main Street West, Hamilton, Canada L8N 3Z5
    11. Institute of Health Sciences, University of Oxford, Old Road, Headington, Oxford OX3 7LF
    12. Department of Clinical Epidemiology and Biostatistics
    13. Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Canada, L8N 3Z5
    14. Foresight Consultants, Dundas, Ontario, Canada L9H 2R5

      EDITOR—Senn1 [previous letter] is critical of the conclusion of Jadad et al in their critical evaluation that “most reviews published in peer reviewed journals or funded by industry …

      View Full Text

      Sign in

      Log in through your institution

      Free trial

      Register for a free trial to thebmj.com to receive unlimited access to all content on thebmj.com for 14 days.
      Sign up for a free trial

      Subscribe