Letters

The quality of systematic reviews

Review is biased

High quality reporting of both randomised trials and systematic reviews should be priority

Criticism is unjustified

Cochrane Collaboration should ensure equitable participation in management and policy

Authors' reply

Review is biased

EDITORIn discussing meta-analyses in the treatment of asthma, Jadad et al state that most reviews published in peer reviewed journals or funded by industry have serious methodological flaws.¹ This summary is misleading and could have been put more succinctly as, "most reviews published in peer reviewed journals have serious methodological flaws," since the industry reviews in their paper were a (similar) subset of the published papers.

Jadad et al are, however, right in drawing attention to the inherent bias in their Cochrane approved quality index. Cochrane reviews are excellent in many respects but grossly deficient in at least onenamely, the reliance on software, RevMan, that is incapable of satisfying an essential and elementary requirement placed by drug regulators on sponsors, that "the particular model chosen should reflect the state of medical knowledge about the variables to be analysed as well as the statistical design of the trial."² RevMan cannot deal appropriately with covariates nor with multicentre, cluster randomised, minimised, or crossover trials. It is thus a suitable tool for single centre, randomised, parallel group trials in which no covariates are measured: a type of trial that is rather rare in my experience. On the other hand, the pharmaceutical industry has long employed doctors and statisticians capable of dealing with the complications of real clinical trials. For a good illustration, see the paper by Richardson and Bablok,³ which Jadad et al did not include.

The biased and one sided review by Jadad et al cannot be taken as showing the superiority of Cochrane reviews to pharmaceutical industry reviews.

Stephen Senn, professor of pharmaceutical and health statistics. 
University College London, London WC1E 6BT stephens{at}public-health.ucl.ac.uk

Competing interests: Professor Senn is a consultant to the pharmaceutical industry.

High quality reporting of both randomised trials and systematic reviews should be priority

EDITORSenn¹ [previous letter] is critical of the conclusion of Jadad et al in their critical evaluation that "most reviews published in peer reviewed journals or funded by industry have serious methodological flaws."² We agree that this remark was overly strong. Firstly, Jadad et al considered the reporting quality as well as the quality of the review, and we would expect (although regret) poorer reporting in journals, where limitations on space might prevent the publication of key information. This does not invalidate the comparison, however, as it is important to know how well research is reported in medical journals. Secondly, although the contrast between Cochrane and journal reviews was clear, only six of the 38 reviews in journals were industry sponsored, too few to make a safe generalisation.

We do not agree, however, with Senn's description of the article of Jadad et al as "biased and one-sided." Jadad et al used a quality score to assess systematic reviews in asthma published in medical journals and in the Cochrane Library. This scoring system accords well with the recently published QUOROM Statement.³ Also, it should be of concern that the authors of fewer than half of the 38 reviews published in journals reported factors such as how they searched the literature, the criteria they used to determine which studies to include in their systematic review, or the statistical methods used to combine the data.

Senn's letter also discusses the Cochrane software RevMan, a topic not addressed by Jadad et al. Unfortunately, he gives the impression that Cochrane reviewers are allowed to use only RevMan in their analysisthis is not true. Also, and crucially, he does not recognise that nearly all Cochrane reviews are performed using summary statistics from published (and sometimes unpublished) papers, whereas reviews performed within the industry would have access to individual patient data. It is entirely appropriate to take covariate information into account in the latter, but it is usual (not just in the Cochrane Collaboration) to perform meta-analysis of the former using unadjusted data.

Lastly, there are indeed problems associated with incorporating crossover and cluster trials into meta-analyses, but these are largely resulting from inadequate reporting of these types of trial. We should seek to correct these inadequacies and insist on high quality reporting of both randomised trials⁴ and systematic reviews,³ and this was the concern of Jadad et al.²

Douglas G Altman, professor of statistics in medicine. 
altman{at}icrf.icnet.uk

Jonathan J Deeks, senior medical statistician. 
Centre for Statistics in Medicine, Institute of Health Sciences, Headington, Oxford OX3 7LF

Mike Clarke, associate director (research). 
UK Cochrane Centre, Oxford OX2 7LG

Christopher Cates, general practitioner. 
Bushey, Hertfordshire WD2 2NN

Competing interests: All the authors have substantial involvement in the work of the Cochrane Collaboration.

Criticism is unjustified

EDITORIn their systematic review Jadad et al have critically evaluated systematic reviews and meta-analyses of the treatment of asthma and judged 40/50 papers as having serious or extensive flaws that limit their value to guide decisions.¹

The Oxman and Guyatt scale used as a criterion is well recognised as useful in assessing the quality of systematic reviews and meta-analyses where extensive selection, by the authors, of studies from the worldwide literature is made.² The reviews judged by Jadad et al as most rigorous all had extensive selection processesfor example, the Cochrane review of Gibson et al reviewed only 11 studies from a possible 53 selected from 156 source hits.³ In cases such as this it is essential that authors state clearly and in detail the methods they used to justify their extremely small sample from the total population. The Oxman scale is particularly sensitive to reviews with selection bias and rightly condemns them as variously flawed.

One of the papers criticised did not involve any selection at all.⁴ The paper clearly stated that all the controlled worldwide studies with data available and meeting the given criteria were used. The applicability of the Oxman scale to this particular meta-analysis may therefore be called into question.

Jadad has subsequently provided us with the individual components of the assessment, which showed that our paper had serious or extensive flaws. This was based on us not stating our reasons for believing that all the studies used were valid. We agree that some readers may not have understood the validity implications of the word "controlled" and accept that a more explicit statement was needed. The facts are that all 14 studies included in the analysis out of a total of 14 studies completed worldwide at the time point stated were standard randomised drug comparisons. They were all conducted according to European standards of good clinical practice and are valid according to the criteria underlying the Oxman index.

All Cochrane reviews, including those studies described by Jadad et al as rigorous, contact authors of the reviewed papers before publication to clarify matters of fact. It is unfortunate that this omission by Jadad et al to adopt the same procedures has resulted in the science of our paper being inappropriately classified as being severely or extensively flawed on the basis of our failing to detail the validation process.

N C Barnes, consultant physician. 
The London Chest Hospital, London E2 9JX

C Hallett, consultant statistician. 
Fast Cycle Sciences Limited, PO Box 221, Epsom KT17 2WF ch90810{at}GlaxoWellcome.co.uk

T A J Harris, professional relations manager. 
GlaxoWellcome, Uxbridge, Middlesex UB11 1BT

Competing interests: Dr Barnes has received research funding, sponsorship to attend meetings, and lecture fees from GlaxoWellcome; Mr Harris is employed by and Mr Hallett provides consultancy services to GlaxoWellcome.

Cochrane Collaboration should ensure equitable participation in management and policy

EDITORThe article by Jadad et al illustrates the success of the Cochrane Collaboration in minimising bias.¹ The collaboration scrupulously implements the science of data synthesis. We believe that the collaboration should apply the same scrupulous approach to ensure equitable participation in its management and policy. In our experience, men dominate editorial groups, so we examined the evidence to support or refute this initial impression.

We analysed the composition of editorial teams from the Cochrane Library, Issue 2, 1999, when there were 48 collaborative review groups, with data available on 45.² Each review group generally has one coordinating editor who has overall responsibility for the group, other editors who contribute to policy and content, and coordinators, who are full time employees organising the day to day editorial work.

We found that one quarter of collaborative review groups (11/45) had no women editors at all. Ten out of 45 coordinating editors were women (21%), and only 61 women were other editors (24%, 61/196). There were no women editors from developing countries. In contrast, the analysis showed that most coordinators were women (78%, 37/47). Although we did not collect data, editors are generally in secure posts, whereas coordinators are usually funded by short term grants, with contracts between three months and two years.

We believe that this imbalance was not intended by people organising review groups but is the result of several related factors. Cochrane contributors usually come from academia or medicine, both of which traditionally discriminate against women. The collaboration depends on individuals making considerable efforts on top of their existing jobs over long periods of time, which will discriminate against women, who are the main carers of children and have less flexibility to work outside office hours. In addition, the collaboration depends heavily on networking in the workplace, which is traditionally a male practice. Our findings raise interesting research questions about whether the predominance of men affects what reviews are done, what outcomes are chosen, and how results are interpreted.

The Cochrane Collaboration steering group has considered these findings. It is currently canvassing the views of collaboration members with respect to adding a principle concerning equity in relation to sex and other barriers to full participation at all levels. We look forward to explicit methods of how collaborative review groups will address the current inequities.

Carole Wilson, research fellow. 
Department of Sociology, Social Policy and Social Work Studies, University of Liverpool, Liverpool L69 7ZA

Paula Waugh, divisional secretary. 
Paul Garner, coordinating editor, Cochrane Infectious Diseases Group. 
International Health Division, Liverpool School of Tropical Medicine, Liverpool L3 5QA

Competing interests: None declared.

Authors' reply

EDITORThis issue of the BMJ contains four letters in response to our recent critical evaluation of systematic reviews and meta-analyses on the treatment of asthma.¹

Senn used most of the space available for his letter to address the limitations of RevMan software, an issue not related to our article. His assessment of our report as "biased" and "one sided" was based entirely on one sentence in the abstract and on our decision to use the Oxman and Guyatt index to assess the quality of the reviews. We agree that the first sentence of the conclusion of our abstract could be misleading if it were taken out of context. This sentence, which referred to the articles included in the review, reflected our findings accurately: all six reviews associated with industry had low quality scores. Our decision to use the Oxman and Guyatt index was based on the fact that it is still the only validated tool to appraise review articles. This index was first published in a medical journal² and, as Altman et al pointed out, includes questions that are likely to be part of any instrument to assess the quality of review articles.³ Lack of description of the literature search, selection criteria, and the methods used to synthesise the data are regarded as serious deficiencies, in any review article, by most standards.⁴

Barnes et al provided reasons for the low scores given to their article. They accept that they should have provided a more explicit statement on the validity of the "controlled" trials included in their review. A mere description of a study as a randomised controlled trial does not guarantee its validity.⁵ They did not feel the need to describe their literature search process because they stated that they had included all trials available, worldwide. This strong claim could only be verified by following a detailed description of the process to locate the studies. Replicability should be one of the essential features of a rigorous review.

The letter by Wilson et al makes two important, albeit unrelated, points: that there is sex bias within the Cochrane Collaboration and that it may have an effect on Cochrane reviews. We were glad to learn that the collaboration is acting upon their findings. Similar efforts may be required to ensure adequate balance and equity of the sexes in the generation of new health related knowledge throughout the world.

Alejandro R Jadad, professor. 
Department of Clinical Epidemiology and Biostatistics, McMaster University, 1200 Main Street West, Hamilton, Canada L8N 3Z5

Michael Moher, Royal College of General Practitioners research training fellow. 
Institute of Health Sciences, University of Oxford, Old Road, Headington, Oxford OX3 7LF

George P Browman, professor. 
Lynda Booker, research assistant. 
Department of Clinical Epidemiology and Biostatistics

Christopher Sigouin, doctoral student. 
Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Canada, L8N 3Z5

Mario Fuentes, research assistant. 
Robert Stevens, research assistant. 
Foresight Consultants, Dundas, Ontario, Canada L9H 2R5

Competing interests: Professor Jadad is codirector of the Canadian Cochrane Network and Centre.