Re: Overlapping meta-analyses on the same topic: survey of published studies
The critical analysis by Siontis et al. (1) is timely and most welcome.
We agree that unnecessary double work should be avoided. Looking for solutions we have suggested an open scientific wiki-type database for critically appraised original studies (life story, features, entity of a study) to be freely accessible for use by guideline developers and other researchers (2).
However, as referees and associate editors we think that selective replication of systematic reviews could play an important quality ensuring role. While handling manuscripts, we are increasingly confronted with the problem of quality assessment of the accelerating number of meta-analyses.
Although Siontis et al. stress that the main results of their analysed meta-analyses remained the same irrespective of substantial variation and incompleteness of included studies we consider quality assessment of included and excluded studies a core element of meta-analyses which needs more attention. This seems particularly relevant for meta-analyses of complex interventions (3) as well as the increasing number of meta-analyses of cohort studies due to their inherent threats to validity by bias and confounding.
Apart from the Cochrane Collaboration, referees and editors of clinical journals usually are not equipped to meticulously control each item of the submitted manuscript, which would mean coming close to replicating the whole systematic review including systematic literature search, selection of eligible papers, critical appraisal, analysis and synthesis of included publications. This is simply beyond what a journal or an individual reviewer can or would be prepared to contribute.
Therefore, other approaches should be considered in order to guarantee that the data published are valid. One possibility would be to design and perform the meta-analysis as a project by two independent working groups and therefore planned replication right from the beginning. Already the protocol could be developed by two independent research groups. A consented protocol should be registered and published before initiation of the main work, the systematic review should be performed independently by the two research groups and both could be published or alternatively they could be consented and merged into one high quality review. Funding should consider costs for replication and the journal could promise conditional publication. It would be worth testing such an approach.
1) Siontis KC, Hernandez-Boussard T, Ioannidis JPA. Overlapping meta-analyses on the same topic: survey of published studies. BMJ 2013; 347: f4501 doi: 10.1136/bmj.f4501 (Published 19 July 2013).
2) Mühlhauser I, Meyer G. Evidence base in guideline generation in diabetes. Diabetologia 2013; 56: 1201-1209.
3) Lenz M, Steckelberg A, Richter B, Mühlhauser I. Meta-analysis does not allow appraisal of complex interventions in diabetes and hypertension self-management. A methodological review. Diabetologia 2007; 50: 1375–1383.
Ingrid Mühlhauser, MD, PhD, professor
Unit of Health Sciences and Education
University of Hamburg
Martin-Luther-King Platz 6
Gabriele Meyer, PhD, professor
Faculty of Medicine
Institute for Health and Nursing Science
Magdeburger Straße 8
D-06110 Halle (Saale)
Competing interests: No competing interests