Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database surveyBMJ 2012; 344 doi: https://doi.org/10.1136/bmj.d7762 (Published 03 January 2012) Cite this as: BMJ 2012;344:d7762
- Ikhlaaq Ahmed, postgraduate student1,
- Alexander J Sutton, professor of medical statistics2,
- Richard D Riley, senior lecturer in medical statistics3
- 1MRC Midlands Hub for Trials Methodology Research, School of Health and Population Sciences, University of Birmingham, Birmingham B15 2TT, UK
- 2Department of Health Sciences, University of Leicester, Leicester LE1 7RH, UK
- 3School of Health and Population Sciences, University of Birmingham
- Correspondence to: R D Riley
- Accepted 7 November 2011
Objective To examine the potential for publication bias, data availability bias, and reviewer selection bias in recently published meta-analyses that use individual participant data and to investigate whether authors of such meta-analyses seemed aware of these issues.
Design In a database of 383 meta-analyses of individual participant data that were published between 1991 and March 2009, we surveyed the 31 most recent meta-analyses of randomised trials that examined whether an intervention was effective. Identification of relevant articles and data extraction was undertaken by one author and checked by another.
Results Only nine (29%) of the 31 meta-analyses included individual participant data from “grey literature” (such as unpublished studies) in their primary meta-analysis, and the potential for publication bias was discussed or investigated in just 10 (32%). Sixteen (52%) of the 31 meta-analyses did not obtain all the individual participant data requested, yet five of these (31%) did not mention this as a potential limitation, and only six (38%) examined how trials without individual participant data might affect the conclusions. In nine (29%) of the meta-analyses reviewer selection bias was a potential issue, as the identification of relevant trials was either not stated or based on a more selective, non-systematic approach. Investigation of four meta-analyses containing data from ≥10 trials revealed one with an asymmetric funnel plot consistent with publication bias, and the inclusion of studies without individual participant data revealed additional heterogeneity between trials.
Conclusions Publication, availability, and selection biases are a potential concern for meta-analyses of individual participant data, but many reviewers neglect to examine or discuss them. These issues warn against uncritically viewing any meta-analysis that uses individual participant data as the most reliable. Reviewers should seek individual participant data from all studies identified by a systematic review; include, where possible, aggregate data from any studies lacking individual participant data to consider their potential impact; and investigate funnel plot asymmetry in line with recent guidelines.
Contributorship: RDR conceived and supervised the study, alongside AJS. IA identified relevant articles for the survey, checked by RDR and AJS. IA performed data extractions and initial meta-analyses, checked by RDR and AJS. RDR obtained the aggregate data from the two non-individual participant data trials in example 2 and extended the meta-analysis accordingly. IA drafted the first version of the article, and this was revised by RDR and AJS.
Funding: IA is funded by the MRC Midlands Hub for Trials Methodology Research, of which RDR is its deputy director.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approcal: Not required.
Data sharing: No additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.