BMJ 2000;320:1574-1577 ( 10 June )

Papers

Empirical assessment of effect of publication bias on meta-analyses

A J Sutton, lecturer in medical statistics, ^a S J Duval, assistant professor, ^b R L Tweedie, head, ^c K R Abrams, senior lecturer in medical statistics, ^a D R Jones, professor in medical statistics. ^a

^a Department of Epidemiology and Public Health, University of Leicester, Leicester LE1 6TP, ^b Division of Epidemiology, School of Public Health, University of Minnesota, Minneapolis, USA, ^c Department of Biostatistics, School of Public Health, University of Minnesota

Correspondence to: A J Sutton ajs22{at}le.ac.uk

Objective: To assess the effect of publication bias on the results and conclusions of systematic reviews and meta-analyses.
Design: Analysis of published meta-analyses by trim and fill method.
Studies: 48 reviews in Cochrane Database of Systematic Reviews that considered a binary endpoint and contained 10 or more individual studies.
Main outcome measures: Number of reviews with missing studies and effect on conclusions of meta-analyses.
Results: The trim and fill fixed effects analysis method estimated that 26 (54%) of reviews had missing studies and in 10 the number missing was significant. The corresponding figures with a random effects model were 23 (48%) and eight. In four cases, statistical inferences regarding the effect of the intervention were changed after the overall estimate for publication bias was adjusted for.
Conclusions: Publication or related biases were common within the sample of meta-analyses assessed. In most cases these biases did not affect the conclusions. Nevertheless, researchers should check routinely whether conclusions of systematic reviews are robust to possible non-random selection mechanisms.

---

© BMJ 2000
CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Review of publication bias in studies on publication bias: Meta-research on publication bias does not help transfer research results to patient care

Peter Kranke
BMJ 2005 331: 638. [Extract] [Full Text]

Review of publication bias in studies on publication bias: Here's a proposal for editors that may help reduce publication bias

M Maria Glymour and Ichiro Kawachi
BMJ 2005 331: 638. [Extract] [Full Text]

Passive smoking: Wider evidence needs to be interpreted

Julia Critchley
BMJ 2003 327: 501. [Extract] [Full Text]

Half of meta-analyses may contain publication bias
BMJ 2000 320: 0. [Full Text]

This article has been cited by other articles:

• Carriquiry, M., Weber, W. J., Crooker, B. A. (2008). Administration of Bovine Somatotropin in Early Lactation: A Meta-Analysis of Production Responses by Multiparous Holstein Cows. J DAIRY SCI 91: 2641-2652 [Abstract] [Full text]  
• Boluyt, N., van der Lee, J. H., Moyer, V. A., Brand, P. L. P., Offringa, M. (2007). State of the Evidence on Acute Asthma Management in Children: A Critical Appraisal of Systematic Reviews. Pediatrics 120: 1334-1343 [Abstract] [Full text]  
• Leaper, C., Ayres, M. M. (2007). A Meta-Analytic Review of Gender Variations in Adults' Language Use: Talkativeness, Affiliative Speech, and Assertive Speech. Pers Soc Psychol Rev 11: 328-363 [Abstract]  
• Sung, L., Nathan, P. C., Alibhai, S. M.H., Tomlinson, G. A., Beyene, J. (2007). Meta-analysis: Effect of Prophylactic Hematopoietic Colony-Stimulating Factors on Mortality and Outcomes of Infection. ANN INTERN MED 147: 400-411 [Abstract] [Full text]  
• Iqbal, A., Schirmer, H., Lunde, P., Figenschau, Y., Rasmussen, K., Jorde, R. (2007). Thyroid Stimulating Hormone and Left Ventricular Function. J. Clin. Endocrinol. Metab. 92: 3504-3510 [Abstract] [Full text]  
• Hall, R., de Antueno, C., Webber, A. (2007). Publication bias in the medical literature: a review by a Canadian Research Ethics Board: [Biais de publication dans la litterature medicale : un compte rendu d'un Comite d'ethique de recherche canadien]. Canadian J. Anesthesia 54: 380-388 [Abstract] [Full text]  
• Strazzullo, P., Kerry, S. M., Barbato, A., Versiero, M., D'Elia, L., Cappuccio, F. P. (2007). Do Statins Reduce Blood Pressure?: A Meta-Analysis of Randomized, Controlled Trials. Hypertension 49: 792-798 [Abstract] [Full text]  
• Braye, S., Preston-Shoot, M. (2007). On Systematic Reviews in Social Work: Observations from Teaching, Learning and Assessment of Law in Social Work Education. Br J Soc Work 37: 313-334 [Abstract] [Full text]  
• Wiesbauer, F., Schlager, O., Domanovits, H., Wildner, B., Maurer, G., Muellner, M., Blessberger, H., Schillinger, M. (2007). Perioperative {beta}-Blockers for Preventing Surgery-Related Mortality and Morbidity: A Systematic Review and Meta-Analysis. Anesth. Analg. 104: 27-41 [Abstract] [Full text]  
• Flores-Mir, C., Major, P. W. (2006). A systematic review of cephalometric facial soft tissue changes with the Activator and Bionator appliances in Class II division 1 subjects. Eur J Orthod 28: 586-593 [Abstract] [Full text]  
• Dekker, A., Bulley, S., Beyene, J., Dupuis, L. L., Doyle, J. J., Sung, L. (2006). Meta-Analysis of Randomized Controlled Trials of Prophylactic Granulocyte Colony-Stimulating Factor and Granulocyte-Macrophage Colony-Stimulating Factor After Autologous and Allogeneic Stem Cell Transplantation. JCO 24: 5207-5215 [Abstract] [Full text]  
• Oliveira, G. J., Leles, C. R. (2006). Critical appraisal and positive outcome bias in case reports published in brazilian dental journals.. J Dent Educ 70: 869-874 [Abstract] [Full text]  
• Bagos, P G, Nikolopoulos, G, Ioannidis, A (2006). Chlamydia pneumoniae infection and the risk of multiple sclerosis: a meta-analysis. Mult Scler 12: 397-411 [Abstract]  
• Abou-Setta, A. M., Mansour, R. T., Al-Inany, H. G., Aboulghar, M. A., Kamal, A., Aboulghar, M. A., Serour, G. I. (2006). Intrauterine insemination catheters for assisted reproduction: a systematic review and meta-analysis. Hum Reprod 21: 1961-1967 [Abstract] [Full text]  
• Smith-Warner, S. A., Spiegelman, D., Ritz, J., Albanes, D., Beeson, W. L., Bernstein, L., Berrino, F., van den Brandt, P. A., Buring, J. E., Cho, E., Colditz, G. A., Folsom, A. R., Freudenheim, J. L., Giovannucci, E., Goldbohm, R. A., Graham, S., Harnack, L., Horn-Ross, P. L., Krogh, V., Leitzmann, M. F., McCullough, M. L., Miller, A. B., Rodriguez, C., Rohan, T. E., Schatzkin, A., Shore, R., Virtanen, M., Willett, W. C., Wolk, A., Zeleniuch-Jacquotte, A., Zhang, S. M., Hunter, D. J. (2006). Methods for Pooling Results of Epidemiologic Studies: The Pooling Project of Prospective Studies of Diet and Cancer. Am J Epidemiol 163: 1053-1064 [Abstract] [Full text]  
• Athanasiou, T., Jones, C., Jin, R., Grunkemeier, G. L., Ross, D. N. (2006). Homograft Implantation Techniques in the Aortic Position: To Preserve or Replace the Aortic Root?. Ann. Thorac. Surg. 81: 1578-1585 [Abstract] [Full text]  
• Smetana, G. W., Lawrence, V. A., Cornell, J. E. (2006). Preoperative pulmonary risk stratification for noncardiothoracic surgery: systematic review for the American College of Physicians.. ANN INTERN MED 144: 581-595 [Abstract] [Full text]  
• Dentali, F., Crowther, M., Ageno, W. (2006). Thrombophilic abnormalities, oral contraceptives, and risk of cerebral vein thrombosis: a meta-analysis. Blood 107: 2766-2773 [Abstract] [Full text]  
• Delgado-Rodriguez, M. (2006). Systematic reviews of meta-analyses: applications and limitations. J. Epidemiol. Community Health 60: 90-92 [Full text]  
• Lagravere, M. O., Heo, G., Major, P. W., Flores-Mir, C. (2006). Meta-analysis of immediate changes with rapid maxillary expansion treatment. Journal of the American Dental Association 137: 44-53 [Abstract] [Full text]  
• Kalus, J. S., Coleman, C. I., White, C. M. (2006). The Impact of Suppressing the Renin-Angiotensin System on Atrial Fibrillation. J Clin Pharmacol 46: 21-28 [Abstract] [Full text]  
• Akobeng, A K, Ramanan, A V, Buchan, I, Heller, R F (2006). Effect of breast feeding on risk of coeliac disease: a systematic review and meta-analysis of observational studies. Arch. Dis. Child. 91: 39-43 [Abstract] [Full text]  
• Collins, J. A., Blake, J. M., Crosignani, P. G. (2005). Breast cancer risk with postmenopausal hormonal treatment. Hum Reprod Update 11: 545-560 [Abstract] [Full text]  
• Palma, S., Delgado-Rodriguez, M. (2005). Assessment of publication bias in meta-analyses of cardiovascular diseases. J. Epidemiol. Community Health 59: 864-869 [Abstract] [Full text]  
• Kranke, P. (2005). Review of publication bias in studies on publication bias: Meta-research on publication bias does not help transfer research results to patient care. BMJ 331: 638-638 [Full text]  
• Glymour, M M., Kawachi, I. (2005). Review of publication bias in studies on publication bias: Here's a proposal for editors that may help reduce publication bias. BMJ 331: 638-638 [Full text]  
• Sin, D. D., Wu, L., Man, S. F. P. (2005). The Relationship Between Reduced Lung Function and Cardiovascular Mortality: A Population-Based Study and a Systematic Review of the Literature. Chest 127: 1952-1959 [Abstract] [Full text]  
• Juffer, F., van IJzendoorn, M. H. (2005). Behavior Problems and Mental Health Referrals of International Adoptees: A Meta-analysis. JAMA 293: 2501-2515 [Abstract] [Full text]  
• Tinmouth, A., MacDougall, L., Fergusson, D., Amin, M., Graham, I. D., Hebert, P. C., Wilson, K. (2005). Reducing the Amount of Blood Transfused: A Systematic Review of Behavioral Interventions to Change Physicians' Transfusion Practices. Arch Intern Med 165: 845-852 [Abstract] [Full text]  
• Moles, D.R., Needleman, I.G., Niederman, R., Lau, J. (2005). Introduction to Cumulative Meta-analysis in Dentistry: Lessons Learned from Undertaking a Cumulative Meta-analysis in Periodontology. J. Dent. Res. 84: 345-349 [Abstract] [Full text]  
• Streppel, M. T., Arends, L. R., van 't Veer, P., Grobbee, D. E., Geleijnse, J. M. (2005). Dietary Fiber and Blood Pressure: A Meta-analysis of Randomized Placebo-Controlled Trials. Arch Intern Med 165: 150-156 [Abstract] [Full text]  
• Sung, L., Nathan, P. C., Lange, B., Beyene, J., Buchanan, G. R. (2004). Prophylactic Granulocyte Colony-Stimulating Factor and Granulocyte-Macrophage Colony-Stimulating Factor Decrease Febrile Neutropenia After Chemotherapy in Children With Cancer: A Meta-Analysis of Randomized Controlled Trials. JCO 22: 3350-3356 [Abstract] [Full text]  
• Cappuccio, F. P, Kerry, S. M, Forbes, L., Donald, A. (2004). Blood pressure control by home monitoring: meta-analysis of randomised trials. BMJ 329: 145- [Abstract] [Full text]  
• Healy, D G, Abou-Sleiman, P M, Lees, A J, Casas, J P, Quinn, N, Bhatia, K, Hingorani, A D, Wood, N W (2004). Tau gene and Parkinson's disease: a case-control study and meta-analysis. J. Neurol. Neurosurg. Psychiatry 75: 962-965 [Abstract] [Full text]  
• Casas, J. P., Bautista, L. E., Humphries, S. E., Hingorani, A. D. (2004). Endothelial Nitric Oxide Synthase Genotype and Ischemic Heart Disease: Meta-Analysis of 26 Studies Involving 23028 Subjects. Circulation 109: 1359-1365 [Abstract] [Full text]  
• Nathan, P. C., Sung, L., Crump, M., Beyene, J. (2004). Consolidation Therapy With Autologous Bone Marrow Transplantation in Adults With Acute Myeloid Leukemia: A Meta-analysis. JNCI J Natl Cancer Inst 96: 38-45 [Abstract] [Full text]  
• Critchley, J. (2003). Passive smoking: Wider evidence needs to be interpreted. BMJ 327: 501-501 [Full text]  
• Hoh, B. L., Rabinov, J. D., Pryor, J. C., Carter, B. S., Barker, F. G. II (2003). In-Hospital Morbidity and Mortality after Endovascular Treatment of Unruptured Intracranial Aneurysms in the United States, 1996-2000: Effect of Hospital and Physician Volume. Am. J. Neuroradiol. 24: 1409-1420 [Abstract] [Full text]  
• Masson, L. F, McNeill, G., Avenell, A. (2003). Genetic variation and the lipid response to dietary intervention: a systematic review. Am. J. Clin. Nutr. 77: 1098-1111 [Abstract] [Full text]  
• Critchley, J A, Unal, B (2003). Health effects associated with smokeless tobacco: a systematic review. Thorax 58: 435-443 [Abstract] [Full text]  
• Astin, J. A., Shapiro, S. L., Eisenberg, D. M., Forys, K. L. (2003). Mind-Body Medicine: State of the Science, Implications for Practice. J Am Board Fam Med 16: 131-147 [Abstract] [Full text]  
• Gaertner, R R W, Theriault, G P (2002). Risk of bladder cancer in foundry workers: a meta-analysis. Occup. Environ. Med. 59: 655-663 [Abstract] [Full text]  
• Nelson, H. D., Humphrey, L. L., Nygren, P., Teutsch, S. M., Allan, J. D. (2002). Postmenopausal Hormone Replacement Therapy: Scientific Review. JAMA 288: 872-881 [Abstract] [Full text]  
• Miller, J., Chan, B. K.S., Nelson, H. D. (2002). Postmenopausal Estrogen Replacement and Risk for Venous Thromboembolism: A Systematic Review and Meta-Analysis for the U.S. Preventive Services Task Force. ANN INTERN MED 136: 680-690 [Abstract] [Full text]  
• Klassen, T. P., Wiebe, N., Russell, K., Stevens, K., Hartling, L., Craig, W. R., Moher, D. (2002). Abstracts of Randomized Controlled Trials Presented at the Society for Pediatric Research Meeting: An Example of Publication Bias. Arch Pediatr Adolesc Med 156: 474-479 [Abstract] [Full text]  
• Song, F., Khan, K. S, Dinnes, J., Sutton, A. J (2002). Asymmetric funnel plots and publication bias in meta-analyses of diagnostic accuracy. Int J Epidemiol 31: 88-95 [Abstract] [Full text]  
• Thoennissen, J., Herkner, H., Lang, W., Domanovits, H., Laggner, A. N., Mullner, M. (2001). Does bed rest after cervical or lumbar puncture prevent headache? A systematic review and meta-analysis. CMAJ 165: 1311-1316 [Abstract] [Full text]  
• Copas, J B, Shi, J Q (2001). A sensitivity analysis for publication bias in systematic reviews. Stat Methods Med Res 10: 251-265 [Abstract]  
• McQueen, M. J. (2001). Overview of Evidence-based Medicine: Challenges for Evidence-based Laboratory Medicine. Clin. Chem. 47: 1536-1546 [Abstract] [Full text]  
• Sterne, J. A C, Egger, M., Smith, G. D. (2001). Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis. BMJ 323: 101-105 [Full text]  
• Ross, S. D. (2001). Drug-Related Adverse Events: A Readers' Guide to Assessing Literature Reviews and Meta-analyses. Arch Intern Med 161: 1041-1046 [Full text]  
• Schwartz, M. D (2001). Review: prokinetics and histamine-2 receptor antagonists improve symptom scores in non-ulcer dyspepsia. Evid. Based Med. 6: 47-47 [Full text]  
• Sutton, A. J, Song, F., Gilbody, S. M, Abrams, K. R (2000). Modelling publication bias in meta-analysis: a review. Stat Methods Med Res 9: 421-445 [Abstract]  

Rapid Responses:

Read all Rapid Responses

High false positive rate for trim and fill method

Jonathan A C Sterne
bmj.com, 21 Jul 2000 [Full text]

Re: High false positive rate for trim and fill method

Alex J Sutton
bmj.com, 22 Sep 2000 [Full text]