BMJ  2005;330:753 (2 April), doi:10.1136/bmj.38356.424606.8F (published 28 January 2005)

Paper

Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors

¹ University Health Network, Department of Medicine, Suite RFE 3-805, 190 Elizabeth Street, Toronto, ON M5G 2C4, Canada, ² Cancer Research UK/NHS Centre for Statistics in Medicine, Oxford

Abstract

Objective To examine the extent and nature of outcome reporting bias in a broad cohort of published randomised trials.

Design Retrospective review of publications and follow up survey of authors.

Cohort All journal articles of randomised trials indexed in PubMed whose primary publication appeared in December 2000.

Main outcome measures Prevalence of incompletely reported outcomes per trial; reasons for not reporting outcomes; association between completeness of reporting and statistical significance.

Results 519 trials with 553 publications and 10 557 outcomes were identified. Survey responders (response rate 69%) provided information on unreported outcomes but were often unreliable—for 32% of those who denied the existence of such outcomes there was evidence to the contrary in their publications. On average, over 20% of the outcomes measured in a parallel group trial were incompletely reported. Within a trial, such outcomes had a higher odds of being statistically non-significant compared with fully reported outcomes (odds ratio 2.0 (95% confidence interval 1.6 to 2.7) for efficacy outcomes; 1.9 (1.1 to 3.5) for harm outcomes). The most commonly reported reasons for omitting efficacy outcomes included space constraints, lack of clinical importance, and lack of statistical significance.

Conclusions Incomplete reporting of outcomes within published articles of randomised trials is common and is associated with statistical non-significance. The medical literature therefore represents a selective and biased subset of study outcomes, and trial protocols should be made publicly available.

Introduction

Outcome reporting bias refers to the selective reporting of some results but not others in trial publications. Direct evidence of such bias has recently been shown in two cohort studies that compared trial publications with the original protocols.^{1 2} However, it is unknown whether selective outcome reporting can be identified when protocols are unavailable.

We used a large representative sample of publications of randomised trials indexed on PubMed to determine the prevalence of incomplete outcome reporting; the reasons for omitting outcomes; and the degree of association between completeness of reporting and statistical significance for trial outcomes.

Methods

Study selection
We identified primary publications of randomised trials published in December 2000 and included in PubMed by August 2002. A primary publication was the first report of final trial results. (See bmj.com for details of search strategy.)

We identified extra journal publications for these trials through a survey of authors, as well as literature searches of PubMed, Embase, the Cochrane Controlled Trials Register, and PsychINFO. For each trial, we reviewed the primary and any subsequent publications to extract the number and characteristics of reported outcomes.

Reporting of outcomes
We identified unreported outcomes if they were described in the methods section but not the results section of any publication. Using a pre-piloted questionnaire, we also asked the authors to list and describe any outcomes that were not reported in the published papers and the reasons why.

For each identified outcome, we recorded the level of reporting as one of four levels based on the amount of data presented in any of the journal publications: fully reported if sufficient data were provided for inclusion in a meta-analysis; partially reported if the publications provided only some of the data necessary for meta-analysis; qualitatively reported if the publications presented only a P value or some indication of the presence or absence of statistical significance; and unreported if no data were provided in any of the publications despite being identified in the methods section or the authors' responses to our survey.

We used two further terms to describe composite levels of reporting. "Reported outcomes" referred to those with some data presented in any of the publications (full, partial, and qualitative). "Incompletely reported outcomes" referred to those with inadequate data for meta-analysis (partial, qualitative, and unreported).

Statistical analyses
We conducted analyses at the trial level stratified by efficacy and harm outcomes. Primary variables of interest included the proportion of incompletely reported outcomes per trial and the reasons given by authors for not reporting outcomes. We also examined the association between the level of outcome reporting and statistical significance. For each trial, we created a 2x2 table for the outcomes, relating the level of reporting (full v incomplete) to statistical significance at the P < 0.05 level. We calculated odds ratios for each trial and pooled these using a random effects meta-analysis to provide an overall estimate of outcome reporting bias. We conducted sensitivity analyses by excluding trials without survey responses as well as excluding physiological and pharmacokinetic trials. We also assessed the impact of using a different cut-off point for dichotomising the level of reporting (fully or partially reported v qualitatively reported or unreported). We used exploratory meta-regression to evaluate the effect of different factors on the size of bias. (See bmj.com for details.)

Results

We identified 519 trials (383 parallel group, 116 crossover, and 20 other designs) with 553 publications and 10 557 outcomes (8325 efficacy and 2232 harm outcomes). Median sample sizes were 80 (10-90th centile range 25-369) and 15 (8-38) for parallel group and crossover trials respectively. A detailed report of the cohort characteristics has been published separately.³

For our questionnaire survey of contact authors, 69% (356/519) responded. Among the 466 trials with identified funding sources, we obtained lower response rates for those funded solely by industry (65% (108/167)) compared with those with partial industry funding (75% (46/61)), non-industry funding (80% (147/184)), or no funding (100% (54/54)).

Prevalence of incompletely reported outcomes
From publications and survey responses, we identified a median of 11 (10-90th centile range 3-36) efficacy outcomes per trial (n = 505) and 4 (1-17) harm outcomes per trial (n = 308). Of these trials, 75% (380/505) and 64% (196/308) respectively did not fully report all their efficacy and harm outcomes in any journal publications (table 1). Of the 232 trials (45%) that defined primary outcomes in their publications, 83 (36%) presented at least one that was incompletely reported.

View this table: [in this window] [in a new window]   Table 1 Median proportion of incompletely reported efficacy and harm outcomes per trial, stratified by trial characteristics, among 519 randomised trials published in December 2000 and cited in PubMed

 

Parallel group trials contained much lower percentages of incompletely reported efficacy and harm outcomes than crossover trials (table 1). We found little difference between specialty and general medical journals, but greater deficiencies for reporting of harm outcomes among trials that were solely funded by industry.

Prevalence of unreported outcomes
Among 356 survey responders, 281 stated that there were no unreported outcomes. However, for 32% (90/281) of these responses, we found evidence of outcomes that were mentioned in the methods section but not the results section of individual publications.

Using combined data from survey responses and publications, we identified at least one unreported efficacy outcome in 33% (169/505) of trials that measured efficacy data, and 28% (85/308) of trials with unreported harms data. A median of 2 (10-90th centile range 1-7) efficacy and 2 (1-6) harm outcomes were unreported for each of these trials.

Characteristics of unreported outcomes based on survey responses
Fifty three survey responders provided data on the clinical importance of 238 unreported efficacy outcomes (table 2). Of these trials, 26% (14/53) had unreported outcomes that were categorised as having high clinical importance. According to survey responses, the important efficacy outcomes for three of these trials were to be reported in future publications. All unreported harm outcomes were classified as having low or moderate clinical importance (table 2); 13 authors provided the statistical significance of their unreported harm outcomes, all of which were non-significant.

View this table: [in this window] [in a new window]   Table 2 Randomised trials with at least one unreported outcome for which survey responders provided data on clinical importance and specification

 

Fifty four survey responders indicated the specification of their unreported efficacy outcomes (table 2). Primary efficacy outcomes were unreported for 13 trials; according to authors, the primary outcomes for six of these trials were to be reported in future manuscript submissions. Three out of 18 trials (17%) had at least one unreported primary harm outcome listed in the survey responses.

The most common reasons given by authors for not reporting efficacy outcomes were journal space restrictions (47%), lack of clinical importance (37%), and lack of statistical significance (24%). For harm outcomes, the commonest reasons were lack of clinical importance (75%) or of statistical significance (50%).

Association between completeness of reporting and statistical significance
Statistically significant outcomes had a higher odds of being fully reported than those that were non-significant (figure). The pooled odds ratio for outcome reporting bias in all trials was 2.0 (95% confidence interval 1.6 to 2.7) for efficacy outcomes and 1.9 (1.1 to 3.5) for harms. Across study designs, the size of bias was similar for efficacy outcomes. We found greater variation between study designs with harm outcomes. (See bmj.com for details.)

View larger version (35K): [in this window] [in a new window]   Odds ratios (black squares) with 95% confidence intervals for outcome reporting bias in randomised trials published in December 2000 and cited in PubMed. Size of the square reflects the weight of the trial in calculating the pooled odds ratio (diamond and dotted line)

 

The overall odds ratios were not greatly affected by our sensitivity analyses. Dichotomising the level of outcome reporting differently (fully or partially reported v qualitatively reported or unreported) produced greater bias.

The final exploratory model in our exploratory multivariate analysis revealed that multicentre trials were associated with significantly less bias than single centre trials (odds ratio 0.44 (95% confidence interval 0.24 to 0.80)). Those that defined primary outcomes in their publications were associated with greater bias than those not specifying any (1.8 (1.0 to 3.2)).

Discussion

We identified deficiencies in outcome reporting in a large sample of randomised trials unrestricted by study location or funding source. Two recent cohort studies compared study protocols with publications to show similar sizes of outcome reporting bias.^{1 2} Other evidence of selective outcome reporting is limited to case reports.^4-6

Contacting authors for information about outcomes
Our survey results indicate that a response rate of almost 70% is achievable when asking authors for unreported outcomes. However, these studies were published recently, which is often not the case for systematic reviews. In addition, many authors provided responses that contradicted evidence within their publications. Response rates were lower for industry funded trials.

How common are incompletely reported outcomes?
We have shown that trial outcomes are often reported inadequately for inclusion in meta-analysis. Over a third of trials had at least one primary outcome that was incompletely reported. Publications for crossover trials were particularly deficient. The revised CONSORT statement and its extension for harms recommend full reporting of data for all primary and secondary outcomes.^{7 8}

Why are outcomes unreported?
Unreported outcomes were common, with over a third of trials omitting an average of two or more outcomes each. The decision to omit outcomes seems to be based on a combination of journal space restrictions, the importance of the outcome, and the statistical results. However, space constraints and a lack of clinical importance may well be directly associated with a lack of statistical significance.

Is outcome reporting associated with statistical significance?
On average, the completeness of outcome reporting was biased to favour statistically significant outcomes. The sizes of bias for efficacy and harm outcomes are similar to those reported in other studies.^{1 2} However, we identified fewer unreported outcomes because we did not review trial protocols, so we may have underestimated reporting deficiencies.

Limitations of study
Response bias was expected in our study, as we relied on self reported data from questionnaires and publications. For 32% of authors who denied the existence of unreported outcomes there was evidence to the contrary in their publications. In addition, we found lower response rates for trials funded solely by industry. A non-response or inaccurate response may arise from a reluctance to reveal biased practices, and we may therefore have underestimated the deficiencies in outcome reporting.

Implications for health care and research
Outcome reporting bias acts in addition to and in the same direction as publication bias of entire studies to produce inflated estimates of treatment effect.⁹ At its worst, the suppression of non-significant findings could lead to the use of harmful interventions. Perhaps more commonly, a treatment may be considered to be of more value than it merits.

What is already known on this topic Selective reporting of some measured outcomes but not others within published trials has been shown in cohorts restricted by geography and funding source Outcome reporting bias limits the critical interpretation of individual trials as well as the conclusions of literature reviews What this study adds Outcome reporting bias exists in published trials indexed on PubMed Contacting authors for a list of unreported outcomes has the potential to identify important omissions from publications, although responses are often unreliable Clinically important trial outcomes are often inadequately reported Trials should be registered, and protocols should be made publicly available

To limit outcome reporting bias, researchers and journal editors should ensure that complete data are provided for all pre-specified trial outcomes, independent of their results. Discrepancies between outcomes in the methods and results sections of publications can also be addressed during peer review. Journal internet sites will help to alleviate concerns over space restrictions.

Trials should be registered and protocols should be made available in the public domain before trial completion. At the least, they should be submitted with manuscripts and reviewed when being considered for publication by journals.

---

This is the abridged version of an article that was posted on bmj.com on 28 January 2005: http://bmj.com/cgi/doi/10.1136/bmj.38356.424606.8F

Contributors: See bmj.com

Funding: AWC was funded by the Rhodes Trust; DGA is funded by Cancer Research UK. The funding sources had no role in any aspect of the study.

Competing interests: None declared.

Ethical approval: None required.

References

1. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291: 2457-65.[Abstract/Free Full Text]
2. Chan AW, Krlea-Jeri K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004;171: 735-40.[Abstract/Free Full Text]
3. Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet (in press).
4. West RR, Jones DA. Publication bias in statistical overview of trials: example of psychological rehabilitation following myocardial infarction [abstract]. In: Proceedings of the 2nd International Conference on the Scientific Basis of Health Services and 5th Annual Cochrane Colloquium; 1997 October 8-12, Amsterdam: 82.
5. McCormack K, Scott NW, Grant AM. Outcome reporting bias and individual patient data meta-analysis: a case study in surgery [abstract]. In: Abstracts for workshops and scientific sessions, 9th International Cochrane Colloquium; 2001 October 9-13, Lyons: 34-5.
6. Felson DT. Bias in meta-analytic research. J Clin Epidemiol 1992;45: 885-92.[CrossRef][ISI][Medline]
7. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001;357: 1191-4.[CrossRef][ISI][Medline]
8. Ioannidis JP, Evans SJ, Gøtzsche PC, O'Neill RT, Altman DG, Schulz K, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141: 781-8.[Abstract/Free Full Text]
9. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess 2000;4(10): 1-115.[Medline]

(Accepted 20 December 2004)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Believability of relative risks and odds ratios in abstracts: cross sectional study

Peter C Gøtzsche
BMJ 2006 333: 231-234. [Abstract] [Full Text] [PDF]

Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1)

Karmela Krleza-Jeric, An-Wen Chan, Kay Dickersin, Ida Sim, Jeremy Grimshaw, Christian Gluud for the Ottawa Group
BMJ 2005 330: 956-958. [Extract] [Full Text] [PDF]

Reporting of trial outcomes is incomplete and biased
BMJ 2005 330: 0. [Full Text]

This article has been cited by other articles:

• Mathoulin-Pelissier, S., Gourgou-Bourgade, S., Bonnetain, F., Kramar, A. (2008). Survival End Point Reporting in Randomized Cancer Clinical Trials: A Review of Major Journals. JCO 26: 3721-3726 [Abstract] [Full text]  
• Hackbarth, D. (2008). Research Reporting and Evidence of Effectiveness: Why "No Difference" Matters. Am J Crit Care 17: 218-220 [Full text]  
• Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., Rosenthal, R. (2008). Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy. NEJM 358: 252-260 [Abstract] [Full text]  
• Patsopoulos, N. A., Tatsioni, A., Ioannidis, J. P. A. (2007). Claims of Sex Differences: An Empirical Assessment in Genetic Associations. JAMA 298: 880-893 [Abstract] [Full text]  
• Cleophas, T. J., Zwinderman, A. H. (2007). Meta-Analysis. Circulation 115: 2870-2875 [Full text]  
• Ioannidis, J. P., Trikalinos, T. A (2007). Authors' response to V Johnson and Y Yuan. Clin Trials 4: 256-257  
• Coultas, D. (2007). Ethical Considerations in the Interpretation and Communication of Clinical Trial Results. Proc Am Thorac Soc 4: 194-199 [Abstract] [Full text]  
• Hall, R., de Antueno, C., Webber, A. (2007). Publication bias in the medical literature: a review by a Canadian Research Ethics Board: [Biais de publication dans la litterature medicale : un compte rendu d'un Comite d'ethique de recherche canadien]. Canadian J. Anesthesia 54: 380-388 [Abstract] [Full text]  
• Ioannidis, J. P.A., Trikalinos, T. A. (2007). The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ 176: 1091-1096 [Abstract] [Full text]  
• Jefferson, T., Zarra, L., Stoica, L. (2006). Attenti Alle Bufale ('Beware of Red Herrings'), or, How to Make Evidence-Based Medicine Work for You. JRSM 99: 625-627 [Full text]  
• Bero, L A (2006). Clinical trial registration at Tobacco Control. Tobacco Control 15: 417-418 [Full text]  
• Gotzsche, P. C (2006). Believability of relative risks and odds ratios in abstracts: cross sectional study. BMJ 333: 231-234 [Abstract] [Full text]  
• Gluud, L. L. (2006). Bias in Clinical Intervention Research. Am J Epidemiol 163: 493-501 [Abstract] [Full text]  
• Williamson, P R, Gamble, C, Altman, D G, Hutton, J L (2005). Outcome selection bias in meta-analysis. Stat Methods Med Res 14: 515-524 [Abstract]  
• (2005). From the Library. Br. J. Ophthalmol. 89: 930-930 [Full text]  
• Krleza-Jeric, K., Chan, A.-W., Dickersin, K., Sim, I., Grimshaw, J., Gluud, C., for the Ottawa Group, (2005). Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). BMJ 330: 956-958 [Full text]