Elsevier

Journal of Clinical Epidemiology

Volume 128, December 2020, Pages 29-34
Journal of Clinical Epidemiology

Original Article
Public availability and adherence to prespecified statistical analysis approaches was low in published randomized trials

https://doi.org/10.1016/j.jclinepi.2020.07.015Get rights and content

Abstract

Background and Objective

Prespecification of statistical methods in clinical trial protocols and statistical analysis plans can help to deter bias from p-hacking but is only effective if the prespecified approach is made available.

Study Design and Setting

For 100 randomized trials published in 2018 and indexed in PubMed, we evaluated how often a prespecified statistical analysis approach for the trial's primary outcome was publicly available. For each trial with an available prespecified analysis, we compared this with the trial publication to identify whether there were unexplained discrepancies.

Results

Only 12 of 100 trials (12%) had a publicly available prespecified analysis approach for their primary outcome; this document was dated before recruitment began for only two trials. Of the 12 trials with an available prespecified analysis approach, 11 (92%) had one or more unexplained discrepancies. Only 4 of 100 trials (4%) stated that the statistician was blinded until the SAP was signed off, and only 10 of 100 (10%) stated the statistician was blinded until the database was locked.

Conclusion

For most published trials, there is insufficient information available to determine whether the results may be subject to p-hacking. Where information was available, there were often unexplained discrepancies between the prespecified and final analysis methods.

Introduction

The statistical methods used to analyze a randomized trial can affect the results; for instance, excluding different participants or using different statistical models can change the size of the estimated treatment effect or P-value [[1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14]]. Selective reporting of the results is problematic in randomized trials [4,6,7,13,15], and choosing or modifying the planned analysis approach after seeing the trial data to obtain a more favorable result can introduce bias; this is commonly referred to as “p-hacking.” [[1], [2], [3], [4], [5],11,12,15] Prespecification of statistical methods is recommended, both to act as a deterrent and help to identify p-hacking [[1], [2], [3], [4], [5],11,12,15]. However, in order for prespecification to be an effective tool to prevent and identify p-hacking, the prespecified analysis approach must be publicly accessible; otherwise, there is no way for people who are not involved in the trial to verify whether investigators followed their prespecified approach.

Previous reviews have looked at how often protocols and statistical analysis plans (SAPs) are available for trials published in high-impact general medical journals [15,16]. However, these results are unlikely to be generalizable across all trials; for instance, some high-impact medical journals have policies requiring investigators to include protocols and SAPs as supplementary material alongside the trial publication, which is not the policy at most medical journals. We therefore undertook this study to evaluate, in a random sample of trials published in all journals indexed in PubMed, how often a prespecified analysis approach was publicly available for the trial's primary outcome, and how often this approach was modified without explanation.

Section snippets

Methods

We followed the same protocol as a previous review which focused on trials published in high-impact general medical journals [15], although we used different inclusion or exclusion criteria and used a different search strategy (described below). The protocol is available in the supplementary material.

Search results and characteristics of included studies

Our search identified 1,382 results (Figure 1). We then identified 100 eligible trials after reviewing 327 randomly ordered articles. Trial characteristics are shown in Table 1.

Protocols were available for 15 trials (15%) (8 published, 7 as supplementary material with publication, and 1 on a website; one trial had two protocols available from different sources). SAPs were available for 3 trials (3%) (0 published, 2 as supplementary material with publication, and 1 on a website). The three

Discussion

In this review of trials indexed in PubMed, we found that very few (12%) had a publicly available prespecified analysis approach. Furthermore, because most documents containing the prespecified analysis approach were dated after the trial had begun (or, in some cases, after the trial was completed), it was often difficult to ascertain whether the analysis approach in these documents had already been modified from the pretrial protocol.

Of the trials that did have a prespecified analysis approach

Conclusion

For most published trials, there is insufficient information available to determine whether results may be subject to bias due to p-hacking. Where information was available, there were often unexplained discrepancies between the prespecified and final analysis methods.

CRediT authorship contribution statement

Brennan C. Kahan: Conceptualization, Methodology, Investigation, Writing - original draft, Supervision, Project administration. Tahania Ahmad: Writing - review & editing, Investigation. Gordon Forbes: Conceptualization, Methodology, Investigation, Writing - review & editing. Suzie Cro: Conceptualization, Methodology, Formal analysis, Investigation, Data curation.

References (16)

There are more references available in the full text version of this article.

Cited by (0)

Funding statement: None.

Conflict of interest: None.

View full text