Original ArticlePublic availability and adherence to prespecified statistical analysis approaches was low in published randomized trials
Introduction
The statistical methods used to analyze a randomized trial can affect the results; for instance, excluding different participants or using different statistical models can change the size of the estimated treatment effect or P-value [[1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14]]. Selective reporting of the results is problematic in randomized trials [4,6,7,13,15], and choosing or modifying the planned analysis approach after seeing the trial data to obtain a more favorable result can introduce bias; this is commonly referred to as “p-hacking.” [[1], [2], [3], [4], [5],11,12,15] Prespecification of statistical methods is recommended, both to act as a deterrent and help to identify p-hacking [[1], [2], [3], [4], [5],11,12,15]. However, in order for prespecification to be an effective tool to prevent and identify p-hacking, the prespecified analysis approach must be publicly accessible; otherwise, there is no way for people who are not involved in the trial to verify whether investigators followed their prespecified approach.
Previous reviews have looked at how often protocols and statistical analysis plans (SAPs) are available for trials published in high-impact general medical journals [15,16]. However, these results are unlikely to be generalizable across all trials; for instance, some high-impact medical journals have policies requiring investigators to include protocols and SAPs as supplementary material alongside the trial publication, which is not the policy at most medical journals. We therefore undertook this study to evaluate, in a random sample of trials published in all journals indexed in PubMed, how often a prespecified analysis approach was publicly available for the trial's primary outcome, and how often this approach was modified without explanation.
Section snippets
Methods
We followed the same protocol as a previous review which focused on trials published in high-impact general medical journals [15], although we used different inclusion or exclusion criteria and used a different search strategy (described below). The protocol is available in the supplementary material.
Search results and characteristics of included studies
Our search identified 1,382 results (Figure 1). We then identified 100 eligible trials after reviewing 327 randomly ordered articles. Trial characteristics are shown in Table 1.
Protocols were available for 15 trials (15%) (8 published, 7 as supplementary material with publication, and 1 on a website; one trial had two protocols available from different sources). SAPs were available for 3 trials (3%) (0 published, 2 as supplementary material with publication, and 1 on a website). The three
Discussion
In this review of trials indexed in PubMed, we found that very few (12%) had a publicly available prespecified analysis approach. Furthermore, because most documents containing the prespecified analysis approach were dated after the trial had begun (or, in some cases, after the trial was completed), it was often difficult to ascertain whether the analysis approach in these documents had already been modified from the pretrial protocol.
Of the trials that did have a prespecified analysis approach
Conclusion
For most published trials, there is insufficient information available to determine whether results may be subject to bias due to p-hacking. Where information was available, there were often unexplained discrepancies between the prespecified and final analysis methods.
CRediT authorship contribution statement
Brennan C. Kahan: Conceptualization, Methodology, Investigation, Writing - original draft, Supervision, Project administration. Tahania Ahmad: Writing - review & editing, Investigation. Gordon Forbes: Conceptualization, Methodology, Investigation, Writing - review & editing. Suzie Cro: Conceptualization, Methodology, Formal analysis, Investigation, Data curation.
References (16)
- et al.
Pre-specification of statistical analysis approaches in published clinical trial protocols was inadequate
J Clin Epidemiol
(2018) - et al.
Discordance between reported intention-to-treat and per protocol analyses
J Clin Epidemiol
(2007) - et al.
Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews
J Clin Epidemiol
(2013) - et al.
SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
BMJ
(2013) - et al.
Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols
BMJ
(2008) - et al.
Guidelines for the content of statistical analysis Plans in clinical trials
JAMA
(2017) Statistical principles for clinical trials. International conference on harmonisation E9 expert working group
Stat Med
(1999)- et al.
Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials
PLoS Med
(2014)
Cited by (0)
Funding statement: None.
Conflict of interest: None.