- An-Wen Chan, assistant professor and Phelan scientist
The high frequency and negative impact of selective reporting of data from clinical trials are well documented.1 The widespread occurrence of data suppression means that healthcare practitioners and policy makers largely make decisions on the basis of an incomplete and biased subset of trial results. Selective reporting can often be identified by reviewing trial protocols and publications; it can be mitigated by defining standard core outcome sets for trials, and by ensuring access to all unpublished and published data⇓.2 3
However, two linked articles highlight major difficulties in obtaining access to protocols and unpublished data for both industry and non-industry trials and provide new insight into trialists’ reasons for suppressing data.4 5 These types of challenges have been described before,6 7 and they reinforce the core principle that full knowledge of both the methods and results for all trials, independent of publication status, is essential for a complete and unbiased evaluation of an intervention.
Sources of information that should be consulted when appraising an intervention include trial protocols, registry records, regulatory agency documents, trial datasets, and journal publications. Smyth and colleagues (doi:10.1136/bmj.c7153) also show the potential value of communicating with trialists and sponsors, although the yield was suboptimal. The public availability, reliability, and completeness of these information sources vary, but they provide complementary information across the stages from trial inception to dissemination. Unfortunately, the two most reliable sources—the raw dataset and the protocol approved by the research ethics committee—are usually not publicly available.
To improve this lack of transparency, Jefferson and colleagues (doi:10.1136/bmj.c7258) propose that the International Committee of Medical Journal Editors (ICMJE) should require submission of the most detailed anonymised dataset along with manuscripts.4 This policy would be a positive initial step, but several factors need to be considered. Firstly, the criteria defining what constitutes an adequately detailed trial dataset need to be clearly established to ensure that data are submitted in an appropriate format.8 Secondly, the policy would need to extend beyond ICMJE to capture a substantive proportion of published trials, most of which appear in specialty journals. Editorial policy would also not improve data availability for unpublished trials. Thirdly, journal editors and peer reviewers may not be in the best position to reanalyse a raw dataset. Most have limited time, funds, and statistical expertise.
Fourthly, access to the raw dataset is of little use without access to the full protocol. Protocols and their amendments provide key methodological information that places trial results in their proper context and enables critical appraisal of whether a given dataset is complete, unbiased, and applicable to a given patient population.3 The trial protocol is the most reliable source of methodological information because it is written before the study is started and cannot be influenced by the trial data. However, empirical evidence has shown that protocols are often incomplete.3 For example, primary outcomes were defined in only half of protocols in the cohort reviewed by Smyth and colleagues.5 This lack of prespecification is rarely acknowledged in trial reports and often leads to major discrepancies between protocols and publications.2 5 The ongoing SPIRIT Initiative (Standard Protocol Items for Randomized Trials) aims to improve the completeness and quality of protocols by providing guidance on the key areas needed in the protocol.9
Finally, the difficulties encountered by both sets of authors4 5 in obtaining information voluntarily from sponsors and trialists support the need for incentives to encourage adherence to policies that aim to increase transparency. Better adherence to trial registration and disclosure of summary results in recent years has been largely driven by policies enforced by journal editors and legislation.10 In contrast, public access to trial protocols and raw datasets is lagging behind.
If the ICMJE policy proposed by Jefferson and colleagues was broadened and extended, these challenges could be tackled. Journal editors and other stakeholders with the ability to enforce change—including funding agencies, research ethics committees, regulatory bodies, and legislators—could adopt and enforce a data sharing policy where, as a condition of approval, trialists commit to making both the protocol and anonymised raw dataset available to interested parties. Some journals have taken the first step of publishing protocols prospectively or requiring that they be submitted in confidence with manuscripts, but public access to protocols has yet to be made mandatory. Logical venues to house full protocols for public access would be the trial registries that record basic protocol information and post summary results, such as ClinicalTrials.gov.
Calls for submission and publication of raw data are not new,11 and guidelines for data sharing have been proposed.8 12 Some journals and funders have adopted data sharing requirements, but promoting adherence is a challenge once the study has been published or funded. One solution is for journals to require posting of the anonymised raw dataset on their website or other internet repository at the time of study publication. Other considerations for implementation of data sharing policies include ensuring patient privacy, obtaining patient consent or waivers, and providing for restricted data access in limited circumstances.8 12
Public access to the full trial protocol and raw dataset is essential for the value or harm of an intervention to be adequately appraised. Stakeholders have a responsibility to try to prevent the consequences of data suppression. Failure to act will perpetuate the status quo of partially informed decision making and will compromise patient care.
Cite this as: BMJ 2011;342:d80
Competing interests: All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review: Commissioned; not externally peer reviewed.