Data sharing in medical researchBMJ 2018; 360 doi: https://doi.org/10.1136/bmj.k510 (Published 14 February 2018) Cite this as: BMJ 2018;360:k510
All rapid responses
Response to the Rapid Response: Why Cochrane should prioritise sharing its data by Farhad Shokraneh, Clive Adams, Mike Clarke, Ben Goldacre and colleaguess
I’m grateful to the authors of the Rapid Response to Packer for raising the issue of Cochrane’s leadership role within the area of open data. As Shokraneh and colleagues report, Cochrane supports initiatives aimed at encouraging clinical trial transparency, the AllTrials initiative, and as a community is committed to moving towards the greatest possible transparency and openness. The debate over both open access and open data is current and important within Cochrane, and therefore this Rapid Response is a welcome addition to the conversation.
It is worth restating the progress Cochrane has already made in respect of open data and open access, which reflects the commitment of our community. It is also consistent with our mission to promote evidence-informed health decision-making by producing high-quality, relevant, accessible systematic reviews. Readers of Cochrane reviews in the Cochrane Library have, for many years, been able to download the data included within an individual review in a structured form. This facility will continue when our new platform is launched, and indeed it will be more visible to readers. In addition, and as implied within the response, Cochrane has a liberal policy towards permitting researchers to use the data it holds for methodological research. Many researchers have benefited from this and the publications that it has supported. Similarly, in the field of open access, which is arguably of greater importance to most readers, Cochrane has made significant strides in recent years with its green and gold open access schemes. Under the former, all new and updated reviews published since February 2013 are now freely available worldwide 12 months after publication. This currently applies to over 50% of the reviews in the Cochrane Database of Systematic Reviews, a percentage that will continue to increase. Reviews made open under the green OA scheme are also automatically deposited in PubMed Central once the author team has given its permission. All protocols are also made freely available worldwide on publication. Under the gold open access scheme author teams can choose to pay an article publication charge to have their reviews published under a creative commons licence. Waivers can be applied when the lead author is based in a low income country. Both the Cochrane Database of Systematic Reviews and the CENTRAL Register of Controlled Trials are fully searchable to all.
All of this represents progress, yet we recognise that it falls short of where we, along with the authors of the response, would like us to be. As the authors acknowledge, an important consideration is the risk that more liberal open access and open data pose to Cochrane’s financial sustainability. Cochrane has used its licensing income in different ways to support the production of its reviews. This includes substantial investments in information technology, in developing open collaborative platforms, in translation of Abstracts and Plain Language Summaries, in developing and publishing online interactive learning modules and support, and extensive editorial, quality assurance and knowledge translation activities. These activities are all funded through the licensing income, which represents a large percentage of the charity’s income. Despite our best efforts over several years, as yet we have not been able to identify sustainable alternative funding sources to replace this income. However, funding is not the only barrier. We are also investigating the legal and contractual issues that impact on making the data sets available, globally, without restriction.
In summary, my colleagues and I support the authors of the response in their call for Cochrane to provide leadership in respect of open science. We will continue to work to identify ways that will permit us to extend our openness, but hope that readers will recognise the significant steps we have made to date, which ensures that readers in 133 countries have free access to the full text of Cochrane Reviews without any need to log in, and also permits readers to access the structured data within individual reviews.
Editor in Chief, Cochrane Library
Competing interests: DT is a full time employee of Cochrane and leads the Editorial and Methods Department within the Central Executive Team. His salary and that of the members of his team is funded through licensing income. The views expressed in this article are his own and do not represent official Cochrane policy.
Why Cochrane should prioritise sharing data
Open sharing is vital for collaboration, innovation, and reproducibility: Cochrane could show leadership.
Packer1 discusses that the one who submits a research for public good should be ready to receive a request for data sharing for examination and re-analysis and tax payers assume that a national agency is checking such data and analysis. Here we discuss Cochrane’s practice on data sharing.
Open science, as endorsed by the G7, 2 includes sharing data, computer code and materials. It is essential for reproducibility, collaboration, and innovation. We support the work of Cochrane, but are concerned Cochrane is not sharing all its reviews’ data. These data should be fully accessible for re-use by third parties.
Cochrane, a non-profit private company3 and registered charity, produces and maintains systematic reviews in health and social care. Its work is undertaken by a global network of thousands of people,4 and its support largely comes from public funding.5 Most people producing Cochrane reviews are volunteers, not specifically funded for this work6 7 and Cochrane encourages ‘crowdsourcing’ of work.8-10
Cochrane Editorial bases help volunteers obtain study reports and manually extract the wealth of data needed to generate systematic reviews.11-13 Cochrane teams use RevMan software14 to produce files in standard format (XML), storing information on the studies, their methods and results for publication in the Cochrane Library.
Benefits of sharing extracted data from trials and systematic reviews are well known, as are the costs of not sharing.13 15-17 Sharing maximises transparency, reliability of data extraction, and syntheses. It improves access to data - saving time and money - and opens new avenues of inquiry.18 Sharing is associated with increased citations,19 more publications,20 and re-use for new purposes.16
Structured data from Cochrane should be fully accessible for download, re-use and review (Box 1). Currently, they are not. Although Cochrane supports transparency initiatives such as AllTrials,21 and is explicit about this within its policy,22 it has no similar clear principles on opening full access to the data within Cochrane reviews. Cochrane does provide access to results data from reviews but, crucially, these cannot be readily re-used; and the available information is an incomplete set of the data generating these reviews, comes in a technically problematic format and can only be viewed by those with access to the full content of the Cochrane Library.23-25
Box 1. Structured data and associated metadata
- All data from within Cochrane Central Register of Controlled Trials (CENTRAL) excluding copyrighted abstracts (so creating OPEN CENTRAL)
- All data from within Cochrane Register of Studies (CRS) excluding copyrighted abstracts (so creating OPEN CRS)
- Links to ‘parent’ study
- Links to ‘parent’ reviews
- Links to ‘child’ references
- Links to ‘parent’ reviews
Characteristics of studies:
- Methods, participants, interventions, outcomes
- Qualitative data on risk of bias
- Quantitative data on outcomes
- Qualitative and quantitative derived data
- Meta-analysis results, grading of quality of outcomes
Small amounts of Cochrane data have been released with bespoke arrangements for specific individuals. This sharing is welcome, but there is a lack of an organisational culture, policy, or process regarding data release; there is no appeals process. For example, OpenTrials aggregates all accessible documents on all trials in an open database and makes it free for public re-use.26 27 Thus far, OpenTrials have been unable to persuade Cochrane to share data for re-use. The Trip Database28 is a searchable library of evidence that asked to re-present structured data from Cochrane but also encountered barriers to access.29 Open sharing could foster collaborative ecosystems of digital innovation going beyond academic publications, with outputs which might include live, interactive presentations of summaries and results of trials produced by teams around the world, interactive decision support tools and many more.
Cochrane’s non-release of data is unlikely to reflect the preferences of funders, publishers, the thousands of Cochrane volunteers, participants in trials, or patients. For example, when asked, 83% of the members of the Cochrane Individual Participant Data (IPD) Meta-analysis Methods Group supported sharing systematic review data via a central repository (recognising that the IPD might require some form of moderated access).30 Many funders now require that data arising from their grants are shared.31-34 Cochrane volunteer authors give tacit consent for use of their work within reviews but may not be aware of the restrictions placed on access to the data they worked so hard to prepare.25 This is morally and ethically questionable, potentially eroding public trust.16 35
This issue of Open Science is now pressing, following recent moves by Cochrane to create more information and become a hub for systematic review data. This has potential to improve evidence and patient care, but while the Cochrane Linked Data Project aims to share re-usable data in some form,36 37 as yet, there is no information on how or when this will happen.38 39 Furthermore, Cochrane is making efforts towards ‘living’ systematic reviews, with updates from data in real-time.40 This is important work, but progress is slow. Opening up this work with shared data resources, and in collaboration with the open source software community - where all can contribute - would accelerate progress and best reflect the culture of collaboration in science.
Open data offers a transformative, collaborative future for the systematic review community. Cochrane has enabled a vast workforce to painstakingly extract information for great benefit. Cochrane could act as a hub, harmonising data collected across groups and sharing these widely, reflecting the collective funding and volunteer workforce that produces them. This could involve the conversion of the morass of free text trial reports into machine-readable curated data, in archived, citable, accessible, inter-operable and re-usable formats, as set out in the FAIR Principles.41 42 Cochrane could show leadership in supporting innovation and open science for clinical trials with full credit to all data extractors before43 and after review publication44 and, in this way harness greatest broadest impact. This reflects the exciting current move towards better use of data to produce digital tools of direct value to clinicians, rather than academic publications alone.
Open data is a route to success and impact in the 21st century. We have raised these issues with Cochrane,45 and understand that the organisation is considering whether to commence a process of reviewing its approach to sharing data.46 We hope that our setting out the benefits of open data is a helpful contribution to open that discussion.
We appreciate Cochrane must focus on making itself sustainable and that open data sharing may be commercially sensitive.47 However, making Cochrane a champion for openness, transparency and sharing can only be beneficial for the organisation’s reputation - and finances. We encourage Cochrane leadership to create a policy that allows open data sharing and to make explicit any concerns they have on open data sharing so that these can be resolved.
Box 2. Key Messages
● Cochrane could lead and set standards for open data sharing from systematic reviews.
● Availability of data from Cochrane reviews would:
○ give opportunities for collaboration, innovation, scientific replication, novel research and clinical decision making.
○ reduce the considerable waste of the current duplication of effort in systematic reviewing.
We thank the Co-ordinating Editors at Cochrane groups who supported internal discussion within Cochrane on sharing data. We express our gratitude for the time they spent studying and commenting on earlier versions of this manuscript and replying to our communications.
We thank David Tovey, Editor in Chief of the Cochrane Library, for his thoughtful and helpful response to our written communication.
We are grateful to the Cochrane co-ordinating editors: Gianni Virgili, Carlos Grillo Ardila, Juan-Pablo Casas, Jos Verbeek, Richard Wormald, and editor of Cochrane Methodology Review Group, Karen Robinson, for supporting the idea of Cochrane sharing the data.
1. Packer M. Data sharing in medical research. BMJ 2018;360:k510.
2. Science GEGoO. Annex 4: G7 Expert Group on Open Science: Executive Summary. Italy: G7, 2017.
3. Companies House. The Cochrane Collaboration Companies House2018 [Available from: http://bit.ly/2GmVIxo.
4. Cochrane Collaborarion. About us: Cochrane Collaboration; 2018 [Available from: http://bit.ly/2sFZRZn.
5. Cochrane Collaborarion. Our funders: Cochrane Collaborarion; 2018 [Available from: http://bit.ly/2Inh6TL.
6. Tharyan P. Evidence of the people, by the people, and for the people. Cochrane Database Syst Rev 2010;2011:ED000013.
7. Wilson M. Help us improve the health of people everywhere: Cochrane Collaboration,; 2018 [Available from: http://join.cochrane.org/.
8. Cochrane Collaborarion. Cochrane Crowd: Cochrane Collaboration,; 2018 [Available from: http://crowd.cochrane.org/index.html.
9. Cochrane Collaborarion. Cochrane TaskExchange: Cochrane Collaborarion; 2018 [Available from: http://taskexchange.cochrane.org/.
10. Wallace BC, Noel-Storr A, Marshall IJ, et al. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc 2017;24(6):1165-68.
11. Shokraneh F, Adams CE. Study-based registers of randomized controlled trials: Starting a systematic review with data extraction or meta-analysis. Bioimpacts 2017;7(4):209-17.
12. Shokraneh F, Adams CE. Increasing value and reducing waste in data extraction for systematic reviews: tracking data in data extraction forms. Syst Rev 2017;6(1):153.
13. Wolfenden L, Grimshaw J, Williams CM, et al. Time to consider sharing data extracted from trials included in systematic reviews. Syst Rev 2016;5(1):185.
14. Review Manager (RevMan) [program]. Copenhagen: Nordic Cochrane Centre, Cochrane Collaboration, 2014.
15. Mayo-Wilson E, Li T, Fusco N, et al. Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study). Res Synth Methods 2017.
16. Nordic Trial Alliance Working Group on Transparency and Registration. Transparency and registration in clinical research in the Nordic countries. Oslo, Norway: Nordic Trial Alliance, 2015.
17. Uhlir PF, Schröder P. Open Data for Global Science. Data Science Journal 2007;6:OD36-OD53.
18. Agency for Healthcare Research and Quality. About the Systematic Review Data Repository: Agency for Healthcare Research and Quality, U.S. Department of Health & Human Services; 2015 [Available from: https://srdr.ahrq.gov/about.
19. Angraal S, Ross JS, Dhruva SS, et al. Merits of Data Sharing: The Digitalis Investigation Group Trial. J Am Coll Cardiol 2017;70(14):1825-27.
20. Piwowar HA, Day RS, Fridsma DB. Sharing detailed research data is associated with increased citation rate. PLoS One 2007;2(3):e308.
21. Brown T. It's time for alltrials registered and reported. Cochrane Database Syst Rev 2013(4):ED000057.
22. Cochrane Collaborarion. Access to data from AllTrials. An official Cochrane policy: Cochrane Supports Access to Data from All Trials: Cochrane Collaboration; 2018 [Available from: http://bit.ly/2FzS9qN.
23. Cochrane Collaborarion. Open Access: Cochrane Collaboration,; 2018 [Available from: http://bit.ly/2FBjUPC.
24. Cochrane Library. Access Options for Cochrane Library: John Wiley & Sons, Inc.; 2018 [Available from: http://bit.ly/1WskJvg.
25. Soares-Weiser K. Cochrane and conflict of interest: Addendum - 28 June 2017: Cochrane Collaboration; 2017 [updated 28 June 2017. Available from: http://bit.ly/2pbNC2J.
26. Goldacre B, Gray J. OpenTrials: towards a collaborative open database of all available information on all clinical trials. Trials 2016;17:164.
27. Goldacre B, Turner E, OpenTrials t. You can now search FDA approval documents easily at fda.opentrials.net. BMJ 2017;356:j677.
28. Trip Database. Trip Medical Database 2018 [Available from: https://www.tripdatabase.com/.
29. Brassey J. Cochrane’s EMBASE screening project - did you participate? In: firstname.lastname@example.org, ed., 2016.
30. Tudur Smith C, Dwan K, Altman DG, et al. Sharing individual participant data from clinical trials: an opinion survey regarding the establishment of a central repository. PLoS One 2014;9(5):e97886.
31. Bill & Melinda Gates Foundation. Open Access Policy: Bill & Melinda Gates Foundation; 2018 [Available from: http://gates.ly/2Ab4hHw.
32. Medical Research Council. Data sharing: Medical Research Council,; 2018 [Available from: http://bit.ly/2x1fSeq.
33. National Health and Medical Research Council. NHMRC Statement on Data Sharing: National Health and Medical Research Council; 2017 [updated 17 April 2015. Available from: http://bit.ly/2FLjMfL.
34. National Institute for Health Research. Data sharing: National Institute for Health Research,; 2018 [Available from: http://bit.ly/2FP973G.
35. Coyne JC. Why I am formally requesting the data set from a Cochrane review 2017 [updated 13 April 2017. Available from: http://bit.ly/2FD5v1a.
36. Li T, Vedula SS, Hadar N, et al. Innovations in data collection, management, and archiving for systematic reviews. Ann Intern Med 2015;162(4):287-94.
37. Slaughter L, Berntsen CF, Brandt L, et al. Enabling Living Systematic Reviews and Clinical Guidelines through Semantic Technologies. D-Lib Magazine 2015;21(1/2).
38. Cochrane Collaborarion. Cochrane Strategy to 2020, 2017.
39. Cochrane Collaborarion. Cochrane | Trusted evidence. Informed decisions. Better health 2018 [Available from: http://www.cochrane.org/.
40. Elliott JH, Turner T, Clavisi O, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med 2014;11(2):e1001603.
41. Data Citation Synthesis Group. Joint Declaration of Data Citation Principles. San Diego, CA: FORCE11, 2014.
42. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 2016;3:160018.
43. Chalmers I, Glasziou P. Should there be greater use of preprint servers for publishing reports of biomedical science? F1000Res 2016;5:272.
44. Bierer BE, Crosas M, Pierce HH. Data Authorship as an Incentive to Data Sharing. N Engl J Med 2017;376(17):1684-87.
45. Adams CE, Goldacre B, Clarke M, et al. RE: Sharing data. In: Tovey D, ed., 2017.
46. Tovey D. Re: Sharing data. In: Adams CE, Goldacre B, Clarke M, et al., eds., 2017.
47. Senior Management Team. Cochrane Strategy to 2020 - in 2017: Definition of success by 2020, an assessment of progress, and a framework for work remaining: Cochrane Collaboration, 2017.
Competing interests: FS is the Information Specialist of Cochrane Schizophrenia and has voluntarily extracted data for 14 Cochrane groups. CEA promotes Cochrane extensively to the public and policy makers; trains hundreds of reviewers per year, is Co-ordinating Editor of Cochrane Schizophrenia, is PI on randomised trials testing the effects of disseminating Cochrane reviews in different forms, is PI on NIHR infrastructure grant for Cochrane Schizophrenia. MC promotes Cochrane to the public, practitioners and policy makers; provides training in the conduct of randomised trials and systematic reviews, is Co-ordinating Editor of the Cochrane Methodology Review Group, and seeks funding and conducts research into the methods using in systematic reviews and other evaluations of health and social care. BG has promoted Cochrane extensively to the public and policy makers; is PI on OpenTrials.net, who have had a data sharing request rejected by Cochrane; has received research funding from the Laura and John Arnold Foundation, the Wellcome Trust, the NHS NIHR School of Primary Care, the Health Foundation, NHS England, the NIHR Oxford Biomedical Research Centre, and the WHO; receives personal income from speaking and writing for lay audiences on the misuse of science; and has a longstanding commitment to open science. LA promotes Cochrane to the public and policy makers; is Co-ordinating Editor of Cochrane Drugs and Alcohol Group; has received grant funding from the WHO, EMCDDA, the Italian National Institute of Health and AIFA (Italian Medicines Agency). HB has received access to Cochrane data for projects and services. JB is director and shareholder in the Trip Database, a limited company, and actively involved in evidence synthesis and there is the potential for Trip to benefit from better access to the data Cochrane currently restricts. RB promotes Cochrane extensively to the public, clinicians and policy makers; trains several reviewers per year, is Joint Co-ordinating Editor of Cochrane Musculoskeletal, is PI on grants developing two living Cochrane reviews, is PI on NHMRC Editorial base funding for Cochrane Musculoskeletal, and has received research funding from NHMRC, Cabrini Foundation, MRC and PCORI. She is funded by an NHMRC Senior Principal Research Fellowship. CDM has received 1) consultancy fees/honoraria from National Prescribing Service MedicineWise (NPSMedicineWise), the RACGP’s ‘Red Book’ preventive guidelines committee; Therapeutic Guidelines (eTG); Remote Primary Health Care Manuals Editorial Committee for expert advice; Editorial work (MJA Deputy Editor; ACP Journal Club; BMJ); Consultation work for BUPA (UK) on shared decision making: Australian Medicine Handbook; 2) Royalties for 3 books (Wileys and BMJ Books) on EBM, and clinical thinking; 3) Grants from NHMRC (Australia) two Centres for Research Excellence; NIHR (UK); HTA (UK); from a private donor (for the Cochrane Collaboration ARI Group); Australian Commission on Safety and Quality in Health Care. MD is Co-ordinating Editor of Cochrane Drugs and Alcohol Group; has received grant funding from the WHO, EMCDDA, the Italian National Institute of Health and AIFA (Italian Medicines Agency), and disseminates Cochrane review results to the public and policy makers. PG is a member of editorial group of the Cochrane Acute Respiratory Infections group. CH has received grant funding from the WHO, the NIHR and the NIHR School of Primary Care. MJ is an editor at Cochrane Schizophrenia Group. DM is on Cochrane Oversight Committee. RSS is Joint Co-ordinating Editor of the Cochrane Schizophrenia group. LV holds an NIHR Systematic Reviews Grant for the Cochrane Incontinence. He holds grants from: EU2020, Wellcome, ESRC, MRC, Health Foundation, NIHR for research using systematic review methods. EB, CG, TH, JPAI, JK, and EO have declared no conflict of interests.
Research is the backbone of scientific development. Medical research is essential to evolve new diagnostic methods, better understanding of the disease process and advanced therapeutics. It is the responsibility of every clinician to contribute their observations. This will finally help society.
The statistical principle is that the study which has a large sample size and is high powered has more impact. Conclusions become more specific, validated and reliable in the presence of alarge sample size.
The database is not the property of researchers, but the public at large should have authority to access data. Research is not meant for researchers but for the welfare of humankind.
So, it is prudent to share data, but it can only be executed if peer groups have better understanding of each other.
Competing interests: No competing interests
Yes, there is an unwritten "social contract" when it comes to clinical (human subject) research data.
Most patients will be appalled to learn that they are being treated on the basis of unverified (and probably fake) data.
Would doctors be happy to fly on a plane where a lot of the pre-flight checks have been taken on trust with no verification?
Currently, data-sharing is not enforcible because we don't have a system of National Clinical Research Councils with teeth to register and regulate trained clinical researchers like we do trained clinical practitioners.
Anonymised and audited raw data should lie in such Research Council repositories and not with individual researchers.
I have always wondered about the wisdom of using published research without access to the underlying data when writing Clinical Practice Guidelines (CPG). CPGs are quasi-legal documents in the sense that they are written by national professional bodies. These bodies usually have some kind of government blessings or recognition. Doctors feel pressured to adhere to them as far as possible. So, they should be based (as far as possible) on actual audited data and not on published statistics.
On the other hand, systematic reviews and meta-analysis are more in the nature of "surveying" the field. No one is obliged to adhere to any particular systematic review.
Trained clinical researchers will not have any problems with data sharing because they will know how to collect, audit and store data - the field is known as Clinical Data Managment (CDM). So, the question of a "witch-hunt" does not arise.
Competing interests: No competing interests