Why Cochrane should prioritise sharing data
Why Cochrane should prioritise sharing data
Open sharing is vital for collaboration, innovation, and reproducibility: Cochrane could show leadership.
Packer1 discusses that the one who submits a research for public good should be ready to receive a request for data sharing for examination and re-analysis and tax payers assume that a national agency is checking such data and analysis. Here we discuss Cochrane’s practice on data sharing.
Open science, as endorsed by the G7, 2 includes sharing data, computer code and materials. It is essential for reproducibility, collaboration, and innovation. We support the work of Cochrane, but are concerned Cochrane is not sharing all its reviews’ data. These data should be fully accessible for re-use by third parties.
Cochrane, a non-profit private company3 and registered charity, produces and maintains systematic reviews in health and social care. Its work is undertaken by a global network of thousands of people,4 and its support largely comes from public funding.5 Most people producing Cochrane reviews are volunteers, not specifically funded for this work6 7 and Cochrane encourages ‘crowdsourcing’ of work.8-10
Cochrane Editorial bases help volunteers obtain study reports and manually extract the wealth of data needed to generate systematic reviews.11-13 Cochrane teams use RevMan software14 to produce files in standard format (XML), storing information on the studies, their methods and results for publication in the Cochrane Library.
Benefits of sharing extracted data from trials and systematic reviews are well known, as are the costs of not sharing.13 15-17 Sharing maximises transparency, reliability of data extraction, and syntheses. It improves access to data - saving time and money - and opens new avenues of inquiry.18 Sharing is associated with increased citations,19 more publications,20 and re-use for new purposes.16
Structured data from Cochrane should be fully accessible for download, re-use and review (Box 1). Currently, they are not. Although Cochrane supports transparency initiatives such as AllTrials,21 and is explicit about this within its policy,22 it has no similar clear principles on opening full access to the data within Cochrane reviews. Cochrane does provide access to results data from reviews but, crucially, these cannot be readily re-used; and the available information is an incomplete set of the data generating these reviews, comes in a technically problematic format and can only be viewed by those with access to the full content of the Cochrane Library.23-25
Box 1. Structured data and associated metadata
- All data from within Cochrane Central Register of Controlled Trials (CENTRAL) excluding copyrighted abstracts (so creating OPEN CENTRAL)
- All data from within Cochrane Register of Studies (CRS) excluding copyrighted abstracts (so creating OPEN CRS)
- Links to ‘parent’ study
- Links to ‘parent’ reviews
- Links to ‘child’ references
- Links to ‘parent’ reviews
Characteristics of studies:
- Methods, participants, interventions, outcomes
- Qualitative data on risk of bias
- Quantitative data on outcomes
- Qualitative and quantitative derived data
- Meta-analysis results, grading of quality of outcomes
Small amounts of Cochrane data have been released with bespoke arrangements for specific individuals. This sharing is welcome, but there is a lack of an organisational culture, policy, or process regarding data release; there is no appeals process. For example, OpenTrials aggregates all accessible documents on all trials in an open database and makes it free for public re-use.26 27 Thus far, OpenTrials have been unable to persuade Cochrane to share data for re-use. The Trip Database28 is a searchable library of evidence that asked to re-present structured data from Cochrane but also encountered barriers to access.29 Open sharing could foster collaborative ecosystems of digital innovation going beyond academic publications, with outputs which might include live, interactive presentations of summaries and results of trials produced by teams around the world, interactive decision support tools and many more.
Cochrane’s non-release of data is unlikely to reflect the preferences of funders, publishers, the thousands of Cochrane volunteers, participants in trials, or patients. For example, when asked, 83% of the members of the Cochrane Individual Participant Data (IPD) Meta-analysis Methods Group supported sharing systematic review data via a central repository (recognising that the IPD might require some form of moderated access).30 Many funders now require that data arising from their grants are shared.31-34 Cochrane volunteer authors give tacit consent for use of their work within reviews but may not be aware of the restrictions placed on access to the data they worked so hard to prepare.25 This is morally and ethically questionable, potentially eroding public trust.16 35
This issue of Open Science is now pressing, following recent moves by Cochrane to create more information and become a hub for systematic review data. This has potential to improve evidence and patient care, but while the Cochrane Linked Data Project aims to share re-usable data in some form,36 37 as yet, there is no information on how or when this will happen.38 39 Furthermore, Cochrane is making efforts towards ‘living’ systematic reviews, with updates from data in real-time.40 This is important work, but progress is slow. Opening up this work with shared data resources, and in collaboration with the open source software community - where all can contribute - would accelerate progress and best reflect the culture of collaboration in science.
Open data offers a transformative, collaborative future for the systematic review community. Cochrane has enabled a vast workforce to painstakingly extract information for great benefit. Cochrane could act as a hub, harmonising data collected across groups and sharing these widely, reflecting the collective funding and volunteer workforce that produces them. This could involve the conversion of the morass of free text trial reports into machine-readable curated data, in archived, citable, accessible, inter-operable and re-usable formats, as set out in the FAIR Principles.41 42 Cochrane could show leadership in supporting innovation and open science for clinical trials with full credit to all data extractors before43 and after review publication44 and, in this way harness greatest broadest impact. This reflects the exciting current move towards better use of data to produce digital tools of direct value to clinicians, rather than academic publications alone.
Open data is a route to success and impact in the 21st century. We have raised these issues with Cochrane,45 and understand that the organisation is considering whether to commence a process of reviewing its approach to sharing data.46 We hope that our setting out the benefits of open data is a helpful contribution to open that discussion.
We appreciate Cochrane must focus on making itself sustainable and that open data sharing may be commercially sensitive.47 However, making Cochrane a champion for openness, transparency and sharing can only be beneficial for the organisation’s reputation - and finances. We encourage Cochrane leadership to create a policy that allows open data sharing and to make explicit any concerns they have on open data sharing so that these can be resolved.
Box 2. Key Messages
● Cochrane could lead and set standards for open data sharing from systematic reviews.
● Availability of data from Cochrane reviews would:
○ give opportunities for collaboration, innovation, scientific replication, novel research and clinical decision making.
○ reduce the considerable waste of the current duplication of effort in systematic reviewing.
We thank the Co-ordinating Editors at Cochrane groups who supported internal discussion within Cochrane on sharing data. We express our gratitude for the time they spent studying and commenting on earlier versions of this manuscript and replying to our communications.
We thank David Tovey, Editor in Chief of the Cochrane Library, for his thoughtful and helpful response to our written communication.
We are grateful to the Cochrane co-ordinating editors: Gianni Virgili, Carlos Grillo Ardila, Juan-Pablo Casas, Jos Verbeek, Richard Wormald, and editor of Cochrane Methodology Review Group, Karen Robinson, for supporting the idea of Cochrane sharing the data.
1. Packer M. Data sharing in medical research. BMJ 2018;360:k510.
2. Science GEGoO. Annex 4: G7 Expert Group on Open Science: Executive Summary. Italy: G7, 2017.
3. Companies House. The Cochrane Collaboration Companies House2018 [Available from: http://bit.ly/2GmVIxo.
4. Cochrane Collaborarion. About us: Cochrane Collaboration; 2018 [Available from: http://bit.ly/2sFZRZn.
5. Cochrane Collaborarion. Our funders: Cochrane Collaborarion; 2018 [Available from: http://bit.ly/2Inh6TL.
6. Tharyan P. Evidence of the people, by the people, and for the people. Cochrane Database Syst Rev 2010;2011:ED000013.
7. Wilson M. Help us improve the health of people everywhere: Cochrane Collaboration,; 2018 [Available from: http://join.cochrane.org/.
8. Cochrane Collaborarion. Cochrane Crowd: Cochrane Collaboration,; 2018 [Available from: http://crowd.cochrane.org/index.html.
9. Cochrane Collaborarion. Cochrane TaskExchange: Cochrane Collaborarion; 2018 [Available from: http://taskexchange.cochrane.org/.
10. Wallace BC, Noel-Storr A, Marshall IJ, et al. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc 2017;24(6):1165-68.
11. Shokraneh F, Adams CE. Study-based registers of randomized controlled trials: Starting a systematic review with data extraction or meta-analysis. Bioimpacts 2017;7(4):209-17.
12. Shokraneh F, Adams CE. Increasing value and reducing waste in data extraction for systematic reviews: tracking data in data extraction forms. Syst Rev 2017;6(1):153.
13. Wolfenden L, Grimshaw J, Williams CM, et al. Time to consider sharing data extracted from trials included in systematic reviews. Syst Rev 2016;5(1):185.
14. Review Manager (RevMan) [program]. Copenhagen: Nordic Cochrane Centre, Cochrane Collaboration, 2014.
15. Mayo-Wilson E, Li T, Fusco N, et al. Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study). Res Synth Methods 2017.
16. Nordic Trial Alliance Working Group on Transparency and Registration. Transparency and registration in clinical research in the Nordic countries. Oslo, Norway: Nordic Trial Alliance, 2015.
17. Uhlir PF, Schröder P. Open Data for Global Science. Data Science Journal 2007;6:OD36-OD53.
18. Agency for Healthcare Research and Quality. About the Systematic Review Data Repository: Agency for Healthcare Research and Quality, U.S. Department of Health & Human Services; 2015 [Available from: https://srdr.ahrq.gov/about.
19. Angraal S, Ross JS, Dhruva SS, et al. Merits of Data Sharing: The Digitalis Investigation Group Trial. J Am Coll Cardiol 2017;70(14):1825-27.
20. Piwowar HA, Day RS, Fridsma DB. Sharing detailed research data is associated with increased citation rate. PLoS One 2007;2(3):e308.
21. Brown T. It's time for alltrials registered and reported. Cochrane Database Syst Rev 2013(4):ED000057.
22. Cochrane Collaborarion. Access to data from AllTrials. An official Cochrane policy: Cochrane Supports Access to Data from All Trials: Cochrane Collaboration; 2018 [Available from: http://bit.ly/2FzS9qN.
23. Cochrane Collaborarion. Open Access: Cochrane Collaboration,; 2018 [Available from: http://bit.ly/2FBjUPC.
24. Cochrane Library. Access Options for Cochrane Library: John Wiley & Sons, Inc.; 2018 [Available from: http://bit.ly/1WskJvg.
25. Soares-Weiser K. Cochrane and conflict of interest: Addendum - 28 June 2017: Cochrane Collaboration; 2017 [updated 28 June 2017. Available from: http://bit.ly/2pbNC2J.
26. Goldacre B, Gray J. OpenTrials: towards a collaborative open database of all available information on all clinical trials. Trials 2016;17:164.
27. Goldacre B, Turner E, OpenTrials t. You can now search FDA approval documents easily at fda.opentrials.net. BMJ 2017;356:j677.
28. Trip Database. Trip Medical Database 2018 [Available from: https://www.tripdatabase.com/.
29. Brassey J. Cochrane’s EMBASE screening project - did you participate? In: email@example.com, ed., 2016.
30. Tudur Smith C, Dwan K, Altman DG, et al. Sharing individual participant data from clinical trials: an opinion survey regarding the establishment of a central repository. PLoS One 2014;9(5):e97886.
31. Bill & Melinda Gates Foundation. Open Access Policy: Bill & Melinda Gates Foundation; 2018 [Available from: http://gates.ly/2Ab4hHw.
32. Medical Research Council. Data sharing: Medical Research Council,; 2018 [Available from: http://bit.ly/2x1fSeq.
33. National Health and Medical Research Council. NHMRC Statement on Data Sharing: National Health and Medical Research Council; 2017 [updated 17 April 2015. Available from: http://bit.ly/2FLjMfL.
34. National Institute for Health Research. Data sharing: National Institute for Health Research,; 2018 [Available from: http://bit.ly/2FP973G.
35. Coyne JC. Why I am formally requesting the data set from a Cochrane review 2017 [updated 13 April 2017. Available from: http://bit.ly/2FD5v1a.
36. Li T, Vedula SS, Hadar N, et al. Innovations in data collection, management, and archiving for systematic reviews. Ann Intern Med 2015;162(4):287-94.
37. Slaughter L, Berntsen CF, Brandt L, et al. Enabling Living Systematic Reviews and Clinical Guidelines through Semantic Technologies. D-Lib Magazine 2015;21(1/2).
38. Cochrane Collaborarion. Cochrane Strategy to 2020, 2017.
39. Cochrane Collaborarion. Cochrane | Trusted evidence. Informed decisions. Better health 2018 [Available from: http://www.cochrane.org/.
40. Elliott JH, Turner T, Clavisi O, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med 2014;11(2):e1001603.
41. Data Citation Synthesis Group. Joint Declaration of Data Citation Principles. San Diego, CA: FORCE11, 2014.
42. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 2016;3:160018.
43. Chalmers I, Glasziou P. Should there be greater use of preprint servers for publishing reports of biomedical science? F1000Res 2016;5:272.
44. Bierer BE, Crosas M, Pierce HH. Data Authorship as an Incentive to Data Sharing. N Engl J Med 2017;376(17):1684-87.
45. Adams CE, Goldacre B, Clarke M, et al. RE: Sharing data. In: Tovey D, ed., 2017.
46. Tovey D. Re: Sharing data. In: Adams CE, Goldacre B, Clarke M, et al., eds., 2017.
47. Senior Management Team. Cochrane Strategy to 2020 - in 2017: Definition of success by 2020, an assessment of progress, and a framework for work remaining: Cochrane Collaboration, 2017.
Competing interests: FS is the Information Specialist of Cochrane Schizophrenia and has voluntarily extracted data for 14 Cochrane groups. CEA promotes Cochrane extensively to the public and policy makers; trains hundreds of reviewers per year, is Co-ordinating Editor of Cochrane Schizophrenia, is PI on randomised trials testing the effects of disseminating Cochrane reviews in different forms, is PI on NIHR infrastructure grant for Cochrane Schizophrenia. MC promotes Cochrane to the public, practitioners and policy makers; provides training in the conduct of randomised trials and systematic reviews, is Co-ordinating Editor of the Cochrane Methodology Review Group, and seeks funding and conducts research into the methods using in systematic reviews and other evaluations of health and social care. BG has promoted Cochrane extensively to the public and policy makers; is PI on OpenTrials.net, who have had a data sharing request rejected by Cochrane; has received research funding from the Laura and John Arnold Foundation, the Wellcome Trust, the NHS NIHR School of Primary Care, the Health Foundation, NHS England, the NIHR Oxford Biomedical Research Centre, and the WHO; receives personal income from speaking and writing for lay audiences on the misuse of science; and has a longstanding commitment to open science. LA promotes Cochrane to the public and policy makers; is Co-ordinating Editor of Cochrane Drugs and Alcohol Group; has received grant funding from the WHO, EMCDDA, the Italian National Institute of Health and AIFA (Italian Medicines Agency). HB has received access to Cochrane data for projects and services. JB is director and shareholder in the Trip Database, a limited company, and actively involved in evidence synthesis and there is the potential for Trip to benefit from better access to the data Cochrane currently restricts. RB promotes Cochrane extensively to the public, clinicians and policy makers; trains several reviewers per year, is Joint Co-ordinating Editor of Cochrane Musculoskeletal, is PI on grants developing two living Cochrane reviews, is PI on NHMRC Editorial base funding for Cochrane Musculoskeletal, and has received research funding from NHMRC, Cabrini Foundation, MRC and PCORI. She is funded by an NHMRC Senior Principal Research Fellowship. CDM has received 1) consultancy fees/honoraria from National Prescribing Service MedicineWise (NPSMedicineWise), the RACGP’s ‘Red Book’ preventive guidelines committee; Therapeutic Guidelines (eTG); Remote Primary Health Care Manuals Editorial Committee for expert advice; Editorial work (MJA Deputy Editor; ACP Journal Club; BMJ); Consultation work for BUPA (UK) on shared decision making: Australian Medicine Handbook; 2) Royalties for 3 books (Wileys and BMJ Books) on EBM, and clinical thinking; 3) Grants from NHMRC (Australia) two Centres for Research Excellence; NIHR (UK); HTA (UK); from a private donor (for the Cochrane Collaboration ARI Group); Australian Commission on Safety and Quality in Health Care. MD is Co-ordinating Editor of Cochrane Drugs and Alcohol Group; has received grant funding from the WHO, EMCDDA, the Italian National Institute of Health and AIFA (Italian Medicines Agency), and disseminates Cochrane review results to the public and policy makers. PG is a member of editorial group of the Cochrane Acute Respiratory Infections group. CH has received grant funding from the WHO, the NIHR and the NIHR School of Primary Care. MJ is an editor at Cochrane Schizophrenia Group. DM is on Cochrane Oversight Committee. RSS is Joint Co-ordinating Editor of the Cochrane Schizophrenia group. LV holds an NIHR Systematic Reviews Grant for the Cochrane Incontinence. He holds grants from: EU2020, Wellcome, ESRC, MRC, Health Foundation, NIHR for research using systematic review methods. EB, CG, TH, JPAI, JK, and EO have declared no conflict of interests.