Why Cochrane should prioritise sharing dataBMJ 2018; 362 doi: https://doi.org/10.1136/bmj.k3229 (Published 30 July 2018) Cite this as: BMJ 2018;362:k3229
- Farhad Shokraneh, information specialist1,
- Clive E Adams, professor, chair of mental health services research2,
- Mike Clarke, professor2,
- Laura Amato, psychiatrist3,
- Hilda Bastian, former coordinator4,
- Elaine Beller, associate professor5,
- Jon Brassey, director6,
- Rachelle Buchbinder, professor7,
- Marina Davoli, director3,
- Chris Del Mar, professor5,
- Paul Glasziou, professor and director5,
- Christian Gluud, head of department8,
- Carl Heneghan, professor of evidence-based medicine and director9,
- Tammy Hoffmann, professor of clinical epidemiology5,
- John PA Ioannidis, professor of medicine and metrics co-director10,
- Mahesh Jayaram, senior lecturer11,
- Joey Kwong, research assistant professor12,
- David Moher, director13,
- Erika Ota, professor14,
- Rebecca Syed Sheriff, honorary lecturer15,
- Luke Vale, professor16,
- Ben Goldacre, senior clinical research fellow9
- 1Institute of Mental Health, University of Nottingham, Nottingham, UK
- 2Centre for Public Health, Queen's University Belfast, Northern Ireland
- 3Department of Epidemiology, Lazio Region-ASL Rome, Italy
- 4Cochrane Consumer Network
- 5Centre for Evidence Based Practice, Bond University, Gold Coast, Australia
- 6Trip Database
- 7Monash University and Cabrini Institute, Australia
- 8The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Rigshospitalet; Copenhagen University Hospital, Copenhagen, Denmark
- 9Centre for Evidence Based Medicine, University of Oxford, UK
- 10Stanford University, USA
- 11Department of Psychiatry, University of Melbourne, Australia
- 12JC School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong
- 13Centre for Journalology, Ottawa Hospital Research Institute, Canada
- 14Global Health Nursing, Graduate School of Nursing Science, St. Luke’s International University, Tokyo, Japan
- 15Child and Adolescent Psychiatric Unit, Department of Psychiatry, Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand
- 16Cochrane Incontinence, Newcastle University, UK
Packer1 says that the one who submits a research for public good should be ready to receive a request for data sharing for examination and re-analysis and that tax payers assume that a national agency is checking such data and analysis. Here we discuss Cochrane’s practice on data sharing.
Open science, as endorsed by the G7,2 includes sharing data, computer code, and materials. It is essential for reproducibility, collaboration, and innovation. We support the work of Cochrane, but are concerned that Cochrane is not sharing all its reviews’ data. These data should be fully accessible for reuse by third parties.
Cochrane, a non-profit private company3 and registered charity, produces and maintains systematic reviews in health and social care. Its work is undertaken by a global network of thousands of people,4 and its support largely comes from public funding.5 Most people producing Cochrane reviews are volunteers not specifically funded for this work,67 and Cochrane encourages “crowdsourcing” of work.8910
Cochrane editorial bases help volunteers obtain study reports and manually extract the wealth of data needed to generate systematic reviews.111213 Cochrane teams use RevMan software14 to produce files in standard format (XML), storing information on the studies, their methods, and results for publication in the Cochrane Library.
Benefits of sharing extracted data from trials and systematic reviews are well known, as are the costs of not sharing.13151617 Sharing maximises transparency, reliability of data extraction, and syntheses. It improves access to data—saving time and money—and opens new avenues of inquiry.18 Sharing is associated with increased citations,19 more publications,20 and reuse for new purposes.16
Structured data from Cochrane should be fully accessible for download, reuse, and review (box 1). Currently, they are not. Although Cochrane supports transparency initiatives such as AllTrials21 and is explicit about this in its policy,22 it has no similar clear principles on opening full access to the data in Cochrane reviews. Cochrane does provide access to results data from reviews but, crucially, these cannot be readily reused, and the available information is an incomplete set of the data generating these reviews, comes in a technically problematic format, and can only be viewed by those with access to the full content of the Cochrane Library.232425
Structured data and associated metadata
All data in the Cochrane Central Register of Controlled Trials (CENTRAL) excluding copyrighted abstracts (so creating OPEN CENTRAL)
All data in the Cochrane Register of Studies (CRS) excluding copyrighted abstracts (so creating OPEN CRS)
Links to “parent” study
Links to “parent” reviews
Links to “child” references
Links to “parent” reviews
Characteristics of studies
Methods, participants, interventions, outcomes
Qualitative data on risk of bias
Quantitative data on outcomes
Qualitative and quantitative derived data
Meta-analysis results, grading of quality of outcomes
Small amounts of Cochrane data have been released with bespoke arrangements for specific individuals. This sharing is welcome, but organisational culture, policy, and process regarding data release are lacking; there is no appeals process. For example, OpenTrials aggregates all accessible documents on all trials in an open database and makes it free for public reuse.2627 Thus far, OpenTrials has been unable to persuade Cochrane to share data for reuse. The Trip Database28 is a searchable library of evidence that asked if it could re-present structured data from Cochrane and also encountered barriers to access.29 Open sharing could foster collaborative ecosystems of digital innovation going beyond academic publications, with outputs that might include live, interactive presentations of summaries and results of trials produced by teams around the world, interactive decision support tools, and many more.
Cochrane’s non-release of data is unlikely to reflect the preferences of funders, publishers, the thousands of Cochrane volunteers, participants in trials, or patients. When asked, 83% of the members of the Cochrane Individual Participant Data Meta-analysis Methods Group supported sharing systematic review data through a central repository (recognising that these data might require some form of moderated access).30 Many funders now require that data arising from their grants are shared.31323334 Cochrane volunteer authors give tacit consent for use of their work in reviews but may not be aware of the restrictions placed on access to the data they worked so hard to prepare.25 This is morally and ethically questionable, potentially eroding public trust.1635
This issue of open science is now pressing, after recent moves by Cochrane to create more information and become a hub for systematic review data. This has the potential to improve evidence and patient care, but although the Cochrane Linked Data Project aims to share reusable data in some form,3637 there is not yet any information on how or when this will happen.3839 Furthermore, Cochrane is working towards “living” systematic reviews, with updates from data in real time.40 This is important work, but progress is slow. Opening up this work with shared data resources and in collaboration with the open source software community—where all can contribute—would accelerate progress and best reflect the culture of collaboration in science.
Open data offers a transformative, collaborative future for the systematic review community. Cochrane has enabled a vast workforce to painstakingly extract information for great benefit. It could act as a hub, harmonising data collected across groups and sharing these widely, reflecting the collective funding and volunteer workforce that produces them. This could include converting the morass of free text trial reports into machine readable curated data, in archived, citable, accessible, interoperable and reusable formats, as set out in the FAIR principles.4142 Cochrane could show leadership in supporting innovation and open science for clinical trials with full credit to all data extractors before43 and after review publication44 and, in this way, harness the greatest broadest impact. This reflects the exciting current move towards better use of data to produce digital tools of direct value to clinicians, rather than academic publications alone.
We have raised these issues with Cochrane and understand that the organisation is considering whether to start reviewing its approach to sharing data (D Tovey, personal communication, 2017). We hope that our setting out the benefits of open data is a helpful contribution to open that discussion.
We appreciate that Cochrane must focus on making itself sustainable and that open data sharing may be commercially sensitive.45 But making Cochrane a champion for openness, transparency, and sharing can only be beneficial for the organisation’s reputation—and finances. We encourage Cochrane leadership to create a policy that allows open data sharing and to make explicit any concerns they have on open data sharing so that these can be resolved.
Cochrane could lead and set standards for open data sharing from systematic reviews
Availability of data from Cochrane reviews would give opportunities for collaboration, innovation, scientific replication, novel research, and clinical decision making
It would also reduce the considerable waste of the current duplication of effort in systematic reviewing
We thank the coordinating editors at Cochrane groups who supported internal discussion within Cochrane on sharing data. We are grateful for the time they spent studying and commenting on earlier versions of this manuscript and replying to our communications. We thank David Tovey, editor in chief of the Cochrane Library, for his thoughtful and helpful response to our written communication. We are grateful to the Cochrane coordinating editors: Gianni Virgili, Carlos Grillo Ardila, Juan-Pablo Casas, Jos Verbeek, Richard Wormald, and editor of Cochrane Methodology Review Group, Karen Robinson, for supporting the idea of Cochrane sharing the data.
Competing interests. FS is the information specialist of Cochrane Schizophrenia and has voluntarily extracted data for 12 Cochrane groups. CEA promotes Cochrane extensively to the public and policy makers; trains hundreds of reviewers per year, is coordinating editor of Cochrane Schizophrenia, is principal investigator on randomised trials testing the effects of disseminating Cochrane reviews in different forms and on the National Institute for Health Research infrastructure grant for Cochrane Schizophrenia. MC promotes Cochrane to the public, practitioners, and policy makers; provides training in the conduct of randomised trials and systematic reviews, is coordinating editor of the Cochrane Methodology Review Group, and seeks funding and conducts research into the methods using in systematic reviews and other evaluations of health and social care. BG has promoted Cochrane extensively to the public and policy makers; is principal investigator on OpenTrials.net, which has had a data sharing request rejected by Cochrane; has received research funding from the Laura and John Arnold Foundation, the Wellcome Trust, the NHS NIHR School of Primary Care, the Health Foundation, NHS England, the NIHR Oxford Biomedical Research Centre, and the World Health Organization; receives personal income from speaking and writing for lay audiences on the misuse of science; and has a longstanding commitment to open science. LA promotes Cochrane to the public and policy makers; is coordinating editor of Cochrane Drugs and Alcohol Group; has received grant funding from WHO, the European Monitoring Centre for Drugs and Drug Addiction (EMCDDA), the Italian National Institute of Health, and AIFA (Italian Medicines Agency). HB has received access to Cochrane data for projects and services. JB is director and shareholder in the Trip Database, a limited company, and is actively involved in evidence synthesis. Trip has the potential to benefit from better access to the data Cochrane currently restricts. RB promotes Cochrane extensively to the public, clinicians, and policy makers; trains several reviewers a year, is joint coordinating editor of Cochrane Musculoskeletal, is principal investigator on grants developing two living Cochrane reviews and on National Health and Medical Research Council (NHMRC) editorial base funding for Cochrane Musculoskeletal, and has received research funding from the NHMRC, Cabrini Foundation, Medical Research Council, and Patient Centered Outcomes Research Institute. She is funded by an NHMRC Senior Principal Research Fellowship. CDM has received consultancy fees/honorariums from National Prescribing Service MedicineWise , the Royal Australian College of General Practitioners’ “red book” preventive guidelines committee; Therapeutic Guidelines (eTG); Remote Primary Health Care Manuals Editorial Committee for expert advice; editorial work (deputy editor of the Medical Journal of Australia; American College of Physicians’ journal club; The BMJ); Consultation work for Bupa (UK) on shared decision making: Australian Medicine Handbook; royalties for three books (Wiley and BMJ Books) on evidence based medicine and clinical thinking; grants from NHMRC (Australia) two centres for research excellence; NIHR (UK); Human Tissue Authority (UK); from a private donor (for the Cochrane Collaboration Acute Respiratory Infections Group); Australian Commission on Safety and Quality in Health Care. MD is coordinating editor of Cochrane Drugs and Alcohol Group; has received grant funding from WHO, EMCDDA, the Italian National Institute of Health and AIFA, and disseminates Cochrane review results to the public and policy makers. PG is a member of editorial group of the Cochrane Acute Respiratory Infections group. CH has received grant funding from WHO, NIHR and the NIHR School of Primary Care. MJ is an editor at Cochrane Schizophrenia Group. DM is on Cochrane Oversight Committee. RSS is joint coordinating editor of the Cochrane Schizophrenia group. LV holds an NIHR systematic reviews grant for the Cochrane Incontinence. He holds grants from: EU2020, Wellcome, Economic and Social Research Council, MRC, Health Foundation, NIHR for research using systematic review methods. EB, CG, TH, JPAI, JK, and EO have declared no conflict of interests.