Intended for healthcare professionals


The automation of systematic reviews

BMJ 2013; 346 doi: (Published 10 January 2013) Cite this as: BMJ 2013;346:f139

Re: The automation of systematic reviews

We would like to congratulate Tsafnat and colleagues for a rich overview of automation in systematic review production [1]. We support the authors’ central argument that automation has the potential to transform the processes involved in producing systematic reviews. Indeed, technological innovation has played a central role since the birth of systematic review [2] and machine processes are embedded throughout current systematic review workflow from evidence retrieval to meta-analysis. Nevertheless, significant inefficiency and redundancy remain and current methods are not sustainable in the face of expanding demands for high quality evidence and the data deluge of primary research [3].

Tsafnat and colleagues highlight the substantial innovation that has occurred in this field in recent years, yet there has been a dearth of real world applications implemented and widely available for review authors. A key challenge has been achieving performance perceived to be adequate by users who often prioritise methodological rigor over efficiency. Furthermore, many promising innovations are yet to develop into viable services and relevant advances in related fields have not been translated into applications for systematic review. We need vibrant environments for systematic review innovation and incentive structures for rapid and broad release.

Whilst a systematic review remains dependent on the analysis of unstructured data and text, human input is likely to remain critical at every step of review workflow. We should therefore focus not just on ‘making the machines work harder’, but in creating the best partnership between people and machines [4]. For example, the Cochrane Collaboration is working to optimise the value of human effort by utilising the PICO structure to link reports, studies, reviews and external data sources in linked data repositories based on semantic technologies [5]. The relationship between the new Cochrane Register of Studies and Cochrane’s review writing software, RevMan, continues to develop towards a vision of semi-automated inclusion of extracted data into reviews from a common data repository.

Achieving efficient production of high quality evidence reviews is an important public good. With ongoing and diverse innovations, such as those described by Tsafnat and colleagues, we believe the trade-offs in evidence synthesis between methodological rigour and review currency can be eroded, resulting ultimately in ‘living systematic reviews’: high quality online evidence summaries that are dynamically updated as new evidence becomes available [6].

1. Tsafnat G, Dunn A, Glasziou P, Coiera E. The automation of systematic reviews. BMJ 2013;346:f139.

2. Chalmers I. Electronic publications for updating controlled trial reviews. Lancet 1986;328:287.

3. Bastian H, Glaziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how we will we ever keep up? PLoS Med 7(9):e1000326.

4. Kasparov G. The chess master and the computer. The New York Review of Books February 11, 2010.

5. Mavergames C. Becker L. Cochrane Linked Data Project: From “Star Trek” to the present. December 2012.

6. Elliott J. Exploiting innovations in technology to improve the efficiency of review production. 20th Cochrane Colloquium, Auckland, October 2012.

Competing interests: All authors are either employed by or are active contributors to the Cochrane Collaboration.

17 January 2013
Julian H Elliott
Infectious Diseases Physician
Chris Mavergames, Lorne Becker, Jörg Meerpohl, Jessica Thomas, Russell Gruen, David Tovey
Alfred Hospital, Monash University and Australasian Cochrane Centre
Commercial Road, Melbourne, Australia 3004