The automation of systematic reviews

Cochrane reviews are conducted in accordance with peer-reviewed published protocols. This is a major determinant of the quality of reviews published in The Cochrane Library. Published protocols serve as a public record of the pre-specified design of a systematic review and reduce the possibility for bias by preventing review authors from making post hoc decisions unless they are explicitly described and justified within the review.

In a timely editorial exploring ways that the systematic reviewing process may be automated, Tsafnat et al dream of being able to create, edit and execute protocols at the push of a button.[1] We may be a couple of steps away from full automation, but we at the editorial office Cochrane Airways Group (a Cochrane Review Group of the Cochrane Collaboration) have been thinking about ways in which we can standardise our approach and have identified a method to effectively automate parts of the production of the Cochrane protocol thus “freeing the reviewer to focus on developing and validating the review question and protocol”. We believe this process will streamline protocol production without compromising the quality of our product.

The Cochrane Collaboration recently agreed on and published a set of methodological expectations for Cochrane Reviews: the Methodological Expectations of Cochrane Intervention Reviews (MECIR) project.[2] We saw an opportunity to develop a standard template protocol based on these expectations. We call this a ‘prepared protocol’.

The prepared protocol consists of three types of text. The first type of text should be read and understood by the author but mainly left unaltered. This text comprises the minimum standards we would expect from the majority of reviews (i.e. boiler plate text), and adheres to the Cochrane Style Guide sparing time and effort during the editorial process. For example “We will include randomised controlled trials (RCTs). We will include studies reported as full-text, those published as abstract only, and unpublished trials/data.” There is also text to describe what action must be taken if the author wishes to deviate from that standard text: [You must include RCTs in your review, but if you want to include any additional study designs, you must justify the choice of eligible study deigns]. There is also text to delete where appropriate: “We will use a (*delete as appropriate: random-effects/fixed-effect) model and perform a sensitivity analysis with (*the other random/fixed model).”

The prepared protocol text provides the author with the bulk of the text needed to describe the methodology, but leaves them free to add additional text to describe methods that they need to answer their particular review question. We also provide appendices to describe standard search strategies and suitable references for the review author. Authors must provide their own text for the background section and objectives, as well as pre-specify primary and secondary outcomes, outcomes for use in a summary of findings table and approaches to subgroup and sensitivity analyses. We believe that this approach enables authors to focus on tailoring the protocol to suit their question, whilst providing some assurance that they are not engaging in a box-ticking exercise.

Time-saving advantages are not unique to the author team and we have also begun using Cochrane’s own internal IT infrastructure to supplement this process. Because the prepared protocol text is inserted into the software file in which the Cochrane Review will be written[3], differences between versions can be identified by editorial staff by comparing them on the Cochrane Collaboration server, Archie. This means that editorial staff can easily identify to those parts that have changed.

We believe this approach will save both review author and editorial staff time and free us to focus on the best way to design and implement methods that will address important review questions rather than drafting and editing methods on which the Collaboration as a whole has already agreed.

e-mail: ewelsh@sgul.ac.uk

References
1. Tsafnat G, Dunn A, Glasziou P, Coiera E. The automation of systematic reviews, BMJ 2013;346:f139
2. Churchill R, Higgins J, Chandler J, Tovey D, Lasserson T. Methodological Expectations of Cochrane Intervention Reviews (MECIR), http://www.editorial-unit.cochrane.org/mecir
3 Review Manager (RevMan) [Computer program]. Version 5.2. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2012 (http://ims.cochrane.org/revman)

I read with interest the Editorial by Tsafnat et al.1 The idea of a complete automation of systematic reviews is attractive but hardly sustainable. Whilst some degree of automation would be welcome, aspects involved in the elaboration of a review article ask necessarily for the human work. For example, the reporting of original studies does not always completely or precisely describe all the methodological aspects employed by the researchers, which can be particularly problematic for defining if a study met the inclusion criteria of a review and for the evaluation of the risk of bias; that is why authors of reviews often need to contact the authors of the original studies to clarify aspects that were not clearly reported in the published paper.

Moreover, Tsafnat et al.1 highlight that there are four basic tasks that underpin systematic review and that “technology can help in each”. However, they forgot the most important aspect of a review, i.e., to give meaning to the evidence synthesised. Therefore, interpreting data is the most crucial aspect in a review; human work at the interpretative stage of a review is irreplaceable.

In conclusion, whilst automation can positively impact the way systematic reviews are performed, by transforming updating in a “immediate, and universal” process, creation and update of reviews will never be “effortless”. And we have all the interest that it continues effortful.

e-mail: philipebarreto81@yahoo.com.br

References
1. Tsafnat G, , Dunn A, Glasziou P, Coiera E. The automation of systematic reviews. BMJ 2013;346:f139