The automation of systematic reviews

Re: The automation of systematic reviews: Get real

1 February 2013

Optimism is important but reality matters most. We agree with this editorial (1) : comprehensive, complete and up to date systematic reviews are desirable. However, automating systematic reviews to do this is a utopian aim that is flawed in its dismissal of language, its scientism, and a mis-placed faith in technology.(2)

That the authors describe systematic reviews as involving ‘basic tasks’ of retrieval, appraisal, synthesis and publication lays bear simplistic misconceptions. Language in systematic review is inescapable and necessarily involves interpretation.(3) For example, descriptions of health interventions are often messy, inconsistent, and incomplete. These descriptions themselves are interpreted when re-described, classified or quantified.(4) Appraisal of bias in studies, though protocol-led, still involves all manner of interpretations and a fusion of judgment, nuance and fairness that computers cannot provide. Finally, conclusions are never inherent in pooled data: do null pooled findings indicate there is ‘no evidence to support an intervention’ or should this intervention ‘not be practiced?’(5) Arguments that these difficulties can be addressed via superior technology, better or more rigorous methods and / or the removal of humans altogether are predictable and amiss. The belief that method and technology can conquer the challenges created by language, as the authors do, is historically common but wrong.(6)

Humans should always be involved in systematic reviews. While ‘technology’ can assist them identify relevant studies, classify these using taxonomies, appraise their quality via tools and calculate synthesized results software – what studies, data and conclusions mean is always about interpretation and needs to involve humans. To do so is not to admit failure in method or science or technology but to recognize properly the nature of language and reality.

1 Tsafnat G, Dunn A, Glasziou P, Coiera E. The automation of systematic reviewsWould lead to best currently available evidence at the push of a button. BMJ. 2013;346(3): f139. doi:10.1136/bmj.f139.
2 Postman N. Technopoly: The surrender of culture to technology. Londong: Vintage; 1993.
3 Palmer R. Hermeneutics: Interpretation Theory in Schleiermacher, Dilthey, Heidegger, and Gadamer. Evanston: Northwestern University Press; 1969.
4 Pawson R. Evidence-based policy: A realist perspective. London: Sage; 2006.
5 Haykowsky M, Liang Y, Pechter D, Jones L, McAlister F, Clark AM. A Meta-Analysis of the Effect of Exercise Training on Left Ventricular Remodeling in Heart Failure Patients: The Benefit Depends on the Type of Training Performed. J Am Coll Cardiol. 2007;49:2329-36.
6 Gadamer HG. Truth and Method. 2 Revised ed. London: Sheed and Ward; 1989.

Competing interests: None declared

Alexander M Clark, Professor

Tom Briffa, Lis Neubeck, Lorraine Thirsk, Julie Redfern

University of Alberta, Faculty of Nursing, 11405 87 Avenue University of Alberta Edmonton, Alberta, Canada, T6G 1C9

Click to like: