We welcome the editorial by Tsafnat and colleagues, which is a timely statement of the potential for new information technologies to increase systematic review efficiency, and support the Rapid Responses by Barreto and Elliott et al. However, while the potential for automation is apparent, further methodological and evaluative work is needed to develop methods and an evidence base for their use; in particular, the amount of empirical data to support many of the purposes outlined in the editorial is very limited. [1] Moreover, there are likely to be some situations in which such procedures operate better than others. For example, natural language processing may work well in highly technical literatures that contain considerable jargon and regularly employed acronyms, but operate less effectively in non-clinical literature, such as public health, where the same meaning can be expressed in multiple ways. With this in mind, developing fit for purpose tools can be best achieved through user engagement and cross-disciplinary collaboration between computer scientists, information specialists, and systematic reviewers.
We currently hold an MRC grant that is aiming to develop methods for using text mining for use in systematic reviews – and to evaluate the potential of such technologies to increase the efficiency of screening studies for eligibility. Part of the work involves simulations on existing review data, and part on using text mining in current reviews. We would like to encourage reviewers to contact us if they have data that we could use; might like to participate in a ‘live’ review; or would like to hear more about the study.[2]
Rapid Response:
Re: The automation of systematic reviews
We welcome the editorial by Tsafnat and colleagues, which is a timely statement of the potential for new information technologies to increase systematic review efficiency, and support the Rapid Responses by Barreto and Elliott et al. However, while the potential for automation is apparent, further methodological and evaluative work is needed to develop methods and an evidence base for their use; in particular, the amount of empirical data to support many of the purposes outlined in the editorial is very limited. [1] Moreover, there are likely to be some situations in which such procedures operate better than others. For example, natural language processing may work well in highly technical literatures that contain considerable jargon and regularly employed acronyms, but operate less effectively in non-clinical literature, such as public health, where the same meaning can be expressed in multiple ways. With this in mind, developing fit for purpose tools can be best achieved through user engagement and cross-disciplinary collaboration between computer scientists, information specialists, and systematic reviewers.
We currently hold an MRC grant that is aiming to develop methods for using text mining for use in systematic reviews – and to evaluate the potential of such technologies to increase the efficiency of screening studies for eligibility. Part of the work involves simulations on existing review data, and part on using text mining in current reviews. We would like to encourage reviewers to contact us if they have data that we could use; might like to participate in a ‘live’ review; or would like to hear more about the study.[2]
E-mail: j.thomas@ioe.ac.uk
[1] Thomas J, McNaught J, Ananiadou S (2011) Applications of text mining within systematic reviews. Research Synthesis Methods. 2(1): 1-14
[2] http://www.ioe.ac.uk/research/63969.html
Competing interests: Some of the above are authors of the paper cited and investigators on the project mentioned.