Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sourcesBMJ 2005; 331 doi: https://doi.org/10.1136/bmj.38636.593461.68 (Published 03 November 2005) Cite this as: BMJ 2005;331:1064
All rapid responses
Greenhalgh et al (1) describe their experience reviewing published
evidence for health care topics other than straightforward questions of
clinical efficacy and safety. While we concur that such topics pose a
greater challenge to researchers' ability to anticipate exact search and
selection criteria prospectively, we believe that approaches can be
developed which remain systematic rather than opportunistic. Manual checks
of references in accepted papers are, we agree, a key source of important
data – they can also serve as a test of the original electronic search
We suggest and have used an iterative approach (2) whereby all
manually or "serendipitously" found relevant articles are checked to
identify the reasons they were not captured in the initial search
(publication in a journal not indexed in the electronic databases
searched, not indexed under search terms used, etc.). A systematic remedy
can then be put into place, via protocol amendment, ensuring that
similarly relevant papers (perhaps with different findings) also have the
highest chance of being found. This allows confidence that appropriate
efforts to reduce bias have been made in the review.
Incidentally, the good news for researchers is that papers not found
due to complete lack of electronic indexing are increasingly uncommon.
More typically in our experience, search terms need to be broadened to
become more comprehensive – resulting in large numbers of abstracts to
weed through, but maintaining the systematic nature of the research.
1. Greenhalgh T, Peacock R. Effectiveness and efficiency of search
methods in systematic reviews of complex evidence: audit of primary
sources. BMJ 2005 Nov 5;331(7524):1064-5.
2. Frame D, Klawansky S. Increasing PubMed yield for pharmacoeconomic
research using iterative search strategies. ISPOR Connections 2003; 9
Competing interests: No competing interests