Papers

Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews

R J McManus, clinical research fellow, ^a S Wilson, research fellow, ^a B C Delaney, senior lecturer, ^a D A Fitzmaurice, senior lecturer, ^a C J Hyde, director, ^b R S Tobias, research associate, ^a S Jowett, research associate, ^a F D R Hobbs, professor. ^a

^a Department of Primary Care and General Practice, University of Birmingham, Birmingham B15 2TT, ^b Aggressive Research Intelligence Facility, Edgbaston, Birmingham B16 9PA

Correspondence to: Dr McManus r.j.mcmanus{at}bham.ac.uk

	Introduction

Top Introduction Methods and results Comment References

Identifying relevant studies is "the most fundamental challenge" when compiling a systematic review.¹ Electronic databases, such as Medline, may detect only about half of papers identified by the gold standard of hand searching journals.¹ Hand searching requires a focus, usually the specialist literature,² which may not exist for newly developed fields or those that cross boundaries with other areas. We examined the usefulness of contacting other experts when searching for relevant references for a systematic review of a field where such a specialist focus does not exist.

	Methods and results

Top Introduction Methods and results Comment References

As part of a systematic review undertaken in 1996, all published literature relating to "near patient testing" (any investigation performed in a clinical setting where the result is available without a sample being sent to a laboratory for analysis) in primary care was identified for 1986-95.³ Electronic databases were searched and secondary citations were collected from identified publications (see table). The search strategy is reported elsewhere.³ Indexes of abstracts from major international primary care scientific conferences were hand searched. We sent a questionnaire to 194 academics in the United Kingdom (heads of academic departments of general practice and clinical chemistry and researchers identified from the previously mentioned abstracts) and to 152 commercial companies known to have an interest in near patient testing. The questionnaire requested key references from journals, unpublished data, and names of other workers in the field.

Overall, 156 (45%) questionnaires were returned completed (103 (53%) of those sent to academics and 53 (35%) of those sent to commercial companies). No unpublished data were offered.

Articles that did not report original data, were not relevant to primary care, or were not in English were excluded. Remaining articles were then judged by both an external and internal reviewer against standard appraisal criteria,⁴ with discrepancies being adjudicated on by a third reviewer (BCD). A verifying search did not identify any further key words or references.

The searches yielded 1286 citations comprising 1057 unique references potentially eligible for inclusion in the review. The table shows a breakdown of the results by source102 references were relevant, of which 29 were of high quality. Of the 102 unique eligible references, 50 (49%) were identified by one of the electronic databases, 40 (39%) by people working in the field, and 31 (30%) by hand searches. Each of these sources produced a similar proportion of high quality papers.

The 1057 potential references came from 418 different publications, of which only 48 were found to contain relevant articles. Only one journal, the BMJ, contained more than 10% (11 articles) of the relevant references.

	Comment

Top Introduction Methods and results Comment References

This study confirms that searching electronic databases may uncover only half of all relevant studies¹ and shows the importance of contacting other experts when compiling a systematic review. Over 300 of the references in this review were identified by experts working in the field, of which 40 were found subsequently to be eligible. Twenty four references (24% of the total number of eligible references) would have been missed entirely without the input of people working in the field.

The lack of sensitivity of electronic databases may be due to problems with indexing⁵: articles from journals that lie outside of the mainstream (which currently includes primary care) take some years to be electronically indexed; furthermore, near patient testing is an expanding field without unique medical subject headings (MeSH terms). The continuing work of the Cochrane Collaboration is improving this situation,⁵ but at the time of this study, making up the shortfall in articles by hand searching would not have been justifiable in terms of either time or money.

This study has quantified the contribution of experts to a systematic review and has found them to be an essential source for identifying literature. We suggest that appropriate experts should be consulted when performing a systematic review in a developing field that does not have a clearly defined specialist literature.

	Acknowledgments

Contributors: RJMcM participated in the analysis, wrote the first draft, and coordinated the redrafting and the editing of the paper. SW participated in the study design and coordinated the analysis. BCD participated in the study design, reviewed the initial search results and supervised the project. DAF participated in the study design and reviewed the initial search results. CJH particpated in the study design and the analysis. RST participated in the design of the search strategy and in identifying and obtaining relevant citations from the searches. SJ participated in identifying and obtaining citations, designed the database used, enetered the data and participated in the data analysis. FDRH participated in the study design and in obtaining national competitive funding to perform the work. All the authors contributed to redrafting and editing of the final paper. FDRH will act as guarantor.

Funding: Health Technology Assessment Programme.

Competing interests: BCD has received research funding and sponsorship to attend meetings from Cortecs. DAF has received research funding from Roche Diagnostics and has been sponsored to attend conferences by Nycomed UK and Roche Diagnostics. FDRH has received research grants from Nycomed UK, Roche Diagnostics, and Cortecs.

	References

Top Introduction Methods and results Comment References

1. Chalmers I, Dickersin K, Chalmers TC. Getting to grips with Archie Cochrane's agenda. BMJ 1992; 305: 786-788.
2. Jadad AR, McQuay HJ. Searching the literature. Be systematic in your searching. BMJ 1993; 307: 66.
3. Hobbs FDR, Delaney BC, Fitzmaurice DA, Wilson S, Hyde CJ, Thorpe GH, et al. A review of near patient testing in primary care. Health Technol Assessment 1997;1(5).
4. Jaeschke R, Guyatt G, Sackett DL. Users' guides to the medical literature. III. How to use an article about a diagnostic test. A. Are the results of the study valid? JAMA; 271:389-91.
5. Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. In: Chalmers I, Altman DG, eds. Systematic reviews. London: BMJ Publishing , 1995.