PDF- Easy Read
- Abridged PDF
- Data supplement
- Respond to this article
- Versions
- bmj.38336.804167.47v1
- 330/7482/68 most recent
Recent rapid responses
Rapid responses are electronic letters to the editor. They enable our users to debate issues raised in articles published on bmj.com. Although a selection of rapid responses will be included as edited readers' letters in the weekly print issue of the BMJ, their first appearance online means that they are published articles. If you need the url (web address) of an individual response, perhaps for citation purposes, simply click on the response headline and copy the url from the browser window.
Displaying 1-6 out of 6 published
18 February 2005
We have tested the search strategy proposed by Montori et al. [Montori VM, Wilczynski NL, Morgan D, Haynes RB, for the Hedges Team. BMJ 2005;330:68] against our own database of systematic reviews in laboratory medicine [http://www.ifcc.org/divisions/EMD/C-EBLM/database.asp]
PubMed was searched up to January 2005, using two predefined search strategies for systematic reviews. The first one, indicated by Horvath and Pewsner [Clin Chim Acta, 2004; 342: 23-39], combines the following terms: "Clinical Laboratory Techniques"[MESH] AND systematic[sb] OR "Laboratory Techniques and Procedures"[MESH] AND systematic[sb]. The second one, indicated by Montori et al., combines the following terms: ”MEDLINE[Title/Abstract] OR (systematic[Title/Abstract] AND review[Title/Abstract]) OR meta-analysis[Publication Type]”. Using these search terms, we respectively retrieved 5058 and 29640 unselected references.
In order to compare the value of these two search strategies, we tested both of them up to Janurary 2005, starting on September 2004. Horvath and Pewsner’s strategy thus enabled us to retrieve 14 systematic reviews, whereas 55 systematic reviews could be identified using Montori et al.’s strategy, among which only nine had been identified using Horvath and Pewsner’s strategy. Therefore, Montori et al.’s search strategy appears to be more effective than Horvath and Pewsner’s, but cannot replace it.
Competing interests: None declared
Competing interests: None declared
Hôpital de Rodez, France
Click to like:
+15 February 2005
We appreciate the interest of Schulman and Nelson in our work and the kind words they offer. We hear their lament about difficulties in identifying a reproducible definition to categorize systematic reviews as a publication type. However, we reported in our paper that our research staff were able to reliably categorize systematic reviews as such (chance- adjusted inter-observer agreement of 89% (95% confidence interval 78% to 99%)) using the following operational definition: “For an article to be considered a systematic review, the authors had to 1) clearly state the clinical topic of the review and 2) how the evidence was retrieved and 3) from what sources, and they had to 4) provide explicit inclusion and exclusion criteria and 5) include at least one study that passed methodological criteria for the purpose category. For example, reviews of interventions had to have at least one study with random allocation of participants to comparison groups and assessment of at least one clinical outcome.” The fifth of these criteria would likely be too difficult for indexers to apply, given their time constraints, but for retrieval purposes it is unnecessary (see below). It would be easy to test the reliability of indexers in applying the first 4 criteria.
Our most sensitive search strategy has high recall (~100%) even with the current indexing, but low precision (~3%). Improvements in indexing would likely result in substantive improvements in precision, an important problem for users, who must spend valuable time sorting systematic reviews from the rest of citations retrieved. For that reason, we hope that the NLM will reconsider its procedures for indexing systematic reviews.
In our paper, we evaluated the performance of the NLM review retrieval strategy implemented in the Clinical Queries front end for PubMed. It performs similarly to our “optimal” review strategy, and is not as sensitive as our most sensitive strategy, nor as specific as our most specific strategy. As Schulman and Nelson indicate, their strategy retrieves “systematic reviews, meta-analyses, reviews of clinical trials, evidence-based medicine, consensus development conferences, guidelines, and citations to articles from journals specializing in review studies of value to clinicians.” Thus, this strategy retrieves not just systematic reviews, and should be considered as complementary to our strategy, rather than as a substitute for it. Further, the NLM strategy should have a name that better matches its purpose: it is not the best strategy for retrieving systematic reviews.
We are grateful to the National Library of Medicine for many things, including its ongoing provision of worldwide online access to the medical literature, its detailed indexing of the literature, its commitment to continuous quality improvement, its funding of information retrieval research (including ours!), and its openness to innovation in retrieval strategies for specialized needs via Clinical Queries. We think that PubMed Clinical Queries users will be well served by a strategy that has been rigorously developed and validated to identify systematic reviews and respectfully request that it be added to the validated strategies for retrieval of studiers of treatment, diagnosis, prognosis, and etiology, with renaming of the NLM strategy to avoid confusion.
Competing interests: None declared
Competing interests: None declared
McMaster University, 1200 Main St W, Hamilton, Ontario L8N 3Z5 Canada
Click to like:
+11 February 2005
Editor
Montori et al. suggest the need for a new publication type in the National Library of Medicine’s Medical Subject Heading (MeSH) vocabulary. 1
The National Library of Medicine (NLM) recognizes the importance of systematic reviews. At the same time, we in the Medical Subject Headings (MeSH) section strive for a controlled vocabulary, with items clearly named and distinct in their use. An important criterion is that the indexer who is assigned the article be able to assign appropriate indexing terms in a reproducible manner.
The MeSH staff has considered several times suggestions to add “Systematic Review” as a publication type. However, we have been unable to find a suitable and reliable definition which would guide the indexer to a reproducible use of that publication type. Accordingly, we have not implemented those suggestions.
The NLM has devised a search strategy that is designed to retrieve systematic reviews. This stored strategy retrieves citations identified as systematic reviews, meta-analyses, reviews of clinical trials, evidence- based medicine, consensus development conferences, guidelines, and citations to articles from journals specializing in review studies of value to clinicians.
This subset can be used in a search as a Boolean expression anded with “systematic [sb].” For example, a search in PubMed with the expression:
“exercise hypertension AND systematic [sb]”
would retrieve citations of systematic reviews and exercise hypertension.
The subset is also available on the Clinical Queries screen of PubMed. The strategy is available here: http://www.nlm.nih.gov/bsd/pubmed_subsets/sysreviews_strategy.html MeSH welcomes the efforts and observations such as that described by Montori et al. and considers carefully all suggestions for improvements to this important controlled indexing vocabulary.
Jacque-Lynne Schulman
Stuart J. Nelson, MD, FACMI
MeSH Section, National Library of Medicine, NIH, DHHS, Bethesda, Maryland USA
Reference 1. Montori, V. et al. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ 2005;330-68.
Competing interests: None declared
Competing interests: None declared
National Library of Medicine, Bethesda, MD 20894 USA
Click to like:
+31 January 2005
Search strategy |
Sensitivity (%) 95% CI |
Specificity (%) 95% CI |
Precision (%) 95% CI |
Accuracy (%) 95% CI |
Top sensitivity search:.tw. OR
risk:.tw. OR
outcome.tw. OR
meta-analys:.tw |
93.2 (90.8 to 95.5) |
78.4 (77.9 to 78.9) |
6.1 (5.6 to 6.7) |
78.6 (78.1 to 79.1) |
Top specificity search
strateg:.tw. |
58.8 (54.2 to 63.4) |
99.95 (99.93 to 99.98) |
94.9 (92.2 to 97.5) |
99.3 (99.2 to 99.4) |
Optimal difference search:.tw. OR
meta-analys:.tw. |
87.0 (83.9 to 90.2) |
98.6 (98.5 to 98.8) |
49.3 (45.8 to 52.8) |
98.5 (98.3 to 98.6) |
Competing interests: None declared
Competing interests:
Health Information Research Unit, McMaster University, Hamilton, Ontario, Canada L8N 3J5
Click to like:
18 January 2005
The paper of the Hedge Team (1) represents a very good benchmark for approaching to MEDLINE search via Pubmed or Ovid interfaces. However, I advise the authors to reconsider some inaccuracies that could act as confounders for unskilled reader about this subject.
First of all, the strategy developed by Shojania and Bero (2) is not only a search strategy programmed into the PUBMED Clinical Queries web-page, but it is actually one of the Pubmed subset strategies. That is, one can search for systematic reviews simply typing “systematic [sb]” into the Pubmed query box where one can enter search terms.
Secondly, the authors report, through the text and in table 3, “cochrane database of systematic reviews.jn." as a top precision performer to find out Cochrane Systematic Reviews via Ovid interface, but they do not say anything about Pubmed translation of that string. Using Pubmed, one can do the same thing using a tag (that is, within square brackets) named “journal name”: “[jour]” instead of the above mentioned “.jn.”. In this case, one have to type into the Pubmed query box as little as a significative different string, “Cochrane Database Syst Rev [jour]”, to achieve the same purpose. However, some troubles exist using that string. Indeed, Pubmed allow to retrieve all the Cochrane Systematic Reviews including updates, so that we have redundant information that right now amount to 3019 retrieved citations against only 2170 complete reviews of the Cochrane Library database (3).
1) Victor M Montori, Nancy L Wilczynski, Douglas Morgan, R Brian Haynes for the Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ 2005; 330: 68
2) Shojania KG, Bero LA. Taking advantage of the explosion of systematic reviews: an efficient Medline search strategy. Effective Clin Pract 2001;4: 157-62.
3) http://www3.interscience.wiley.com/cgi-bin/mrwhome/106568753/HOME
Competing interests: None declared
Competing interests: None declared
Civico e Benfratelli Nation Hospital Trust - Piazza Nicola Leotta, 2 - 90127 Palermo
Click to like:
13 January 2005
MEDLINE In-Process citations have been submitted to the National Library of Medicine for indexing, but have not yet been assigned MeSH terms or complete publication types (2). The length of time a citation may spend in-process varies by journal title and time of year. A search of PUBMED today (Jan 12, 2005), limited to articles having the MeSH term "human," the most recent articles from the New England Journal of Medicine and the Journal of General Internal Medicine are from last week and four months ago, respectively. These delays are important as most hedges include MeSH terms.
Recent changes in medical publishing have increased the importance of searching these in-process citations by diverse groups of health care workers. Regarding the needs of clinicians, new online references such as UpToDate and PIER are increasingly able to cover literature published up to the most recent few months. Regarding the needs of researchers such as meta-analysts, or for any researcher submitting important grants, a search for articles just before submission may retrieve important content. Lastly, resources such as UpToDate, PIER, or Clinical Evidence that strive to identify recent advances also need hedges that allow their editors to reliably search the for most recent publications.
We suggest that the Hedges team report the ability of the new hedges to search MEDLINE In-Process citations. Although the existing hedges contain text words that will retrieve content from the In-Process Citations, the text words were chosen based on how they complement the MeSH terms and publication types when searching MEDLINE. If the hedges meaningfully decay, we urge the team to create parallel hedges for In-Process citations.
References:
1. Montori VM, Wilczynski NL, Morgan D, Haynes RB; Hedges Team.
Optimal search strategies for retrieving systematic reviews from Medline: analytical
survey. BMJ. 2005;330:68. PMID: 15619601 [full
text]
2. Tybaert S. MEDLINE®/PubMed® End-of-Year Activities.
NLM Tech Bull. 2004 Sep-Oct;(340):e4. (available at: http://www.nlm.nih.gov/pubs/techbull/so04/so04_end_year.html)
Sincerely,
Robert G Badgett, MD
badgett@uthscsa.edu
Janna C. Lawrence, MLIS, AHIP
jlawrence@uthscsa.edu
Veterans Evidence-based Research, Dissemination, and Implementation Center
(VERDICT)
ALMD/South Texas Veterans Health Care System
7400 Merton Minter Blvd (11C6)
San Antonio, Texas 78229-4404
Briscoe Library
The University of Texas Health Science Center
7703 Floyd Curl Dr, MSC 7940
San Antonio, TX 78229-3900
Competing interests: None declared
Competing interests: None declared
Veterans Evidence-based Research, Dissemination, and Implementation Cen; San Antonio, TX 78229-440
Click to like:
Latest comments and most commented
Replacement of lead by depleted uranium: no reasons for radiophobia.
Published 23 May 2013
Please could we have some fact.
Published 23 May 2013
Re: CT radiation risks coming into clearer focus
Published 23 May 2013
Re: Should we sequence everyone’s genome?
Published 23 May 2013