Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey
BMJ 2004; 328 doi: https://doi.org/10.1136/bmj.38068.557998.EE (Published 29 April 2004) Cite this as: BMJ 2004;328:1040
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Dear Dr. Verbeek and Dr. Salmi,
We thank Drs. Verbeek and Salmi for their letter. Drs. Verbeek and
Salmi tested in PubMed our highly specific strategy (‘specificity.tw.’)
reported in our paper, and then tested our highly specific strategy
(‘specificty[tiab]’) as displayed on the Clinical Queries screen of
PubMed. Please note, that the search strategies reported in our article
are in OVID syntax. Our strategies have been translated for use in PubMed
and can be utilized through the Clinical Queries screen of PubMed. When
searching in Ovid you should use the search strategies as reported in the
paper (i.e., ‘specificity.tw’) and when searching in PubMed you should use
the search strategies reported on the Clinical Queries page. In the
example Drs. Verbeek and Salmi describe, the PubMed equivalent of Ovid’s
‘.tw.’ is ‘[tiab]’ not ‘[text word]’ (which would yield a much larger
retrieval). The narrower yield that results from using our search strategy
is achieved because the search strategy filters out diagnosis articles
that are not methodologically sound, and looks for “specificity” in the
title and abstract only rather than searching anywhere in the article.
As Drs. Verbeek and Salim note, the most specific strategy still
retrieved too many articles to handle in a couple of minutes. It is
important to note, however, that the retrieval has been significantly
reduced compared with the retrieval when searching without this filter.
When developing the most specific search strategy to identify
scientifically sound, clinically relevant diagnosis articles in MEDLINE,
we tested up to three ‘ORed’ search terms (both MeSH and textwords) and
found that the single term ‘specificity.tw.’ (Ovid syntax) yielded the
highest specificity at 98%. Thus, we are not sure how Drs. Verbeek and
Salim combined strategies to increase the specificity.
It would be desirable to have strategies that perform better than those
that have been developed. In our soon to be published work, we add age-
specific search strategies to our current search filters for MEDLINE, to
further narrow the search.
Clinicians searching for a diagnosis article will likely be best
served by the most specific search strategy. The overall yield will be
reduced by the content term(s) used. The retrieval may still be
substantive but it is significantly reduced from when searching without
the filter while carrying the guarantee that scientifically sound studies
will be included.
Best regards,
Nancy L. Wilczynski, MSc and R. Brian Haynes, MD, PhD
Competing interests:
None declared
Competing interests: No competing interests
Dear Dr Falck-Ytter and Edith Motschall,
Thank you for your letter regarding our search strategies for
detecting diagnostic studies in MEDLINE. As indicated by Dr. Falck-Ytter
and Edith Motschall (F-Y M) there is no direct translation of Ovid terms
tagged with “.mp.” to PubMed. In translating our Ovid strategies to PubMed
syntax we worked with the National Library of Medicine and display on the
Clinical Queries screen the best possible translations currently
available.
Prior to developing our search strategies, we compiled a list of
4,862 unique terms which were derived by consulting numerous sources as
outlined in the paper. The list of terms is not exhaustive and thus there
are many additional ways to search MEDLINE using a number of tags
available through Ovid (e.g., .ot., .hw.). All methods of term searching
were not tested in our database. However, we did test over 17,000 search
strategies for detecting diagnostic studies. Although we are currently
unable to test the F-Y M strategy proposed for Ovid in our database we are
able to conduct a test to compare retrieval.
We tested the F-Y M Ovid search strategy “AND”ing it to the MeSH Term
"Altitude Sickness" for a disease concept and limited the search to the
publishing year 2002. (This tactic makes the search yield tractable for
comparisons.) This search yielded 37 citations. We did the same for our
most sensitive diagnosis search strategy, yielding 34 citations, all of
which were picked up by the F-Y M proposed Ovid strategy. Thus, the F-Y M
strategy picked up three additional citations. Our most sensitive strategy
correctly eliminated these three citations as one would be classified as
etiology, one would be classified as “something else”, and one does not
pass the methods filter for diagnosis.
Additionally, we ran the F-Y M PubMed translation and our PubMed
translation in PubMed using the MeSH Term "Altitude Sickness" for a
disease concept and limited the search to the publishing year 2002. Both
searches yielded the same 40 citations, including all 37 from the F-Y M
Ovid strategy with three additional citations; all 34 from our Ovid
retrieval with six additional citations.
From this test it appears that the PubMed strategy purposed by Falck-
Ytter and Motschall performs much the same as the one posted on the
Clinical Queries page of PubMed and that the proposed Ovid strategy picks
up more false positives than our strategy posted on the Ovid “Limits” page
resulting in lower precision.
Best regards,
Nancy Wilczynski, MSc and Brian Haynes, MD, PhD
Competing interests:
None declared
Competing interests: No competing interests
Dear Dr. Haynes,
We read with great interest your article about your new search
strategy to identify diagnostic accuracy studies. Since its implementation
at PubMed as a search filter within clinical queries, it has been a great
service to everybody with an interest in evidence-based medicine. However,
when running the sensitive version of the strategy within PubMed, we
noticed differences from the version published in BMJ (Ovid version). We
are wondering whether a slightly modified version would be even better in
both systems.
Here is what we think: your new sensitive search filter
(sensitiv:.mp. OR diagnos:.mp. OR di.fs.) is straight forward in Ovid, but
because of the tag ".mp.", it is not easily portable to PubMed. For
example, "diagnos:.mp." has been translated into an array of terms
including the search term "diagnostic * [MeSH:noexp]". However, running
this term does not perform as one would expect: only 4 instead of the
possible 18 MeSH headings are found and it appears that this PubMed syntax
is unable to identify MeSH headings that include a comma or an "and", i.e.
"diagnostic techniques, cardiovascular" or "diagnostic techniques and
procedures". But your original Ovid notation actually finds all these
MeSHs resulting in a different cover range than the PubMed version.
Conversely, the search "sensitivity and specificity[MeSH Terms]" as part
of the translated version in PubMed includes the important terms
"predictive value of tests" and "ROC curve". However, it does not look
like these terms are actually included in your published strategy
(sensitiv:.mp. OR diagnos:.mp. OR di.fs.), since "sensitiv:.mp." will not
be able to pick them up. Maybe adding these terms would actually increase
the accuracy of the search?
We, therefore, performed the following tests (details not shown) to
analyze whether a more congruent version between Ovid and PubMed is
possible. First, we did a "back translation" of the PubMed strategy into
Ovid syntax and compared the resulting two versions within Ovid. Then we
constructed a PubMed compatible version with Ovid syntax within Ovid.
Finally, we took the best of both worlds (Ovid and PubMed, that is) to
construct a version that may potentially improve overall accuracy. So,
based on your work (and without going into lengthy details), the following
is a slightly modified version that we would suggest as a potential
alternative:
For PubMed: sensitiv*[Title/Abstract] OR sensitivity and
specificity[MeSH Terms] OR diagnos*[tw]
For Ovid: sensitiv:.tw. or exp "sensitivity and specificity"/ or
diagnos:.tw,ot,hw,rw. or (di or du).fs.
We would be very curious (and extremely thankful) to hear whether
these alternate strategies may in fact perform more accurately when tested
on your reference set.
With kind regards,
Yngve Falck-Ytter, M.D. and Edith Motschall
Competing interests:
None declared
Competing interests: No competing interests
I agree with Prof Haynes and his colleague that clinicians lack the
time to search the literature and often fail to search effectively.
However, they have not mentioned one of the obvious solutions to these
problems. Hospitals in the UK and elsewhere have libraries and librarians.
Librarians have the expertise and experience to enable them to search
effectively on clinicians' behalf.
The practice of medicine and healthcare requires team work. Every
clinician relies on a network of colleagues from other professions and
specialities to complete the picture of patient care. The wise clinician
looks upon colleagues in the library as another essential part of that
network of expertise.
Competing interests:
None declared
Competing interests: No competing interests
Dear Dr Haynes and Dr Wilczynski,
We are very happy with your work on search strategies, because it helps us
to spend our time on searching more efficiently. At the moment, we are
studying on an optimal search strategy for occupational health
intervention studies. Therefore, we read your article with more than
ususal interest. Although the message from the article was clear, we were
still confused about how we could best use the search strategies that you
found to be the most optimal.
We used the example from your article on
arthritis to find the meaning of the search strategies in practice. You
found 3451 articles with 'diagnosis' AND 'arthritis', which motivated you
to find better strategies. Using your best search strategy with highest
specificity and accuracy '(specificity.tw)' in PubMed, we found 3210
articles. Not a very spectacular improvement. When we use the PubMed
filter 'specificity[tiab]' from the Clinical Queries box, the number of
articles decreases to 1475. That seems to be more helpful for a busy
clinician, but still far too many articles to handle in a couple of
minutes. In addition, it is confusing that this is a different search term
than the one that you propose in your latest article. Combining more of
the advocated best search strategies reduces the number of articles to a
number that is better to manage in a short time, apparently also with
increasing the specificity. For the busy clinician, it would be helpful to
know the implications of using such a strategy. We feel that it would
increase the possibilities for implementation of your work in practice if
you could shed some light on this problem.
Competing interests:
None declared
Competing interests: No competing interests
Response to letter: Network of expertise
Dear Sara Stock,
We agree that a network of health care professionals are important in
distributing information to a busy physician, and we certainly support the
vital role that librarians play in assisting clinicians with their
searching needs. Unfortunately, there are not enough librarians to deal
with all clinical queries, especially during off-hours. Our search
strategies are presented as an option for clinicians who wish to or have
no other option but to perform a search on their own. Additionally, they
are presented as an option for clinicians who lack critical appraisal
skills as these strategies will help filter out studies that are not
scientifically sound. We hope that they are also useful for librarians.
Best regards,
Nancy L. Wilczynski, MSc and R. Brian Haynes, MD, PhD
Competing interests:
None declared
Competing interests: No competing interests