Googling for a diagnosis—use of Google as a diagnostic aid: internet based study
BMJ 2006; 333 doi: https://doi.org/10.1136/bmj.39003.640567.AE (Published 30 November 2006) Cite this as: BMJ 2006;333:1143All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
I am very disturbed by the suggestions in this article. Basic Internet search is different & Basic Internet search for medical literature & references is totally different. It requires lot of skill & basic medical knowledge in the medical field. It is an art & not an easy task. I would hope that a medical professional would be able to always tell the difference between a good resource and a poor one, and that they would know when it made more sense to search an online database such as Medline, rather than rely on a general search engine such as Google. Finally, it should be remembered that search engines are simply not reliable. I would strongly urge any medical professional who requires accurate and reliable data to consult a suitably qualified information professional; that is what they are there for.
Competing interests: None declared
Competing interests: No competing interests
In the mid 1980's Marsden Blois at Stanford did a study to see if a useful list for a differential diagnosis could be generated using a few key words related to observations of a patient. The premis was that there is a structure to medical knowledge and certain abnormalities would cluster around a specific diagnosis. An extensive medical reference source was used but the search tool did not have any medical intelligence embedded. It was found that this program called RECONSIDER was able to produce a list of potential diagnoses that could be used for diagnostic prompting.(1,2)
The relevance of search results depends on the search terms that are chosen. A useful approach is to use one or more terms that casts a broad net and then include a term that would likely eliminate a lot of the chaff. Another technique is to use terms that in day to day like have little to do with each other yet the terms are relevant to the specific task at hand. An example is one search that was done using "10", "gram" and "monofilament". Google found over 100,000 hits but the top ten produced exactly what was being search for, the 10 gram monofilament used for testing for decreased sensation. Putting quotations around "10 gram monofilament" resulted in 155 hits.
Searchs without any medical intelligence embedded in the search tool can still produce worthwhile results as long as there is some intelligence in the selection of the search terms.
1) Evaluating RECONSIDER: a computer program for diagnostic prompting Journal of Medical Systems, Volume 9 , Issue 5-6 (December 1985) http://www.springerlink.com/content/n023043841n9j65t/
2) "Expertness" from Structured Text? Reconsider: A diagnostic Prompting Program http://ucrel.lancs.ac.uk/acl/A/A83/A83-1021.pdf
Competing interests: None declared
Competing interests: No competing interests
Evaluating Google-assisted Diagnosis versus Unassisted Human Performance : the Missing Control Group
Abstract :
Comparing Google-assisted and/or Pubmed assisted diagnosis with unassisted human performance is the only way to properly evaluate their effectiveness. A suitable controlled trial is proposed.
Text :
Tang and Ng’s work found that 58 % of Google-Assisted diagnosis attempts led to correct diagnoses on their set of 26 NEJM cases.(1) Wentz’s replication reports an 88 % correct diagnosis rate with PubMed assisted search. (2) Sceptics have questioned the benefits of such web- database assisted diagnosis approaches, and have suggested that competent medical practitioners would not need or benefit from such assistance. 58 % might not sound particularly impressive, on the other hand the cases were challenging and unusual enough to merit publication in NEJM.
How would a cohort of registrars, GPs, or specialist consultants do on these case ? Who is correct ?
The best way to find out is –naturally - to use a randomized trial.
I am proposing the following design :
Select a series of cases. (Perhaps NEJM cases from a previous year).
Select a cohort of doctors. (Perhaps GPs, or attendees at a medical conference - ideally all of them should be already familiar with how to use Google).
Randomly assign the doctors to one of two groups.
Group 1 : (Experimental Group)
These doctors first receive quick instructions on how they should use Google or Pubmed to assist their diagnosis.
They are provided with several (perhaps 3) randomly chosen cases from the case series, and a reasonable amount of time – say 20 minutes each – to come up with a differential diagnosis list.
Group 2 : (Control Group)
These doctors first receive quick instructions specifying that they should not use any online source or database or PDA based program to assist their diagnosis.
They are provided with several (perhaps 3) randomly chosen cases from the case series, and a reasonable amount of time – say 20 minutes each – to come up with a differential diagnosis list.
Measurement
A panel of 2 or more independent evaluators – who are blinded to the group allocation will evaluate each differential diagnosis list. A diagnosis is scored as being correct if the diagnosis from the official case report is mentioned on the subject’s differential diagnosis list for that case. Evaluators will have to agree that it is sufficiently specific a diagnosis to count as being correct.
Null Hypothesis
There is no significant difference in the assisted versus unassisted correct differential diagnosis rates.
Hypothesis
Doctors assisted with PubMed or Google searches will have higher rates of correct diagnoses.
Concluding Remarks
Web-database assistance in diagnosis has been compared to a diagnostic test. As such, it is extremely inexpensive, non-invasive and safe. The question is – does it improve diagnosis?
It seems likely that web-database assisted approaches will improve the ability of doctors to diagnose challenging cases. Despite the face validity of this proposition, however, the only proper way to evaluate this is with a randomized trial.
This topic is important to the ongoing practice of medicine and deserves to be treated with appropriate scientific diligence.
References
(1) Hangwi Tang and Jennifer Hwee Kwoon Ng Googling for a diagnosis—use of Google as a diagnostic aid: internet based study BMJ 2006; 333: 1143-1145
(2) PubMed versus Google – a brief comparison Reinhard Wentz bmj.com, 19 Nov 2006
Competing interests: Former computer science and psychology lecturer.
Competing interests: No competing interests
The article by Tang et al [1] illustrates the potential value of the internet as a resource for health information. Search engines, such as Google, allow users to openly access health information that was previously restricted to health professionals. However, users must also have the ability, knowledge and expertise required to search and appraise medical information online to benefit from such a resource and not be misled.
‘Googling’ for health information has become quite popular due to its perceived ease at identifying websites that offer seemingly relevant information. Whilst Google may claim to offer the broadest index of websites, other types of search engines, including general, meta and medical portals, may offer as good, if not better return of relevant websites. [2-3]
Since the internet is an unregulated medium, users must be aware of the variable quality of information. Tang et al state that health professionals and patients are becoming more efficient at sourcing online health information. However, it can not be assumed that both types of users can efficiently search and distinguish between websites that offer good quality evidence based health information from poor websites.[4]
Providing users with the skills to effectively search and appraise information can improve their ability to efficiently sift through the good, bad and ugly health information that is so readily available on the internet.[5]
1. Tang H, Ng J. Googling for a diagnosis – use of Google as a diagnostic aid: internet based study. BMJ 2006;333:1143-1145.
2. Ilic D, Maloney S, Green S. Guiding Users to Quality Information about Osteoarthritis on the Internet: A Pilot Study. Telemedicine and e- Health 2005;11:703-706
3. Ilic D, Risbridger G, Green S. Searching the Internet for information on prostate cancer screening: An assessment of quality. Urology 2004:64;112–116
4. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002;324:573-577
5. Maloney S, Ilic D, Green S. Accessibility, nature and quality of health information on the Internet: a survey on osteoarthritis. Rheumatology 2005:44:382–385
Competing interests: None declared
Competing interests: No competing interests
It is undeniable that Google as a starting point can lead one to discover large amounts of information on an array of topics, (although it can also mislead and divert) and can be used by experts to ‘fish’ for information on a rare clinical problem before confirming by more rigorous search methods.
Tang and Ng acknowledge the role of librarians in their response on 30 November 2006. We would also like to point out that after identifying citations to potentially answer the diagnostic problems they would have accessed the full text articles to complete their decision making.
Why are some full text articles readily available via Google Scholar? What lies behind the access to articles when using Google? Did you know – it’s often your library sitting in the background!
“Google Scholar is collaborating with university libraries to develop a way to access full-text journals through institutional subscriptions” (Henderson 2004), so that researchers and physicians affiliated with a university can go directly from a Scholar search to a full-text journal article if their university has a subscription to that journal”. Although it's not obvious on the Internet full text availability is often due to the fact that your library pays for a subscription to the journal and the IP is recognised - hence access is granted. Provision of full text subscriptions and ready reference resources such as the BMJ Clinical Evidence and evidence based journals are not free in most countries; they are probably available to answer your clinical queries because your research library has integrated them seamlessly on your behalf.
Tang and Ng make the point that Google "has the advantage of being easier to use and is freely available on the internet”. We would argue that the PubMed interface matches Google’s ease of use and of course is also freely available on the Internet. PubMed has many added benefits among which, is the identification of more potentially relevant studies than Google. As Wentz R. showed in his response to BMJ on 19 November 2006, the PubMed search interface for diagnostic answers to the clinical scenarios increased the success rate of retrieving potentially relevant citations from 58% to 88%.
There is no way of knowing what resources Google searches and what has been omitted. It has been estimated that even Google Scholar indexes only 1 million of the 15 million citations appearing in PubMed (Giustini and Barsky, 2005). It does harvest from PubMed, but does not include all relevant articles as Wentz shows. Although Google Scholar is an excellent tool, currency does not appear to be a factor in the retrieval of records, with recent articles appearing many pages from the first page, or in fact not appearing in the search results at all. Using Google, results are displayed in a relevancy ranked list, partially based on how often the article is cited and the frequency of the search terms appearing on the pages retrieved. Pubmed allows searchers to sort and filter results by date or other criteria.
Google is merely a quick and easy preliminary research tool and provides direction for further confirmation. Freely available resources such as PubMed provide high quality indexing that once mastered, will ensure greater precision and higher calibre search results for every type of clinical query.
References Giustini, D and Barsky E, (2005) A look at Google Scholar, PubMed, and Scirus: comparisons and recommendations JCHLA / JABSC 26: 85–89
Henderson J. Google Scholar: a source for clinicians? CMAJ. 172(12):1549–50
Competing interests: None declared
Competing interests: No competing interests
Dear Sir
I read with some interest the article, “Googling for a diagnosis –use of Google as a diagnostic aid: internet based study” by authors Tang and Ng1. They examine the diagnostic usefulness of Google in the elusive clinical episodes of the New England Journal of Medicine case records. I fear that the authors are being simply too modest and self-deprecating. In the same way a bad workman must not blame his tools a good workman equally must not lend all the praise to his tools. There is little mention of the means by which the authors selected terms to be processed by the Google search engine. I suspect however that there was a level of “pre- processing” in the selection of the search terms. For example if in a given case, a patient presented with chest pain and shortness of breath; it is unlikely that the authors would use exclusively these two terms. Any clinician would know they would be overwhelmed with results on myocardial infarction, angina, pulmonary embolism, pneumonia, pneumothorax etc. This however would not necessarily be obvious to the lay user. If it is known that none of these are the diagnosis, then the clinician will tailor the search to ensure that a combination of symptoms is selected which is liable to exclude these possibilities flooding the results field. Hence even before the Google-search there is a level of surreptitious clinical pre-processing which subtly boost Google's diagnostic accuracy.
Yet in the right hands Google may potentially be a more powerful diagnostic tool than the authors suggest. The search may be an iterative process. It need not be an isolated search using a single set of search terms. Tentative diagnoses can be reprocessed by Google to identify their comparative merits as the definitive diagnosis. There are two important but simple premises which underlie the use of the “Google Doctor”. The first premise is that virtually all if not all diseases are currently on the internet, however obtuse or protean the mode of presentation. Google has the ability to find the relevant websites. The second premise is Occam’s razor principle of a single rather multiple causality. Both assumptions are highly reliable but not infallible, most notably the second premise2.
Undoubtedly Google is a powerful tool in the right hands; provided that the information is “pre-processed” before it is fed to Google and the user bears in mind the premises which underpin the search process.
1. Tang H, Ng J Googling for a diagnosis – use of Google as a diagnostic aid: internet based study. BMJ 2006;1143-45
2. Neira MI, Sanchez J, Moreno I, Chiaraviglio A, Rayo A, Gutierrez J, Erice A. Occam can be wrong: a young man with lumbar pain and acute weakness of the legs. Lancet 11;367:540
Competing interests: None declared
Competing interests: No competing interests
The article by H. Tang and J.H.K Ng is an "interesting" article. Whether such an "interesting" article warrants the front page of the BMJ is a matter of debate.
The BMJ is a medical journal and not a magazine wherein juicy information is provided to readers. Would I be wrong if I said that most people are aware of Google and use it routinely to search for information.
I do however agree with the authors that should one know the proper seach terms he will be able to find "almost anything on the web". However, the accuracy of information obtained will vary and so will the inter- "searcher" variablity.
If I was to take the same cases and search it using search terms chosen by me, I will most likely never get a value as quoted by the authors. And if I dont, then, is there any use for this study, since it may not be replicable.
We need to be wary, little knowledge is dangerous and tools are only as good as the person using them.
Competing interests: None declared
Competing interests: No competing interests
This google co-op beta version is just beginning and potentially may be quite useful. Quick access to it might help practitioners or patients, but it is unwieldy to type in a long address such as: http://64.233.167.99/ (which was provided by one of the previous correspondents.) This numerical type of web address is impractical to use, and furthermore, it does not link to google's health co-op beta search engine.
For this reason I have prepared a web site which does nothing but link directly to google's new health co-op beta search engine. This web site is now active: the user need only type in " www.searchmed.net " (or click on the searchmed links in this text). Typing your search terms directly into the "SearchMedicum" google custom search box will take you directly to your results, using the google health specialty co-op beta engine. You can further refine the search by clicking on the options available (prognosis, treatment, etc.).
I will attempt to keep this link active, even if google changes its engine's URL. This allows me - and any other practitioner so inclined - to easily access this new tool.
The web site with its link, already functional, is: www.searchmed.net
I have picked the names "searchmed.net" and "SearchMedicum" because they are easy to remember and use. A "www.searchmedicum.net" link will also be created soon.
Respectfully,
Lance Montauk
Competing interests: My son has worked for google for 18 months and my daughter has worked there for 4 months.
Competing interests: No competing interests
As a layman who lived with an undiagnosed condition for decades, and who has participated in several different online "patient communities", I've seen the good, the bad, and the ugly of Internet medicine.
First the ugly: There are unfortunately many commercial (or simply crackpot) sites selling miracle cures, from noni juice to mysterious electrical devices. There are people who will tell you they can cure AIDS with radio waves, treat cancer with a macrobiotic diet, etc. I think we'll all agree that it's unfortunate that folks fall for this sort of thing, but desperate people often do foolish things, and, it seems, there's always someone more than willing to profit from the desperation of others.
(It may behove the medical community to undertake some patient education in this area, and encourage legislation to crack down on the worst offenders. However, I perceive great peril in attempting to regulate online medical information in some more general way.)
The bad: The classical case of this is the person who Googles "twitching" and comes up with the MedlinePlus page on the topic. Skipping down past the list of common causes, the anxious patient's eyes stop on "ALS", and an ALS fixation sets in. No amount of doctor's visits, EMG testing, or reassurance from others will assuage the anxiety, and the anxiety, it seems, only makes the twitching worse.
The good: Many people with rare or unusual disorders have either found an initial diagnosis or, after conventional diagnosis, found a wealth of information on treating their disease or dealing with its symptoms. (Though, I'm sad to report, there are numerous examples of doctors rejecting any and all patient-presented, Internet-derived information, regardless of its merits.)
Others with both rare and not-so-rare conditions have found "communities" of fellow-sufferers who have among them a combined knowledge and experience beyond that of even the most accomplished "experts" in the field. Perhaps more importantly, these communities often serve to relieve the isolation that many patients feel, especially when dealing with chronic conditions that lead to some degree of disability.
Based on what I've seen and experienced, I believe that the Internet (and, in particular, Google-style full-text searching) can be a valuable resource for doctors, but I also recognize that the existing tools are far too inefficient for the average (ie, busy) diagnostician to use regularly. However, it seems likely that some relatively minor customization of existing search engines would greatly improve their utility in this area, and "forward thinking" doctors would be well-advised to begin incorporating Google and its kin into their practice on an experimental basis.
Competing interests: None declared
Competing interests: No competing interests
Googling a Diagnosis 10 years Later
In 2006 Tang and Ng published this much promulgated study about the use of internet search in making a diagnosis. They used case reports from the New England Journal of medicine, and identified the unique or prominent features to use as search terms, and found that an internet search on Google was able to correctly identify the diagnosis more than half the time (58%). I recall learning about this study as a medical student 10 years ago, and since then I have heard it quoted in academic meetings and other learning environments.
We sought to see how much the internet has changed in the past 10 years. We surmised that internet search techniques may have improved during this time, or perhaps the internet has become more overloaded with unhelpful forums and opinion based content. As best we could, we duplicated the original study by using the same search terms and strategy to develop a differential diagnosis for the same cases, while blinded to the final diagnosis. Of course, we disregarded any webpages that referred to this study or the original cases - all published online.
We found similar results to the original investigation 10 years ago. The correct diagnosis was identified from Google in 15 of 26 cases (58%). 20 of 26 internet searches returned the same results as the original study, 3 cases did better in 2016 and 3 cases did worse. So, we conclude that the utility of basic internet search engines for making a diagnosis has not changed substantially in the past decade.
Scott Haines
Nabil Khandker
Competing interests: No competing interests