Evaluation of symptom checkers for self diagnosis and triage: audit studyBMJ 2015; 351 doi: https://doi.org/10.1136/bmj.h3480 (Published 08 July 2015) Cite this as: BMJ 2015;351:h3480
- Hannah L Semigran, research assistant1,
- Jeffrey A Linder, associate professor 2,
- Courtney Gidengil, instructor3, natural scientist4,
- Ateev Mehrotra, associate professor15
- 1Department of Health Care Policy, Harvard Medical School, Boston, MA 02115, USA
- 2Division of General Medicine and Primary Care, Brigham and Women’s Hospital & Harvard Medical School, Boston, MA, USA
- 3 Division of Infectious Diseases, Boston Children’s Hospital, Boston, MA, USA
- 4RAND Corporation, Boston, MA, USA
- 5Division of General Internal Medicine and Primary Care, Beth Israel Deaconess Medical Center, Boston, MA, USA
- Correspondence to: A Mehrotra email@example.com
- Accepted 15 June 2015
Objective To determine the diagnostic and triage accuracy of online symptom checkers (tools that use computer algorithms to help patients with self diagnosis or self triage).
Design Audit study.
Setting Publicly available, free symptom checkers.
Participants 23 symptom checkers that were in English and provided advice across a range of conditions. 45 standardized patient vignettes were compiled and equally divided into three categories of triage urgency: emergent care required (for example, pulmonary embolism), non-emergent care reasonable (for example, otitis media), and self care reasonable (for example, viral upper respiratory tract infection).
Main outcome measures For symptom checkers that provided a diagnosis, our main outcomes were whether the symptom checker listed the correct diagnosis first or within the first 20 potential diagnoses (n=770 standardized patient evaluations). For symptom checkers that provided a triage recommendation, our main outcomes were whether the symptom checker correctly recommended emergent care, non-emergent care, or self care (n=532 standardized patient evaluations).
Results The 23 symptom checkers provided the correct diagnosis first in 34% (95% confidence interval 31% to 37%) of standardized patient evaluations, listed the correct diagnosis within the top 20 diagnoses given in 58% (55% to 62%) of standardized patient evaluations, and provided the appropriate triage advice in 57% (52% to 61%) of standardized patient evaluations. Triage performance varied by urgency of condition, with appropriate triage advice provided in 80% (95% confidence interval 75% to 86%) of emergent cases, 55% (47% to 63%) of non-emergent cases, and 33% (26% to 40%) of self care cases (P<0.001). Performance on appropriate triage advice across the 23 individual symptom checkers ranged from 33% (95% confidence interval 19% to 48%) to 78% (64% to 91%) of standardized patient evaluations.
Conclusions Symptom checkers had deficits in both triage and diagnosis. Triage advice from symptom checkers is generally risk averse, encouraging users to seek care for conditions where self care is reasonable.
Contributors: All authors conceived and designed the study. HLS acquired the data and drafted the manuscript. HLS and AM analysed and interpreted the data. CG, JAL, and AM critically revised the manuscript for important intellectual content. HLS and AM carried out the statistical analysis. AM provided administrative, technical, and material support and supervised the study. AM acts as guarantor.
Funding: This study was funded by the US National Institute of Health (National Institute of Allergy and Infectious Disease grant No R21 AI097759-01).
Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare: all authors are affiliated with Harvard Medical School. Harvard Medical School’s Family Health Guide is used as the basis for one of the symptom checkers evaluated. This symptom checker is available both in print and online (www.health.harvard.edu/family_health_guide/symptoms). None of the authors have been or plan to be involved in the development, evaluation, promotion, or any other facet of a Harvard Medical School related symptom checker; the authors have no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required
Data sharing: No additional data available.
Transparency: The guarantor (AM) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.