Intended for healthcare professionals

CCBY Open access
Research

Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies

BMJ 2020; 368 doi: https://doi.org/10.1136/bmj.m127 (Published 10 February 2020) Cite this as: BMJ 2020;368:m127

Linked Editorial

The poor performance of apps assessing skin cancer risk

  1. Karoline Freeman, methodologist1 2,
  2. Jacqueline Dinnes, senior methodologist1 3,
  3. Naomi Chuchu, NIHR fellow in test evaluation and research fellow in cancer epidemiology1 4,
  4. Yemisi Takwoingi, senior research fellow in biostatistics1 3,
  5. Sue E Bayliss, information specialist1,
  6. Rubeta N Matin, consultant dermatologist5,
  7. Abhilash Jain, associate professor of plastic and hand surgery6 7,
  8. Fiona M Walter, principal researcher in primary care cancer research8,
  9. Hywel C Williams, professor of dermato-epidemiology9,
  10. Jonathan J Deeks, professor of biostatistics1 3
  1. 1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham B15 2TT, UK
  2. 2Warwick Medical School, University of Warwick, Coventry, UK
  3. 3NIHR Birmingham Biomedical Research Centre, University Hospitals Birmingham NHS Foundation Trust and University of Birmingham, Birmingham, UK
  4. 4London School of Hygiene and Tropical Medicine, London, UK
  5. 5Department of Dermatology, Churchill Hospital, Oxford, UK
  6. 6Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
  7. 7Department of Plastic and Reconstructive Surgery, Imperial College Healthcare NHS Trust, St Mary’s Hospital, London, UK
  8. 8Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK
  9. 9Centre of Evidence Based Dermatology, University of Nottingham, Nottingham, UK
  1. Correspondence to: J Deeks j.deeks{at}bham.ac.uk
  • Accepted 17 December 2019

Abstract

Objective To examine the validity and findings of studies that examine the accuracy of algorithm based smartphone applications (“apps”) to assess risk of skin cancer in suspicious skin lesions.

Design Systematic review of diagnostic accuracy studies.

Data sources Cochrane Central Register of Controlled Trials, MEDLINE, Embase, CINAHL, CPCI, Zetoc, Science Citation Index, and online trial registers (from database inception to 10 April 2019).

Eligibility criteria for selecting studies Studies of any design that evaluated algorithm based smartphone apps to assess images of skin lesions suspicious for skin cancer. Reference standards included histological diagnosis or follow-up, and expert recommendation for further investigation or intervention. Two authors independently extracted data and assessed validity using QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies 2 tool). Estimates of sensitivity and specificity were reported for each app.

Results Nine studies that evaluated six different identifiable smartphone apps were included. Six verified results by using histology or follow-up (n=725 lesions), and three verified results by using expert recommendations (n=407 lesions). Studies were small and of poor methodological quality, with selective recruitment, high rates of unevaluable images, and differential verification. Lesion selection and image acquisition were performed by clinicians rather than smartphone users. Two CE (Conformit Europenne) marked apps are available for download. SkinScan was evaluated in a single study (n=15, five melanomas) with 0% sensitivity and 100% specificity for the detection of melanoma. SkinVision was evaluated in two studies (n=252, 61 malignant or premalignant lesions) and achieved a sensitivity of 80% (95% confidence interval 63% to 92%) and a specificity of 78% (67% to 87%) for the detection of malignant or premalignant lesions. Accuracy of the SkinVision app verified against expert recommendations was poor (three studies).

Conclusions Current algorithm based smartphone apps cannot be relied on to detect all cases of melanoma or other skin cancers. Test performance is likely to be poorer than reported here when used in clinically relevant populations and by the intended users of the apps. The current regulatory process for awarding the CE marking for algorithm based apps does not provide adequate protection to the public.

Systematic review registration PROSPERO CRD42016033595.

Footnotes

  • Contributors: KF and JD contributed equally to this work. KF, JD, and NC undertook the review. All authors contributed to the conception of the work and interpretation of the findings. KF, JD, and JJD drafted the manuscript. All authors critically revised the manuscript and approved the final version. JJD acts as guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: This paper presents independent research supported by the National Institute for Health Research (NIHR) Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham (grant reference No BRC-1215-20009) and is an update ofone of a collection of reviews funded by the NIHR through its Cochrane Systematic Review Programme Grant (13/89/15). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the National Institute for Health Research (NIHR) Systematic Reviews Programme and the NIHR Birmingham Biomedical Research Centre for the submitted work; KF is funded by the NIHR through a doctoral research fellowship; JD, NC, YT, SEB, and JJD were employed by the University of Birmingham under an NIHR Cochrane Programme grant to produce the original Cochrane review which this paper updates; JD, JJD, and YT are supported by the NIHR Birmingham Biomedical Research Centre; RNM received a grant for a BARCO NV commercially sponsored study to evaluate digital dermoscopy in the skin cancer clinic and received payment from Public Health England for “Be Clear on Cancer Skin Cancer” report and royalties for Oxford Handbook of Medical Dermatology (Oxford University Press); HCW is director of the NIHR Health Technology Assessment Programme, part of the NIHR which also supports the NIHR systematic reviews programme from which this work is funded; no other relationships or activities that could appear to have influenced the submitted work.

  • Ethical approval: Ethical approval not required.

  • Data sharing: No additional data available

  • The lead authors and manuscript’s guarantor affirm that the manuscript is an honest, accurate and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.

  • Dissemination to participants and related patient and public communities: No study participants were involved in the preparation of this systematic review. The results of the review will be summarised in media press releases from the Universities of Birmingham and Nottingham and presented at relevant conferences (including the Centre of Evidence-Based Dermatology 2020 Evidence Based Update Meeting on Keratinocyte carcinomas).

http://creativecommons.org/licenses/by/4.0/

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

View Full Text