Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studiesBMJ 2020; 368 doi: https://doi.org/10.1136/bmj.m127 (Published 10 February 2020) Cite this as: BMJ 2020;368:m127
All rapid responses
SkinVision response to: Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies
We are writing regarding your recently published research on SkinVision. We are proud that the authors have taken the time to research how technology can positively impact the quality of skin cancer care. At SkinVision, we take the health of our users seriously. Therefore, research has always been a cornerstone of our business. The latest study, not included in the authors’ overall assessment, proves that SkinVision can detect 95% of skin cancer cases (https://onlinelibrary.wiley.com/doi/10.1111/jdv.15935). The sensitivity of general practitioners ranges from 61% and 66%, while the sensitivity of dermatologists is between 75% and 92%.[1-6] SkinVision’s specificity is also on par with the standard of care and is reported in multiple published studies and on our website.
It’s estimated that one in five people may be diagnosed with skin cancer in their lifetime and that, when detected in an early stage, the vast majority of skin cancer cases are cured. Therefore, we aim to raise awareness of skin cancer at an individual level and to provide a regulated medical device to help users assess their risk and get to the doctor in time. We believe that for the benefit of public health, all algorithm-based smartphone Apps should be researched and would like to continue to lead by example, as we continue supporting further clinical research on our algorithm. SkinVision’s service is proven as safe, and its effectiveness continuously benefits National Healthcare Systems (e.g. NHS), hospitals (e.g. Erasmus Rotterdam) and insurers, by having a positive impact on both cancer and non-cancer cases.
SkinVision goes beyond legal and regulatory minimum requirements.
Specifically, as reported in the article, our device is currently determined by regulatory requirements as being Class I. However, SkinVision recognises that provisions for Class I may be supplemented in the interest of public health, and therefore goes beyond them, and voluntarily calls an independent Notified Body to certify its operations. In fact, the BMJ article fails to acknowledge that SkinVision decided to take safety seriously in all areas, for example by achieving voluntary certification to ISO 13485 (best practice quality systems for ensuring safety of medical devices), by achieving voluntary certification to ISO 27001 (best practice for information security systems), and by complying to ISO 14971 (risk management for medical devices).
Additionally, based on current regulations, SkinVision may have been able to market the device based on a smaller set of clinical studies than what we did; however, proving our commitment to safety, we (once again) do more than required, and we continuously invest on new studies, which investigate safety and performance of our device in clinical environments.
Furthermore, SkinVision would like to point out that the article does not consider key factors. Specifically:
1) The most recent study, which proves that SkinVision has a sensitivity of 95%, is not factored in the BMJ article. The study was published in 2019, so the argument of being too recent to be included does not hold ground for an up to date journal like the BMJ.
2) The BMJ article does not emphasise the fact that, even if no apps are used, the standard of care for skin cancer (e.g. doctors or other medical devices) has a large number of false negatives and missed cancers.
3) The use of SkinVision also facilitates access to healthcare globally and reduces unnecessary referrals to dermatologists. This helps healthcare professionals to concentrate on value-added activities.
4) The BMJ article does not consider that decisions on medical devices are not merely done on risk, but rather on risk associated with risk-benefit analysis, which is standardised by ISO 14971 - to which SkinVision complies.
5) The BMJ article compares large studies with positive outcomes with analysis (conducted in unclear circumstances) on 15 people, which allegedly has negative outcomes.
6) The BMJ article may give the impression that no third party certifies SkinVision, which is instead certified by an official EU Notified Body, through multiple on-site assessments and audits per year.
While there are risks associated with all medical devices, research also shows that the use of algorithm-based apps alongside healthcare professionals can provide a greater benefit for patients and the healthcare system. We have assisted in finding over 40,000 cases of skin cancer already. SkinVision believes that a health system with Apps and doctors together can reduce risk for the overall population, and SkinVision is focusing its future research on providing additional data on this.
These objective facts prove that the clinical benefits of the service outweigh the potential risks.
SkinVision welcomes research on its application. However, it is regrettable that article authors did not even contact SkinVision, which could have provided input to help their research to be more effective, and based on real-life data, rather than being a theoretical essay on potential risks that may already exist with current (App free) standard of care for skin cancer.
Detecting all cases of melanoma and skin cancer has never been the objective or the claim of SkinVision, which instead clearly advertises that its sensitivity is not 100%. No medical device is free of risk, and no standard of care can detect all cases of melanoma and skin cancer.
The article conclusions (‘Current algorithm based smartphones apps cannot be relied on to detect all cases of melanoma or other skin cancer. [...]”) have been written in a way that maybe (and has been) easily misinterpreted by general press in a way that press may believe that there is scientific evidence proving a negative impact on public health when using smartphone Apps, which is not the case.
Therefore, SkinVision finds the BMJ article and its conclusion as potentially: a) not giving a fair representation of facts; b) misleading and c) not in the interest of public health.
Considering the need for improving the efficiency and reducing the workload on global healthcare systems, SkinVision looks forward to further supporting population health and to achieving our mission of saving 250,000 lives in the next decade.
Whichever party would like to reach out to SkinVision for further information or statements, is welcome to do so by emailing firstname.lastname@example.org.
Erik de Heus, CEO
1. Ahmadi K, Prickaerts E, Smeets JGE et al. Current approach of skin lesions suspected of malignancy in general practice in the Netherlands: a quantitative overview. J Eur Acad Dermatol Venereol. 2018;32(2):236-241.
2. Beecher SM, Keogh C, Healy C. Dedicated general practitioner education sessions can improve diagnostic capabilities and may have a positive effect on referral patterns for common skin lesions. Ir J Med Sci. 2018;187(4):959-963.
3. Corbo MD, Vender R, Wismer J. Comparison of dermatologists’ and nondermatologists’ diagnostic accuracy for malignant melanoma. J Cutan Med Surg. 2012;16(4):272-80.
4. Kroemer S, Frühauf J, Campbell TM, et al. Mobile teledermatology for skin tumour screening: diagnostic accuracy of clinical and dermoscopic image tele-evaluation using cellular phones. Br J Dermatol. 2011;164(5):973-9.
5. Rosendahl C, Tschandl P, Cameron A et al. Diagnostic accuracy of dermatoscopy for melanocytic and nonmelanocytic pigmented lesions. J Am Acad Dermatol. 2011;64(6):1068-73.
6. Sinz C, Tschandl P, Rosendahl C Accuracy of dermatoscopy for the diagnosis of nonpigmented cancers of the skin. J Am Acad Dermatol. 2017;77(6):1100-1109.
Competing interests: No competing interests
With great interest we have read the systematic review of Freeman et al. Although on a weekly basis articles on artificial intelligence (AI) algorithms in dermatology are published, very limited evidence exists about the implementation and use of these tools in clinical practice and the lay population. The big challenge is not to refine the algorithms, but to implement them in health care.
Freeman et al. worry that ‘current algorithm based smartphone apps cannot be relied on to detect all cases of melanoma or other skin cancer’. There is no perfect diagnostic test that captures all skin cancers. Physicians are not perfect in diagnosing skin cancer either. The sensitivity of melanoma detection by primary care physicians ranges from 38% to 91% with a number needed to biopsy (NNB) of 28.[2-3] Dermatologists do better and demonstrated 80% to 95% sensitivity and a NNB of 14.[3-4] Pathologists have an estimated accuracy of 83% in classifying melanocytic lesions, which is as low as 40% for severe atypia and melanoma in situ. So if physicians are less than perfect, is it fair to expect AI to perform flawlessly? It is paradoxical that the Dutch and UK health care systems accepts general practitioners (GPs) with a modest diagnostic accuracy for melanocytic lesions to be the gatekeepers, whereas the much discussed apps need to be perfect. It seems that double standards apply when comparing our own abilities with those of machines.
Ideally, a diagnostic mHealth app is compared to the gold standard. However, the gold standard (i.e., standard of care) is a moving target depending on where the app in the patient journey is being used. When lay people use an app, its accuracy should be compared to GPs. When GPs implement the app it should be compared to dermatologists and if dermatologists would use it to support their decision making process, the gold standard is the pathologist. In addition to different gold standards, the optimal diagnostic accuracy also depends on the prevalence of skin cancer in the population that is being targeted.
It is a noble aim to pursue 100% sensitivity because missing potentially lethal cancers has to be avoided, but it comes with a cost. To push for increasing sensitivity above a certain level, the specificity decreases which leads to more false positives. We feel that a sensitivity of 95% or more is acceptable for a diagnostic test to be incorporated in clinical care as long as the specificity is also acceptable (e.g., >80%). Perfection must not stand in the way of the good.
The systematic review of Freeman et al. shows that there is a need for high quality studies, as is the conclusion of most systematic reviews. The results should encourage us to perform better designed studies as recently described so that we avoid jumping to conclusions. High quality studies on AI in other medical specialties can serve as an example. Technological innovation, including mHealth, deserves a fair assessment of its qualities.
1 Freeman, K., et al., Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. Bmj, 2020. 368: p. m127.
2 Jones, O.T., et al., Dermoscopy for melanoma detection and triage in primary care: a systematic review. BMJ open, 2019. 9(8): p. e027529-e027529.
3 Privalle, A., et al., Number of skin biopsies needed per malignancy: Comparing the use of skin biopsies among dermatologists and nondermatologist clinicians. Journal of the American Academy of Dermatology, 2020. 82(1): p. 110-116.
4 Vestergaard, M.E., et al., Dermoscopy compared with naked eye examination for the diagnosis of primary melanoma: a meta-analysis of studies performed in a clinical setting. British Journal of Dermatology, 2008. 159(3): p. 669-676.
5 Elmore, J.G., et al., Pathologists’ diagnosis of invasive melanoma and melanocytic proiferations: observer accuracy and reproducibility study. Bmj, 2017. 357: p. j2813.
6 Angus DC. Randomized Clinical Trials of Artificial Intelligence. JAMA. Published online February 17, 2020. doi:10.1001/jama.2020.1039
Competing interests: TS, MW and TN received an unrestricted research grant from SkinVision.
Re: Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies
My name is Zeljko Ratkaj, and I am CEO of TeleSkin ApS, the company that developed the mobile application skinScan.
While I totally support the paper finding and conclusions that you have based on thorough analysis, I must notice one thing. It was mentioned in the paper the following:
“SkinScan was evaluated in a single study of 15 moles with five melanomas. The app did not identify any of the melanomas.”
Based on the following reference:
Chadwick X, Loescher LJ, Janda M, Soyer HP. Mobile medical applications for melanoma risk assessment: false assurance or valuable tool? 47th Hawaii International Conference on System Sciences; 6-9 January, 2014;2675-84.
Application skinScan in the form it is known was mostly developed during the year 2014 and 2015, and it was released in Denmark (only in Denmark) in August 2015. In 2016, we released in Norway and other Scandinavian countries, and in NZ, Australia and UK we released the app in 2017 and 2018. We had this huge window of releases because, even though we had Class I Medical CE certification, we wanted to be compliant with the medical rules in the countries where we released the app. That is why we, for example, still have not released the app in Germany, because being compliant with German medical rules is rather complex and for us, as a small company, very hard to achieve.
That is also why is rather impossible that our app was the study subject in the paper that was being referenced to. This paper, since it was released officially in January 2014 (in Australia) was probably conducted during the year 2012/2013, and at that time, our app was just an idea because we were still in the process of app analysis and applications for our internal use, primarily as a follow up tool for doctors and patients in Serbia.
I was one of the authors that published the paper:
“Melanoma screening with skinScan, Windows mobile phone application, 8th World Congress of Melanoma, July 17 – 20, 2013, Hamburg, Germany, Congress Center Hamburg (CCH)”
and we had a pilot app for internal use (not published) where we discussed the possibility of using this kind of app for screening purposes. We conducted the study internally on the Dr. Jadran Bandic (one of the IDS board members at that time) private clinic “ORS Hospital”, Belgrade, Serbia. And we still believe that the strongest side for this kind of app is in the medical screening area as a telemedicine and follow up tool, but there are still lot of obstacles for this idea to come to life.
In 2013, we also received Microsoft Health Users Group 2013 Innovation Awards, Innovation in Flexible Mobile Workstyle Solutions, for this solution that was an image gathering tool with ABCDE questionnaire without any kind of image analysis, coupled with an “Ask our Doctor” service, where the users were sending the cases to certified doctors. Again, in that period of time we were still primarily following the idea of a screening tool.
Because of all of the mentioned above, it is really impossible that our app was the subject of the study in Australia, since they mention the SkinScan app developed for iOS platform and our iOS version of the app was released in 2015.
We had a similar situation with the paper:
Ferrero et al. ” Skin scan: A demonstration of the need for FDA regulation of medical apps on iPhone. JAAD (2012)68,3,515
And we had several unpleasant situations where we needed to explain that we are not the company and the app that was mentioned and that everyone can check us and the company on the internet. It is just that the “skinScan” name is really catchy and lot of companies were (and some still are) using the name without right to use it.
Zeljko Ratkaj, CEO
Competing interests: No competing interests
We have been made aware that the app named “SkinScan” in the Chadwick et al 2014 study is not related to the current “skinScan” app produced by TeleSkin (https://teleskin.org/) as we infer in the paper. It appears that there was more than one app using the name SkinScan at the point in time when the study was done. Chadwick et al indicate that the version of the “SkinScan” app they evaluated later became SkinVision. Thus we now believe that the data we described as relating to SkinScan would better be described as being for the an early version of the SkinVision app. We apologise to Teleskin for this error.
The review currently contains no data on the accuracy of the any version of the SkinScan app from TeleSkin. We have been made aware of an unpublished study of the TeleSkin SkinScan app which we will endeavour to add to the review. The TeleSkin SkinScan app is CE marked and is available for download as described in Table 1 of the paper.
Competing interests: No competing interests