3
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Poor agreement between the automated risk assessment of a smartphone application for skin cancer detection and the rating by dermatologists

      research-article

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Background

          Several smartphone applications (app) with an automated risk assessment claim to be able to detect skin cancer at an early stage. Various studies that have evaluated these apps showed mainly poor performance. However, all studies were done in patients and lesions were mainly selected by a specialist.

          Objectives

          To investigate the performance of the automated risk assessment of an app by comparing its assessment to that of a dermatologist in lesions selected by the participants.

          Methods

          Participants of a National Skin Cancer Day were enrolled in a multicentre study. Skin lesions indicated by the participants were analysed by the automated risk assessment of the app prior to blinded rating by the dermatologist. The ratings of the automated risk assessment were compared to the assessment and diagnosis of the dermatologist. Due to the setting of the Skin Cancer Day, lesions were not verified by histopathology.

          Results

          We included 125 participants (199 lesions). The app was not able to analyse 90 cases (45%) of which nine BCC, four atypical naevi and one lentigo maligna. Thirty lesions (67%) with a high and 21 with a medium risk (70%) rating by the app were diagnosed as benign naevi or seborrhoeic keratoses. The interobserver agreement between the ratings of the automated risk assessment and the dermatologist was poor (weighted kappa = 0.02; 95% CI −0.08‐0.12; P = 0.74).

          Conclusions

          The rating of the automated risk assessment was poor. Further investigations about the diagnostic accuracy in real‐life situations are needed to provide consumers with reliable information about this healthcare application.

          Related collections

          Most cited references13

          • Record: found
          • Abstract: found
          • Article: not found

          Diagnostic inaccuracy of smartphone applications for melanoma detection.

          To measure the performance of smartphone applications that evaluate photographs of skin lesions and provide the user with feedback about the likelihood of malignancy. Case-control diagnostic accuracy study. Academic dermatology department. PARTICIPANTS AND MATERIALS: Digital clinical images of pigmented cutaneous lesions (60 melanoma and 128 benign control lesions) with a histologic diagnosis rendered by a board-certified dermatopathologist, obtained before biopsy from patients undergoing lesion removal as a part of routine care. Sensitivity, specificity, and positive and negative predictive values of 4 smartphone applications designed to aid nonclinician users in determining whether their skin lesion is benign or malignant. Sensitivity of the 4 tested applications ranged from 6.8% to 98.1%; specificity, 30.4% to 93.7%; positive predictive value, 33.3% to 42.1%; and negative predictive value, 65.4% to 97.0%. The highest sensitivity for melanoma diagnosis was observed for an application that sends the image directly to a board-certified dermatologist for analysis; the lowest, for applications that use automated algorithms to analyze images. The performance of smartphone applications in assessing melanoma risk is highly variable, and 3 of 4 smartphone applications incorrectly classified 30% or more of melanomas as unconcerning. Reliance on these applications, which are not subject to regulatory oversight, in lieu of medical consultation can delay the diagnosis of melanoma and harm users.
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            Expert-Level Diagnosis of Nonpigmented Skin Cancer by Combined Convolutional Neural Networks

            Convolutional neural networks (CNNs) achieve expert-level accuracy in the diagnosis of pigmented melanocytic lesions. However, the most common types of skin cancer are nonpigmented and nonmelanocytic, and are more difficult to diagnose.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Teledermatology for diagnosing skin cancer in adults

              Early accurate detection of all skin cancer types is essential to guide appropriate management and to improve morbidity and survival. Melanoma and squamous cell carcinoma (SCC) are high‐risk skin cancers which have the potential to metastasise and ultimately lead to death, whereas basal cell carcinoma (BCC) is usually localised with potential to infiltrate and damage surrounding tissue. Anxiety around missing early curable cases needs to be balanced against inappropriate referral and unnecessary excision of benign lesions. Teledermatology provides a way for generalist clinicians to access the opinion of a specialist dermatologist for skin lesions that they consider to be suspicious without referring the patients through the normal referral pathway. Teledermatology consultations can be 'store‐and‐forward' with electronic digital images of a lesion sent to a dermatologist for review at a later time, or can be live and interactive consultations using videoconferencing to connect the patient, referrer and dermatologist in real time. To determine the diagnostic accuracy of teledermatology for the detection of any skin cancer (melanoma, BCC or cutaneous squamous cell carcinoma (cSCC)) in adults, and to compare its accuracy with that of in‐person diagnosis. We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials, MEDLINE, Embase, CINAHL, CPCI, Zetoc, Science Citation Index, US National Institutes of Health Ongoing Trials Register, NIHR Clinical Research Network Portfolio Database and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles. Studies evaluating skin cancer diagnosis for teledermatology alone, or in comparison with face‐to‐face diagnosis by a specialist clinician, compared with a reference standard of histological confirmation or clinical follow‐up and expert opinion. We also included studies evaluating the referral accuracy of teledermatology compared with a reference standard of face‐to‐face diagnosis by a specialist clinician. Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS‐2). We contacted authors of included studies where there were information related to the target condition of any skin cancer missing. Data permitting, we estimated summary sensitivities and specificities using the bivariate hierarchical model. Due to the scarcity of data, we undertook no covariate investigations for this review. For illustrative purposes, we plotted estimates of sensitivity and specificity on coupled forest plots for diagnostic threshold and target condition under consideration. The review included 22 studies reporting diagnostic accuracy data for 4057 lesions and 879 malignant cases (16 studies) and referral accuracy data for reported data for 1449 lesions and 270 'positive' cases as determined by the reference standard face‐to‐face decision (six studies). Methodological quality was variable with poor reporting hindering assessment. The overall risk of bias was high or unclear for participant selection, reference standard, and participant flow and timing in at least half of all studies; the majority were at low risk of bias for the index test. The applicability of study findings were of high or unclear concern for most studies in all domains assessed due to the recruitment of participants from secondary care settings or specialist clinics rather than from primary or community‐based settings in which teledermatology is more likely to be used and due to the acquisition of lesion images by dermatologists or in specialist imaging units rather than by primary care clinicians. Seven studies provided data for the primary target condition of any skin cancer (1588 lesions and 638 malignancies). For the correct diagnosis of lesions as malignant using photographic images, summary sensitivity was 94.9% (95% confidence interval (CI) 90.1% to 97.4%) and summary specificity was 84.3% (95% CI 48.5% to 96.8%) (from four studies). Individual study estimates using dermoscopic images or a combination of photographic and dermoscopic images generally suggested similarly high sensitivities with highly variable specificities. Limited comparative data suggested similar diagnostic accuracy between teledermatology assessment and in‐person diagnosis by a dermatologist; however, data were too scarce to draw firm conclusions. For the detection of invasive melanoma or atypical intraepidermal melanocytic variants both sensitivities and specificities were more variable. Sensitivities ranged from 59% (95% CI 42% to 74%) to 100% (95% CI 48% to 100%) and specificities from 30% (95% CI 22% to 40%) to 100% (95% CI 93% to 100%), with reported diagnostic thresholds including the correct diagnosis of melanoma, classification of lesions as 'atypical' or 'typical, and the decision to refer or to excise a lesion. Referral accuracy data comparing teledermatology against a face‐to‐face reference standard suggested good agreement for lesions considered to require some positive action by face‐to‐face assessment (sensitivities of over 90%). For lesions considered of less concern when assessed face‐to‐face (e.g. for lesions not recommended for excision or referral), agreement was more variable with teledermatology specificities ranging from 57% (95% CI 39% to 73%) to 100% (95% CI 86% to 100%), suggesting that remote assessment is more likely recommend excision, referral or follow‐up compared to in‐person decisions. Studies were generally small and heterogeneous and methodological quality was difficult to judge due to poor reporting. Bearing in mind concerns regarding the applicability of study participants and of lesion image acquisition in specialist settings, our results suggest that teledermatology can correctly identify the majority of malignant lesions. Using a more widely defined threshold to identify 'possibly' malignant cases or lesions that should be considered for excision is likely to appropriately triage those lesions requiring face‐to‐face assessment by a specialist. Despite the increasing use of teledermatology on an international level, the evidence base to support its ability to accurately diagnose lesions and to triage lesions from primary to secondary care is lacking and further prospective and pragmatic evaluation is needed. What is the diagnostic accuracy of teledermatology for the diagnosis of skin cancer in adults? Why is improving the diagnosis of skin cancer important? There are different types of skin cancer. Melanoma is one of the most dangerous forms and it is important to identify it early so that it can be removed. If it is not recognised when first brought to the attention of doctors (also known as a false‐negative test result) treatment can be delayed resulting in the melanoma spreading to other organs in the body and possibly causing early death. Cutaneous squamous cell carcinoma (cSCC) and basal cell carcinoma (BCC) are usually localised skin cancers, although cSCC can spread to other parts of the body and BCC can cause disfigurement if not recognised early. Calling something a skin cancer when it is not really a skin cancer (a false‐positive result) may result in unnecessary surgery and other investigations that can cause stress and worry to the patient. Making the correct diagnosis is important. Mistaking one skin cancer for another can lead to the wrong treatment being used or lead to a delay in effective treatment. What is the aim of the review? The aim of this Cochrane Review was to find out whether teledermatology is accurate enough to identify which people with skin lesions need to be referred to see a specialist dermatologist (a doctor concerned with disease of the skin) and who can be safely reassured that their lesion (damage or change of the skin) is not malignant. We included 22 studies to answer this question. What was studied in the review? Teledermatology means sending pictures of skin lesions or rashes to a specialist for advice on diagnosis or management. It is a way for primary care doctors (general practitioners (GPs)) to get an opinion from a specialist dermatologist without having to refer patients through the normal referral pathway. Teledermatology can involve sending photographs or magnified images of a skin lesion taken with a special camera (dermatoscope) to a skin specialist to look at or it might involve immediate discussion about a skin lesion between a GP and a skin specialist using videoconferencing. What are the main results of the review? The review included 22 studies, 16 studies comparing teledermatology diagnoses to the final lesion diagnoses (diagnostic accuracy) for 4057 lesions and 879 malignant cases and five studies comparing teledermatology decisions to the decisions that would be made with the patient present (referral accuracy) for 1449 lesions and 270 'positive' cases. The studies were very different from each other in terms of the types of people with suspicious skin cancer lesions included and the type of teledermatology used. A single reliable estimate of the accuracy of teledermatology could not be made. For the correct diagnosis of a lesion to be a skin cancer, data suggested that less than 7% of malignant skin lesions were missed by teledermatology. Study results were too variable to tell us how many people would be referred unnecessarily for a specialist dermatology appointment following a teledermatology consultation. Without access to teledermatology services however, most of the lesions included in these studies would likely be referred to a dermatologist. How reliable are the results of the studies of this review? In the included studies, the final diagnosis of skin cancer was made by lesion biopsy (taking a small sample of the lesion so it could be examined under a microscope) and the absence of skin cancer was confirmed by biopsy or by follow‐up over time to make sure the skin lesion remained negative for melanoma. This is likely to have been a reliable method for deciding whether people really had skin cancer. In a few studies, a diagnosis of no skin cancer was made by a skin specialist rather than biopsy. This is less likely to have been a reliable method for deciding whether people really had skin cancer*. Poor reporting of what was done in the study made it difficult for us to say how reliable the study results are. Selecting some patients from specialist clinics instead of primary care along with different ways of doing teledermatology were common problems. Who do the results of this review apply to? Studies were conducted in: Europe (64%), North America (18%), South America (9%) or Oceania (9%). The average age of people who were studied was 52 years; however, several studies included at least some people under the age of 16 years. The percentage of people with skin cancer ranged between 2% and 88% with an average of 30%, which is much higher than would be observed in a primary care setting in the UK. What are the implications of this review? Teledermatology is likely to be a good way of helping GPs to decide which skin lesions need to be seen by a skin specialist. Our review suggests that using magnified images, in addition to photographs of the lesion, improves accuracy. More research is needed to establish the best way of providing teledermatology services. How up‐to‐date is this review? The review authors searched for and used studies published up to August 2016. *In these studies, biopsy, clinical follow‐up or specialist clinician diagnosis were the reference comparisons.
                Bookmark

                Author and article information

                Contributors
                n.a.kukutsch@lumc.nl
                Journal
                J Eur Acad Dermatol Venereol
                J Eur Acad Dermatol Venereol
                10.1111/(ISSN)1468-3083
                JDV
                Journal of the European Academy of Dermatology and Venereology
                John Wiley and Sons Inc. (Hoboken )
                0926-9959
                1468-3083
                12 September 2019
                February 2020
                : 34
                : 2 ( doiID: 10.1111/jdv.v34.2 )
                : 274-278
                Affiliations
                [ 1 ] Dutch Society of Dermatology and Venereology Utrecht The Netherlands
                [ 2 ] Dermapark Uden The Netherlands
                [ 3 ] Department of Dermatology Academic Medical Center and Vrije University Medical Center Amsterdam The Netherlands
                [ 4 ] Department of Dermatology Erasmus Medical Center Rotterdam The Netherlands
                [ 5 ] Department of Dermatology Maastricht University Medical Center Maastricht The Netherlands
                [ 6 ] Department of Dermatology Leiden University Medical Center Leiden The Netherlands
                Author notes
                [*] [* ]Correspondence:N. Kukutsch. E‐mail: n.a.kukutsch@ 123456lumc.nl
                [†]

                These authors share senior authorship

                Article
                JDV15873
                10.1111/jdv.15873
                7027514
                31423673
                8605daae-141a-4736-9123-30433e6867ee
                © 2019 The Authors. Journal of the European Academy of Dermatology and Venereology published by John Wiley & Sons Ltd on behalf of European Academy of Dermatology and Venereology

                This is an open access article under the terms of the http://creativecommons.org/licenses/by-nc-nd/4.0/ License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non‐commercial and no modifications or adaptations are made.

                History
                : 27 February 2019
                : 19 July 2019
                Page count
                Figures: 0, Tables: 3, Pages: 5, Words: 3967
                Categories
                Original Article
                Original Articles and Short Reports Oncology
                Custom metadata
                2.0
                February 2020
                Converter:WILEY_ML3GV2_TO_JATSPMC version:5.7.5 mode:remove_FC converted:18.02.2020

                Dermatology
                Dermatology

                Comments

                Comment on this article