Smartphone Apps Not Reliable for Diagnosing Melanoma

Roxanne Nelson

January 16, 2013

The accuracy of smartphone applications (apps) in the evaluation of melanoma risk appears to be highly variable. Three of 4 applications evaluated in a new study incorrectly classified at least 30% of melanomas as "unconcerning."

The study was published online today in JAMA Dermatology.

There has been a large increase in the use of smartphone apps by both healthcare professionals and the general public. The growing number of healthcare-related apps include those designed to track calories and weight loss, and those used to educate patients about the adverse effects of medication.

The researchers tested 4 smartphone apps marketed for analyzing skin lesions. The user uploads a photograph of a cutaneous lesion, and the app determines whether it is a potential melanoma or a cause for concern.

The sensitivity of the apps ranged from 6.8% to 98.1%. There was also a large variance in specificity, which ranged from 30.4% to 93.7%. Positive predictive values ranged from 33.3% to 42.1%, and negative predictive values ranged from 65.4% to 97.0%.

Not surprisingly, the highest sensitivity for melanoma diagnosis was in an app that sent the images directly to a board-certified dermatologist to be analyzed. The lowest sensitivity was in apps that used automated algorithms to analyze images.

Currently No Regulation

The researchers note that some of the apps they evaluated are of particular concern, especially because 3 of 4 did not involve a physician at any point in the evaluation process. In addition, the fact that some apps go beyond helping patients catalog and track lesions, and give an assessment of the risk or probability that a lesion is benign or malignant, might lead people to substitute a smartphone evaluation for a medical consultation.

Physicians need to be aware that these are out there.

"Physicians need to be aware that these are out there," said researcher Laura K. Ferris, MD, PhD, from the Department of Dermatology at the University of Pittsburgh in Pennsylvania.

"Right now, the FDA does not have oversight of smartphone apps, although they are starting to get greater awareness," Dr. Ferris told Medscape Medical News. "They are planning to regulate smartphone applications that pair with medical devices that are already regulated by the FDA. Congress also approved the FDA Safety and Innovation Act, and the first step for that is to look at what's out there and see where some kind of regulation is needed," she explained.

It is unclear what type of regulation is needed at this point, she noted. "Is it needed in terms of a very formal warning on any application, or is it that they need to meet some basic degree of sensitivity to know how many melanomas they are going to miss?"

She pointed out that although the apps state that they are not meant to replace medical care, none of them really give a more direct warning, such as "the implication for delaying treatment for melanoma is that it becomes more deadly," she said.

Dr. Ferris noted that physicians need to adapt to the times. "Ten years ago we never would have thought about smartphone apps, and 10 years from now our practices are going to be a lot different," she said. "There is a place for technology, but applications that give false reassurance are a concern."

Study Details

Dr. Ferris and her colleagues used clinical images of 188 lesions; 60 were melanoma (44 invasive and 16 in situ) and the remainder were benign.

They evaluated the sensitivity, specificity, and positive and negative predictive values of 4 smartphone apps designed to aid the general public in deciding whether or not a skin lesion is benign or malignant.

App 1 used an automated algorithm to detect the border of the lesion, but allowed for manual input to confirm or change the detected border. It provided 1 of 3 assessments: "problematic," which was considered to be a positive result; "okay," which was considered to be a negative result; or "error," if the image could not be assessed by the application.

App 2 used an automated algorithm to evaluate the uploaded image. It provided 1 of 2 assessments: "melanoma," which was considered to be a positive result; or "looks good," which was considered to be a negative result.

App 3 asked the user to position the uploaded image in a box to ensure that the correct lesion was analyzed. It provided 1 of 3 assessments: "high risk," which was considered to be a positive result; and "medium risk" and "low risk," which were considered to be negative results.

App 4 can be run on a smartphone or from a Web site. It differed from the others in that it did not use an automated analysis algorithm to evaluate images. Instead, each image was sent to a board-certified dermatologist for evaluation; an assessment was returned to the user within 24 hours.

Of the 3 apps that did not involve a physician, even the "best-performing" classified 18 of 60 melanomas (30%) as benign.

The study was supported by grants from the National Institutes of Health. Dr. Ferris reports having serving as an investigator and consultant for MELA Sciences, Inc.

JAMA Dermatol. Published online January 16, 2013. Abstract