1. Introduction
Melanoma is the most deadly form of skin cancer and accounts for about 75% of deaths associated with skin cancer [
1]. Accurate recognition of melanoma in early stage can significantly increase the survival rate of patients. However, the manual detection of melanoma produces huge demand of well-trained specialists, and suffers from inter-observer variations. A reliable automatic system for melanoma recognition, increasing the accuracy and efficiency of pathologists, is worthwhile to develop.
The dermoscopy technique has been developed to improve the diagnostic performance of melanoma. Dermoscopy is a noninvasive skin imaging technique of acquiring a magnified and illuminated image of skin region for increased clarity of the spots [
2], which enhances the visual effect of skin lesion by removing surface reflection. Nevertheless, automatic recognition of melanoma from dermoscopy images is still a difficult task, as it has several challenges. First, the low contrast between skin lesions and normal skin region makes it difficult to segment accurate lesion areas. Second, the melanoma and non-melanoma lesions may have high degree of visual similarity, resulting in the difficulty for distinguishing melanoma lesion from non-melanoma. Third, the variation of skin conditions, e.g., skin color, natural hairs or veins, among patients produce different appearance of melanoma, in terms of color and texture, etc.
Skin lesion segmentation is the essential step for most classification approaches. Recent review of automated skin lesion segmentation algorithms can be found in [
3]. Accurate segmentation can benefit the accuracy of subsequent lesion classification. Extensive studies [
4,
5,
6,
7,
8,
9,
10,
11,
12] have been made to produce decent lesion segmentation results. For example, Gomez et al. proposed an unsupervised algorithm, named Independent Histogram Pursuit (IHP), for the segmentation of skin lesion [
13]. The algorithm was tested on five different dermatological datasets, and achieved a competitive accuracy close to 97%. Zhou developed several mean-shift-based approaches for segmenting skin lesions in dermoscopic images [
14,
15,
16]. Garnavi et al. proposed an automated segmentation approach for skin lesion using optimal color channels and hybrid thresholding technique [
17]. In more recent research, Pennisi et al. employed Delaunay Triangulation to extract binary masks of skin lesion regions, which does not require any training stage [
18]. Ma proposed a novel deformable model using a newly defined speed function and stopping criterion for skin lesion segmentation, which is robust against noise and yields effective and flexible segmentation performance [
19]. Yu used a deep learning approach, i.e., a fully convolutional residual network (FCRN), for skin lesion segmentation in dermoscopy images [
20].
Based on the segmentation results, hand-crafted features can be extracted for melanoma recognition. Celebi et al. extracted several features, including color and texture from segmented lesion region for skin lesion classification [
21]. Schaefer used an automatic border detection approach [
22] to segment the lesion area and then assembled the extracted features, i.e., shape, texture and color, for melanoma recognition [
23]. On the other hand, some investigations [
24] have attempted to directly employ hand-crafted features for melanoma recognition without a segmentation step. Different from approaches using hand-crafted features, deep learning networks use hierarchical structures to automatically extract features. Due to the breakthroughs made by deep learning in an increasing number of image-processing tasks [
25,
26,
27,
28], some research has started to apply deep learning approaches for melanoma recognition. Codella et al. proposed a hybrid approach, integrating convolutional neural network (CNN), sparse coding and support vector machines (SVMs) to detect melanoma [
29]. In recent research, Codella and his colleagues established a system combining recent developments in deep learning and machine learning approaches for skin lesion segmentation and classification [
30]. Kawahara et al. employed a fully convolutional network to extract multi-scale features for melanoma recognition [
31]. Yu et al. applied a very deep residual network to distinguish melanoma from non-melanoma lesions [
20].
Although lots of work has been proposed, there is still a margin of performance improvement for both skin lesion segmentation and classification. The International Skin Imaging Collaboration (ISIC) is a cooperation focusing on the automatic analysis of skin lesion, and has continuously expanded its datasets since 2016. In ISIC 2017, annotated datasets for three processing tasks related to skin lesion images, including lesion segmentation, dermoscopic feature extraction and lesion classification, were released for researchers to promote the accuracy of automatic melanoma detection methods. Different from the extensively studied lesion segmentation and classification, dermoscopic feature extraction is a new task in the area. Consequently, few studies have been proposed to address the problem.
In this paper, we proposed deep learning frameworks to address the three main processing tasks of skin lesion images proposed in ISIC 2017. The main contribution of this paper can be summarized as follows:
- (1)
Existing deep learning approaches commonly use two networks to separately perform lesion segmentation and classification. In this paper, we proposed a framework consisting of multi-scale fully-convolutional residual networks and a lesion index calculation unit (LICU) to simultaneously address lesion segmentation (task 1) and lesion classification (task 3). The proposed framework achieved excellent results in both tasks. Henceforth, the proposed framework is named as Lesion Indexing Network (LIN).
- (2)
We proposed a CNN-based framework, named Lesion Feature Network (LFN), to address task 2, i.e., dermoscopic feature extraction. Experimental results demonstrate the competitive performance of our framework. To the best of our knowledge, we are not aware of any previous work proposed for this task. Hence, this work may become the benchmark for the following related research in the area.
- (3)
We made detailed analysis of the proposed deep learning frameworks in several respects, e.g., the performances of networks with different depths; and the impact caused by adding different components (e.g., batch normalization, weighted softmax, etc.). This work provides useful guidelines for the design of deep learning networks in related medical research.
5. Conclusions
In this paper, we proposed two deep learning frameworks, i.e., the Lesion Indexing Network (LIN) and the Lesion Feature Network (LFN), to address three primary challenges of skin lesion image processing, i.e., lesion segmentation, dermoscopic feature extraction and lesion classification.
The Lesion Indexing Network was proposed to simultaneously address lesion segmentation and classification. Two very deep fully convolutional residual networks, i.e., FCRN-88, trained with different training sets, are adopted to produce the segmentation result and coarse classification result. A lesion indexing calculation unit (LICU) is proposed to measure the importance of a pixel for the decision of lesion classification. The coarse classification result is refined according to the distance map generated by LICU.
The Lesion Feature Network was proposed to address the task of dermoscopic feature extraction, and is a CNN-based framework trained by the patches extracted from the dermoscopic images. To the best of our knowledge, we are not aware of any previous work available for this task. Hence, this work may become a benchmark for subsequent related research.
Our deep learning frameworks have been evaluated on the ISIC 2017 dataset. The JA and AUC of LIN for lesion segmentation and classification are 0.753 and 0.912, which outperforms the existing deep learning frameworks. The proposed LFN achieves the best average precision and sensitivity, i.e., 0.422 and 0.693, for dermoscopic feature extraction, which demonstrates its excellent capacity for addressing the challenge.