Multi-Class CNN for Classification of Multispectral and Autofluorescence Skin Lesion Clinical Images

Round 1

Reviewer 1 Report

An interesting experiment using modern techniques, where the authors also discuss the limitations of the method.
One doubt: The methodology mentions the names of people responsible for the data collection, not all of them are the authors of the work? Are their names necessary here, in my opinion it should sufficient to state their competences / specializations and not their surnames.

Author Response

Response to Reviewer 1 Comments.

 

Point 1: English language and style are fine/minor spell check required. 

Response 1: The English language and style was revised and minor changes were made to the manuscript.

Point 2: The methodology mentions the names of people responsible for the data collection, not all of them are the authors of the work? Are their names necessary here, in my opinion it should sufficient to state their competences / specializations and not their surnames.

Response 2: Thank you for the comment. The English language was revised and minor changes were made to the main text. The sentence that mentioned the names of the data collectors in subsection “2.2. Description of multispectral data” was rewritten: 

“Multispectral data were collected from individuals with Caucasian skin type at the Oncology Center of Latvia (Riga, Latvia) and at Semmelweis University (Budapest, Hungary) under the supervision of medical physicists, dermatologists and dermato-oncologists.”

Reviewer 2 Report

Computer-based analyses such as Convolutional Neural Networks (CNN) using multimodal spectral imaging is a new method that is different from RGB CNN, which has been widely used and rarely reported. This manuscript is a valuable clinical research paper that examines the usefulness of deep learning for the classification of multispectral reflectance (MSR) and autofluorescence (AF), and is considered worthy of further promotion.

Author Response

Response to Reviewer 2 Comments.

Thank you for the supportive comment. The English language was revised and minor changes were made to the manuscript.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

This paper shows that how Differentiable Architecture Search can be used to find a good model for the application, and a simple network works better than InceptionV3.

 

(1) "A total of 17124 (1427 original and 15697 augmented) multispectral datasets of skin 212 lesions were used."

The dataset is not small in number, but to cover all kinds of Lesions in Table 1, more data are needed.  Are there open-source multispectral datasets that can be used?

 

(2) What software package is used for Differentiable Architecture Search? How difficult is it to do the search ?  What is the time cost?

 

(3) In Figure 1, I see round objects (two circles) on the skin. What were they used for ?

 

(4) to get saliency maps, a new method GradCam could be used

https://arxiv.org/abs/1610.02391

implementation is here

https://github.com/utkuozbulak/pytorch-cnn-visualizations

 

(5) Area Under the Receiver Operating Characteristics (ROC-AUC)

The abbreviation is AUROC

 

 

 

 

 

 

 

 

 

 

Author Response

Thanks to the reviewer for the comments, they definitely helped us to improve the quality of the article! Here are our answers. Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Dear Authors,

This is a very important and interesting research. The manuscript was written well, however, there are some issues that need to address before it may consider for publication. Please find my concerns and suggestions below:

1. Line 28-34, 39-40 needs valid references.
2. Where are the CNN architecture details used in this research? As this research is based on CNN and they have claimed that their network outperformed "InceptionV3", authors should consider a detailed architectural discussion and illustration. I understand that there is a small indication of the architecture in line 264-267, but, that is not enough to understand.
3. What is the main contribution/novelty of this research? This needs to clearly state.
4. You need to validate your claim why your CNN outperform. What is the logic behind your CNN architecture (layers)?
5. Also, why the authors have chosen to compare their performance only with "InceptionV3"? There are many other networks available I believe. And, most importantly, why they compare only with a network rather than a full process? For example, only CNN is just architecture, but, the full proposed method is considered a full proposed study.
6. You need to compare your results with similar proposed methods, not just with another CNN model.

I believe after addressing those important issues, this manuscript will be serving its purpose to the readers. Good luck!

Author Response

Thanks to the reviewer for the comments, they definitely helped us to improve the quality of the article! Here are our answers. Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Authors propose essentially a CNN to classify AF skin lesions imaging with few samples to train.

In the reviewer’s opinion the paper has the following weaknesses:

• The introduction is a mixture of many concepts. Authors introduce there the melanoma detection problem, the state-of-the-art (it could be another section), some concerns about using certain measures to assess the methods (ROC-AUC), etc. I would recommend using the introduction to state the problem, show what the literature is missing, and which is the author's contribution to fill the gap between what has been done and the actual needs.
• This reviewer does not understand why a simple 5-layer CNN can improve Inception model with pre-trained data. If the multispectral nature of the data is a problem with respect to the rich nature of visual data, why authors do not use the same training approach in both cases (no pretrain on the Inception case)? Or pretrain both networks using the same data, and fine-tune them on the particular problem they face.
• My main concern is regarding to the paper novelty. Authors use a well-known approach (DARTS) to set the parameters of the CNN and test it with a particular problem (skin lesion). If the contribution is not the method but the application of CNN to this data, then the experimental validation should improve in order to make the paper useful for the community. Particularly, I recommend: (i) to use other CNNs in the comparative (ResNET, VGG, …), if the goal is to validate that this particular architecture is the one that must be adopted by the community for this sort of data, (ii) to use other public benchmarks that show the accuracies on public data (if there are similar sets available), that would show that the proposed method is consistently better than the state-of-the-art in deep learning for this particular tasks (beyond the authors’ own database), (iii) to use specific architectures that deal with small sample size problem and multispectral data in the comparative.

 

In summary, if the paper’s contribution is to show to the community that skin lesion segmentation is better approached by this particular CNN, this needs to be experimentally validated in the general case (other data sets) and with respect to the current state-of-the-art in vision. A CNN with this few number of layers may not be the most appropriate to deal with the problem’s complexity.

Author Response

Thanks to the reviewer for the comments, they definitely helped us to improve the quality of the article! Here are our answers. Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

I thank the authors for answering my questions.

Three additional comments:

(1) In Fig.1, "produced by a correctly trained network (b) and network with noticeable overfitting effect (d).". Please provide detailed information about the correctly trained network and the overfitting network.

(2) The advantage of using multispectral data over RGB images should be discussed, e.g., high accuracy, etc

(3) will the imaging parameters affect the image quality, which then affects the accuracy?

 

 

 

 

 

 

Author Response

Thanks to the reviewer for the comments, they helped us to improve the quality of the article! Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

After the revision based on the reviewers’ comments, the quality of the manuscript has been greatly improved. I would like to recommend the manuscript for publication.

Author Response

Thank you very much!

 

Reviewer 3 Report

The authors’ response solved the following issues I pointed out:

• Reorganization of the introduction and motivation
• The motivation of what is missing and the contribution: essentially the first time a CNN is applied to this particular problem.

 

Nevertheless, in my opinion, the experimental validation for a Q1 journal needs to be very robust. 

 

• A 5-layer CNN seems to improve Inception and VGG (one of the most used architectures in vision). Authors use pre-trained models (as I understand from the sentence: “that InceptionV3 was trained on feature rich RGB images (photos of the surrounding world), and trained filters were not applicable to our dataset”). But do they fine-tune also the filters to adapt to the particular use of their images? Using filters trained on natural images may be the main cause of model failure. The training protocol should be the same in all the cases, use pre-trained filters (instead of random ones, which is a common practice in vision), and fine-tune in all cases to adjust the model to the data.
• The novelty is limited, both DARTS and CNNs are commonly used, and the novelty here is just the application of them to skin lesion classification. CNNs have repeatedly applied to skin lesion classification. To make the contribution useful for the community, either the experimental validation is vast and complete, or if the method is the key contribution, it must be tested with other equivalent public DBs, proving that DARTS methodology is consistently strong when using public data (if there are similar sets available), that would show that the proposed method is consistently better than the state-of-the-art in deep learning for this particular tasks (beyond the authors’ own database).

Author Response

Thanks to the reviewer for the comments, they helped us to improve the quality of the article! Please see the attachment.

Author Response File: Author Response.pdf