Chain Graph Explanation of Neural Network Based on Feature-Level Class Confusion

Round 1

Reviewer 1 Report

the paper is well written. They respect also the rules of open sciences. The conclusions are sound and safe. The research idea is interesting and also they do a correct SWAT over the proposed method. The work is at the beginning but it is promising.

Author Response

Thank you for your general opinion on our paper.  

Reviewer 2 Report

The paper is about a very hot topic in CNN: how to transform a good performance in something understandable by a human.

The paper is well written and very interesting with a lot of experiments. But authors should clarify these issues:

• CNN are very time and memory consuming. Authors must provide some time calculations in order to realize if his approach is affordable or not (how long to obtain the confusion graphs) and if the new models speed the classification process in anyway.
• It is a pity that authors use Imagenet database only. There are plenty of more interesting applications where CNN is applied but people are reluctant to use it. I am thinking in Medical Image community; it is very difficult that a diagnosis based on CNN is accepted by a real physician.
• The pseudo code is hard to read. Try to extend the explanation in the text or make some connections with the pseudo code in the text to make it clear.
• What happens if the CNN architecture has some feedback? how can affect the graph? There many ideas to explore about how to define the connections in the graph.

 

Author Response

Please refer attached file for our response.

Author Response File: Author Response.pdf

Reviewer 3 Report

This paper tackles the explainability of CNN by identifying a chain graph that plays a vital role in decision changes of the model. The proposed approach first suppresses a channel on a layer by turning its value to zero and then examines its impact on classification changes or confusion. The approach classifies classification confusion into violation confusion and correction confusion. The key channels are selected if they cause common confusion for each layer. A neighbor matrix is constructed for neighboring channels. Edges are extracted according to the highest weight in the neighbor matrix, indicting the highest similarity between nodes. 

Studying explainability is essential for people to better understand CNN's performance and mechanism. This topic would be of interest to the audience of Applied Sciences. The proposed method is relatively easy to understand, and its general idea is sound. The experimental results show that the proposed method can identify critical channels more than random selections. 

My main concern with this paper is its novelty and the computational complexity of the proposed approach:

Generally, it is not a new idea to greedily deactivate or activate individual neurons and examine their impacts on CNN's classification results. Such approaches have been widely used. However, as pointed out in [28], as a network typically has many neurons, these approaches can incur high complexities if they want to go through neurons exhaustively. 

It can be easily seen that the proposed method follows the above idea. Algorithm 1 clearly involves three nested loops in Step 1, implying that it needs to go through all the model inputs, all layers for each input, and all channels for each layer, thus leading to at least a cubic complexity and a prohibitively high computational cost for a real-world network.

To address this concern, the paper needs to clarify the algorithm's complexity, and provide the running times with respect to the numbers of inputs, layers, and channels considered in the algorithm. Without such a complexity analysis and the timing results, it is not convincing that the proposed method can be feasible to handle real-world networks.

In addition, the paper needs to compare the approach and the results presented in [28] that also quantifies individual neuron's contribution
to the network's performance. The paper needs to show if there would be any similarities or differences between the proposed method and the method in [28]. Without such a comparison study, it is unclear if this work would be superior to the-state-of-the-art. 

Overall, the approach presented in this paper is easy to understand. However, the idea appears standard, and its novelty is limited. Compared to the existing work, the complexity of the proposed approach seems high and may lead to prohibitive computation costs for real-world networks. The paper needs to justify the complexity and feasibility of the approach and compare it to the existing work. 

Author Response

Please refer attached file for our response.

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

Thank the authors for their effort to improve the paper by addressing my previous comments, particularly by comparing their work with [28]. I don't have any further comments.