Explanations for Neural Networks by Neural Networks

Round 1

Reviewer 1 Report

This paper shows to generate explanations via another neural network (called Interpretation Network, or I-Net), which maps network parameters to a symbolic representation of the network function.

The contribution of the paper is clear and well explained.

This paper should be improved as follows:

• ‘5. Although ‘Related Approches’ is included, In ‘1. Introduction’, the problem of the existing approach is difficult to understand.
• In Table 3, the proposed approach uses four variables (n=1, 5, 10, 15). Why does Figure 5 show 3 variables? In addition, the number of variables seems small.
• Although the paper introduces the “real-time” approach, In Sections 2 and 3, there is no mention of “real-time”. Further explanation is needed.

Author Response

Thanks for your feedback. (1) We understand that the connection to the problem of related approaches can be hard to understand in the introduction. Therefore, we adjusted the second paragraph in the introduction (line 32-37) for clarification. (2) Figure 5 does not include the case n=1 since that case was just included for a visual comparison of the generated functions (in Figure 4). We left the case of n=1 out for the performance evaluation since we wanted to focus on more complex and realistic scenarios here. (3) The “real-time” aspect of our approach is linked to the fact that generating an explanation only requires a single prediction of an I-Net (which can be achieved in nearly real-time), while existing approaches always include an optimization to generate an explanation (which cannot be achieved in real-time). However, we understand that it will support the understanding if we include this in Section 2/3. Therefore, we adjusted our manuscript in the end of Section 2.1 (line 106-109) as well as at the end of section 2.2 (line 113-114 and line 141-144) for clarification.

Reviewer 2 Report

- The author introduces I-Net to quickly explain the mathematical model of the Neural Network. I-Net uses a learning understanding function in the neural network, generating network patterns to represent network functions. This I-Net model is evaluated against several cases and provides competitive results for applying multiple data and functional complexity. 
- This paper provides a significant added value for researchers who conduct research related to neural networks. 
- The author provides a clear, detailed, and gradual description of I-Net. - The dataset specifications and methods used in the evaluation are clearly explained, including the setup environment. 
- Using a neural network and genetic programming as a comparison method. 
- Experimental results using datasets are described and evaluated using several graphics, and the authors also carry out additional evaluations using noise. 

Author Response

Thanks for your feedback. We will upload a revised version of the manuscript.

Reviewer 3 Report

This paper proposes a machine learning approach for the real-time extraction of mathematical functions from already trained neural networks as an explanation of how the neural network works. Generating explainable functions for a neural network from its parameters is essentially useful in explainable machine learning. The paper is overall in a good shape however can be further improved in several aspects.

1) The dataset description is not clear. More datasets need to be tested. 

2) The linearity of the pretrained black-box neural network needs to be discussed. If the original model is highly non-linear, such as a deep convolution network trained on image data, in what extend can it be explained by polynomial functions?

3) The proposed method should be compared with existing explainable AI models, such as LIME, SHAP. Or at least a discussion about their comparison should be provided.

4) The Greek symbols used in the paper can be organized into a table for better introduction. Otherwise readers may get lost while reading.

 

Author Response

Thanks for your feedback.

1. The datasets are synthetically generated according to Algorithm 1 to cover reasonable scenarios where explanations using mathematical functions can be performed. Furthermore, we added noise to the datasets in 4.2.3. to show that I-Nets can also be applied in realistic scenarios where noise is inherently contained in the data. An evaluation within a real-world scenario is subject to further work. However, for clarification, we adjusted the details for the data generation at the end of Section 3 (line 201-204 and line 210-212) and clarified why this is a reasonable setting for the evaluation at the beginning of Section 4.1 (line 237-240).
2. By using polynomials with degree>1, we can represent non-linear functions to some extent. The reviewer is of course right, that it is difficult to explain models like deep convolutional neural networks with polynomial functions while still achieving a considerable fidelity. However, this is not a weakness of the presented approach (the I-Nets) but rather of the function family chosen for the explanation (i.e., polynomials). The general approach presented in this paper can also be applied to further function families that are more suitable to represent the model learned by a deep convolutional neural network which is subject to further work. We added a short discussion this within the conclusion (line 415-420).
3. The focus of our paper is on global explanation methods. XAI techniques like LIME and SHAP however do not provide global explanations, but just local explanations for single samples which precludes an experimental comparison. We clarified this in the beginning of the related work section (line 353-359).
4. We added the table in the Appendix B.

Reviewer 4 Report

This is an interesting paper on understanding of the neural network. It has a huge potential today and research in this field is very important. In this situation is providing the first approach that attempts to learn the mapping from neural networks. The manuscript is well written and fully understandable. There is clear novelty in the manuscript and the overall quality is good and well written.

Author Response

Thanks for your feedback. We will upload a revised version of the manuscript.