PARS: Proxy-Based Automatic Rank Selection for Neural Network Compression via Low-Rank Weight Approximation

Round 1

Reviewer 1 Report

Summary:

The authors propose an automatic proxy-based rank search method, which uses BN to alleviate distribution shift and Bayesian optimization to find ranks. The motivation is clear and the theoretical explanation is easy to understand. The experiments on different architectures also verify the effectiveness of the proposed method. However, the comparisons could be further improved.

 

Strengths:

1. The proposed method provides a good idea to alleviate the main shortcomings of existing rank search algorithms.

2. Experiments show that the proposed method can accelerate the optimal rank search procedure and improve performance.

 

Weaknesses:

1. In Section 5.3, the comparison of the search cost taken by each method should be considered in Table 4.

2. Besides low-rank approximation, channel pruning is one of the main approaches to compressing deep networks. It is interesting to see whether the proposed method also outperforms the state-of-the-art pruning methods.

3. Is it possible to combine the proposed method with other compression methods, e.g., channel pruning or quantization? It would be also good to include some discussions into the paper if the empirical results are hard to obtain.

4. There are some typos in the manuscript. For example, in line 316, “... and 160 iteration of Bayesian optimization ...” should be “... and 160 iterations of Bayesian optimization ...”.

 

 

Author Response

We would like to thank the reviewer for positive feedback on our paper and suggested improvements of the paper. We address each point below:

1. We added number of evaluations required for each method in the Table 4 and extended experiment description in Section 5.3.
2. Some methods presented in Table 3 are channel pruning methods. We added column Type to highlight DNN compression approach type. We also added several new method results from recent publications.
3. Generally, PARS selects a compression ratio for each model layer. This allows PARS to work with each type of weight decomposition model. Thus, it can be combined neural network compression methods, such as structured or unstructured pruning. Unfortunately, at the moment it is difficult for us to obtain results from this experiment. However we added this information into Discussion section.
4. We thank the reviewer for this comment. We conducted a proofreading of the manuscript and hope that the number of typographical errors is reduced to a minimum.

We would also like to point out other considerable changes in the manuscript:

1. We revised Introduction and added a Related Work section with broader Literature review.
2. Added diagram that visualizes Algorithm 1.
3. Provided more detailed PARS hyper-parameters selection explanation.
4. Revised Conclusion and Discussion section and splitted it into two separate sections.

Author Response File: Author Response.pdf

Reviewer 2 Report

The authors propose a Proxy-based Automatic Tensor Rank Selection method (PARS) that uses a Bayesian optimization approach to find the best combination of ranks for neural network compression, in which they examine how well the decomposition of weight tensors with different ranks impacts feature distribution within a neural network.

The topic of this paper is very interesting. However, the authors need to address the following raised points in order to improve the presentation and quality of their paper:

1- In the abstract section: Answer the following questions: What problem did you investigate, and why is it important? What techniques did you employ? What were your primary findings? What conclusions can you draw from your findings? Please make your abstract more descriptive and quantitative in order to reach a wider readership.
2- An updated and complete literature review should be conducted to present the state-of-the-art and knowledge gaps of the research with strong relevance to the topic of the paper, and the authors must position their work according to the literature review.
3- The authors need to re-organize their work as follows: Abstract; Introduction; Related works; Materials and methods; Results and discussions; Conclusion; References.
4- All statistical metrics used must be explained with their associated equations.
5- In Section 2, the authors should present the equations separately from the text with their associated numbers.
6- Figures 4a and 4b should be presented with their y-labels, then the authors should clearly interpret the results obtained in these figures.
7- The authors should justify how they defined the PARS hyper-parameters. The trial and error method will be time-consuming in this case and won't be accurate, I recommend authors to use an optimizer (metaheuristic) at this point, complexity will increase just in the process of finding suitable hyperparameters but this will be more effective than the trial and error strategy.
8- Discussion and conclusion must be presented separately and avoid citing figures and references in the conclusion section.
9- Please present your paper in order to have a fluent transition between sections and subsections.
10- I recommend that the authors use the Taylor diagram to present a clear and visual comparison between their proposal and their comparatives for all datasets used.
11- The authors should mention what data sampling and validation methods are used for their proposed model and for the models used for comparison.
12- Please present the "Algorithm 1 PARS" in more detail + add a descriptive flowchart of the proposed PARS.
13- The authors need to provide a detailed complexity analysis of their proposal against their comparatives.
14- Please provide a complete list of acronyms and abbreviations.
15- Please avoid using the pronoun "we" in your writing.
16- Proofreading is required to enhance the presentation and the language of the paper.

Author Response

We would like to thank the reviewer for his hard work and extensive suggestions for paper improvement. We tried to take into account as many comments as possible. Below, we provide a summary of performed changes:

1. We slightly edited the abstract to make it more descriptive and easier to understand for broader audience.

2. We revised Introduction and added a Related Work section with broader Literature review.

3. We reorganized the manuscript. Now it has following structure: Abstract; Introduction; Related works; Low-Rank Approximation of Neural Network Weight; Method; Feature Distribution Analysis; Results; Discussion; Conclusion; References. Sections Low-Rank Approximation of Neural Network Weight; Method; Feature Distribution Analysis can be considered as parts of Materials and methods block. Conclusion and Discussion section and splitted it into two sections.

4. It is not clear what statistical metrics were assumed, but we tried to make statistical part connected with Bayessian optimization as clear as possible.

5. In Section 2, equations are now presented separately from the text with their associated numbers.

6. We added y-labels for Figures 4a and 4b.

7. We provided and extensive explanation for PARS hyper-parameters selection.

8. Conclusion and Discussion section is revised and splitted it into two sections.

9. We tried to make transition between sections more fluent

10. For data sampling we used standard Pytorch dataloaders, for evaluation we used classification accuracy calculation.

11. We added a diagram that visualizes Algorithm 1.

12. We added number of evaluations required for each method in the Table 4 and extended experiment description in Section 5.3.

13. We extended list of acronyms and abbreviations.

14. We reduced using the pronoun "we" in manuscript to minimum.

15. We conducted a proofreading of the manuscript and hope that the number of typographical errors is reduced to a minimum.

Author Response File: Author Response.pdf

Reviewer 3 Report

This paper proposes a Proxy-based Automatic tensor Rank Selection method (PARS) that utilizes a Bayesian optimization approach to find the best combination of ranks for neural network (NN) compression. The idea is relatively novel, and the experiments support the conclusion. I have the following comments:

 

1, The parameter and computation reduction rate of the proposed method needs to be analyzed and compared with CPD-3, Spatial-SVD, and TKD-2.

 

2, The related work part is not enough; the relation between the proposed method and existing works needs to be analyzed.

 

3, Line 72. The summary of the contribution is too simple. A concrete and compact summary are needed.

 

4, Line 254. The outlier point at about iteration 90 in Figure 6 needs explanation.

 

5. The effect of batch size needs to be analyzed.

 

6. The relation between the PARS and CPD-EPC should be clearly explained. Can the proposed method be applied to other frameworks?

 

7. Line 361. The abbreviations should be compact since many abbreviations are not listed.

 

8. Line 313. The search time settings should be specific, such as which layer and the paper should add the comparison to other methods.

 

 

9. Line 208. Figure 4. The paper should use some kind of metric to evaluate the channel-wise error distributions with or without bn calibration.

Author Response

We thank the reviewed for the comments. We address each point below:

1. PARS does not propose a novel decomposition scheme for neural network layers. It is an automatic rank selection method that selects best sets of ranks for DNN to compression and can work with different matrix/tensor decomposition, such as CPD-3, Spatial-SVD, and TKD-2.
2. We revised Introduction and added a Related Work section with broader Literature review.
3. We provided a more detailed summary of the contribution.
4. We added explanation for this point in Section 6.1.1.
5. We conducted additional experiments on dependance of post-calibration model accuracy on calibration batch size in Appendix E.
6. CPD-EPC is stable version of CPD (described in Appendix C), PARS is rank search method that can be applied to find decomposition rank for different matrix/tensor decomposition for DNN compression task.
7. We extended list of acronyms and abbreviations.
8. PARS finds decomposition ranks for all layers simultaneously performing model-wise rank search, that is why we can not add how much time each layer takes. To compare with other methods, we added number of evaluations required for each method in the Table 4 and extended experiment description in Section 5.3.
9. We added y-labels for Figures 4a and 4b.

We would also like to point out other considerable changes in the manuscript:

1. Conclusion and Discussion section is revised and splitted it into two sections.
2. Added diagram that visualizes Algorithm 1.
3. Provided more detailed PARS hyper-parameters selection explanation.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

With another proofreading for the entire manuscript, the latter will have fulfilled the requirements for publication.

Author Response

We would like to thank the reviewer for his positive feedback on our paper. We have conducted an extensive proofreading for the entire manuscript. The details can be seen in the attached pdf file.

Author Response File: Author Response.pdf

Reviewer 3 Report

The authors have addressed the concerns, I recommend acceptance for this paper.

Author Response

We would like to thank the reviewer for his positive evaluation of our paper. We also would like to note that we have conducted additional round of proofreading for the entire manuscript. The details can be seen in the attached pdf file.

Author Response File: Author Response.pdf