Optimization Based Layer-Wise Pruning Threshold Method for Accelerating Convolutional Neural Networks

Abstract

Among various network compression methods, network pruning has developed rapidly due to its superior compression performance. However, the trivial pruning threshold limits the compression performance of pruning. Most conventional pruning threshold methods are based on well-known hard or soft techniques that rely on time-consuming handcrafted tests or domain experience. To mitigate these issues, we propose a simple yet effective general pruning threshold method from an optimization point of view. Specifically, the pruning threshold problem is formulated as a constrained optimization program that minimizes the size of each layer. More importantly, our pruning threshold method together with conventional pruning works achieves a better performance across various pruning scenarios on many advanced benchmarks. Notably, for the $L_1$-norm pruning algorithm with VGG-16, our method achieves higher FLOPs reductions without utilizing time-consuming sensibility analysis. The compression ratio boosts from 34% to 53%, which is a huge improvement. Similar experiments with ResNet-56 reveal that, even for compact networks, our method achieves competitive compression performance even without skipping any sensitive layers.

1. Introduction

Deep neural networks have achieved great success in various applications [1,2] ranging from image classification [3,4] to image segmentation [5,6] and object detection [7,8], but their limitations are gradually emerging. Extreme model complexity hinders large-scale deployment on constrained edge devices. For instance, to classify an image with $224\times 224$ pixels, the VGG-16 model consumes 550 M of storage space and 15.5 billion floating point operations (FLOPs). For embedded applications, these resource demands are prohibitive. Thus, how to deploy deep neural networks on embedded systems has become an urgent problem for both academia and industry.

To reduce the storage and energy requirements to run inference of large networks on mobile devices, many different compression and acceleration methods [9,10] have been proposed. Among them, pruning is a classic technique in machine learning. Pruning decision trees [11] can reduce model complexity and improve model generalization. To achieve high performance, network pruning [12,13] reduces storage and computational costs by removing unimportant connections and achieves excellent compression performance. Nowadays, network pruning can be roughly grouped into three categories, namely, metric-based pruning, training-based pruning, and reconstruction-based pruning. Metric-based pruning methods [14,15,16] typically propose a metric to measure whether a filter/channel is critical or not then retrain the pruned network to recover accuracy drops. There are three commonly used metrics in metric-based pruning methods, i.e., the weight, activation and gradient. Weight pruning methods evaluate the importance of connections using the weight magnitude. In 2015, unstructured weight pruning [17] proposed a simple yet effective pruning strategy based on weight, where weight absolute values below a given threshold are removed. $L_1$-norm pruning [18] extends this idea to structured pruning by removing a certain percentage of filters with smaller $L_1$-norms. Activation pruning methods [19,20] focus on the neuron importance in terms of activation values. Network trimming [20] finds that most neurons have zero output regardless of what the input data are. These neurons with zero output only increase the risk of overfitting. Therefore, the Average Percentage of Zeros (APoZ) criterion is proposed to evaluate the importance of different neurons. Fundamentally, the two previous pruning methods are heuristics and do not guarantee the effectiveness of pruning. Gradient pruning [21,22] starts directly from the loss function and prunes neurons that have the smallest impact on the loss function. The most representative approach is Taylor expansion pruning [22], which adopts the gradient score to determine which filters are pruned.

There have been other attempts to prune networks. Training-based pruning methods [23,24] add regularization terms to different network parameters during training. Different from metric-based pruning methods, training-based pruning methods focus more on sparse terms and then employ relatively simple pruning criteria. Reconstruction-based pruning methods [25,26,27] mainly evaluate the parameter importance by minimizing the feature map reconstruction error of the next layer. Given the pruning threshold, the reconstruction method selects a subset of channels as the pruned network.

Apart from pruning algorithms, the pruning performance of these methods is limited due to a common challenge. That is, they always need to determine the pruning threshold either by hand or using handcrafted heuristic rules. However, determining the layer-wise threshold is a difficult task. If the pruning threshold is too small, the network compression is insufficient, but if the pruning threshold is too large, it is difficult to recover the accuracy loss by fine-tuning. With a well-chosen pruning threshold, conventional pruning algorithms can achieve significant reductions in the number of parameters and FLOPs. A question arises: how to determine the pruning threshold for each layer? Currently, the commonly used pruning threshold methods are divided into the following three types: sensitivity analysis [18], the automatic method [28] and the soft threshold [29]. We summarize these methods in Figure 1. Sensitivity analysis methods mainly select the one with the smallest accuracy drop as the final pruning threshold for a specific layer. To obtain dynamic layer-wise pruning thresholds, the automatic method determines the pruning degree of each layer based on a single global pruning threshold, such as network slimming [24], which trains the scale factor of batch normalization (BN) layer by adding an $L_1$ regular constraint, which automatically determines the pruning degree of each layer.

Nevertheless, these pruning threshold methods still suffer from several problems and limitations. The sensitivity analysis requires a large number of search experiments to obtain the optimal pruning threshold setting, and the time cost grows exponentially with an increasing layer depth. Assuming a network with L layers and each layer with M threshold candidate testing, the sensitivity analysis requires $M^L$ experiments, which cause the prohibitively expensive time cost. Another method, the automatic method, has difficulty in assessing the local parameters’ importance from different layers since the contributions of each layer to the model are significantly different. This allows some layers with few parameters but significant FLOPs to be removed, such as an initial convolutional layer. Hence, while such models may significantly reduce the number of non-zeros, their FLOPs may still be large, making the pruning works based on a single global threshold insufficient and non-robust. To mitigate these shortcomings, the various soft threshold methods [29,30,31,32] are proposed to automatically learn the pruning thresholds for each layer using a soft thresholding operator or a close variant of it. However, most of the discussed pruning threshold techniques conventionally require handcrafted heuristics or domain experts to explore a large design space. More seriously, the aforementioned pruning threshold methods lack a corresponding recoverability guarantee. Thus, how to obtain pruning thresholds offline and give reasonable interpretability is a critical issue that needs to be addressed. It is an appealing idea to design the pruning threshold of each layer from an optimization point of view.

Unlike the traditional pruning threshold algorithms, we propose a general layer-wise pruning threshold method for metric-based pruning algorithms from an optimization point of view. Concretely, we transform the pruning threshold problem into a constrained optimization program, whose solution reveals that how many connections are pruned depends on the pruning metric of the specific layer and the statistical information computed from the next layer. Based on the time cost savings, the proposed pruning threshold method enhances the interpretability of the pruned network. Moreover, the proposed method can update the performance of most metric-based pruning algorithms. Extensive experiments on a variety of datasets demonstrate that our pruning threshold method together with the conventional metric-based pruning works can achieve higher compression performance. Obtaining pruning thresholds via optimization problems and, in turn, improving the compression performance of conventional pruning algorithms is the main contribution of this paper.

2. Related Work

Different methods have been proposed to compress neural network models, which can be roughly categorized as follows: (i) network quantization; (ii) knowledge distillation; (iii) network pruning. Currently, network pruning methods have emerged as an efficient strategy for compressing models with limited accuracy degradation, which can be broadly categorized into unstructured and structured pruning. Early pruning works focused on unstructured pruning, which removes individual weights from the neural network. While unstructured pruning methods achieve much higher weight sparsity, unstructured pruning is considered less hardware friendly, as irregular sparsity often makes it difficult to leverage the speedup provided by commodity hardware during training and inference. Structured pruning takes structure into account, making the model scalable on commodity hardware with standard computation techniques/architectures.

Although fruitful pruning algorithms have been proposed, most unstructured and structured pruning methods generally require hand-tuned thresholds to achieve an excellent performance. These pruning methods iteratively prune weights smaller than a certain threshold and retrain the network to regain the performance lost during pruning. However, when the wrong pruning threshold is set, the accuracy of the pruned model degrades considerably. In general, a small pruning threshold is needed for certain “sensitive” layers of the network and a big pruning threshold for “non-sensitive” layers. Thus, the key challenge in pruning is to find an optimal setting for these pruning thresholds. For simplicity, some methods set the same threshold for all layers, which may not be appropriate since the distribution and range of weights in each layer can be very different. Also, different layers may have varying sensitivities to pruning, depending on their location in the network or their type. The optimal setting of the threshold should take these layer-wise features into account. However, most pruning threshold methods are based on the well-known hard [17,18,26] and soft threshold [29,30,31,32] techniques, which rely on handcrafted features or require domain experts to explore the large design space trade-off between model size, speed, and accuracy, which is often suboptimal and time-consuming. More seriously, existing pruning threshold methods lack theoretical guarantees on compression performance. Unlike the aforementioned methods, our approach is a general pruning threshold method that achieves a better performance across various pruning scenarios.

3. Methodology

The goal of pruning is to compress the network as much as possible with little accuracy drop. To achieve higher compression, it is necessary to design appropriate pruning thresholds for each layer. In this section, we will give a comprehensive introduction to our pruning threshold method, which prunes redundant filters/channels using a three-step strategy. First, we define our optimization problem for obtaining a more reasonable pruning threshold for each layer. Next, we relax the optimization condition for solving easily. Finally, we obtain a suboptimal solution and discuss the impact of the error bound.

3.1. Optimization Problem

Suppose the l-th layer output is defined as follows:

$$F^{\left(l\right)}\left(x\right)=ReLU(w^{\left(l\right)}x+b^{\left(l\right)})$$

(1)

where x denotes the input data of the ReLU activation function, $F^{\left(l\right)}$ is the output tensor, and the parameters $w^{\left(l\right)}$ and $b^{\left(l\right)}$ represent the weight and bias, respectively.

Our goal is to maximize the l-th layer pruning threshold for metric-based pruning methods, which means that we should keep the number of channels as small as possible. So, our final objective can be set to minimize the number of surviving channels. Given the error bound before and after pruning, we formulate our optimization problem as follows:

$$\underset{\tau }{arg\; min}||s_l{||}_1$$

$$s.t.\sum _{m=1}^M||F^{\left(l\right)}\left(x_l^{\left(m\right)}\right)-F^{\left(l\right)}(s_l\ast x_l^{\left(m\right)}){||}_1\le ϵ$$

where $||·{||}_1$ denotes the $L_1$-norm, M denotes the number of test samples, $x_l^{\left(m\right)}$ is the input tensor of the l-th layer, $ϵ$ is the tolerable error bound before and after pruning and $s_l$ is a binary vector, which denotes the channel prune indicator of the l-th layer. Suppose $N_l$ channels are to be kept in the l-th layer after pruning: $s_{l,i}=1$ if and only if $s_{l,i}$ is among the top $N_l$ values in the l-th layer.

3.2. Relaxing Optimization Condition

We have formulated the pruning threshold problem as an optimization program, but, due to the ReLU function, it is difficult to obtain an analytical solution through direct optimization. In order to obtain a suboptimal solution, we derive an upper bound on the original optimization condition

$$|ReLU(x)-ReLU(y)|\le |x-y|$$

(2)

where $|·|$ denotes the element-wise absolute value. Then, we can obtain

$$|F^{\left(l\right)}\left(x\right)-F^{\left(l\right)}\left(y\right)|\le |w^{\left(l\right)}x-w^{\left(l\right)}y|\le |w^{\left(l\right)}|·|x-y|$$

(3)

By substituting $x=x_l^{\left(m\right)},y=s_l\ast x_l^{\left(m\right)}$ into (3),

$$||F^{\left(l\right)}\left(x_l^{\left(m\right)}\right)-F^{\left(l\right)}(s_l\ast x_l^{\left(m\right)}){||}_1\le \sum _i\sum _j|w_{i,j}^{\left(l\right)}|·(1-s_{l,j})·|x_{l,j}^{\left(m\right)}|$$

(4)

Introducing all M test samples,

$$\sum _{m=1}^M||F^{\left(l\right)}\left(x_l^{\left(m\right)}\right)-F^{\left(l\right)}(s_l\ast x_l^{\left(m\right)}){||}_1\le \sum _{m=1}^M\sum _i\sum _j|w_{i,j}^{\left(l\right)}|·(1-s_{l,j})·|x_{l,j}^{\left(m\right)}|$$

(5)

Since the input data $|x_{l,j}^{\left(m\right)}|$ are bounded, thus there exist a constant C such that ${\displaystyle \sum _{m=1}^M}|x_{l,j}^{\left(m\right)}|\le C$. The optimization condition is written as

$$\sum _{m=1}^M||F^{\left(l\right)}\left(x_l^{\left(m\right)}\right)-F^{\left(l\right)}(s_l\ast x_l^{\left(m\right)}){||}_1\le C·\sum _i\sum _j|w_{i,j}^{\left(l\right)}|·(1-s_{l,j})\le ϵ$$

(6)

Defining that $\delta$ is a new error bound as follows, we can finally state the optimization condition as follows:

$$\sum _i\sum _j|w_{i,j}^{\left(l\right)}|·s_{l,j}\ge \sum _i\sum _j|w_{i,j}^{\left(l\right)}|-ϵ/C=\delta$$

(7)

where $s_{l,j}=0$ denotes the j-th feature map be pruned; otherwise, it should be preserved. Equation (7) shows that, when given a suitable error bound, the pruning threshold is entirely up to the pruning metric of the specific layer and the weight information computed from the next layer. The detailed steps will be shown in the following sections.

3.3. Solution

Now, if we further define $k_j={\displaystyle \sum _i}|w_{i,j}^{\left(l\right)}|$, the optimization condition (7) can be simplified as follows:

$$\sum _j{k}_j·s_{l,j}\ge \delta$$

(8)

Metric-based pruning methods typically propose a metric to evaluate whether a channel is critical or not. Then, the unimportant channels are pruned. To minimize the objective function, i.e., $|s_l{|}_1$, we rank channels’ importance via the proposed metric and select the corresponding next layer $k_j$ such that the optimization condition (8) is satisfied. Among the remaining importance factors, the maximum or minimum value is the pruning threshold of the l-th layer. The detailed pruning procedure of the l-th layer is summarized below:

• For each feature map $x_{l,j}$ from the l-th layer, calculate the importance as $a_j$ using the proposed metric.
• For each filter $w_{n,j}$ from the $(l+1)$-th layer, calculate $k_j={\displaystyle \sum _n}|w_{n,j}|$.
• According to the distribution of $k_j$, determine the error bound $\delta$.
• Rank the $a_j$ by importance.
• Select the first m filters by $a_j$, then choose a corresponding $k_j$ such that the optimization condition (8) is satisfied. Among the remaining importance score $a_j$, the maximum or minimum value is the pruning threshold of the l-th layer.

Generally, other metrics include but are limited to the $L_1$-norm of filters, which also leads to the removal of the corresponding feature maps. Thus, any pruning works that result in the removal of the feature maps can use this method, such as [24,33,34], whose pruning performance can be updated using our proposed pruning threshold method.

3.4. Analysis of Choice of Error Bound

In this subsection, we investigate the effect of error bound on the pruning performance. The error bound $\delta$ determines the degree of pruning. Thus, we need to choose the error bound carefully. In effect, it is hard to set the error bound $\delta$ because of the uniqueness of each layer. In 2019, a study on filter pruning via geometric median (FPGM) [34] pointed out the problem of the criterion “smaller-norm-less-important” adopted by previous works. They believe that the large norm deviation and small minimum norm can ensure the effectiveness of pruning. Inspired by FPGM, we find that the error bound is related to the statistics information computed from the next layer. Therefore, we set the error bound as $r·{\displaystyle \sum _i}{\displaystyle \sum _j}|w_{i,j}^{\left(l\right)}|$, where r is related to the Std/Min from the distribution of $k_j$. (Std and Min denote the standard derivation and minimum of the $k_j$, respectively).

4. Experiments

In this section, we evaluate our method for the popular VGG Net [35], ResNet [36], on ImageNet and CIFAR-10 datasets. There are two reasons why we use the VGG model and CIFAR-10 dataset. The first reason is that the VGG model and CIFAR-10 dataset are very redundant, so almost all pruning algorithms show VGG and CIFAR-10 results to verify the effectiveness of the algorithm. The second reason is that all five different metric-based pruning works use the VGG model and CIFAR-10 dataset, so we also compare them in the experimental section. We organize the experiments according to different pruning metrics, including the $L_1$ norm and $L_2$-norm of weights, activations, BN factors, and geometric median, which come from five different works on metric-based pruning. We first introduce these different pruning works and use their default settings. We then update their pruning performance with the proposed pruning threshold method. If we do not report the accuracy of the experimental table, it indicates that the accuracy of the pruned model decreased by less than 0.1 compared to the unpruned model.

4.1. $L_1$-Norm-Based Filter Pruning [18]

$L_1$-norm-based filter pruning [18] is one of the earliest works on structured pruning, which selects the $L_1$-norm as the pruning metric and prunes redundant filters using the three-step method in Figure 2. First, the unpruned network is trained to learn which filters are important. Next, the filters with small $L_1$-norms are removed. Finally, the pruned network is retrained to fine-tune the remaining filters. We update this pruning work with the proposed pruning threshold method. The model parameters and FLOPs before and after pruning are shown in

Table 1. Overall results. The accuracy, parameters and FLOPs of different models are reported. We achieved higher FLOPs reductions on VGG-16 and ResNet-56. Although the FLOPs are slightly higher on ResNet-34, our method does not use time-consuming sensibility analysis and does not skip any sensitive layers.

Dataset	Model	Method	Accuracy	Params	FLOPs
CIFAR-10	VGG-16	Unpruned	93.63	1.50 $\times {10}^7$	3.14 $\times {10}^8$
		Haoli	93.41	5.40 $\times {10}^6$	2.07 $\times {10}^8$
		Ours	93.54	3.80 $\times {10}^6$	1.83 $\times {10}^8$
	ResNet-34	Unpruned	93.14	8.50 $\times {10}^5$	1.27 $\times {10}^8$
		Haoli	92.67	7.40 $\times {10}^5$	9.22 $\times {10}^7$
		Ours	92.97	6.20 $\times {10}^5$	8.29 $\times {10}^7$
ImageNet	ResNet-34	Unpruned	73.31	2.18 $\times {10}^7$	3.67 $\times {10}^9$
		Haoli	72.29	1.95 $\times {10}^7$	2.79 $\times {10}^9$
		Ours	72.57	1.80 $\times {10}^7$	2.87 $\times {10}^9$

. The detailed procedure of pruning from the l-th layer is as follows:

• For each filter $w_{j,m}$ from the l-th layer, calculate the $L_1$-norm $a_j={\displaystyle \sum _m}|w_{j,m}|$.
• For each filter $w_{n,j}$ from the $(l+1)$-th layer, calculate $k_j={\displaystyle \sum _n}|w_{n,j}|$.
• According to the distribution of $k_j$, determine the error bound $\delta$.
• Rank the $a_j$ in a descending order.
• Select the first m filters by $a_j$, then choose corresponding $k_j$ such that the optimization condition (8) is satisfied. Among the remaining importance score $a_j$, the maximum value is the pruning threshold of the l-th layer.

Detailed experiment results are shown as follows:

• VGG-16 on CIFAR-10: According to the value of the Std/Min, we can distinguish different layers’ pruning sensibilities.

  Table 2. Pruning factor and surviving maps of VGG-16 on CIFAR-10. The values of Std/Min from Conv-1 and Conv-8 to 13 are significantly higher than Conv-2 to 7, so we only prune Conv-1 and Conv-8 to 13.

  Layer Type	Used Layer	Std/Min	r	Maps
  Conv-1	Conv-2	485.87	0.5	15
  Conv-2	Conv-3	0.209	0	64
  Conv-3	Conv-4	0.247	0	128
  Conv-4	Conv-5	0.102	0	128
  Conv-5	Conv-6	0.112	0	256
  Conv-6	Conv-7	0.129	0	256
  Conv-7	Conv-8	0.179	0	256
  Conv-8	Conv-9	0.581	0.5	214
  Conv-9	Conv-10	1.039	0.5	199
  Conv-10	Conv-11	1.597	0.5	175
  Conv-11	Conv-12	1.667	0.5	178
  Conv-12	Conv-13	1.609	0.5	181
  Conv-13	Linear	1.329	0.5	175

  shows our results. We observe that the values of Std/Min from Conv-1 and Conv-8 to 13 are significantly higher than Conv-2 to 7, so we only prune Conv-1 and Conv-8 to 13. The find is consistent with $L_1$-norm-based filter pruning [18], but we do not use the sensibility analysis, which is time-consuming. For simplicity, we set pruning factor r as 0.5 for these pruned layers. Exactly how many filters are pruned depends on the $L_1$-norm and corresponding $k_j$. As shown in

  Table 3. Other pruning factor and FLOPs of VGG-16 on CIFAR-10. We achieved higher FLOPs reduction.

  Prune Layer	r	Maps	r	Maps
  Conv-1	0.5	15	0.6	11
  Conv-2	0.1	56	0	64
  Conv-3	0.1	112	0	128
  Conv-4	0.1	114	0	128
  Conv-5	0.1	227	0	256
  Conv-6	0.1	227	0	256
  Conv-7	0.1	226	0	256
  Conv-8	0.5	214	0.6	167
  Conv-9	0.5	199	0.6	150
  Conv-10	0.5	175	0.6	132
  Conv-11	0.5	178	0.6	133
  Conv-12	0.5	181	0.6	136
  Conv-13	0.5	175	0.6	129
  FLOPs		0.148G		0.173G

  and

  Table 4. Pruning results of VGG-16 on CIFAR-10. We achieved higher FLOPs reduction than the original method.

  Layer Type	Maps			FLOPs
  	VGG-16	Haoli	Ours	VGG-16	Haoli	Ours
  Conv-1	64	32	15	1.77 $\times {10}^6$	8.85 $\times {10}^5$	4.15 $\times {10}^5$
  Conv-2	64	64	64	3.77 $\times {10}^7$	1.89 $\times {10}^7$	8.85 $\times {10}^6$
  Conv-3	128	128	128	1.89 $\times {10}^7$	1.89 $\times {10}^7$	1.89 $\times {10}^7$
  Conv-4	128	128	128	3.77 $\times {10}^7$	3.77 $\times {10}^7$	3.77 $\times {10}^7$
  Conv-5	256	256	256	1.89 $\times {10}^7$	1.89 $\times {10}^7$	1.89 $\times {10}^7$
  Conv-6	256	256	256	3.77 $\times {10}^7$	3.77 $\times {10}^7$	3.77 $\times {10}^7$
  Conv-7	256	256	256	3.77 $\times {10}^7$	3.77 $\times {10}^7$	3.77 $\times {10}^7$
  Conv-8	512	256	214	1.89 $\times {10}^7$	9.44 $\times {10}^6$	7.89 $\times {10}^6$
  Conv-9	512	256	199	3.77 $\times {10}^7$	9.44 $\times {10}^6$	6.13 $\times {10}^6$
  Conv-10	512	256	175	3.77 $\times {10}^7$	9.44 $\times {10}^6$	5.01 $\times {10}^6$
  Conv-11	512	256	178	9.44 $\times {10}^6$	2.36 $\times {10}^6$	1.12 $\times {10}^6$
  Conv-12	512	256	181	9.44 $\times {10}^6$	2.36 $\times {10}^6$	1.16 $\times {10}^6$
  Conv-13	512	256	175	9.44 $\times {10}^6$	2.36 $\times {10}^6$	1.14 $\times {10}^6$
  Total				3.14 $\times {10}^8$	2.07 $\times {10}^8$	1.83 $\times {10}^8$

  , we achieved higher FLOPs reductions on the basis of a lower time cost. Especially in Table 3, the compression ratio boosts from 34% to 53%, which is a huge improvement. To achieve a higher compression performance, we can also carefully set different pruning factor r values for different layers based on the value of the Std/Min. Maybe the sigmoid function is a good choice, which is the direction of our future work.
• ResNet-56 on CIFAR-10: Similar to VGG-16 in the CIFAR-10 experiment, we can distinguish different layers’ pruning sensibilities according to the Std/Min.

  Table 5. The Std/Min of ResNet-56 on CIFAR-10. We find that the value of Std/Min from ResNet-56 is very small, so we consider Std/Min as the pruning factor r of each layer.

  Layer	Used Layer	Std/Min	Layer	Used Layer	Std/Min
  Conv-2	Conv-3	0.22	Conv-30	Conv-31	0.41
  Conv-4	Conv-5	0.47	Conv-32	Conv-33	0.38
  Conv-6	Conv-7	0.58	Conv-34	Conv-35	0.41
  Conv-8	Conv-9	0.47	Conv-36	Conv-37	0.47
  Conv-10	Conv-11	0.24	Conv-38	Conv-39	0.13
  Conv-12	Conv-13	0.39	Conv-40	Conv-41	0.15
  Conv-14	Conv-15	0.36	Conv-42	Conv-43	0.11
  Conv-16	Conv-17	0.32	Conv-44	Conv-45	0.12
  Conv-18	Conv-19	0.27	Conv-46	Conv-47	0.13
  Conv-20	Conv-21	0.15	Conv-48	Conv-49	0.11
  Conv-22	Conv-23	0.12	Conv-50	Conv-51	0.13
  Conv-24	Conv-25	0.32	Conv-52	Conv-53	0.10
  Conv-26	Conv-27	0.27	Conv-54	Conv-55	0.12
  Conv-28	Conv-29	0.39

  shows our results. Unlike the VGG-16 experiment, the value of the Std/Min from ResNet-56 is very small, so we consider the Std/Min as the pruning factor r of each layer. As the stage grows, the average value of the Std/Min decreases, so the pruning threshold should also decrease; this finding is consistent with the pruning rate setting in the original paper (0.6, 0.3, 0.1). However, we do not skip any sensitive layers in the pruning process, which avoids the time-consuming sensibility analysis. As shown in Table 1, ResNet-56 also achieves a competitive compression performance.
• ResNet-34 on ImageNet: With promising results on VGG-16 and ResNet-56, we also test our method using a more complicated network, ResNet-34, on the ImageNet dataset. Different from ResNet-56, ResNet-34 uses the projection shortcut when the output feature maps are downsampled, which makes the pruning operation more difficult than that using ResNet-56. Following the pruning strategy from [18], we only prune the first layer of each residual block. Results are shown in Table 1. Although the FLOPs are slightly higher than the pruned result from original paper, our method does not use time-consuming sensibility analysis and does not skip any sensitive layers. Moreover, our pruning threshold method achieves more parameter savings than the $L_1$-norm-based filter pruning [18].

To understand the pruning sensitivity of each layer, the $L_1$-norm-based filter pruning [18] prunes each layer independently and evaluates the pruned network accuracy on the validation set, which needs hundreds of steps of testing and retraining progress. These time demands are prohibitive for network pruning. As shown in Table 1, we observe that in each row our method achieves a better compression performance with little extra experiments, which greatly saves time costs.

4.2. Network Trimming [20]

Network trimming [20] selects the activation output as the pruning metric and prunes redundant neurons based on analysis of network outputs on a large dataset. It defines the Average Percentage of Zeros (APoZ) to measure the percentage of zero activations of a neuron after the ReLU mapping. In pruning experiments, a neuron with mostly zeros in its output will have an extremely small contribution. Thus, a certain percentage of neurons with a large APoZ will be pruned. To avoid pruning too many neurons in one step, an iterative scheme is chosen to prune the network. Instead of an iterative scheme, we use a one-shot way for fast comparison. To decide how many neurons to prune, Network Trimming [20] prunes neurons whose APoZ is larger than one standard derivation from the mean APoZ of the target layer. We compare this pruning scheme with our proposed pruning threshold scheme. Detailed experiment results are shown in

Table 6. Results of network trimming [20] and network slimming [24] on CIFAR-10, which shows that our proposed pruning threshold method can achieve better compression performance than the original pruning scheme.

Dataset	Model	Method	Pruning Threshold	Acc	FLOPs
CIFAR-10	VGG-16	1Network Trimming [20]	Unpruned	93.63	3.14 $\times {10}^8$
			Pruned	93.41	2.34 $\times {10}^8$
			Ours	93.54	1.84 $\times {10}^8$
	VGG-19	Network Slimming [24]	Unpruned	93.53	3.99 $\times {10}^8$
			Pruned	93.60	1.95 $\times {10}^8$
			Ours	93.56	1.74 $\times {10}^8$

, which shows that our proposed pruning threshold method can achieve a better compression performance than the original pruning scheme.

4.3. Network Slimming [24]

Network slimming [24] imposes $L_1$-sparsity on scaling factors from batch normalization layers during training and selects scaling factors as the pruning metric. Since the channel scaling factors are compared across layers, this approach automatically produces the target architecture. However, Network Slimming does not account for layer differences. Evaluating the local parameter importance of each layer with a global threshold is difficult because the parameters and contributions of each layer in the network are significantly different. This approach can lead to some layers with few parameters, but significant FLOPs are removed. In fact, a $70\%$ pruning rate for VGG16 is a limit. If setting the pruning rate as $80\%$, some layers’ remaining channel is zero, which leads to deleting the whole layer. In addition, these pruned filters mainly operate on $2\times 2$ feature maps, which leads to fewer FLOPs in such small dimensions. To avoid these problems and further compress the network, we update this pruning approach with our proposed pruning threshold method. As shown in Table 6, Network Slimming achieves fewer FLOPs by setting the pruning threshold reasonably.

4.4. Soft Filter Pruning [33]

For previous hard filter pruning works, the pruned filters were deleted permanently and were no longer updated during training. Therefore, the model capacity was reduced due to the small model size. To tackle the problem of model capacity reduction, a Soft Filter Pruning (SFP) scheme [33] is proposed to select the $L_2$-norm as the pruning metric. Specifically, the pruned filters are updated again using SFP while training the pruned model, and its effectiveness has been demonstrated for various advanced network architectures. For simplicity, SFP prunes the same percentage of parameters at each layer, which limits the pruning performance of SFP. To achieve a better performance of the pruned model, we update the pruning performance of SFP with our proposed pruning threshold method. Please note that SFP prunes filters with small $L_2$-norms. Detailed results are shown in

Table 7. Comparison of pruned ResNet on CIFAR-10. For pruning the ResNet-20, our method achieves higher compression with a 0.25% accuracy increase. For pruning the pre-trained ResNet-56, our method also achieves better pruning performance than SFP.

Depth	Method	Pre-Trained	Accuracy	FLOPs
20	SFP(10%)	N	92.24	3.54 $\times {10}^7$
	SFP(20%)	N	91.20	2.93 $\times {10}^7$
	SFP(30%)	N	90.83	2.47 $\times {10}^7$
	Ours	N	91.45	2.57 $\times {10}^7$
56	SFP(20%)	N	93.47	9.31 $\times {10}^7$
	SFP(30%)	N	93.10	7.87 $\times {10}^7$
	SFP(40%)	N	92.26	6.31 $\times {10}^7$
	SFP(40%)	Y	93.35	6.31 $\times {10}^7$
	Ours	N	92.73	5.74 $\times {10}^7$
	Ours	Y	93.41	5.74 $\times {10}^7$

. For pruning the ResNet-20, our method achieves higher compression with a 0.25% accuracy increase. For pruning the pre-trained ResNet-56, our method also achieves a better pruning performance than SFP.

4.5. Filter Pruning via Geometric Median [34]

Filter pruning via geometric median (FPGM) points out the problem of the criterion “smaller-norm-less-important” utilized by the previous works, whose effectiveness depends on the minimum norm and the large norm deviation. However, the statistical information collected from the pre-trained model suggests that the two proposed requirements may not always hold. To solve this problem, FPGM rejects norm importance and introduces the geometric median as the pruning metric. Similar to SFP [33], the same pruning threshold for all layers limits the pruning performance of FPGM. We further improve the pruning performance using our proposed pruning threshold method. As shown in

Table 8. Comparison of pruned ResNet on CIFAR-10. For ResNet, our method achieves higher compression performance with competitive and even higher accuracy.

Depth	Method	Pre-Trained	Accuracy	FLOPs
20	FPGM(30%)	N	91.09	2.47 $\times {10}^7$
	FPGM(40%)	N	90.44	1.95 $\times {10}^7$
	Ours	N	91.99	2.32 $\times {10}^7$
56	FPGM(40%)	N	92.93	6.31 $\times {10}^7$
	FPGM(40%)	Y	93.49	6.31 $\times {10}^7$
	Ours	N	93.33	5.54 $\times {10}^7$
	Ours	Y	93.45	5.54 $\times {10}^7$

, for ResNet, our method achieves a higher compression performance with a competitive and even higher accuracy. For example, for ResNet-20, our method without utilizing the pre-trained model achieves a 0.9 accuracy improvement but fewer FLOPs. Comparing to FPGM, when pruning ResNet-56, our method without utilizing the pre-trained model even achieves the same competitive accuracy as using the pre-trained model. These results demonstrate that our pruning threshold method can achieve a state-of-the-art pruning performance.

In the above subsection, we introduced five different works on metric-based pruning and updated their pruning performance with the proposed pruning threshold method. The experiment results show that our proposed pruning threshold method can achieve higher FLOPs reductions while saving time costs. Thus, it is an appealing idea to design the pruning threshold of each layer from an optimization point of view. For more modern networks like efficientnet and mobilenet, we can also use our method to combine with them in the future to achieve higher compression, because the different compression directions can be combined with each other.

5. Conclusions

Traditional pruning threshold methods require extensive search experiments or handcrafted heuristics and domain experts to obtain optimal pruning threshold settings. Unlike the traditional pruning threshold algorithms, which only aim at a single pruning metric, we propose a general pruning threshold method from an optimization point of view. By formalizing the pruning threshold as an optimization problem, we provide a reasonable pruning threshold setting for all metric-based pruning algorithms in terms of time cost savings. Numerical experiments on a range of popular models and datasets demonstrate the effectiveness of our pruning threshold method. Notably, the compression ratio boosts from 34% to 53% on the $L_1$-norm pruning algorithm with VGG-16, which is a huge improvement. In the future, we can set the more refined prune threshold for different layers carefully, which can further improve the compression performance. Perhaps the sigmoid function is a better choice.