Learning Bilateral Clipping Parametric Activation for Low-Bit Neural Networks

Abstract

Among various network compression methods, network quantization has developed rapidly due to its superior compression performance. However, trivial activation quantization schemes limit the compression performance of network quantization. Most conventional activation quantization methods directly utilize the rectified activation functions to quantize models, yet their unbounded outputs generally yield drastic accuracy degradation. To tackle this problem, we propose a comprehensive activation quantization technique namely Bilateral Clipping Parametric Rectified Linear Unit (BCPReLU) as a generalized version of all rectified activation functions, which limits the quantization range more flexibly during training. Specifically, trainable slopes and thresholds are introduced for both positive and negative inputs to find more flexible quantization scales. We theoretically demonstrate that BCPReLU has approximately the same expressive power as the corresponding unbounded version and establish its convergence in low-bit quantization networks. Extensive experiments on a variety of datasets and network architectures demonstrate the effectiveness of our trainable clipping activation function.

1. Introduction

Deep neural networks (DNNs) have achieved great success in various computer vision tasks, such as image classification [1,2], object detection [3,4], and image segmentation [5,6], etc. However, the high model complexity hinders large-scale deployment on resource-constrained devices. For instance, to classify a $224\times 224$ image, the VGG-16 model consumes 550 M storage space and 15.5 billion floating point operations (FLOPs). For embedded applications, these resource demands are prohibitive. Determining how to deploy deep neural networks on embedded systems has therefore become an urgent problem for both academia and industry.

Among a wide range of neural network compression and acceleration methods [7,8], network quantizations [7,9] reduce the storage and computational cost by converting 32-bit floating points to low-precision representation and achieve excellent compression performance. Nowadays, network quantization can be roughly grouped into two categories, namely weight quantization and activation quantization. A plethora of weight quantization methods have been extensively studied and proposed. Han et al. [10] quantized the model jointly with pruning, which reduced the storage required by AlexNet from 240 MB to 6.9 MB. Liu et al. [11] proposed a one-shot mixed-precision quantization method to adaptively adjust bit-width precision.

In addition to weight quantization methods, rectified activation functions also play an important role in both full-precision and quantized networks, whose non-saturated feature alleviates the problem of exploding/vanishing gradient, thereby speeding up the model convergence. Of all these activation functions, the Rectified Linear Unit (ReLU) is one of several keys to the recent success of deep networks. To alleviate the problem of zero gradients, leaky ReLU proposes a fixed slope to negative inputs. PReLU [12] introduces a trainable slope for negative inputs rather than a fixed one. This trainable slope is a key factor in surpassing human-level classification performance on the ImageNet 2012 dataset. Furthermore, RReLU [13], SE [14], and DY-ReLU [15] have also been proposed to better improve classification performance. To better understand the differences between the above-mentioned activation functions, we illustrate them in Figure 1.

While rectified activation functions have achieved great success for floating point networks, the unbounded outputs make activation quantization difficult. The quantization ranges determine the trade-off between model accuracy and quantization error. Specifically, models with a smaller quantization range can mitigate the quantization error but restrict their numerical range for training. On the other hand, models with a larger quantization range have a more flexible quantization range for achieving higher accuracy. Thus, a question arises: how does one obtain a reasonable quantization range? The traditional activation quantization methods [16,17,18,19] directly transform activation into [−1, 1] or [0, 1] to limit the high output range. This trivial clipping strategy limits the performance of activation quantization. So far, several clipping activation functions have been proposed to address the unboundedness problem. The quantization scheme [20] with ReLU6 achieves less accuracy degradation by placing a fixed upper bound on the activation values, which shows that a fixed range is easier to quantize. Half-Wave-Gaussian-Quantization (HWGQ) [21] utilizes the statistics of network activations to propose a variant of ReLU that constrains the unbounded value. Nevertheless, the fixed upper bound during training yields a drastic accuracy drop. In order to improve quantization performance more flexibly, Parameterized Clipping activation (PACT) [22] places a trainable upper bound limit on the output of standard ReLU, which achieves full-precision accuracy with 4-bit precision. However, these conventional clipping functions mainly focus on standard ReLU, which lacks an adaptive consideration of all rectified activation functions.

In this paper, we propose a Bilateral Clipping Parametric Rectified Linear Unit (BCPReLU) as a generalization of ReLU and its variants. The learnable two-sided thresholds and non-zero slopes for activation input are introduced during training. We prove that the expressive ability of BCPReLU is almost the same as the corresponding unbounded one, and we establish the convergence property of the quantization error. Extensive experiments on a variety of popular models and datasets demonstrate that our proposed clipping activation function can achieve better compression performance in both full-precision and quantized networks. Our contributions are summarized below:

• We propose a novel clipping activation function as a generalized version of rectified activation functions. The trainable slopes and thresholds for both positive and negative input are introduced in BCPReLU.
• We theoretically prove that BCPReLU has almost the same expressive ability as the corresponding unbounded function in a full-precision network, and we establish the convergence of BCPReLU in the quantized network.
• Extensive experiments on CIFAR-10 and ImageNet datasets demonstrate the effectiveness of BCPReLU.

The rest of the paper is organized as follows: Section 2 provides a summary of prior work on quantization. Section 3 proposes a novel clipping activation function, in which the expressive ability in the full-precision network is analyzed, and the convergence in the quantized network is established. In Section 4, we demonstrate the effectiveness of our quantization scheme.

2. Related Work

Currently, network quantizations [23,24,25] have become an efficient strategy to compress models with limited accuracy degradation. These quantization approaches can be roughly divided into weight quantization and activation quantization. Early quantization works [26,27,28,29] are mainly concerned with weight quantization, which quantizes weight into one bit (binary) or two bits (ternary). Recently, other weight quantization schemes [30,31,32] based on optimization or approximation have been proposed. For example, Liu et al. [30] formulated weight quantization as a differentiable nonlinear mapping function. Cai et al. [21] used piece-wise backward approximators to overcome the gradient mismatch problem. In addition to weight quantization methods, activation quantization is another essential factor that affects final quantization performance. In order to maximally utilize bit-wise operations, activation quantization methods [16,33] have gained considerable attention. In general, conventional activation quantization methods generally directly utilize the activation distribution or the range of activation output to quantize models, in which the quantization range is essential to quantization performance. The quantization range must be large enough to reduce the clipping error, and at the same time, it should be small enough to ensure the rounding error does not become too large. The quantization scheme of [20] is limited to the ReLU function and chooses a quantization range of $[0,c]$. Outlier Channel Splitting (OCS) [34] exploits channel splitting to avoid containing outliers, which duplicates channels containing outliers, then halves the input activations. Furthermore, ACIQ [35] also suggests limiting the range of activation values, which approximates the optimal clipping value analytically from the distribution of the tensor by minimizing the mean-square-error measure. These works reveal that a fixed clipping range with the right quantization range can achieve state-of-the-art accuracy. However, a fixed quantization range on the output is suboptimal due to the difference in layers/channels. Instead of fixing the quantization range for quantized deep learning models, novel quantization methods [22,36] propose a learnable parameter soft clipping quantization, which automatically learns suitable quantization ranges for each layer and then linearly quantizes the values in the optimized range to M bits.

3. BCPReLU and Quantization Analysis

In this section, we first propose a novel clipping activation function and then prove that the proposed activation function has almost the same representation ability as the corresponding unbounded one in the full-precision network. According to the reasonable assumption of activation distribution, we finally establish the convergence of the quantization error.

3.1. Novel Activation Function

As stated in the previous section, the traditional clipping activation quantization mainly focuses on standard ReLU, which lacks an adaptive consideration of all rectified activation functions. To overcome this limitation, we propose a more comprehensive clipping trainable activation function (BCPReLU) as follows:

$$y=f(x;\mu ,k_1,\alpha ,k_2)=\left\{\begin{array}{ccc}\hfill -k_1\mu & & x\in (-\infty ,-\mu )\hfill \\ \hfill k_{1}x& & x\in [-\mu ,0)\hfill \\ \hfill k_{2}x& & x\in [0,\alpha )\hfill \\ \hfill k_2\alpha & & x\in [\alpha ,+\infty )\hfill \end{array}\right.$$

(1)

The parameter x is the input of the nonlinear activation y, and the parameters $k_1$ and $k_2$ are trainable and control the slopes of the negative input and the positive input, respectively. The parameters $-\mu$ and $\alpha$ are the clipping values of the negative input and the positive input, respectively. In fact, the proposed activation function includes a large number of traditional activations. When $k_1=0$ and $k_2=1$, the clipping activation function becomes PACT (the trainable clipping ReLU); when $k_1$ is a small and fixed value and $k_2=1$, it becomes a clipping form of LReLU; and when $k_1$ is a trainable variable and $k_2=1$, it becomes a clipping form of PReLU.

As shown in Figure 2, when $-\mu \to -\infty$ and $\alpha \to +\infty$, the output of BCPReLU is unbounded, which incurs large quantization error. To limit the range of output, it is necessary to place reasonable clipping values. The clipping values $-k_1\mu$ and $k_2\alpha$ in BCPReLU limit the range of output, and $k_1$ and $\mu$ also enhance the expressive power in negative input to avoid dead neurons.

3.2. Expressive Ability

In this subsection, we consider the effect of clipping values and compare BCPReLU with the corresponding unbounded one in a full-precision network. Following PACT [22], we summarize the theorem as follows:

Theorem 1. 

Assume that x is an activation input, y represents the corresponding output. The network with BCPReLU can be trained to find the same output as the corresponding unbounded function and converge faster than the corresponding unbounded function.

Proof of Theorem 1. 

Assume that $y^{*}$ is the corresponding label of x; the cost function is defined as following the mean-square-error (MSE):

$$L\left(y\right)=0.5·{(y-y^{*})}^2$$

We define the unbounded function corresponding to BCPReLU as g:

$$g=g(x;k_1,k_2)=\left\{\begin{array}{ccc}\hfill k_{1}x& & x\in (-\infty ,0)\hfill \\ \hfill k_{2}x& & x\in [0,+\infty )\hfill \end{array}\right.$$

(2)

If $x\in [-\mu ,\alpha ]$, the network with BCPReLU (Equation (1)) behaves the same as the network with the unbounded function g (Equation (2)).

If $x<-\mu$, then $y=-k_1\mu$, $g=k_{1}x$. $k_2$ and $\alpha$ are not updated because

$$\frac{\partial y}{\partial k_2}=\frac{\partial y}{\partial \alpha }=0.$$

Updating $k_1$ and $\mu$:

$$k_1^{new}=k_1-\eta \frac{\partial L}{\partial k_1}=k_1-\eta \frac{\partial L}{\partial y}·\frac{\partial y}{\partial k_1}=k_1+\eta \mu \frac{\partial L}{\partial y},$$

$${\mu }^{new}=\mu -\eta \frac{\partial L}{\partial \mu }=\mu -\eta \frac{\partial L}{\partial y}·\frac{\partial y}{\partial \mu }=\mu +\eta k_1\frac{\partial L}{\partial y}.$$

The update of $k_1$ and u depend on $\frac{\partial L}{\partial y}$, so we need to consider the different cases of $\frac{\partial L}{\partial y}=y-y^{*}:$

Case 1: If $y^{*}<k_{1}x$, then $y^{*}<y$, $\frac{\partial L}{\partial y}>0$, $k_1$ increases, $\mu$ increases, and thus $y=-k_1\mu$ decreases until $-\mu <x$, i.e., $y=g$. The network with BCPReLU (Equation (1)) behaves the same as the network with the unbounded function g (Equation (2)) in this case.

Case 2: If $k_{1}x<y^{*}<y$, $\frac{\partial L}{\partial y}>0$, $k_1$ increases, $\mu$ increases, and thus $y=-k_1\mu$ decreases and converges to $y^{*}$.

Case 3: If $y<y^{*}$, $\frac{\partial L}{\partial y}<0$, $k_1$ decreases, $\mu$ decreases, and thus $y=-k_1\mu$ increases and converges to $y^{*}$. Note that in case 2 and case 3, the output of BCPReLU converges to the same target $y^{*}$ faster than the corresponding unbounded function g.

If $x>\alpha$, then $y=k_2\alpha$, $k_1$ and $\mu$ are not updated. The analytical method is similar to that when $x<-\mu$. □

From Theorem 1, we know that BCPReLU has almost the same expressive power as the corresponding unbounded function.

3.3. Convergence Analysis

The proposed activation function limits the range of activation to $[-k_1\mu ,k_2\alpha ]$, and then the bounded range of output is linearly quantized to M bits as follows

$$y_q=round(y·\frac{2^M-1}{k_1\mu +k_2\alpha })·\frac{k_1\mu +k_2\alpha }{2^M-1}$$

During the training stage, $k_1$, $\mu$, $k_2$, and $\alpha$, are trainable variables. The gradient with respect to the involved parameters can be computed using the Straight-Through Estimator (STE) [37]. Thus,

$$\frac{\partial y_q}{\partial \mu }=\frac{\partial y_q}{\partial y}·\frac{\partial y}{\partial \mu }=\left\{\begin{array}{ccc}\hfill -k_1& & x\in (-\infty ,-\mu )\hfill \\ \hfill 0& & x\in [-\mu ,+\infty )\hfill \end{array}\right.$$

$$\frac{\partial y_q}{\partial k_1}=\frac{\partial y_q}{\partial y}·\frac{\partial y}{\partial k_1}=\left\{\begin{array}{ccc}\hfill -\mu & & x\in (-\infty ,-\mu )\hfill \\ \hfill x& & x\in [-\mu ,0)\hfill \\ \hfill 0& & x\in [0,+\infty )\hfill \end{array}\right.$$

$$\frac{\partial y_q}{\partial k_2}=\frac{\partial y_q}{\partial y}·\frac{\partial y}{\partial k_2}=\left\{\begin{array}{ccc}\hfill 0& & x\in (-\infty ,0)\hfill \\ \hfill x& & x\in [0,\alpha )\hfill \\ \hfill \alpha & & x\in [\alpha ,+\infty )\hfill \end{array}\right.$$

$$\frac{\partial y_q}{\partial \alpha }=\frac{\partial y_q}{\partial y}·\frac{\partial y}{\partial \alpha }=\left\{\begin{array}{ccc}\hfill 0& & x\in (-\infty ,\alpha )\hfill \\ \hfill k_2& & x\in [\alpha ,+\infty )\hfill \end{array}\right.$$

Next, we consider the quantization error of BCPRLU. Under a reasonable bell-curved distribution assumption [35], the convergence of quantization error is established as follows:

Theorem 2. 

Assume that activation input x is a random variable, with a probability density function $f\left(x\right)=$ Laplace$(0,b)$, and the quantization error of BCPReLU converges to zero when bit-width $M\to \infty .$

A proof of Theorem 2 will be given in Appendix A. This theorem shows that our proposed activation function using the common quantization method satisfies the property of quantization error; i.e., the quantization error converges to zero when the bit-width M approaches infinity ($\to \infty$).

4. Experiments

To demonstrate the effectiveness of BCPReLU, we evaluate it on several well-known models: ResNet20/32 [1] on the CIFAR10 dataset and ResNet18 [1] on the ImageNet dataset. The CIFAR-10 dataset contains 10 different class images, and each image is an RGB image $32\times 32$ in size. There are 50,000 training images and 10,000 testing images. The ImageNet dataset consists of 1000 classes with 1.28 million training images and 50 k validation images. The image from the ImageNet dataset is an RGB image $256\times 256$ in size. All experiments are based on the Pytorch framework. We used an NVIDIA Tesla K20c GPU for training and testing.

4.1. Equivalent Form of BCPReLU

In general, the more trainable parameters are the easier it is for overfitting to occur. To mitigate this problem, with the loss of generality, we consider the case where the convolutions and BCPReLU are fused to derive an equivalent form of BCPReLU. Considering a single-neuron network with BCPReLU, where $(b,y^{*})$ is a sample of training data and w denotes weight, then the activation input is denoted as $x=wb$, and the activation output is denoted as:

$$y=\left\{\begin{array}{ccc}\hfill -k_1\mu & & wb\in (-\infty ,-\mu )\hfill \\ \hfill k_{1}wb& & wb\in [-\mu ,0)\hfill \\ \hfill k_{2}wb& & wb\in [0,+\alpha )\hfill \\ \hfill k_2\alpha & & wb\in [\alpha ,+\infty )\hfill \end{array}\right.$$

$k_{2}w$, $k_2\mu$, and $k_2\alpha$ are denoted by $w^{*}$, ${\mu }^{*}$, and ${\alpha }^{*}$, respectively.

$$y=\left\{\begin{array}{ccc}\hfill -\frac{k_1}{k2}{\mu }^{*}& & w^{*}b\in (-\infty ,-{\mu }^{*})\hfill \\ \hfill \frac{k_1}{k2}{w}^{*}b& & w^{*}b\in [-{\mu }^{*},0)\hfill \\ \hfill w^{*}b& & w^{*}b\in [0,+{\alpha }^{*})\hfill \\ \hfill {\alpha }^{*}& & w^{*}b\in [{\alpha }^{*},+\infty )\hfill \end{array}\right.$$

The slope $k_2$ of the positive input can be integrated into the training of $k_1$, $\mu$, $\alpha$, and weight. The proposed clipping activation function is equivalent to the bounded form of PReLU [12]:

$$y=f(x;\mu ,k,\alpha )=\left\{\begin{array}{ccc}\hfill -k\mu & & x\in (-\infty ,-\mu )\hfill \\ \hfill kx& & x\in [-\mu ,0)\hfill \\ \hfill x& & x\in [0,\alpha )\hfill \\ \hfill \alpha & & x\in [\alpha ,+\infty )\hfill \end{array}\right.$$

(3)

The parameter k is a coefficient controlling the slope of the negative input, and $-k\mu$ and $\alpha$ control the lower bound and upper bound of y, respectively. Note that we use the equivalent function from Equation (3) in the following experiments.

4.2. Trainable Parameters

We have transformed the original function in a simpler way. We use $\alpha =10,k=0.25$, and $\mu =5$ as the initialization with regularization. To better understand the more flexible quantization scale, the learned parameters of BCPReLU are shown in

Table 1. The learned parameters of BCPReLU using a ResNet20 model on the CIFAR10 dataset.

Layer	Learned Parameters
	$\mathit{\alpha }$	$\mathit{k}$	$\mathit{\mu }$
Conv-1	1.5829	0.2151	0.5267
Conv-2	1.3588	0.2425	0.4372
Conv-3	1.1250	0.2336	0.3098
Conv-4	1.0106	0.1555	0.2061
Conv-5	1.5837	0.2528	0.4292
Conv-6	1.1262	0.3445	0.4317
Conv-7	2.0118	0.3648	0.3836
Conv-8	1.6620	0.2455	0.7877
Conv-9	1.3133	0.6584	2.4953
Conv-10	1.7046	0.2471	0.5996
Conv-11	2.9583	0.1456	0.9196
Conv-12	1.7303	0.3160	0.5303
Conv-13	4.3142	0.4195	1.3533
Conv-14	2.0902	0.1666	0.6147
Conv-15	3.1975	0.2384	0.9554
Conv-16	2.5048	0.3540	0.8222
Conv-17	5.2674	0.3605	1.9852
Conv-18	1.9860	0.3511	0.8580
Conv-19	2.2096	0.3535	0.9461

. It can be observed that the trainable parameter $\alpha$ converges to a much smaller value after epochs of training, and other learned parameters k and $\mu$ rarely have a magnitude larger than 1 in our experiment. Thus, our learned activation function limits the dynamic range of activations and reduces the quantization error, leading to better performance in terms of accuracy.

4.3. Comparison of PReLU and BCPReLU

For BCPReLU experiments, we only replace PReLU with BCPReLU but keep the other hyper-parameters the same in the full-precision network. Figure 3 shows the validation error of PReLU and BCPReLU on CIFAR10-ResNet20. The curves show that the BCPReLU has almost the same expressive ability as PReLU in the full-precision network. The performance of BCPReLU is consistent with the theoretical analysis from Theorem 1.

4.4. Different Bit-Width Quantization Performance

In this subsection, we compare the full-precision network and the low-bit (4,8-bit) network. The curves of validation error with different bit-width are shown in Figure 4. We observe that the validation error is gradually reduced as the bit width increases. The result is in line with the theoretical analysis in Theorem 2; i.e., when the bit-width M increases, the quantization error of BCPReLU decreases.

4.5. Accuracy Performance Comparison

To evaluate our method, we compare it with PACT [22] on several well-known CNNs: ResNet [1] for the CIFAR10 and ImageNet datasets. ResNet20/32 on the CIFAR10 dataset uses stochastic gradient descent (SGD) with a momentum of 0.9. The learning rate starts from 0.1 and is scaled by 0.1 at epochs 60 and 120. The mini-batch size of 128 is used, and the maximum number of epochs is 200. ResNet18 on the ImageNet dataset uses the same training parameters as PACT. During the training stage, the initial values of k, $\mu$, and $\alpha$ are set to 0.25, 5, and 10, respectively.

Table 2. The Comparison of top-1 accuracy between PACT and BCPReLU on CIFAR-10 and ImageNet datasets.

Dataset	Model	FullPrec	PACT			BCPReLU
			2b	4b	5b	2b	4b	5b
CIFAR-10	ResNet20	0.916	0.897	0.913	0.917	0.903	0.914	0.915
	ResNet32	0.923	0.907	0.921	0.921	0.912	0.921	0.922
ImageNet	ResNet18	0.702	0.644	0.692	0.698	0.652	0.695	0.698

summarizes quantization accuracy on CIFAR-10 and ImageNet datasets. As shown in Table 2, BCPReLU achieves competitive accuracy consistently across the networks. With 2-bit quantization, BCPReLU achieves higher accuracy. This is reasonable because the parameters of BCPReLU are adapt more flexibly to training than PACT.

5. Conclusions

In this paper, we have proposed a learnable clipping activation function to improve low-bit quantization performance, which dynamically determines the clipping range utilizing adaptive learning capability. By analyzing the representation power on the full-precision network and the convergence of the low-bit network, the quantized model accuracy can be improved while preserving the full-precision model’s accuracy. Extensive experiments on various classification datasets further demonstrate that our proposed activation function can achieve higher quantization accuracy in low-bit networks. In the future, we can also propose corresponding clipping methods for other types of activation functions, such as the sigmoid function. In addition, we plan to work on how to combine our quantization method with other acceleration algorithms to achieve higher compression performance.