Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

Abstract

Deep convolutional neural networks have a large number of parameters and require a significant number of floating-point operations during computation, which limits their deployment in situations where the storage space is limited and computational resources are insufficient, such as in mobile phones and small robots. Many network compression methods have been proposed to address the aforementioned issues, including pruning, low-rank decomposition, quantization, etc. However, these methods typically fail to achieve a significant compression ratio in terms of the parameter count. Even when high compression rates are achieved, the network’s performance is often significantly deteriorated, making it difficult to perform tasks effectively. In this study, we propose a more compact representation for neural networks, named Quantized Low-Rank Tensor Decomposition (QLTD), to super compress deep convolutional neural networks. Firstly, we employed low-rank Tucker decomposition to compress the pre-trained weights. Subsequently, to further exploit redundancies within the core tensor and factor matrices obtained through Tucker decomposition, we employed vector quantization to partition and cluster the weights. Simultaneously, we introduced a self-attention module for each core tensor and factor matrix to enhance the training responsiveness in critical regions. The object identification results in the CIFAR10 experiment showed that QLTD achieved a compression ratio of 35.43×, with less than 1% loss in accuracy and a compression ratio of 90.61×, with less than a 2% loss in accuracy. QLTD was able to achieve a significant compression ratio in terms of the parameter count and realize a good balance between compressing parameters and maintaining identification accuracy.

1. Introduction

Deep convolutional neural networks (DCNNs) have realized great success in many computer vision tasks, such as object identification [1,2,3], object detection [4,5,6] and image segmentation [7,8,9]. The design of DCNNs is becoming increasingly intricate, which is accompanied by a simultaneous enhancement in their performance. However, this progress comes at the cost of escalating the model complexity, leading to heightened storage space requirements. This trend indicates that while we strive for a more powerful performance, we also encounter challenges associated with managing larger models, notably, the increased demand for storage resources. Hence, many methods for compressing DCNNs have been proposed to address this issue and obtain light-weight networks with significantly fewer parameters, including pruning methods, low-rank tensor or matrix decomposition methods, and quantization methods.

Network pruning methods involve trimming away less important weight connections within a model, reducing the network’s complexity, and enhancing the model’s inference speed. Pruning at the layer, feature map, and kernel levels is called structured pruning [10,11], while weight pruning within the kernel itself is unstructured [12,13,14]. The primary advantage of weight pruning lies in the fact that weights are the smallest, most fundamental elements in a network. Therefore, a substantial pruning of these connections can be performed without significantly impacting the network performance. Additionally, widely-used deep learning frameworks such as PyTorch allow easy access to all parameters of a network, making the implementation of this method straightforward.

Pre-trained models often exhibit low-rank characteristics in their convolutional kernels, making it possible to leverage low-rank decomposition methods for model compression. In low-rank tensor or matrix decomposition methods, the weights of convolutional or fully connected layers are compressed via a low-rank tensor or matrix decomposition. By employing such approaches, the parameter count of DCNNs is substantially reduced, leading to a significant improvement in their inference speed.

Network quantization is a method of saving the network storage space by reducing the number of bits required to store weights. Quantization methods are typically categorized into low-bit representations and weight-sharing methods. Weight-sharing quantization methods, which primarily employ clustering techniques, cluster different weights into several clusters representing all weights within a cluster with the centroid of that cluster. During an inference calculation, it is only necessary to look up the index of the centroid to obtain the corresponding weight value.

Even though these DCNN compression methods reduce the parameter count and accelerate the inference speed of networks, they face the challenge of a significant decline in accuracy when employing extremely large compression ratios of over 50× the parameter count. In this study, we propose a Quantized Low-Rank Tensor Decomposition (QLTD) method with self-attention to achieve an extremely large compression rate with only a minimal loss in accuracy. The main scheme of our QLTD method is shown in Figure 1. As pre-trained weights in DCNNs often contain a significant amount of redundant information [15], firstly, we conducted a Tucker-2 decomposition on the pre-trained weights of the convolutional layers. Meanwhile, we introduced a self-attention module on each core tensor and factor matrix through the Tucker-2 decomposition to obtain and focus on the key positions, which mitigated performance losses arising from the permutation and quantization. The self-attention module was a trainable and sparse convolutional layer. Then, to further mine the redundancy in the core tensor and factor matrices, we applied a permutation and quantization approach [16] using concepts from the rate-distortion theory. Finally, the compressed network was fine-tuned for several epochs to recover the accuracy. The identification results in the CIFAR-10 experiment showed that our approach achieved a compression ratio of 35.43× with less than 1% loss in accuracy and a compression ratio of 90.61× with less than 2% loss in accuracy. Additionally, depending on the requirements, our method can achieve a parameter compression ratio of up to 200 times.

Our contributions can be summarized as follows:

(i) We propose a QLTD method with self-attention that is easy to integrate into DCNNs to extremely compress them. Our method incurs minimal losses in the identification accuracy at a parameter compression ratio below 100 times and can achieve a parameter compression ratio of up to 200 times if needed.

(ii) Our QLTD framework unifies network pruning, low-rank decomposition and quantization compression methods to fully leverage their advantages. The framework compresses DCNNs by sequentially performing Tucker decomposition, permutation, and quantization. Meanwhile, a sparse self-attention sparse layer is designed to obtain and focus on the key positions of each core tensor and factor matrix through Tucker decomposition.

2. Related Work

Here, we review the related work on DCNN compression methods, including network pruning methods, low-rank decomposition methods and network quantization methods.

Network pruning. Structured pruning techniques focus on zeroing out organized groups of convolutional kernels at the layer, feature map, and kernel levels. Fang et al. [17] proposed a universal automated method named DepGraph for the structural pruning of neural network architectures. This method explicitly modeled two types of dependencies (inter-layer and intra-layer), constructed a dependency graph, and introduced the general structural pruning through sparse training that constrained the parameter grouping. This approach facilitated the universal structural pruning. Despite compressing the parameter count and accelerating the inference speed, these approaches led to a decline in the identification accuracy.

Unstructured pruning methods involve deactivating connections associated with small weights or applying sparsity regularization to the weights. DEEP-R [18] takes a Bayesian perspective and performs sampling for pruning and regrowth decisions. Sparse evolutionary training (SET) [19] simplifies pruning-regrowth cycles by pruning the smallest and most negative weights and growing new weights in random locations. Dynamic Sparse Reparameterization (DSR) [20] utilizes a pruning–redistribution–regrowth cycle and addresses the limitations of previous techniques, such as their high computation costs and the use of manual configuration for the number of free parameters allocated to each layer. Sparse networks from scratch [21] are more general and use a fully sparse setting. Soft threshold weight reparameterization (STR) [22] smoothly induces the sparsity while learning pruning thresholds. The authors of [23] proposed a novel adversarial training method called inverse weight inheritance, which imposed sparse weight distribution on a large network by inheriting weights from a small network, thereby improving the robustness of the large network. However, these methods are not able to realize an extremely large parameter compression ratio.

Low-rank decomposition. Low-rank matrix and tensor decomposition methods can be directly applied to compress DCNNs. Canonical polyadic (CP) decomposition [24] was proposed to compress the number of parameters and speed up networks. Tucker decomposition [25] was proposed to compress convolutional layers for fast and low-power mobile applications. Tensor-train (TT) decomposition [26] was proposed and found to be effective in solving dense connection problems to avoid the curse of dimensionality. Tensor-ring decomposition [27] can be viewed as a linear combination of TT decomposition, in which the cyclic permutation invariance of the latent cores is maintained through the use of tracing operations and the equitable treatment of potential cores. These methods directly apply a low-rank decomposition to the weights of DCNNs. CP and Tucker decomposition cannot achieve large compression ratios. While the TT and TR decomposition achieve large compression ratios over 10×, they result in a significant decrease in accuracy.

Some other approaches have made improvements to low-rank decomposition methods. Jaderberg et al. [28] proposed a linear combination of a smaller basis set of 2D separable filters to approximate a 2D filter set to speed up the evaluation of convolutional neural networks. The authors of [29] proposed the automatic selection of ranks in a recent study of tensor-ring decomposition in each convolutional layer, which was inspired by reinforcement learning. The authors of [30] derived filter pruning and low-rank decomposition by simply changing the way in which the sparsity regularization was enforced. The authors of [31] proposed a systematic framework for the tensor–decomposition-based model compression using the Alternating Direction Method of Multipliers (ADMM). The Trained Rank Pruning (TRP) [32] alternated between the low-rank approximation and training, and the low-rank approximation and regularization were integrated into the training process. The authors of [33] showed that, with a suitable formulation, determining the optimal rank of each layer was amenable to a mixed discrete continuous optimization jointly over the ranks and matrix elements. The authors of [34] proposed a novel compact design of convolutional layers with a spatial transformation towards a lower-rank representation, and they applied trainable spatial transformations to low-rank convolutional kernels in a predefined Tucker product form to enhance the versatility of convolutional kernels. These approaches can improve the identification accuracy of a compressed network to some extent. However, they cannot achieve very large compression ratios of over 30×.

The original intention of low-rank decomposition methods was to accelerate the network inference and reduce the network storage. The common issue with low-rank decomposition is the difficulty in achieving high compression ratios of over 30×. Our method is based on the Tucker decomposition. Apart from the fact that the Tucker decomposition method cannot achieve very high compression ratios, it is more important to find redundant information in the core tensor and factor matrices obtained through the Tucker decomposition, as these can be further exploited through permutation and quantization methods. Through our QLTD method, network weights achieve a more fundamentally lightweight representation and a high parameter compression ratio of over 30×. The experiment in Section 4.4 “Ablation Study” validates that our QLTD method realizes a better balance between the parameter compression ratio and identification accuracy.

Network quantization. Quantization methods based on weight sharing primarily aggregate different weights into several clusters through techniques such as clustering, and the centroid of each cluster is used to represent the values of all weights within that cluster. During the inference calculation, one only needs to look up the index of the centroid to obtain the corresponding weight value. Stock et al. [35] proposed a vector quantization method based on product quantization (PQ), focusing on the importance of activation values rather than weights. They combined this with knowledge distillation and minimized the reconstruction error between teacher and student networks to achieve compression. The “Permute, Quantize, and Fine-Tune” method [16] establishes a connection with a rate-distortion theory to search for permutations that make a network more amenable to compression. Through a quantization algorithm that was subjected to annealing, this method achieved a high network compression ratio and identification accuracy. The authors of [36] introduced an adaptive method for a data-free quantization (AdaDFQ), and they treated the training process of a generative network in generator-based data-free quantization (GDFQ) as a zero-sum game. This method optimized the adaptability of the generative network by constructing boundaries between inconsistent and consistent samples. The authors of [37] addressed the problem of the module recovery loss oscillation in post-training quantization (PTQ) methods. They proposed a solution by introducing the concept of module topological homogeneity to optimize modules with significantly different capacities, thereby reducing the recovery loss of quantized networks. The authors of [38] presented a zero-shot quantization (ZSQ) method called HAST. It enhanced the difficulty of matching synthetic samples for a quantization model by elevating the difficulty of generating pseudo-samples. This approach mitigated the issue of synthetic samples being easy to fit and ensured a similarity between the quantization model and the full-precision model through feature alignment. While these methods achieve large compression ratios, they often come with a substantial decrease in accuracy.

3. Quantized Low-Rank Tensor Decomposition with Self-Attention

3.1. Preliminary Notations for Low-Rank Tensor Decomposition

A tensor is a multi-dimensional array. A d-order tensor represents a d dimensional multi-way array. Scalars, vectors and matrices are zero-order, one-order and two-order tensors, respectively. In this study, we denote scalars, vectors and matrices with lowercase letters $(x,y,z,\cdots )$, bold lowercase letters $(\mathbf{x},\mathbf{y},\mathbf{z},\cdots )$ and uppercase letters $(X,Y,Z,\cdots )$, respectively. An N-order tensor ($\mathrm{N}\ge 3$) is denoted as $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$, and each element is denoted as $x_{i_1,i_2,\cdots ,i_N}$.

The mode-n matrix form of a tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ is the operation of reshaping a tensor into a matrix $X_{\left(n\right)}\in {\mathbb{R}}^{I_n\times (I_1\cdots I_{n-1}{I}_{n+1}\cdots I_N)}$. For a tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$, its Tucker decomposition is defined as:

$$\begin{array}{cc}\hfill \mathcal{X}& \hfill =\sum _{s_1=1}^{S_1}\times \cdots \times \sum _{s_N}^{S_N}{g}_{s_1{s}_2\dots s_N}(u_{s_1}^{\left(1\right)}\circ u_{s_2}^{\left(2\right)}\circ \cdots \circ u_{s_N}^{\left(N\right)})\hfill \\ \hfill & \hfill =\mathcal{G}{\times }_1{U}^{\left(1\right)}{\times }_2{U}^{\left(2\right)}\cdots {\times }_N{U}^{\left(N\right)},\hfill \end{array}$$

(1)

where $\mathcal{G}\in {\mathbb{R}}^{S_1\times S_2\times \cdots \times S_N}$ denotes the core tensor and $U^{\left(n\right)}=[u_1^{\left(n\right)},u_2^{\left(n\right)},\cdots ,u_N^{\left(n\right)}]\in {\mathbb{R}}^{I_n\times {\mathrm{S}}_n}$ denotes a factor matrix.

3.2. Low-Rank Decomposition on Pre-Trained Weights

A convolution operation maps an input tensor $\mathcal{X}\in {\mathbb{R}}^{H\times W\times C_{in}}$ to an output tensor $\mathcal{Y}\in {\mathbb{R}}^{H^{\prime }\times W^{\prime }\times C_{out}}$ with the following equation:

$$\begin{array}{c}\hfill {\mathcal{Y}}_{h^{\prime },w^{\prime },o}=\sum _{h=1}^H\sum _{w=1}^W\sum _{i=1}^S{\mathcal{W}}_{h-h^{\prime }+1,w-w^{\prime }+1,i,o}{\mathcal{X}}_{h,w,i},\end{array}$$

(2)

where $H\times W$ are the height and width of the input feature map, $C_{in}$ is the number of channels of the input feature map, $H^{\prime }\times W^{\prime }$ are the height and width of the output feature map, $C_{out}$ is the number of channels of the output feature map, and ${\mathcal{Y}}_{h^{\prime },w^{\prime },o}$ denotes a voxel of $\mathcal{Y}$.

$\mathcal{W}\in {\mathbb{R}}^{d\times d\times C_{in}\times C_{out}}$ is a four-order tensor, where $d\times d$ represent the filter width and height, $C_{in}$ is the number of input channels, and $C_{out}$ is the number of output channels. It can be decomposed via theTucker decomposition [39] in Equation (1), as follows:

$$\begin{array}{c}\hfill {\mathcal{W}}_{i,j,s,t}=\sum _{r_1=1}^{R_1}\sum _{r_2=1}^{R_2}\sum _{r_3=1}^{R_3}\sum _{r_4=1}^{R_4}{\mathcal{G}}_{r_1,r_2,r_3,r_4}^{\prime }{U}_{i,r_1}^{\left(1\right)}{U}_{j,r_2}^{\left(2\right)}{U}_{s,r_3}^{\left(3\right)}{U}_{t,r_4}^{\left(4\right)},\end{array}$$

(3)

where ${\mathcal{W}}_{i,j,s,t}$ denotes a voxel of $\mathcal{W}$, $\mathcal{G}$ is the core tensor of size $(R_1\times R_2\times R_3\times R_4)$, and $U^{\left(1\right)}$, $U^{\left(2\right)}$, $U^{\left(3\right)}$ and $U^{\left(4\right)}$ are factor matrices of sizes $(d\times R_1)$, $(d\times R_2)$, $(C_{in}\times R_3)$, and $(C_{out}\times R_4)$, respectively.

As the kernel size $(d\times d)$ of a DCNN is always $(3\times 3)$ or $(1\times 1)$, which is too small to decompose, we only decompose the input and output channels. Then, Equation (3) is reformulated into a variant form called the Tucker-2 decomposition, as follows:

$$\begin{array}{c}\hfill {\mathcal{W}}_{i,j,s,t}=\sum _{r_3=1}^{R_3}\sum _{r_4=1}^{R_4}{\mathcal{G}}_{i,j,r_3,r_4}{U}_{s,r_3}^{\left(3\right)}{U}_{t,r_4}^{\left(4\right)}.\end{array}$$

(4)

After the Tucker-2 decomposition, the ordinary convolution operation in Equation (2) can be replaced with three sequential convolution operations [25] that use the core tensor and factor matrices as convolution kernels. The three convolution operations are formulated, as follows:

$$\begin{array}{ccc}\hfill {\mathcal{Z}}_{h,w,r_3}& \hfill =\hfill & \sum _{s=1}^{C_{in}}{\mathcal{U}}_{s,r_3}^{\left(3\right)}{\mathcal{X}}_{h,w,s},\hfill \\ \hfill {\mathcal{Z}}_{h^{\prime },w^{\prime },r_4}^{\prime }& \hfill =\hfill & \sum _{i=1}^d\sum _{j=1}^d\sum _{r_3=1}^{R_3}{\mathcal{G}}_{i,j,r_3,r_4}{\mathcal{Z}}_{h_i,w_j,r_3},\hfill \\ \hfill {\mathcal{Y}}_{h^{\prime },w^{\prime },t}& \hfill =\hfill & \sum _{r_4=1}^{R_4}{\mathcal{U}}_{t,r_4}^{\left(4\right)}{\mathcal{Z}}_{h^{\prime },w^{\prime },r_4}^{\prime },\hfill \end{array}$$

(5)

where ${\mathcal{Z}}_{h,w,r_3}$ and ${\mathcal{Z}}_{h^{\prime },w^{\prime },r_4}^{\prime }$ are two intermediate feature maps. Equations (2) and (5) are equivalent. The advantages are that the total parameter count of the three sequential convolution operations is much smaller than that of the original convolution operation, and the inference speed is much faster.

3.3. Permuting the Kernels

Vector quantization inevitably comes with a large reconstruction error, so the accuracy of identification drops significantly after vector quantization. The reason for that is that the sub-vectors have a rich diversity, and they are not able to be well represented by the centroid.

To address this problem, we search for permutations applied to the low-rank convolutional kernels of each layer, which results in sub-vectors that are easier to quantize, by minimizing the determinant of the covariance of the resulting sub-vectors [16]. It is worth mentioning that the network is invariant under the permutation of its weights, as long as the same permutation is applied to the output dimension for parent layers and the input dimension for children layers. We refer the reader to Equations (4)–(16) of paper [16] for more details about how to find the permutation matrix for each layer.

3.4. Vector Quantization

In this section, we introduce how to apply the vector quantization to the low-rank convolution kernels ${\mathcal{U}}^{\left(3\right)},\mathcal{G}$, and ${\mathcal{U}}^{\left(4\right)}$.

We learn an encoding $\beta$ that takes considerably less memory to store the low-rank convolution kernels, and it consists of a codebook ${\mathcal{C}}_b$ and a set of codes $\mathcal{B}$. Taking ${\mathcal{U}}^{\left(3\right)}\in {\mathbb{R}}^{C_{in}\times R_3}$ as an example, we need to split the matrix into column sub-vectors $u_{i,j}\in {\mathbb{R}}^{d\times 1}$. Equation (6) shows a divided factor matrix.

$$\begin{array}{c}\hfill {\mathcal{U}}^{\left(3\right)}=\left[\begin{array}{cccc}{u}_{11}& \cdots & \cdots & u_{1R_3}\\ u_{21}& \cdots & \cdots & u_{2R_3}\\ ⋮& ⋮& \ddots & ⋮\\ u_{\widehat{C_{in}}1}& \cdots & \cdots & u_{\widehat{C_{in}}{R}_3}\end{array}\right],\end{array}$$

(6)

where $\widehat{C_{in}}=C_{in}/d$. Notably, $\mathcal{G}$ needs to be reshaped into a 2D matrix before being divided. Instead of storing all of the sub-vectors in Equation (6), we apply K-means to the sub-vectors to obtain k centroids and approximate them with a smaller set, which we call the codebook ${\mathcal{C}}_b$ for the layer, and all centroids are stored, as follows:

$$\begin{array}{cc}\hfill {\mathcal{C}}_b& \hfill =[c\left(1\right),\dots ,c\left(k\right)]\in {\mathbb{R}}^{d\times 1}.\hfill \end{array}$$

(7)

Each sub-vector $u_{i,j}$ is assigned to a centroid by measuring the minimum Euclidean distance as $b_{i,j}\in (1,\cdots ,k)=\underset{t}{argmin}{\parallel u_{i,j}-c\left(t\right)\parallel }_2^2$, which is the index of the element in ${\mathcal{C}}_b$ that is closest to $u_{i,j}$ in the Euclidean space.

Codes $\mathcal{B}$ store the index of each sub-vector as

$$\begin{array}{cc}\hfill \mathcal{B}& \hfill =\left[\begin{array}{cccc}{b}_{11}& \cdots & \cdots & b_{1R_3}\\ b_{21}& \cdots & \cdots & b_{2R_3}\\ ⋮& ⋮& \ddots & ⋮\\ b_{\widehat{C_{in}}1}& \cdots & \cdots & b_{\widehat{C_{in}}{R}_3}\end{array}\right]\in {\mathbb{R}}^{\widehat{C_{in}}\times R_3}.\hfill \end{array}$$

(8)

Then, the low-rank convolutional kernel ${\mathcal{U}}^{\left(3\right)}$ can be reconstructed by decoding ${\mathcal{C}}_b$ and $\mathcal{B}$ for the forward propagation, as follows:

$$\begin{array}{c}\hfill {\mathcal{U}}^{\left(3\right)}=\left[\begin{array}{cccc}c\left(b_{11}\right)& \cdots & \cdots & c\left(b_{1R_3}\right)\\ c\left(b_{21}\right)& \cdots & \cdots & c\left(b_{2R_3}\right)\\ ⋮& ⋮& \ddots & ⋮\\ c\left(b_{\widehat{C_{in}}1}\right)& \cdots & \cdots & c\left(b_{\widehat{C_{in}}{R}_3}\right)\end{array}\right].\end{array}$$

(9)

The schematic diagrams for permutation and vector quantization are shown in Figure 2.

3.5. Self-Attention Module

To further recover from the decrease in accuracy caused by the quantization and permutation of low-rank convolutional kernels, we introduce a self-attention module to focus on the key positions of the low-rank convolutional kernels. Taking ${\mathcal{U}}^{\left(3\right)}\in {\mathbb{R}}^{C_{in}\times R_3}$ as an example, to retain the important information of ${\mathcal{U}}^{\left(3\right)}$, we define a self-attention module ${\mathcal{S}}^{{\mathcal{U}}^{\left(3\right)}}$ and preset the self-attention percentage s of three levels 0.1%, 0.5%, 1% to focus on the key positions of ${\mathcal{U}}^{\left(3\right)}$ that have large absolute values. The self-attention module is a sparse layer whose weight is a trainable sparse tensor that has the same dimensions as the weight of ${\mathcal{U}}^{\left(3\right)}$. The sparse weight of ${\mathcal{S}}^{{\mathcal{U}}^{\left(3\right)}}$ is initialized, as follows.

We first sort the elements in ${\mathcal{U}}^{\left(3\right)}$ by their absolute values. Let $t=\lceil C_{in}\times R_3\times s\rceil$, and the truncation value $\lambda$ is the absolute value of the element with the $t^{th}$ largest absolute value in ${\mathcal{U}}^{\left(3\right)}$. For any position $S_{ij}^{{\mathcal{U}}^{\left(3\right)}}$ of ${\mathcal{S}}^{{\mathcal{U}}^{\left(3\right)}}$, it is initialized as:

$$\begin{array}{c}\hfill S_{ij}^{{\mathcal{U}}^{\left(3\right)}}=\mathrm{sgn}\left({\mathcal{U}}_{ij}^{\left(3\right)}\right)max\left\{\right|{\mathcal{U}}_{ij}^{\left(3\right)}|-\lambda ,0\}.\end{array}$$

(10)

In addition, we design a (0,1) mask that has the same dimension as that of the sparse weight. For the positions of the sparse weights where the values are 0, the values of the corresponding positions on the mask are also 0. The values of the other positions on the mask are set to 1. The mask is multiplied to the sparse weights to block the gradient back-propagation and training of the positions with a value of 0.

3.6. The Overall Framework of QLTD

We summarize the overall framework of our QLTD method in Algorithm 1.    

Algorithm 1 The overall framework of QLTD

Input:  weights $\mathcal{W}\in {\mathbb{R}}^{d\times d\times C_{in}\times C_{out}}$, the channel compression ratio of the Tucker decomposition r, the length of subvectors in vector quantization clustering d, the number of cluster centroids k, and the self-attention percentage s. Output:  Codebook ${\mathcal{C}}_b^{{\mathcal{U}}^{\left(3\right)}}$, ${\mathcal{C}}_b^{{\mathcal{U}}^{\left(4\right)}}$ and ${\mathcal{C}}_b^{\mathcal{G}}$, code ${\mathcal{B}}^{{\mathcal{U}}^{\left(3\right)}}$, ${\mathcal{B}}^{{\mathcal{U}}^{\left(4\right)}}$ and ${\mathcal{B}}^{\mathcal{G}}$, the initial weight of the self-attention module ${\mathcal{S}}^{{\mathcal{U}}^{\left(3\right)}}$, ${\mathcal{S}}^{{\mathcal{U}}^{\left(4\right)}}$ and ${\mathcal{S}}^{\mathcal{G}}$. 1: Obtain ${\mathcal{U}}^{\left(3\right)},\mathcal{G}$ and ${\mathcal{U}}^{\left(4\right)}$ by applying Tucker-2 decomposition to $\mathcal{W}\in {\mathbb{R}}^{d\times d\times C_{in}\times C_{out}}$ as Equation (4). 2: Split ${\mathcal{U}}^{\left(3\right)}$ to sub-vectors by $\widehat{C_{in}}=C_{in}/d$ as Equation (6). Cluster the sub-vectors to k centroids by K-means and obtain the codebook ${\mathcal{C}}_b^{{\mathcal{U}}^{\left(3\right)}}$ and code ${\mathcal{B}}^{{\mathcal{U}}^{\left(3\right)}}$ as Equations (7) and (8). 3: Similarly, obtain the codebook ${\mathcal{C}}_b^{{\mathcal{U}}^{\left(4\right)}}$ and code ${\mathcal{B}}^{{\mathcal{U}}^{\left(4\right)}}$ of ${\mathcal{U}}^{\left(4\right)}$. 4: Similarly, obtain the codebook ${\mathcal{C}}_b^{\mathcal{G}}$ and code ${\mathcal{B}}^{\mathcal{G}}$ of $\mathcal{G}$. 5: Apply the same permutation to the output dimension for parent layers and the input dimension for children layers. 6: Obtain the initial weight of the self-attention module ${\mathcal{S}}^{{\mathcal{U}}^{\left(3\right)}}$ from ${\mathcal{U}}^{\left(3\right)}$ and the self-attention percentage s as Equation (10). 7: Similarly, obtain the initial weight of the self-attention module ${\mathcal{S}}^{{\mathcal{U}}^{\left(4\right)}}$. 8: Similarly, obtain the initial weight of the self-attention module ${\mathcal{S}}^{\mathcal{G}}$.

4. Experiment

In this section, we validate our proposed approach on three datasets, including CIFAR10, CIFAR100 [40], and ImageNet [41]. To validate the effectiveness of our approach, we compare it with several recent low-rank tensor decomposition approaches, vector quantization approaches, scalar quantization approaches and network pruning methods. The low-rank tensor decomposition approaches include SVD [42], LCT [34], TDNR [43], HALOC [44], Maestro [45] and ELRT [46]. The vector quantization approaches include PQF [16] and BGD [47]. The scalar quantization approaches include ABC-Net [48], DC [12], LR-Net [49], HAQ [50] and BWN [51]. The network pruning methods include CLIP-Q [52], TRP [32] and SSS [53].

4.1. Training Settings

Compressing each layer independently causes errors in the activations to accumulate, resulting in a degradation of performance. To address this problem, the compressed model is fine-tuned to recover its performance. During fine-tuning, we fix the codes and permutations and only fine-tune the centroids. Notably, each centroid in the compressed network is differentiable. Therefore, the centroids will be updated, as follows:

$$\begin{array}{c}\hfill c(i+1)=c\left(i\right)-\eta \frac{\partial L}{\partial c\left(i\right)},\end{array}$$

(11)

where i is the step of back propagation, L is the original loss function and $\eta$ is the learning rate.

All of the compressed models were fine-tuned with an Adam optimizer. For the CIFAR-10 and CIFAR-100 experiments, we set the initial learning rate to ${10}^{-3}$ and decreased it by a factor of 10 every 20 epochs (50 epoch in total). For the ImageNet experiments, the initial learning rate was ${10}^{-5}$ and decreased by a factor of 10 every 5 epochs (15 epochs in total). The batch size was 256. We implemented our approach with Pytorch, and all of the following experiments were run on a single Nvidia GeForce RTX 3090 GPU.

4.2. CIFAR-10 and CIFAR-100 Experiment

The CIFAR-10 and CIFAR-100 datasets consisted of colored natural images with 32 × 32 pixels in 10 and 100 classes, respectively. Each dataset contained 50k training images and 10k testing images. All of the CIFAR-10 and CIFAR-100 experiments that followed used the data augmentation method provided in [54] for training: 4 pixels with a value of 0 were padded on each side for a 32 × 32 image, and a 32 × 32 crop was randomly sampled from the padded image or its horizontal flip. We only evaluated the original 32 × 32 images for testing.

We chose a popular DCNN ResNet-18 [55] as the baseline network. The uncompressed baseline ResNet-18 model had 11.2 M parameters and reached 95.09% Top-1 accuracy on the CIFAR-10 dataset and 75.58% Top-1 accuracy on the CIFAR-100 dataset. We applied our compression scheme to all the convolutional layers of ResNet-18 except for the first convolutional layer, as it only had three input channels, making it too small to compress. In order to achieve similar compression ratios with the different network compression approaches used for comparison, we compressed the model under different compression regimes. The compression ratio of the network could be controlled by adjusting the following parameters: (1) the channel compression rate of Tucker decomposition, (2) the length of subvectors in vector quantization clustering, (3) the number of cluster centers in vector quantization clustering and (4) the sparsity of the self-attention module. It is worth mentioning that our approach was able to realize an extremely high compression ratio of over 90×.

Table 1. Experimental results of ResNet-18 on the CIFAR10 dataset. We report the Top-1 accuracy and compression ratios of our approach and the approaches used for comparison. For our approach, we provide the compression scheme. For example, “2-4-256-0.5” denotes that: (1) the channel compression rate of the Tucker decomposition is 2, (2) the length of subvectors in the vector quantization clustering is 4, (3) the number of cluster centers in the vector quantization clustering is 256 and (4) the sparsity of the self-attention module is 99.5%. The following tables follow the same notation.

Method	Compression Scheme	Top-1 Accuracy (%)	Com. Ratio (×)
Maestro [45]	-	93.86	9.1
TDNR [43]	-	92.82	11.82
PQF [16]	-	93.77	40.97
		93.21	60.23
		93.15	79.34
QLTD (ours)	2-2-256-0.5	94.15	35.43
	2-4-256-0.5	93.87	48.13
	4-2-256-0.1	93.51	72.07
	4-4-256-1	93.24	90.61

shows the experimental results of ResNet-18 on the CIFAR10 dataset for our approach and the approaches used for comparison. The uncompressed baseline ResNet-18 model reached 95.09% Top-1 accuracy. Our approach was able to achieve an accuracy drop of less than 1% while achieving a compression ratio of 35.43×. As the PQF [16] used quantization and permutation to compress the network, we conducted a more detailed compared experiment with it. The results showed that our approach achieved a better trade-off between the compression ratio and Top-1 accuracy than PQF did. The results of the last three rows from our approach showed that it achieved a compression ratio that was 10× higher and a higher Top-1 accuracy than those of the corresponding results in the three rows from the PQF approach [16]. Conventional compression approaches such as Maestro [45] and TDNR [43] were only able to achieve a compression ratio of about 10×, and their accuracy was not as high as that of our QLTD approach.

Table 2. Experimental results of ResNet-18 on the CIFAR100 dataset. We report the Top-1 accuracy and compression ratios of our approach and the PQF approach [16]. For our approach, we provide the compression scheme.

Method	Compression Scheme	Top-1 Accuracy (%)	Com. Ratio (×)
PQF [16]	-	72.68	22.73
		70.85	45.57
		69.97	75.12
QLTD (ours)	2-2-256-0.1	74.07	36.27
	4-2-256-0.5	71.59	65.86
	4-4-256-0.5	70.35	88.51

shows the experimental results of ResNet-18 on the CIFAR100 dataset with our approach and the PQF approach. The uncompressed baseline ResNet-18 model reached a Top-1 accuracy of 75.58%. We conducted a more detailed experiment with the PQF approach [16]. The results showed that our approach achieved a better balance between the compression ratio and Top-1 accuracy than PQF did. Our approach was able to reach an accuracy of 74.07% while compressing the parameters by 36.27×. Even when our approach achieved a compression ratio as high as 88.51×, it still maintained an identification accuracy of over 70%.

4.3. ImageNet Experiment

We extensively validate our approach on the most popular benchmark ImageNet. ImageNet contains 1.28 million training images and 50 thousand validation images of 1000 different classes. These images cover various categories found in everyday life. The data augmentation for training included a random resized cropping and a random horizontal flip. The image was center-cropped to match the input size for validation.

We chose the pre-trained ResNet-18 and ResNet-50 from the Pytorch model zoo as baseline networks. The baseline ResNet-18 and ResNet-50 achieved a 69.1% and 76.15% identification accuracy on the ImageNet dataset, respectively. We compressed the baseline ResNet-18 and ResNet-50 and conducted a series of experiments.

Table 3. Experimental results of ResNet-18 on the ImageNet dataset. We report the Top-1 accuracy and compression ratios of our approach and the approaches used for comparison. For our approach, we provide the compression scheme.

Method	Compression Scheme	Top-1 Accuracy (%)	Com. Ratio (×)
HALOC [44]	-	70.65	2.75
LCT [34]	-	67.87	2.57
ELRT [46]	-	68.65	2.17
TRP [32]	-	65.51	2.59
SVD [42]	-	63.10	1.41
ABC-Net [48]	-	62.8	32.05
LR-Net [49]	-	59.9	31.89
BGD [47]	-	64.12	35.21
		61.17	43.23
PQF [16]	-	65.23	35.14
		59.87	56.74
		58.92	59.53
QLTD (ours)	2-2-256-0.1	65.32	36.72
	4-2-512-0.5	61.57	54.35
	4-2-256-1	59.85	63.41

shows the experimental results of ResNet18 on the ImageNet dataset with our approach and the approaches used for comparison. The HALOC [44], LCT [34], and ELRT [46] approached the aim to compress the parameter count while maintaining identification accuracy and, thus, only achieved a compression ratio of less than 3×. Direct low-rank decomposition and pruning approaches such as TRP [32] and SVD [42] were not able to achieve high compression ratios, and their accuracy noticeably decreased. ABC-Net [48] and LR-Net [49] were able to achieve high compression ratios, but the identification accuracy also noticeably decreased. BGD [47] and PQF [16] achieved a good identification accuracy while achieving relatively high compression ratios for the parameter count. Compared with PQF [16], our approach achieved a higher identification accuracy with a similar compression ratio for the parameter count. To sum up, our approach achieved the best balance between compressing parameters and maintaining identification accuracy among all of the approaches shown in Table 3.

Table 4. Experimental results of ResNet-50 on the ImageNet dataset. We report the Top-1 accuracy and compression ratio of our approach and the approaches used for comparison. For our approach, we provide the compression scheme.

Method	Compression Scheme	Top-1 Accuracy (%)	Com. Ratio (×)
SSS [53]	-	72.98	1.20
TRP [32]	-	72.69	2.30
SVD [42]	-	71.80	1.50
CLIP-Q [52]	-	73.77	14.9
HAQ [50]	-	70.63	15.2
DC [12]	-	68.9	15.18
BGD [47]	-	74.81	15.2
		71.53	25.91
PQF [16]	-	75.42	16.82
		70.22	29.37
		69.13	32.16
QLTD (ours)	2-4-256-0.5	75.77	18.53
	4-2-256-1	72.25	34.24
	4-2-256-0.1	71.46	35.72

shows the experimental results of ResNet-50 on the ImageNet dataset with our approach and the approaches used for comparison. ResNet-50 used a bottleneck block that included two $1\times 1$ convolutional layers and one $3\times 3$ convolutional layer. The $1\times 1$ convolutional layer was not very suitable for compression with approaches based on low-ranking decomposition, pruning, and scalar quantization. Therefore, SSS [53], TRP [32], SVD [42], CLIP-Q [52], HAQ [50], HAQ [50], and DC [12] did not achieve very high compression ratios, and the accuracy significantly decreased compared with that of the baseline ResNet-50. The vector quantization approaches, BGD [47] and PQF [16], achieved very high accuracies of 74.81% and 75.42%, respectively, with a compression ratio of around 15× in terms of the parameter count. Compared with BGD [47] and PQF [16], our approach achieved a higher identification accuracy and compression ratio at the same time. Notice that our approach achieved a compression ratio of 18.53× with only an accuracy drop of 0.38% compared with the baseline ResNet-50.

4.4. Ablation Study

We studied the impact on our QLTD approach with/without the (1) Tucker decomposition, (2) quantization, (3) permutation, and (4) self-attention module. For the sake of discussion, we kept the following: (1) the channel compression rate of the Tucker decomposition at 2, (2) the length of sub-vectors in the vector quantization clustering at 2, (3) the number of cluster centers in the vector quantization clustering at 256, and (4) the sparsity of the self-attention module at 99.5%, if the corresponding scheme was used.

We chose the CIFAR-10 dataset and ResNet-18 as the baseline network.

Table 5. Experimental results of the ablation study of our proposed method on the CIFAR-10 dataset. A “✓” notation denotes that the corresponding module is utilized.

Tucker	Quantization	Permutation	Self-Attention	Com. Ratio	Accuracy
✓				3.75	94.47
✓	✓			37.33	92.57
✓	✓	✓		37.33	93.26
✓			✓	3.66	94.81
✓	✓		✓	35.43	93.38
✓	✓	✓	✓	35.43	94.15

shows the results of our QLTD approach with different schemes. In the table, we can observe the following conclusions: (1) Significant parameter compression relies primarily on quantization. (2) Permutation does not introduce parameters and can even enhance the identification accuracy. (3) The self-attention module introduces a small number of parameters, but it contributes significantly to improving the accuracy. (4) When not aiming for an extremely high compression ratio, using only the Tucker decomposition can reduce the loss caused by compression. To sum up, the proposed QLTD approach achieves the best trade-off between the parameter compression ratio and identification accuracy when utilizing all these modules.

5. Discussion

In this section, we discuss the compression potential and limitations of our QLTD method. When using the Tucker decomposition to compress tensors, it is common practice not to compress dimensions that are too small, such as the $3\times 3$ dimension of convolutional kernels, and also not to compress the three input channels when the input is an RGB image. As the low-rankness of small dimensions is poor, using low-ranking decomposition methods for compression will lead to a significant performance decrease, which is not conducive to achieving a better balance between the compression ratio and accuracy. Additionally, the degree of redundancy in network parameters is correlated with the complexity of the dataset and the difficulty of identification tasks. As shown in Table 3, our QLTD method achieved a parameter compression ratio of 54.35× and an accuracy of 61.57%, representing a 7.53% decrease in accuracy compared with that of the baseline ResNet-18 on the ImageNet dataset. As shown in Table 1, on the CIFAR-10 dataset that is relatively simple, our QLTD method was able to achieve a parameter compression ratio of 90.61× and an accuracy of 93.24%, representing a 1.85% decrease in accuracy compared with that of the baseline ResNet-18, which have a higher parameter compression ratio and smaller accuracy decrease than those of the ImageNet dataset.

It is also worth mentioning that early convolutional neural networks use relatively large receptive fields with dimensions of, for example, $11\times 11$ and $7\times 7$. VGGNet [56] achieved the same receptive field as that attained with larger convolutional kernels by stacking multiple layers of $3\times 3$ kernels, thereby reducing the number of network parameters. However, adding more layers to a suitable deep model increases the training error and reduces the accuracy [55,57,58]. An effective strategy for achieving both a large receptive field and parameter efficiency without adding more layers involves employing larger convolutional kernels, such as $11\times 11$ kernels, and compressing them to $3\times 3$ using our QLTD method during the Tucker decomposition. This approach ensures the preservation of a substantial receptive field while mitigating parameter redundancy. We will explore this strategy in future work.

6. Conclusions

In this study, we proposed a compact representation for neural networks named Quantized Low-Rank Tensor Decomposition (QLTD) to super compress deep convolutional neural networks. We found that the parameter redundancy in the low-rank space can be alleviated by performing a vector quantization, which enables the network to realize an ultra-light-weight structure. Furthermore, the self-attention module which is designed as a trainable sparse convolutional layer can contribute significantly to improving accuracy. Extensive experiments on object identification showed that our approach can achieve state-of-the-art results in the super-compression ratio domain and can realize the best balance between compressing parameters and maintaining identification accuracy.