1. Introduction
In recent years, CNNs have played an important role in various computer-vision techniques, such as image classification [
1,
2,
3], target detection [
4,
5,
6,
7,
8], and semantic segmentation [
9,
10]. As the research on CNNs progresses, the scale of CNN models becomes deeper and wider, while increasingly high accuracy is obtained, which leads to an inevitable increase in the amount of computation and parameters in the models. For example, the original VGG-16 model has more than 130 million parameters, has more than 500 MB of memory storage space, and requires more than 30 billion floating-point operations to complete an image-recognition task [
1]. This requires hardware devices with more computing and storage resources as support. In addition, some industrial application areas, such as intelligent cars, unmanned aerial vehicles, and radars, which most often use CNNs for real-time data detection and processing, have the same requirement for the deployment of CNN models in embedded devices: high efficiency, high real-time accuracy, and low energy consumption [
11,
12,
13,
14]. However, embedded devices, such as FPGAs, often provide relatively few computation and storage hardware resources due to their limited size and energy consumption [
15]. Therefore, when CNN models are deployed to FPGA platforms, they often need to reduce computational parallelism or crop parameters to keep devices running properly [
16,
17]. These operations make it difficult to achieve the balance of efficiency, accuracy, and overhead required in industrial applications.
In order to solve the above problems, it is usually necessary to design software and hardware co-optimization in terms of both the compression of CNN models in software development and the improvement of the hardware friendliness of the models from a hardware-design perspective to improve the computational efficiency of CNNs on FPGA platforms. Currently, the main solutions for CNN model compression are model pruning [
18,
19] and parameter quantization [
20,
21]. Parameter quantization is an important and commonly used method, which refers to the sharing [
22,
23,
24] or streamlining [
25] of model parameters, such as weights and activation, without changing the model structure. Several studies [
26,
27] have demonstrated that quantizing the 32-bit floating-point weights in CNN models to 8-bit is sufficient to ensure the accuracy of CNN models without decline. However, the hardware-friendly features invite us to consider both the CNN model’s size and its computational complexity. First, we need to quantize 32-bit floating-point numbers into lower-bit fixed-point numbers to reduce storage-resource usage and improve memory-access efficiency. Second, some previous works [
28,
29,
30] deployed YOLO v2, MobileNet, and VGG-16 models on the ZYNQ XC7Z045 and XC7Z100 platforms; the on-chip DSP resource occupancies reached 88.9%, 95.3%, and 86.7%, while the LUT resource occupancies were all in the range of 50%. These two ZYNQ chips are already higher-end products in the FPGA family with relatively rich on-chip hardware resources. However, the DSP resources on them are almost completely consumed, which not only tends to lead to layout and routing failure of the system and an increase in the difficulty of the work, but it also indirectly proves that DSP resources are the primary constraint on the achievement of high-computational-efficiency CNN models in low-end and mid-end FPGAs. Furthermore, the paper [
31] mentions that the actual computational efficiency of on-chip DSPs is about 1/4 of that of other on-chip logic units. Therefore, it is necessary to use simple logic operations (AND, OR, shift, etc.) instead of complex operations (multiplication, etc.) in the hardware to reduce the use of DSP resources [
32].
Combining the above two points, Courbariaux et al. [
20] and Rastegari et al. [
21] proposed a binary weight approximation neural network and XNOR-Net, which quantize all the weights in CNNs to −1 and +1. Subsequently, the ternary weight network (TWN) [
33] improved the binary quantization method by introducing 0 into the quantization values. These quantization methods allow CNNs to be deployed to FPGAs using a simple XNOR logic operation or additive and subtractive operations to implement convolution computation without occupying on-chip DSP resources at all, which not only significantly compresses the model size and saves FPGA computation and storage resources, but it also improves computational efficiency. However, they share the common drawback of excessive accuracy loss. Although recently developed methods, such as trained ternary quantization (TTQ) [
34], learnable companding quantization (LCQ) [
35], and smart quantization (SQ) [
36], have significantly reduced quantization accuracy loss based on the traditional binary and ternary quantization methods, they are not completely hardware-friendly. As with many other methods, they do not quantize the first convolutional layer and the last fully connected layer in order to maintain the accuracy of quantized CNN models [
20,
21,
33,
34,
37,
38,
39,
40]. This leads to the need to continue using multiplication operations when deploying to FPGAs, and it is very difficult to implement these multiplication operations here because the unquantized layers are still 32-bit floating-point numbers, which require large on-chip computation resources and compromise computational efficiency.
However, incremental network quantization (INQ) [
23] balances quantization accuracy and hardware adaptation capability better. INQ quantizes all the weights in a CNN model from a full-precision floating-point to powers of two or zero. It first proposed an iterative approach of three operations of weight partition, group-wise quantization, and retraining to make CNN models have a lower accuracy loss after all layers are quantized. In addition, weights in the form of powers of two make it possible to deploy CNNs on FPGA platforms using simple shift-logic operations instead of expensive multiplication operations without using DSP resources, resulting in a higher computational efficiency and a greater engineering significance. However, the INQ method still fails to achieve optimal quantization results at low-bit quantization. Some recent methods, such as MPA [
41], also adopt the iterative approach of three operations of weight partition, group-wise quantization, and retraining proposed by INQ and introduce the clustering algorithm, etc., into the iterative to achieve higher quantization results at low-bit quantization. However, because MPA is not a power-of-two quantization method, it still requires a large number of multiplication operations when deployed to FPGAs, and the hardware-adaptation capability is limited.
In addition, unlike traditional power-of-two quantization methods, Gudovskiy et al. [
24] and Chen et al. [
42] both proposed algorithms that quantize the weights in CNNs directly to two power-of-two summations. Li et al. [
37] further proposed the additive power-of-two quantization (APoT) algorithm by quantizing the weights to sums of multiple powers of two, depending on different settings. These quantization methods provide a denser interval of quantization values, solve the problem of the rigid resolution in power-of-two quantization, and have smaller accuracy loss. However, when deploying to FPGAs, the original single multiplication operation needs to be replaced with multiple shift operations and multiple addition operations without using on-chip DSPs and multiplication operations, because each weight is quantized as a multiple power-of-two summation. Compared with traditional power-of-two quantization methods, the consumed computational and storage resources increase exponentially, and the computational efficiency also decreases.
To better balance the quantization accuracy and hardware-adaptation capabilities in current quantization methods, in this paper we focus on the software and hardware co-optimization design method, presenting research on the low-bit “lossless” quantization of CNN models and its hardware-adaptation capability based on traditional power-of-two quantization methods and FPGA platforms. We proposed a more effective generic power-of-two quantization method: global sign-based network quantization (GSNQ). While maintaining the quantization framework comprising weight partition, group-wise quantization, and retraining, GSNQ develops a new weight-quantization strategy and adopts a more effective fine-grained and multi-scale global retraining method for the CNN models, which can significantly reduce the accuracy loss of the CNN models in low-bit quantization and obtain low-bit models with high accuracy.
Cong et al. [
43] analyzed and counted the computational complexity of CNNs, finding that more than 90% of the computational complexity arises from convolutional computation, and the key to CNN deployment is the processing of convolutional computation. Therefore, we used shift operations to replace multiplication operations according to the characteristics of the power-of-two quantization, and we designed a novel convolutional computation module on FPGA platforms for GSNQ quantized models. This module does not occupy on-chip DSP resources and consumes less storage and fewer computational resources. Through this design, we verified that the GSNQ method has strong hardware-adaptation capability.
In summary, this study provides a reference for the industrial application of CNNs from both theoretical and experimental perspectives. The main work of this study is as follows:
We developed a sign-based power-of-two quantization strategy. The quantization value range was reasonably extended and the corresponding quantization rules were formulated according to the signs of weights. This strategy can prevent quantization loss caused by differences in the distribution of positive and negative weights, can ensure a wider quantized value range for each layer in the network, and can reduce the accuracy loss in low-bit quantization.
We proposed a fine-grained and multi-scale global retraining method, which adds vertical-level (inter-layer) iterations to horizontal-level (intra-layer) retraining in the original INQ method. The accuracy loss due to the change in parameters in one layer can be compensated by retraining all the other layers. This global retraining can compensate for the accuracy loss in low-bit quantization more effectively.
We reported on our performance of comparison quantization experiments on three classification datasets, MNIST [
44], CIFAR10 [
45], and Mini-ImageNet [
46], each tested for LeNet-5 [
44], AlexNet [
2], VGG-16 [
1], ResNet-18, ResNet-20, ResNet-56 [
3], and GoogLeNet [
47], to compare the proposed GSNQ method with INQ. We also compared the quantization results for the CIFAR10 dataset with other state-of-the-art quantization methods to comprehensively validate the effectiveness and generality of GSNQ.
We designed a high-performance convolutional computation module using shift operations instead of multiplication operations to complete the deployment of GSNQ quantized models on FPGA platforms. This module only uses abundant hardware resources, such as LUTs, to complete convolutional computation without consuming DSP resources. We compared this module with other mainstream multiplication-based convolutional computation modules to verify and analyze the hardware adaptability and engineering-application value of GSNQ.
The remaining parts of the paper are organized as follows.
Section 2 describes the proposed method.
Section 3 analyzes the details and results of the experiments. Finally,
Section 4 concludes the study.