Advances in the Neural Network Quantization: A Comprehensive Review

Abstract

Artificial intelligence technologies based on deep convolutional neural networks and large language models have made significant breakthroughs in many tasks, such as image recognition, target detection, semantic segmentation, and natural language processing, but also face a conflict between the high computational capacity of the algorithms and limited deployment resources. Quantization, which converts floating-point neural networks into low-bit-width integer networks, is an important and essential technique for efficient deployment and cost reduction in edge computing. This paper analyzes various existing quantization methods, showcases the deployment accuracy of advanced techniques, and discusses the future challenges and trends in this domain.

1. Introduction

With the rapid development of artificial intelligence and machine learning technologies, deep learning models have achieved excellent results in many fields, such as target detection [1,2,3,4], intelligent obstacle avoidance [5], semantic segmentation [6,7,8,9], and situational awareness [10]. However, the high performance of these models often relies on a large number of parameters and complex computational processes, leading to significant challenges in computational resource and energy consumption in their practical applications. Existing deep learning models still face the shortcomings of requiring high storage, memory bandwidth, and computational power [11]. These shortcomings limit their use in resource-limited devices in many practical applications. Therefore, how to reduce model size, improve model running speed, and enable existing deep models to be applied to embedded platforms are key to solving the current challenges of deep learning applications. To address these challenges, neural network model quantization techniques have been extensively investigated in recent years to compress neural network models and accelerate their inference process by converting high-precision data to low-precision data [12,13]. For example, quantizing 32-bit multiplication operations to 8-bit can reduce power consumption by about 94.59% (8-bit multiplication operations consume 0.2 pj, while 32-bit multiplication operations consume 3.7 pj), and the data transfer speed of the quantized model can be increased to 4 times that of the original model [14,15]. Neural network model quantization is an important technology that needs to be solved in the field of artificial intelligence, and has urgent application needs and broad application prospects in the fields of embedded high-speed inference and large model compression.

(1) There is a need for efficient computation of intelligent algorithms on the embedded devices. With the continuous progress of artificial intelligence technology, the application field of neural networks is also expanding. Due to the high computational complexity of intelligent algorithm models represented by neural networks, the contradiction between limited computational resources and the high computational demand of intelligent models seriously restricts the deployment of high-performance neural network models on end-side computing devices, which has become a bottleneck restricting the promotion and application of artificial intelligence technology [16,17,18]. To address this problem, there is an urgent need to carry out research on quantization techniques for neural network models to reduce data storage, data transmission, and computational power. Modern neural networks, while highly effective, often require substantial computational resources and memory, which can be prohibitive for deployment on edge devices with limited hardware capabilities. By reducing the precision of the model’s weights and activations, quantization techniques aim to shrink the model size and enhance efficiency without significantly sacrificing performance.

(2) Deployment Requirements for Large Language Models. In recent years, big models based on ultra-large-scale text data, represented by BERT [19] and GPT [20,21], have flourished and made substantial progress in tasks such as natural language reasoning and human-computer dialogue, reaching or even surpassing the human level on some datasets. Among them, the most famous is the GPT series of models introduced by OpenAI, whose latest version has reached the scale of 100 billion parameters and is regarded as the most advanced language model. With the rapid increase in the scale of large models, the demand for GPU memory and arithmetic power is getting bigger and bigger, resulting in bottlenecks involving storage, access bandwidth, and computation speed; it is getting more and more difficult for high-performance servers to meet the demand for hardware resources of large models. Deloitte Touche Tohmatsu, one of the world’s Big Four accounting firms, predicts that the market for dedicated chips optimized for generative AI will exceed $50 billion in 2024. The quantization of large models allows companies that rely on them to significantly reduce their need for HPC hardware, which in turn significantly reduces chip procurement and operational costs. Quantizing floating-point large models to 8-bit models for inference computation is also a mainstream practice in the industry today [22,23].

Therefore, the research and development of efficient and low-loss neural network model quantization methods are not only of great significance for promoting the wide application of deep learning techniques, but also has a long-term impact on advancing the development of computer science and artificial intelligence technologies. Neural network quantization can reduce the complexity and size of the model, accelerate model inference speed, and reduce resource consumption, which has significant research value in promoting the development of artificial intelligence technology. The main challenge of quantization is how to minimize the loss of computational accuracy caused by the reduction of numerical bit width. A large amount of research has been conducted to try to find quantization strategies suitable for different network architectures and application scenarios, as well as the optimal quantization parameters under different quantization strategies. However, there are still some technical difficulties in this research. The main challenge of quantization at present is how to minimize the loss of accuracy due to the reduction of numerical bit-width. The quantization methods face challenges regarding how to allocate the quantization bit-width, how to adaptively optimize the quantization parameters according to the differences in the model and the data distributions, as well as how to balance the speed of computation with the computational accuracy. Post-training quantization is simpler and faster to implement since it quantizes a model after training without requiring retraining, making it ideal for rapid deployment. However, it can lead to notable accuracy loss, especially in models sensitive to precision. In contrast, quantization-aware training integrates quantization into the training phase, leading to better-optimized models with minimal accuracy degradation, but it requires more computational resources and time due to the need for retraining.

About the neural network model quantization accuracy loss assessment technique, the traditional neural network model quantization accuracy loss assessment criteria tend to use a single metric to measure the neural network model quantization accuracy loss, and this accuracy loss is the cumulative result on a small number of test sets, which often fails to comprehensively reflect the loss of accuracy after neural network model quantization. Research on how to quickly and accurately assess the loss of accuracy after model quantization is crucial to the field of model compression and model quantization.

For neural network model bit-width allocation techniques, existing techniques have made great progress compared to the traditional uniform accuracy quantization methods, automating the bit-width search for different layer weights and activations. However, these methods still have room for improvement. Most of the existing methods do not take into account the loss of accuracy and efficiency of the embedded device in the optimization objective function of the quantization bit width selection, and since the effect of different hardware devices on the operation of the quantization model varies, the establishment of a complete set of co-optimization methods is of great importance. It is necessary to study the feedback of the deployment model on the hardware side of the accuracy as well as the efficiency loss through both the neural model accuracy loss assessment and speed assessment method, optimizing the bit width selection process through the accuracy and efficiency loss of the deployment model. Most of the existing methods tend to be computationally demanding, and some rely on manual configuration to optimize the initial session, posing challenges for different model and device setups, and there is a need to build a fast and initialization-free bit-width search technique to address these issues.

For neural network model quantization parameter generation technology, the current quantization methods are usually only for single application scenarios. Most of them adopt the unified symmetric uniform quantization mapping strategy, but their robustness is poor, and they cannot adapt to different neural network model structures, hardware resource limitations, or computational task requirements, especially for computational tasks with high accuracy requirements or difficult computational tasks, such as small target detection and identification tasks. Existing quantization methods have difficulty meeting accuracy requirements, resulting in a huge loss of performance.

Current comprehensive research on quantization methods includes how to improve the accuracy of the quantized neural network model. This is a key problem to be solved, and it is necessary to study how to automatically select the appropriate quantization threshold according to the distribution of the data, how to combine the quantized features with the quantization error to be considered in the training, how to solve the problem of mismatch between quantized forward propagation and back propagation, and how to make the neural network model more friendly to quantization.

In this paper, we compare and analyze existing neural network model quantization methods, give the deployment accuracy results of the existing advanced methods in the field, and summarize the technical difficulties and development trends of future neural network quantization technology, providing practical guidelines. The rest of the paper is organized as follows: Section 2 describes the quantization fundamentals. Section 3 describes several key techniques of quantification. Section 4 provides details of the experimental results and analysis. Section 5 concludes our work and discusses the future challenges and trends.

2. Quantization Fundamentals

Neural network models use floating-point data types. To achieve high-speed computations, it is necessary to quantize the floating-point neural network models into integer neural networks. As shown in Figure 1, quantization reduces computational cost by decreasing the precision of the data type, which minimizes the bit-width of data storage and the amount of data passing through the deep neural network. Computing and storing data at lower bit-widths enable fast inference and reduces energy consumption. The quantization process of a neural network model involves choosing a quantization mapping strategy, deciding whether to train the model, computing the quantization parameters, generating the inverse quantization parameters, and deploying the inference on resource-limited devices.

When performing the quantization process on a neural network model, the activation and weight values of the model are restricted to a discrete set of numbers that can have different distributions: uniform or non-uniform. The non-uniform quantization method is a logarithmic distribution [24]. Inference of neural network models in hardware platforms requires processing of quantized computational results through certain inverse quantization computational steps to obtain a true floating-point type output. Non-uniform quantization methods have complex inverse quantization computation processes and are not friendly to hardware platforms. Most current hardware supports only uniform quantization methods. Therefore, the most widely used quantization scheme is the uniform distribution [25] method with uniform step size.

The key issue of quantization is to design a proper quantization mapping function and a proper method to calculate quantization parameters. For uniform quantization, most existing approaches use either asymmetric or symmetric quantization mapping functions [26]. The asymmetric quantization mapping function is as follows:

$$r=f\left(Q\right)=s·Q+D$$

(1)

$$Q=f^{-1}(r)=round\left({\displaystyle \frac{r-D}{s}}\right)$$

(2)

where f and $f^{-1}$ are the quantization mapping function, $f^{-1}$ is the inverse function of f, round is the rounding operation, r is the floating-point real value, Q is the integer value after quantization, s and D are quantization parameters, s is the scaling factor, and D is the zero point, chosen such that the 0 value would exactly map to quantized values. Symmetric quantization is a simplified version of the general asymmetric case [27]. The symmetric quantizer restricts the quantization parameter D to 0 [28].

In Figure 2, we take symmetric quantization to 8-bit as an example. We can see that quantization converts continuous floating-point data into discrete integers, which brings accuracy loss. The quantization parameters are very important for both asymmetric and symmetric quantization, and affect the performance of the quantized neural network. The quantization parameters depend on the clipping range, and the scaling factors divide the clipping range into a number of partitions.

The optimal clipping range for the input is [$mi{n}_{in},ma{x}_{in}$], the optimal clipping range for the output is [$mi{n}_{out},ma{x}_{out}$], and the threshold of the weights is $t{h}_w$. The method to compute the quantization parameters according to the clipping thresholds is as follows: The quantization parameters $s_{in}$, $D_{in},s_{out}$, $D_{out}$, $s_w$ of a layer are computed according to Equations (3)~(7).

$$s_{in}={\displaystyle \frac{ma{x}_{in}-mi{n}_{in}}{2^{b{w}_{in}}-1}}$$

(3)

$$D_{in}={\displaystyle \frac{mi{n}_{in}-ma{x}_{in}}{2^{b{w}_{in}}-1}}·\mathrm{round}({\displaystyle \frac{\left(2^{b{w}_{in}-1}-1\right)·mi{n}_{in}+2^{b{w}_{in}-1}·ma{x}_{in}}{mi{n}_{in}-ma{x}_{in}}})$$

(4)

$$s_{out}={\displaystyle \frac{ma{x}_{out}-mi{n}_{out}}{2^{b{w}_{out}}-1}}$$

(5)

$$D_{out}={\displaystyle \frac{mi{n}_{out}-ma{x}_{out}}{2^{b{w}_{out}}-1}}·\mathrm{round}({\displaystyle \frac{\left(2^{b{w}_{out}-1}-1\right)·mi{n}_{out}+2^{b{w}_{out}-1}·ma{x}_{out}}{mi{n}_{out}-ma{x}_{out}}})$$

(6)

$$s_w={\displaystyle \frac{t{h}_w}{2^{b{w}_w-1}-1}}$$

(7)

where $b{w}_{in}$, $b{w}_{out}$, and $b{w}_w$ are the bit width of input, output, and weight, of which 8 is commonly used.

To effectively quantize the convolutional layer, it is essential to understand its underlying computational principles. The fundamental operation of the convolutional layer is:

$${{\displaystyle \sum }}^{ }\left({\mathrm{input}}_{i,j,k}^f\cdot w_{k,n}^f\right)+{\mathrm{bias}}_k^f={\mathrm{output}}_{l,m,n}^f$$

(8)

where ${\mathrm{bias}}_k^f$ represents the k-th bias of the convolutional layer, and ${\mathrm{output}}_{m,n,l}^f$ denotes the output of the convolutional layer. All these data types are in floating-point format.

Based on the computational principles of the convolutional layer and the proposed hybrid asymmetric quantization strategy, the method for quantizing the convolutional layer can be deduced. The activations of the convolutional layer (including both the input and output) use asymmetric quantization mapping, while the weights of the convolutional layer use symmetric quantization mapping. The computation principle for quantizing the convolutional layer is:

$$\left\{\left({\displaystyle \frac{s_{in}\cdot s_w}{s_{out}}}\cdot 2^S\right)\left({{\displaystyle \sum }}^{ }\left({\displaystyle \frac{{\mathrm{input}}_{i,j,k}^f-D_{in}}{s_{in}}}\cdot {\displaystyle \frac{w_{k,n}^f}{s_w}}\right)\right)+{\displaystyle \frac{{\mathrm{bias}}_k^f+D_{in}·{{\displaystyle \sum }}^{ }{w}_{k,n}^f-D_{out}}{s_{out}}}\cdot 2^S\right\}·2^{-S}={\displaystyle \frac{{\mathrm{output}}_{l,m,n}^f-D_{out}}{s_{out}}}$$

(9)

where $s_{out}$ and $D_{out}$ are the quantization parameters for the convolutional layer’s output, and S represents the shift parameter used during the inference process of the convolutional layer.

As shown in Figure 3, the data distribution of neural network model activations and weights is often asymmetric, which poses a significant challenge for selecting clipping range and quantization parameters. A good quantization method should resolve the two following questions to improve the deployment performance. The first question is the trade-off between the accuracy and the difficulty of deployment. The second question is the trade-off between clipping range and quantization resolution, which significantly influences quantization parameters’ computation. There are two main forms of quantization methods: Post-training quantization [14,29] and quantization-aware training [30,31]. Post-training quantization (PTQ) is the quick quantization of floating-point weights and activations after the model has been trained. Quantization-aware training (QAT) is the process of updating the weights of the model by considering the quantization process during the training process. PTQ is time-saving and convenient, while QAT can obtain a higher accuracy. In addition to PTQ and QAT, the selection of quantization bit-width and the evaluation of the accuracy loss of deep learning models after quantization are also very important techniques related to quantization. In this paper, we will analyze the development status and challenges of the above related techniques, respectively.

3. Quantization Techniques

3.1. PTQ

Post-Training Quantization (PTQ) is an optimization technique applied to neural network models after they have been trained. It is designed to reduce the model’s memory footprint and accelerate inference speed while attempting to maintain the model’s accuracy. The steps of PTQ are as shown in Figure 4. When the distribution is not gaussian-like, it is difficult for the quantized model to meet the accuracy requirements, especially for complicated tasks with higher accuracy requirements.

The key point for PTQ is how to calculate the quantization parameters, which depends on the clipping range. Usually, a series of calibrations are used as the input of a neural network to compute the typical range of activations [30,32]. A straightforward choice is to use the min/max of the data for the clipping range [30], which may unnecessarily increase the range and reduce the quantization resolution. One approach is to use the i-th largest/smallest value instead of the min/max value as the clipping range [33]. Another approach is to select the clipping range by some kind of information loss between the original real values and the quantized values [34,35], including KL divergence [36,37], Mean Squared Error (MSE) [38,39,40,41], or entropy [42]. Wei [43] proposes an activation redistribution-based hybrid asymmetric quantization method for neural networks, which takes data distribution into consideration and can resolve the contradiction between the quantization accuracy and the ease of implementation. Choosing the optimal quantization parameters and how to reduce the loss of accuracy of the neural network model after quantization are the key problems to be solved.

3.2. QAT

Quantization-aware training (QAT) is a technique in deep learning that integrates the quantization process into the training phase of a neural network model. Unlike post-training quantization (PTQ), which applies quantization after the model has been trained, QAT allows the model to learn and adapt to the quantization effects during training. The steps of QAT are shown in Figure 5.

QAT is a powerful technique for preparing neural network models for deployment on quantization-sensitive hardware. By training the model to be aware of quantization, it can better adapt and maintain accuracy even with reduced precision. However, most QAT methods use the straight-through gradient estimation (STE) technique, which causes a significant gradient error. To solve this problem, researchers have proposed alternative approaches. PACT [44] explores the impact of activation value trimming on quantization performance. Gong [45] adopts a differentiable tanh function to gradually approximate the quantization function. DoReFa [25] proposes tailoring the weight range prior to quantization. SAT [46] uses the gradient updating process and weight scale adjustment during training to improve quantization performance. Sharpness-aware quantization (SAQ) [47] provides a unified view of both quantization and sharpness-aware minimization (SAM) by treating them as introducing quantization noises and adversarial perturbations to the model weights. Zhuang [48] proposes a solution by training the low-precision network with a full-precision auxiliary module and constructs a mix-precision network by augmenting the original low-precision network with the full precision auxiliary module. It is worth noting that although QAT methods can deliver better quantitative performance, they often require large training datasets and significant computational resources, with training times exceeding 100 GPU hours [49].

3.3. The Selection of Quantization Bitwidth

Traditional model quantization methods involve quantizing the weight parameters and activation values of the whole model to a fixed bit-width. However, while high bit-width quantization ensures high accuracy, it also causes larger memory footprint and computation. Low bit-width quantization has lower accuracy but a smaller memory footprint and computation. Because different layers have different redundancies and computational effort, simply assigning the same bit-width does not guarantee optimal network performance. Therefore, mixed precision quantization (MPQ) is needed to achieve further efficient compression of the model. Wu [50] proposes a search method using a differentiable neural network structure search for the bit-width of each layer with no consideration about the inference delay on hardware. Wang [51] proposes the HAQ algorithm, which combines the hardware latency and energy consumption fed back from the hardware simulator to constrain the search for bit-widths at each layer, resulting in a hardware-aware hybrid accuracy policy. In order to quickly allocate the quantization bit-width of each layer, Dong [52] proposes HAWQ, which uses the second-order information of the model parameters to assess the sensitivity of each layer of the model to quantization, and then allocates the bit-width of each layer based on this sensitivity to improve the search efficiency. Dong [53] proposes the EMQ method, which automates the search for mixed-precision configurations with the help of evolutionary algorithms. Tang [54] proposes the SEAM method using small agent datasets to perform a fast search in order to uncover effective MPQ strategies applicable to large-scale training datasets, thus improving the search efficiency and practicality. HAWQv2 [55] enhances model quantization by using second-order information (the Hessian) to determine optimal quantization levels, thereby preserving accuracy while significantly reducing computational and memory costs. Tang [56] proposes the LIMPQ method that can speed up the indicator training process by parallelizing the original sequential training processes. With these learned importance indicators, the MPQ search problem is treated as a one-time integer linear programming (ILP) problem.

Existing hybrid bit-width allocation methods suffer from the following challenges: Most of the methods do not consider the adaptability to hardware, and since the efficiency and accuracy of the quantized neural network model may vary significantly on different intelligent processors, it is necessary to incorporate hardware feedback on accuracy and speed into the optimization function of the bit-width search. Meanwhile, these methods either still require a lot of computational resources or are very sensitive to hyperparameters or even initialization, so it is important to propose a parsimonious and fast search technique.

3.4. The Accuracy Loss Evaluation of the Quantized Models

The quantized model needs to be tested to ensure that the loss of accuracy is within an acceptable range. Some researchers have designed several quantization accuracy loss assessment criteria from the perspective of statistical data analysis to improve the performance of quantization models. Qualcomm [57] designed the Signal to Quantization Noise Ratio (SQNR) to measure the quantization accuracy of different quantization bit-widths. Pengcheng [58] proposed a method to determine the quantization coefficients using the quantization mean square error as a metric and proposed a method to update the statistical parameters for small networks with serious performance loss. However, the above accuracy loss assessment criteria evaluate the loss from the perspective of data statistics, resulting in a deviation from the actual application data and a lack of practicability in obtaining the accuracy loss of the task. Wang [59] constructs a quantization accuracy predictor based on the highly flexible once for all network [60], which encodes the model structure and quantization strategy to directly predict the accuracy of the quantization model. However, in this method, collecting the quantization dataset takes 16,000 GPU hours, which is quite expensive and time-consuming, and the quantization predictor can only predict neural network models of a predetermined structure.

Existing studies on the quantization of neural network models have less systematically analyzed the mechanism of computational accuracy loss generated by the quantization process, making it difficult to effectively measure the accuracy loss of different neural network models and different computational tasks after quantization. Therefore, how to accurately measure the accuracy loss of quantized models and how to provide a judgement basis for the quantization method of neural network models are challenges that still need to be solved.

3.5. Quantization of LLMs

The LLM (Large Language Model) is a state-of-the-art deep learning model designed for natural language processing tasks [61]. By integrating the latest quantization techniques, these models achieve significant reductions in computational and memory costs while maintaining high accuracy. However, existing solutions may still face challenges in preserving performance for highly complex tasks and ensuring generalizability across diverse datasets.

Existing quantization methods for large language models (LLMs) can be mainly divided into two categories: weight-only quantization and joint weight and activation quantization. The former compresses a large number of weights into lower bit-widths [62], effectively reducing the memory footprint of the models. The latter quantizes both weights and activations into mixed bit-widths [63], which accelerates matrix multiplications and significantly enhances computational speed.

However, when dealing with significant activation outliers, existing methods often exhibit limited improvements or result in unstable gradients. This challenge highlights the need for more robust quantization techniques capable of effectively managing outliers without compromising stability. Future advancements in this field may focus on adaptive quantization strategies that dynamically adjust bit-widths based on activation distributions, ensuring both efficiency and accuracy. Additionally, exploring novel quantization-aware training methods could further enhance the robustness and generalizability of quantized LLMs, paving the way for their broader application in resource-constrained environments.

4. Experimental Results

4.1. Experimental Setting

We perform two sets of experiments, one comparing PTQ methods and the other comparing QAT methods with mixed bit-widths. In order to obtain the experimental results conveniently, software based on fake quantization v1.0 [30] modules is used to simulate the accuracy on the neural network accelerator. Fake-quantization models quantization errors in the forward passes. The biggest reason to apply fake quantization is to quickly simulate the effects of quantization using simulated quantization operations. The purpose of the experiments is to compare the state-of-the-art quantization methods. The CPU is an Intel(R) Core(TM) i7-8700K, 3.70 GHz, and the GPU is a NVIDIA GeForce GTX1070.

In the PTQ experiment, we utilize classic image classification models GoogleNet and VGG16, the YOLOv1 model from the YOLO series for object detection applications, and the classic Unet model for image segmentation applications. GoogleNet and VGG16 are tested using the publicly authoritative ImageNet dataset to highlight the scientific rigor and fairness of our comparative method. The YOLOv1 model is applied to a custom ship detection dataset, while the Unet model is used with a custom remote sensing image segmentation dataset. The aim is to assess the precision loss of quantized models across targets of various sizes.

In the QAT experiment, we utilize the classic image classification ResNet50, which is evaluated through comparative experiments using the publicly authoritative ImageNet dataset.

The accuracy is verified using software. The evaluation metrics are the accuracy metrics of the model. For image classification applications, we use Top-1 Accuracy (the one with the highest probability must be exactly the expected answer). For small target detection applications, we use mAP (Mean Average Precision). The calculation of mAP is the same as in the internationally renowned target detection competition PASCAL VOC Challenge. For image segmentation applications, we use the mean intersection over union (mIoU) metric to evaluate accuracy.

4.2. Results

First, we compare the PTQ methods [41,42,43] on image classification models, the small target detection model, and the segment model. The results of the PTQ methods are shown in

Table 1. Results of the PTQ methods.

Task	Model	PC Accuracy (FP32)	Method [42]	Method [41]	Method [43]
Image Classification	GoogleNet	67.04	65.22	65.91	66.65
Image Classification	VGG16	66.13	64.72	62.53	65.61
Object Detection	YOLOv1	61.99	59.41	60.94	61.27
Image Segmentation	U-net	82.78	82.13	81.71	82.14

. Higher accuracy can be achieved due to the use of hybrid asymmetric quantization [43]. Upon implementing the hybrid asymmetric quantization approach [43] within the GoogleNet classification model, the resultant accuracy loss is minimal, amounting to a mere 0.39% when compared to the traditional symmetric quantization method [41,42]. Similarly, when applied to the VGG16 classification model, this method achieves an accuracy loss of only 0.52%. In the context of object detection, the application of the hybrid asymmetric quantization method [43] to the YOLO v1 model yields an accuracy loss of just 0.72%. Furthermore, when utilized in the U-net segmentation model, the method results in an accuracy loss of a mere 0.64%. The hybrid asymmetric quantization technique effectively reconciles the trade-off between the precision of quantized neural networks and the simplicity of their implementation, striking a balance between clipping range and quantization precision. Therefore, for the PTQ quantization method, using hybrid quantization with symmetric weights and asymmetric activation can improve the accuracy of the quantized model as much as possible while ensuring that it is hardware-friendly.

Then, we compare the QAT methods with the mixed bit-widths on the image classification model. The results of the QAT methods are shown in

Table 2. Results of the mixed-precision allocation and QAT methods for ResNet50 on the ImageNet dataset. “W-C” stands for weight compression rate.

Method	W-Bits	B-Bits	Top-1	W-C
PACT [44]	3	3	67.57	10.67x
HAWQv2 [55]	3-Mixed-Precision	3-Mixed-Precision	68.62	12.2x
LIMPQ [56]	3-Mixed-Precision	4-Mixed-Precision	70.15	12.3x

. We compare three methods for mixed-precision allocation and QAT: PACT [44], HAWQv2 [55], and LIMPQ [56].

It is evident that LIMPQ, due to its ability to assign different precision levels to different layers based on their sensitivity, achieves the best results with minimal loss. The LIMPQ technique intelligently discerns the significance of each neural network layer and dynamically assigns appropriate levels of precision. This sophisticated, adaptive strategy ensures that pivotal layers receive the requisite high precision to preserve functionality, while allowing less crucial layers to operate with reduced precision. This nuanced approach not only maintains the integrity of the model’s performance but also enhances overall efficiency. By precisely gauging and applying the necessary level of precision to each layer, LIMPQ lays the groundwork for the development of more potent and streamlined neural network architectures. The method’s scalability is evident in its applicability across a spectrum of neural network configurations, including but not limited to convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer models. This versatility positions LIMPQ as a multifaceted asset within the realm of deep learning, capable of addressing diverse computational needs and optimizing performance across a wide range of applications. This advantage allows LIMPQ to create a more balanced trade-off between model size reduction and accuracy retention, leading to it being the most effective model among the three methods.

In conclusion, when quantizing neural network models, it can be divided into two technical routes according to different application requirements. When a large amount of training data is not available, neural network models can be quickly quantized using the PTQ method, as in [43]. When a large amount of training data can be obtained, the weights and quantization parameters of the model can be co-optimized using the hybrid bit-width training framework [56].

5. Future Challenges and Trends

Quantization is an essential technique for model deployment and practical application, helping to balance model performance and efficiency. The main challenge of quantization is how to minimize the accuracy loss caused by the reduction of numerical bit-width. From the application perspective, the requirement for the quantization of neural network models is that the bit-width should be set as low as possible to reduce the computational energy consumption and improve the inference speed, while the accuracy loss due to quantization should be as small as possible. To solve this problem, the weights of the model and the quantization parameters need to be co-optimized. Co-optimization leads to problems regarding how to build a quantization operator that can be derived everywhere, how to evaluate the loss of quantization accuracy, and how to improve the robustness of the model after quantization. To solve the problem that the back propagation process of quantization is not derivable during the training process of existing neural network models, it is necessary to model the full-domain differentiable quantization operator. To solve the problem that the existing quantization accuracy evaluation methods adopt the real measurement method, which requires manual participation and is too time-consuming to be integrated into the collaborative optimization algorithm, it is necessary to study the quantization accuracy loss modelling method. To solve the problem of poor performance of existing quantization methods in real intelligent application scenarios, methods to improve the robustness of neural network models after quantization need to be investigated. Therefore, the future development trends of quantization technology are to construct fast adaptive updating of quantization parameters, explore the mechanism of precision loss caused by quantization, and propose automatic compensation and robustness enhancement methods to quantize the neural network models.