Mitigating Quantization Errors Due to Activation Spikes in Gated Linear Unit-Based Large Language Models

Abstract

Modern large language models (LLMs) achieve state-of-the-art performance through architectural advancements but require high computational costs for inference. Post-training quantization is a widely adopted approach to reduce these costs by quantizing weights and activations to lower precision, such as INT8. However, we identify a critical challenge in activation quantization for GLU (Gated Linear Unit) variants, which are commonly used in the feed-forward networks of modern LLMs like the LLaMA family. Specifically, severe local quantization errors arise due to excessively large activation magnitudes, which we refer to as activation spikes, leading to significant degradation in model performance. Our analysis reveals a systematic pattern of these spikes: they predominantly occur in the FFN (feed-forward network) layers at the early and late layers of the model and are concentrated on a small subset of tokens rather than being uniformly distributed across a token sequence. To mitigate this issue, we propose two empirical methods: Quantization-free Module (QFeM) and Quantization-free Prefix (QFeP), which isolate activation spikes during quantization. Extensive experiments demonstrated that our methods effectively improve activation quantization, particularly in coarse-grained quantization schemes, enhancing the performance of LLMs with GLU variants and addressing the limitations of existing quantization techniques. The code for implementing our methods and reproducing the experiments is publicly available our GitHub repository.

1. Introduction

Large language models (LLMs) have become a key paradigm in natural language processing, accelerating the release of variations within the community [1,2]. Furthermore, the latest LLMs establish state-of-the-art performance by training with increased scale, as well as by adopting architectural improvements such as GLU [3], RoPE [4], GQA [5], and MoE [6]. Especially, GLU (Gated Linear Unit) variants (e.g., SwiGLU, GeGLU), which modulate hidden representations through element-wise gating between parallel linear transformations, have been adopted in most modern LLM architectures (e.g., LLaMA family [7]) due to their training efficiency [3,8]. Although LLMs broaden foundational capabilities in natural language tasks and potential for various applications, billions of parameters in the large models impose considerable computational costs on end users in practice. To reduce GPU memory requirements and accelerate inference speed, post-training quantization (PTQ), which reduces bit-widths of weights and activations after training without updating parameters, offers an affordable solution by quantizing weights and activations into a lower precision (e.g., INT8) without a need for expensive retraining steps [9,10,11]. However, recent studies have revealed that large magnitude values at certain coordinates exist in the activations of LLMs, which are often called outliers, posing a key challenge in activation quantization [12,13,14,15]. Another line of works attempts to explain the role of outlier values in the attention mechanism [16,17].

In this paper, we present our discovery that the GLU architecture in the feed-forward network (FFN) generates excessively large activation values, which are responsible for significant local quantization errors. Specifically, we observe that these problematic activation values occur in specific linear modules and are dedicated to a couple of tokens, which will be discussed in Section 3. To distinguish the excessive GLU activations from the outliers, we refer to them as activation spikes. In light of our observations, we propose two empirical methods to mitigate the impact of activation spikes on quantization: Quantization-free Module (QFeM) and Quantization-free Prefix (QFeP). QFeM aims to partially exclude quantization for linear layers (or modules) where large quantization errors occur, instead of quantizing the entire linear modules in the LLM. By scoring the extent of scale disparity, QFeM selects linear modules to exclude. On the other hand, QFeP identifies the prefix that triggers activation spikes and stores its context using a key–value (KV) cache. This cache mechanism retains previously computed attention keys and values, allowing the model to bypass redundant computations during decoding. It is noteworthy that both QFeM and QFeP rely on calibration results to capture activation spikes in advance without any modifications to the target LLM. Thus, our methods can be integrated into any existing quantization methods such as Xiao et al. [13], Wei et al. [15].

In our comprehensive experiments, we demonstrated that recently released LLMs incorporating GLU variants struggled with activation spikes when applying activation quantization. Consequently, the proposed methods, QFeM and QFeP, substantially enhance the performance of the primitive quantization method, the round-to-nearest (RTN) method, which quantizes activations by rounding each value to the closest discrete level based on fixed scaling. Furthermore, we observe that current outlier alleviation methods [13,15] and state-of-the-art quantization methods [18,19] are exposed to the activation spikes and benefit from our proposed methods. Compared to the strong baseline of fine-grained activation quantization [20], our methods show competitive performance, achieving reduced latency and memory footprint. In summary, the contributions of our work are as follows:

• We find that the GLU architecture in modern LLMs systematically generates excessive activation values, which are responsible for significant performance degradation when quantizing the entire model and input tokens.
• Based on our observations, we propose two empirical methods, QFeM and QFeP, which effectively exclude the activation spikes during quantization, with negligible computational overhead and compatibility with any existing quantization techniques.
• Our experiments validate the detrimental impact of the activation spikes on activation quantization, while the proposed methods consistently enhance the quantization performance.

2. Related Works

2.1. Outlier Values in LLMs

Previously, outlier values have been observed in transformer-based language models such as BERT [21] and early GPT [22] models through numerous studies [23,24,25,26,27]. Since the advent of LLMs [28,29] rooted in the GPT [22], recent studies [12,13,14] have tackled the existence of outlier values in LLMs. These outliers exhibit a large magnitude of values at the shared dimensions of hidden states across tokens. More recently, Bondarenko et al. [16], Sun et al. [17] explained that the outliers are attributed to the vertical pattern in the attention mechanism [30,31], which influences the performance of LLMs. In particular, Sun et al. [17] claims a different type of outlier existing in the hidden states of specific tokens. However, prior studies merely focused on the superficial hidden states between the decoder layers. Our work provides a module-level investigation where quantization is applied practically, focusing on different LLM architectures.

2.2. Post-Training Quantization for LLMs

Post-training quantization (PTQ) refers to the quantization of a neural network model to low precision, such as INT8, without additional parameter updates [9,10]. Especially for LLMs, this approach cost-effectively achieves inference with low memory usage and faster inference latency by quantizing the weights and activations used in matrix multiplication (e.g., linear layer). However, because of the challenges in the activation quantization of LLMs, many recent works have mainly focused on the weight-only quantization [19,32,33,34,35,36,37]. Otherwise, the activation quantization faces inherent outliers, which hinder accurate quantization by reducing the representation resolution. To address this challenge, Dettmers et al. [12] proposed a mixed-precision quantization method where the outlier dimensions are computed in high precision. Xiao et al. [13], Wei et al. [15], Shao et al. [19] approached the migration of a scale from activation to weights to alleviate the scale of outlier activations. Most similar to our work, Liu et al. [38], Son et al. [39] utilized a prompt to improve the quantization accuracy. We highlight that our work uncovers how the prefix works on GLU activations and propose an efficient prefix length that is sufficient for handling activation spikes, while they focused on preserving the attention sink [30] pattern.

3. Activation Spikes: Excessive Magnitude of GLU Activations

For clarity, “hidden states” refer to the output tensor of a transformer layer (or block), while “input activations” or “activations” denote the input tensor of a linear layer (or module) in the remainder of this paper. Recent work [17] has investigated a novel type of outlier existing in the hidden states across modern LLMs. Although these outliers of hidden states play a crucial role in the attention mechanism [16,17,30], their relationship with input activations for quantization has not been fully explored. Importantly, because recent LLMs adopt Pre-LN [40,41], which normalizes hidden states before self-attention and feed-forward network (FFN) blocks, the scale of hidden states does not reflect the scale of input activations within the transformer block. Therefore, we focus on the input activations fed into each linear module within the transformer block to connect to activation quantization. Specifically, we examine the four linear (projection) layers: query (parallel to key and value), out, up (parallel to gate), and down modules. For details of the Pre-LN transformer, please see Appendix A.

3.1. Existence of Activation Spikes in GLU Variants

To analyze the input activations, we employed a calibration method, which was used to estimate the quantization factors such as the scale and zero-point. For the calibration data, we used 512 samples randomly collected from the C4 [42] training dataset. Afterwards, we fed each sample into the LLM and monitored each hidden state and input activation through the decoder layers. To estimate the scale factor, we used absolute maximum value. The tested LLMs are listed in

Table A1. Architecture specification of LLMs. We categorize them into two groups depending on whether GLU is implemented in the FFN. All LLMs in the table use Pre-LN for the LayerNorm position.

Model	Size	FFN Activation	Normalization	PE	Vocabulary Size
GLU-implemented LLMs:
LLaMA-2 [47]	7 B, 13 B, 70 B	SwiGLU	RMSNorm	RoPE	32,000
LLaMA-3	8 B, 70 B	SwiGLU	RMSNorm	RoPE	128,256
Mistral [49]	7 B	SwiGLU	RMSNorm	RoPE	32,000
Mixtral [6]	8 × 7 B	SwiGLU	RMSNorm	RoPE	32,000
SOLAR [50]	10.7 B	SwiGLU	RMSNorm	RoPE	32,000
StableLM-2 [58]	12 B	SwiGLU	LayerNorm	RoPE	100,352
Gemma [51]	7 B	GeGLU	RMSNorm	RoPE	256,000
Non GLU-implemented LLMs:
OPT [29]	6.7 B, 13 B, 30 B, 66 B	ReLU	LayerNorm	Learned	50,272
MPT [59]	7 B, 30 B	GeLU	LayerNorm	ALiBi	50,432
Pythia [60]	6.9 B, 12 B	GeLU	LayerNorm	RoPE	50,432, 50,688
Falcon [61]	7 B, 40 B	GeLU	LayerNorm	RoPE	65,024
Phi-2 [62]	2.7 B	GeLU	LayerNorm	RoPE	51,200

.

3.1.1. GLU-Implemented LLMs Exhibit Activation Spikes at Specific Layers

In Figure 1a, we display the calibrated scale factors for the LLMs that implement GLU variants (e.g., SwiGLU, GeGLU). Across models, we observe a shared pattern of scale from the results. Within the early and late layers, the down modules in the FFN show noticeable magnitudes of input activations. Note that these input activations are derived from the Hadamard Product within the GLU [3]. Thus, the GLU variants generate activation spikes at the specific layers. Interestingly, we notice a high correlation between the emergence of activation spikes and intermediate hidden states of a large scale. This indicates that the FFN contributes to amplifying the hidden states via the addition operation in the residual connection [43]. Once the magnitude of the hidden states is exploded, it persists through layers until it encounters the activation spikes at late layers.

3.1.2. Non GLU-Implemented LLMs Show Modest Scale Distribution

Figure 1b illustrates the calibration results for LLMs with the original feed-forward implementation in the transformer [44]. We observe that the LLMs continue to generate the large-scale hidden states regardless of the GLU implementation. This corresponds to the observations in Sun et al. [17]. More importantly, our module-level results elaborate that the scale of hidden states is not transferable to the input activations of inner linear modules. Instead, we reveal that GLU variants are associated with the hidden states and generate activation spikes. This clarifies the quantization challenge of the GLU-implemented LLMs concentrated in the early and late layers. Because excessive scales of activation spikes have the potential to hinder the accurate quantization, we conduct an in-depth analysis to better understand these activation spikes in the following sections.

3.2. Token-Level Scale Analysis Within Activation Spikes

In the previous section, we observe the excessive scale of the input activations derived from GLU activation. When quantizing the input activations, the variance of input activation scales for each token affects the quantization performance [20]. To delve into the disparity between token-wise scales in the activation spikes, we unroll them through the sequence of tokens. Figure 2 illustrates the individual input activation scales where the activation spike appears. Given a token sequence, the large magnitudes of input activations are observed in a couple of tokens, such as the BOS token, newline (\n), and apostrophe (′). These specific tokens coincide with the observations of Sun et al. [17], which suggests that such tokens exhibit massive values in the hidden states. Thus, the activation spike is associated with the process of assigning a special role to these tokens in later transformer layers. However, the excessive scale of a specific token hinders the estimation of the scale factor for the other tokens, such as in per-tensor quantization. Additionally, the largest scale is dedicated to the first instance of the specified token, while the following usage exhibits a modest scale. This phenomenon makes the quantization more complicated, as the activation spikes dynamically occur depending on the current input sequence.

3.3. Effect of Quantization on Activation Spikes

We explore the impact of local quantization errors caused by activation spikes on LLM outputs. To identify the layers where activation spikes occur, we utilize a ratio between the maximum and median values of the token-wise input activation scales instead of using the maximum scale value alone. The max–median ratio for linear layer m can be formulated as $r^{\left(m\right)}={\displaystyle \frac{max\left({\mathbf{S}}^{\left(m\right)}\right)}{median\left({\mathbf{S}}^{\left(m\right)}\right)}}$, where ${\mathbf{S}}^{\left(m\right)}$ represents the token-wise input activation scales coming to module $m\in M$. This max–median ratio captures the extent to which the maximum scale dominates the other token scales. For comparison, we chose the activation quantization targets as the top-4, middle-4, and bottom-4 modules based on the max–median ratio in descending order. Then, we evaluated the perplexity and mean-squared error (MSE) using the calibration dataset. Here, the MSE was calculated for the last hidden states between the original (FP16) and partially quantized LLM. As shown in

Table 1. Perplexity and MSE of partial activation quantization of LLMs. Quantization is applied to the Top 4, Middle 4, and Bottom 4 layers, selected based on the max-to-median ratio of token-wise activation scales. Both perplexity and MSE are measured using the calibration dataset. MSE is computed between the last hidden states of the FP16 and partially quantized models.

Model	Perplexity (↓)				MSE (↓)
	FP16	Top 4	Middle 4	Bottom 4	Top 4	Middle 4	Bottom 4
LLaMA-2-7B	7.37	11.77	7.38	7.40	1908.80	1.03	12.90
LLaMA-2-13B	6.84	15.09	6.84	6.84	4762.11	0.91	10.38
Mistral-7B	8.35	69.45	8.35	8.36	218.60	0.02	0.18
Gemma-7B	10.85	85.83	10.94	10.87	213.93	1.60	1.07

, quantization on the top-4-rated modules solely degrades the LLM performance by significant margins, while the other cases exhibit negligible performance changes. We consider these quantization-sensitive input activations (inter alia activation spikes) to be the quantization bottleneck, which, in this paper, refers to the quantization error caused by outliers.

Furthermore, the activation spikes are conditioned on the specific context of the input sequence as discussed in Section 3.2. Altogether, to address the influence of activation spikes—which vary across layers due to the GLU architecture—we propose in the following sections a layer-wise selective quantization method and a prefix-based approach for mitigating quantization errors.

4. Mitigating Quantization Quality Degradation Based on the Observation

To address the quantization bottleneck, our approach is based on the deterministic occurrence patterns of activation spikes. First, we utilize the observation that bottlenecks occur at a few specific layers. This implies that the naive full quantization of LLMs is affected by these bottlenecks. Second, we exploit the phenomenon that the activation spike is derived from the first occurrence of specific tokens. Thus, the planned occurrence prevents recurrence in the subsequent and possibly future tokens.

4.1. Overview of Adapting Our Techniques for Quantization

In this section, we introduce two practical methods—Quantization-free Module (QFeM) and Quantization-free Prefix (QFeP)—that can be applied independently or jointly to mitigate activation spike-induced quantization errors. An overview of these methods is illustrated in Figure 3, which summarizes the key ideas behind QFeM and QFeP. QFeM is applied in a layer-wise manner: after selecting an LLM with a GLU-based architecture and a layer-wise quantization method, QFeM evaluates the quantization efficiency for each linear layer based on activation spike severity. Layers with strong spikes are excluded from activation quantization to reduce degradation. The scoring and selection process is detailed in Section 4.2. QFeP is a token-level method applicable regardless of GLU usage or layer-wise quantization. It identifies tokens that trigger strong spikes and preemptively inserts them as a prefix to be stored in the KV cache, effectively suppressing spikes during inference. For tested LLMs listed in

Table 2. Specifications for QFeM and QFeP used in experiments. $\left|M\right|$ denotes the total number of linear layers in the LLM, and $|M_{unq}|$ represents the number of unquantized layers for QFeM.

Model	Prefix	$\mathit{\alpha }$	$|{\mathit{M}}_{\mathit{unq}}|/|\mathit{M}|$
LLaMA-2-7B	[BOS] all.	6.68	17/128
LLaMA-2-13B	[BOS] then,	12.91	6/160
LLaMA-2-70B	[BOS] I′	9.16	25/320
Mistral-7B	[BOS] how\n	49.00	3/128
Mixtral-8x7B	[BOS]) .\n	4.03	191/608
SOLAR-10.7B	[BOS] a 1	6.48	11/192
Gemma-7B	[BOS]. Più	10.65	5/112
LLaMA-3-8B	[BOS]-nd	6.64	6/128
LLaMA-3-70B	[BOS] and,	78.37	3/320

, preselected prefixes can be used directly; otherwise, a prefix can be searched and applied according to the technique described in Section 4.3.

4.2. Quantization-Free Module (QFeM)

In the full quantization of LLM, all linear layers within the LLM are quantized. Among these linear layers, we propose omitting the quantization of input activations for linear layers where significant quantization errors are caused by activation spikes. To be noted, increasing the number of unquantized modules exhibits a trade-off between the inference latency and the model performance. Thus, determining which module should be quantized (or left unquantized) is crucial to retaining the efficacy of quantization. Here, we use the max–median ratio $r^{\left(m\right)}$ and define a set of unquantized modules, denoted as $M_{\mathrm{unq}}$, where the ratio $r^{\left(m\right)}$ of each linear layer is larger than the threshold $\alpha$. For instance, all linear layers in M are quantized if $\alpha =\infty$. For clarity, we treat sibling linear layers, such as query-key-value, as a single linear layer. To control the impact of activation quantization only, we leave the weight parameters in unquantized linear layers as INT8 and dequantize them into FP16 during matrix multiplication with the incoming activations, operating as weight-only quantization.

Optimizing the Threshold $\alpha$

To calculate the activation scale ratio for each linear layer, we first gather token-wise input activation scales from the calibration examples discussed in Section 3.1. Exceptionally, for FFN experts in mixture of experts (MoE) architectures like the Mixtral model [6], calibration is performed separately. After determining these ratios, we use binary search to set the threshold value $\alpha$, balancing inference latency and performance degradation. As a metric, we assess performance through perplexity measured on the same calibration examples. For example, the relationship between the threshold value $\alpha$ and its impact on performance is depicted in Figure 4, demonstrating how full quantization can degrade performance. Rather than fully quantizing, we identify an optimal threshold by finding the intersection of two performance curves; in Figure 4, this threshold is approximately 16. Details on the QFeM implementation are provided in Table 2.

4.3. Quantization-Free Prefix (QFeP)

Orthogonal to the QFeM, we propose the Quantization-free Prefix (QFeP), which mitigates the quantization errors by precomputing the prefix (or short prompt) corresponding to activation spikes. This method is based on the observations presented in Section 3.2, which indicate that significant quantization errors result from the overestimated scale factor of the first instance within the restricted token set. Inspired by this occurrence pattern of activation spikes, we aim to construct a prefix that stabilizes the quantization scale factor of the tokens that come after the prefix. In other words, once the prefix is fixed at the beginning, the activation spikes consistently occur within the prefix. Afterward, we employ the key–value (KV) caching mechanism to process the activation spikes in advance. In practice, KV cache is utilized to optimize the decoding speed of causal language models by storing precomputed key and value states of the previous tokens [45,46]. This approach provides a bypass of the quantization including activation spikes, while preserving the context of the prefix through the KV cache. The KV cache for the prefix is precomputed once through the offline inference of the LLM without quantization. Then, this KV cache is exploited in the quantization phases, such as calibration or dynamic quantization, even for quantized inference. The process of QFeP is illustrated in Figure 3. We verified that our prefix design is effective in mitigating the quantization errors through a prefix ablation study (Section 5.4).

4.3.1. Prefix Search

To form a prefix of an explicit activation spike, we first identify a candidate token that represents the activation spike at the linear layer with the highest max–median ratio $r^{\left(m\right)}$. For instance, the candidate token can be an apostrophe (′) token for the LLaMA-2-70B model, as highlighted in red in Figure 2. Once the candidate token is identified, we search the middle context token between the BOS token and the candidate token in the prefix. This middle context provides a dummy context, which is required to activate the candidate token. To find the middle context, we design a template $[B,T_1,C_1,T_2,C_2]$ where B, $T_i$, and $C_i$ denote the BOS token, context token, and candidate token in the vocabulary V, respectively. Then, we select the context token T where $C_1$ triggers an activation spike, whereas the latter instance of the same token $C_2$ does not. When the context token for the activation spikes is varied, we choose the token that maximizes the activation scale ratio between $C_1$ and $C_2$. Finally, we prepare the KV cache for the searched prefix of $[B,T,C]$. Note that the latter sequence in the template can be replaced with sequences from the dataset instead of a repetition.

4.3.2. Implementation Details

During the prefix search phase, we exploit the calibration dataset used in Section 3.1. For the candidate tokens, we consider the tokens with the top three largest input activation magnitudes. Then, we search for the middle context token among the top 200 most frequent tokens in the calibration dataset, which is the subset of the vocabulary V. Finally, with the search result, we prepare the KV cache for the target model in FP16 precision. Exceptionally, for the Mixtral [6] model, we use the scale of output hidden states instead of input activations, as the tokens are divided sparsely in a mixture of experts architecture. Table 2 presents the searched prefix.

5. Experiments

5.1. Experimental Setup

5.1.1. Models and Environment

Our proposed methods, QFeM and QFeP, aim to mitigate the quantization bottleneck, which is discussed in Section 3.3, caused by the activation spikes, especially in the GLU variants. To validate the proposed efficiency methods, we tested publicly released LLMs that were implemented with GLU according to their paper and source code. We recognize that recent LLMs, including LLAMA-2-{7B, 13B, 70B} [47], LLaMA-3-{7B, 70B} [48], Mistral-7B [49], Mixtral-8x7B [6], SOLAR-10.7B [50], and Gemma-7B [51], utilize the GLU architecture. LLMs with the original FFN are not covered, as they suffer from the existing outliers rather than activation spikes. All models were sourced from the Hugging Face model repository (available at: https://huggingface.co/models, accessed on 17 April 2025). We used Python 3.10, PyTorch 2.2, and Transformers 4.38.1 as our primary software environment and employed the lm-eval-harness library for LLM evaluation. Experiments were conducted on a single NVIDIA RTX 4090 and a single NVIDIA A100 GPU, where smaller models were evaluated on the NVIDIA RTX 4090, and larger models were evaluated on the NVIDIA A100 based on their respective sizes.

5.1.2. Quantization

In the experiments, we quantized both the input activations and the weights of linear layers for INT8 matrix multiplication operations. Note that in Table 2, $\left|M\right|$ denotes the total number of linear modules targeted for quantization. In these linear layers, we opted for dynamic per-tensor quantization as the quantization scheme of input activations and per-channel quantization for weights, respectively. Regarding both input activations and weights, we symmetrically quantized the range using the absolute maximum value as the scale estimation function. For comparison, we used FP16 and per-token activation quantization [20] as baselines. We refer the reader to Appendix E for Batch Matrix-Multiplication (BMM) quantization, which involves quantizing tensors in the self-attention.

5.1.3. Evaluations

We evaluated the quantized LLMs with two metrics: zero-shot evaluation accuracy and perplexity. For zero-shot evaluation, we used four datasets: PIQA [52], LAMBADA [53], HellaSwag [54], and WinoGrande [55]. We utilized the lm-evaluation-harness library [56] to evaluate zero-shot tasks. To measure perplexity, we used the WikiText-2 [57] dataset. In all cases, we used the [BOS] token as the starting token for each input sequence by default.

5.2. Main Results

5.2.1. LLaMA-2 Models

We report the evaluation results of quantization on LLaMA-2 models in

Table 3. Perplexity and zero-shot evaluation for the quantization on LLaMA-2 models. FP16 denotes the original model precision, and W8A8 denotes the model quantized to INT8 for both weights and activations. Colored cells indicate results obtained by applying our proposed methods to existing quantization approaches. Column headers represent different benchmarks, with the values in parentheses denoting ppl (perplexity) and acl (accuracy), respectively. Downward arrows (↓) indicate that lower values are better (e.g., for perplexity), while upward arrows (↑) indicate that higher values are better (e.g., for accuracy). Bold text in the main table highlights the improvements over the baseline quantization method (W8A8 without our method).

Method	WikiText-2 (ppl ↓)	PIQA (acc ↑)	LAMBADA (acc ↑)	HellaSwag (acc ↑)	WinoGrande (acc ↑)	Avg (acc ↑)
LLaMA-2-7B
FP16	5.268	78.18%	73.67%	57.13%	69.46%	69.61%
W8A8	8.634	72.80%	62.27%	49.57%	63.69%	62.08%
+QFeM	5.758 [−2.876]	78.02%	73.86%	56.32%	68.35%	69.14% [+7.06]
+QFeP	5.758 [−2.876]	76.44%	73.57%	55.55%	69.22%	68.69% [+6.61]
+QFeM+QFeP	5.573 [−3.061]	77.86%	74.58%	56.05%	69.38%	69.47% [+7.39]
LLaMA-2-13B
FP16	4.789	79.49%	76.54%	60.20%	72.38%	72.15%
W8A8	34.089	70.13%	49.66%	42.65%	58.72%	55.29%
+QFeM	5.241 [−28.848]	77.58%	75.68%	59.13%	72.61%	71.25% [+15.96]
+QFeP	6.000 [−28.089]	77.53%	73.94%	57.23%	70.96%	69.91% [+14.62]
+QFeM+QFeP	5.126 [−28.963]	78.51%	75.86%	59.44%	72.61%	71.61% [+16.32]
LLaMA-2-70B
FP16	3.218	81.45%	79.45%	65.29%	80.43%	76.65%
W8A8	8.055	74.05%	70.27%	55.21%	67.96%	66.87%
+QFeM	3.830 [−4.225]	81.23%	77.66%	64.15%	78.14%	75.30% [+8.43]
+QFeP	6.007 [−2.048]	77.64%	73.26%	63.40%	76.16%	72.62% [+5.75]
+QFeM+QFeP	3.708 [−4.347]	81.23%	77.82%	64.65%	77.11%	75.20% [+8.33]

. Compared to FP16 precision, quantizing both weights and activations (W8A8) degrades the overall performance. The results demonstrate that our proposed methods resolve the activation spikes and, surprisingly, restore the performance of the W8A8 close to that of FP16. For example, the LLaMA-2 7B model achieved less than a 1% performance drop from FP16. It is worth noting that the proposed QFeM and QFeP improved at comparable levels. This indicates that the activation spikes present a direct cause of the significant decrease in quantization performance. Because the proposed methods are orthogonal, the performance slightly increases when incorporating both QFeM and QFeP compared to applying them individually.

5.2.2. Other GLU-Implemented LLMs

For other LLMs that incorporate GLU, we investigated the effectiveness of our methods in mitigating the quantization bottleneck. As can be seen in Figure 5, our methods consistently remedy the performance drop caused by activation spikes. Noticeably, the Mixtral model demonstrates robustness towards the performance degradation. This indicates that the mixture of experts architecture, which divides the MLP experts by tokens, helps to alleviate the impact of the activation spikes. Meanwhile, addressing the activation spikes is not a sufficient complement for the Gemma model compared to other models. We attribute this to the choice of activation function among GLU variants; specifically, Gemma uses GeGLU, while other models employ SwiGLU.

5.3. Combining Outlier Alleviation Methods

While our method focuses on the activation spikes, the inherent outlier values in the input activations remain. Here, we combine the prior outlier alleviation methods, such as SmoothQuant (SQ) [13] and OutlierSuppressionPlus (OSP) [15], to further improve the quantization error.

Table 4. Perplexity on Wikitext-2 for outlier alleviation methods with QFeM and QFeP under different activation quantization granularities. Activation quantization is applied at two levels: fine-grained (per-token) and coarse-grained (per-tensor). We employ per-channel granularity for weight quantization. For low-bit quantization (W6A6 and W4A4), we adopt asymmetric scale estimation. The bold values indicate the best performance within each group. “#Bits” denotes the bit-width configuration used for quantization, representing how much the weights and activations have been quantized.

Model	#Bits	Method	Per-Token Quant.			Per-Tensor Quant.
			Base	QFeM	QFeP	Base	QFeM	QFeP
LLaMA-2-7B (FP16: 5.268)	W8A8	SQ	5.296	5.288	5.302	9.907	5.534	5.715
	W8A8	OSP	5.293	5.288	5.303	38.490	5.493	5.642
	W4A4	OSP	48.199	16.151	44.635	23491	4772	10940
LLaMA-2-13B (FP16: 4.789)	W8A8	SQ	4.809	4.808	4.818	34.869	5.118	6.551
	W8A8	OSP	4.813	4.812	4.819	5.148	5.099	5.144
	W4A4	OSP	21.037	11.535	11.860	9340	8680	9362

demonstrates the evaluation results of applying the outlier alleviation methods solely and combining them with our methods. We find that there are cases where the alleviation method fails to recover the performance when quantizing the activations with per-tensor scheme. In the original papers [13,15], the activations of LLaMA models were quantized using only a dynamic per-token scheme. This indicates that alleviating the outlier scales, including the activation spikes, is challenging. Under 4-bit quantization, we observe that QFeM and QFeP are efficient even when per-token quantization is applied. Similarly, we confirm this increase in our analysis of state-of-the-art quantization methods in Appendix D.

5.4. Prefix Ablation Study

For the QFeP, we designed a length-three prefix for the KV cache, including the BOS token, context token, and extra token for activation spike. Because the KV cache consumes the capacity of the pretrained sequence position, it raises a question about the length of the prefix. Therefore, we conducted an ablation study for different prefixes for the KV cache. For the prefixes, we prepared random, BOS only, and both QFeP without and with the context tokens. Notably, the BOS cache corresponds to $IntactK{V}_{\left[B\right]}$ [38]. We illustrate the results of the ablation study in Figure 6. In all cases, the random prefix showcases the lowest performance. While the KV cache with the BOS token demonstrates inconsistent performance, our QFeP consistently shows significant improvements. Importantly, the results imply that the sufficient prefix for the models exhibits differences. However, we emphasize that our KV design for QFeP shows improvements by large margins across all models. Beyond our QFeP, $IntactK{V}_{\left[P\right]}$ [38] extends the BOS token to the system prompt for supervised fine-tuned models and CushionCache [39] update prefixes through quantization-aware prefix tuning.

5.5. Computational Cost Analysis

The proposed methods require additional resources to evict the activation spikes. Therefore, we analyzed the computational costs of the methods and compared them in various schemes. For comparison, we evaluated different activation quantization schemes: dynamic per-token, dynamic per-tensor, and static per-tensor, denoted as AQ1, AQ2, and AQ3, respectively. This distinction establishes strong baselines and demonstrates the potential of the methods. To calibrate the static scales, we estimated the absolute maximum value using the calibration dataset, which is used in Section 3.1.

5.5.1. Inference Latency

For each setting, we present the accuracy of the zero-shot tasks and inference latency of the fixed token sequence, as shown in Figure 7. While the fine-grained scheme (AQ1) shows a negligible accuracy drop, the counterparts (AQ2, AQ3) degrade with the quantization bottleneck. However, by applying our methods, the coarse-grained schemes achieved a competitive performance gain. For example, the combination of AQ2 and QFeM demonstrates a performance close to AQ1 but with a faster latency. The results signify that addressing the quantization bottleneck is important to accelerate the inference latency with coarser granularity. Specifically, the naive static quantization (AQ3), the fastest scheme, exhibits a significant decline. We hope that our work contributes to future works that address the remaining challenges in static quantization.

5.5.2. Memory Footprint

In

Table 5. Memory footprint (in MiB) for different quantization methods across sequence lengths of 1 K and 2 K. Results are shown for both LLaMA-2-7B and LLaMA-2-70B models.

Method	SeqLen
	1 K	2 K
LLaMA-2-7B
AQ1	8185 MiB	9516 MiB
AQ2	8148 MiB	9474 MiB
+QFeP	8149 MiB	9478 MiB
+QFeM	8148 MiB	9474 MiB
LLaMA-2-70B
AQ1	67,756 MiB	69,037 MiB
AQ2	67,648 MiB	68,820 MiB
+QFeP	67,651 MiB	68,822 MiB
+QFeM	67,838 MiB	68,819 MiB

, we record the maximum memory footprint of our methods. For QFeP, the additional memory is consistently required for the preserved KV cache. However, this memory overhead is much smaller than that used in the fine-grained quantization (AQ1), as QFeM utilizes only three tokens for the cache. Contrary to QFeP, QFeM shows inconsistent memory utilization. For example, the 7B model with QFeM exhibits memory usage similar to AQ2, while the 70B model with QFeM incurs additional consumption for a sequence length of 1K. This is attributed to the use of W8A16 for the unquantization modules in QFeM. To tailor the memory usage or inference speed, an alternative strategy can be utilized for QFeM, such as applying fine-grained activation quantization to the unquantization modules instead of using W8A16.

6. Conclusions

We explored the quantization challenge of GLU activations for modern LLMs. We found that the GLU variants generate excessive activation scales, which cause significant quantization bottlenecks at the specific layers. Based on the systematic generation pattern of the activation spikes, we propose methods that address the spikes in a layer-wise (QFeM) and token-wise manner (QFeP). Our experimental results demonstrate significant improvements, showing approximately a 7–16% increased accuracy in quantized LLaMA-2 models compared to traditional quantization methods. Additionally, our methods achieved a substantial reduction in perplexity—up to 85.4% in LLaMA-2-13B, and over 50% in LLaMA-2-7B—closely approaching full-precision (FP16) performance. Furthermore, our analysis indicates that QFeM optimizes computational efficiency, reducing latency by up to 10.4% in LLaMA-2-13B, by selectively avoiding quantization in layers severely impacted by activation spikes, effectively balancing latency–performance trade-offs. Similarly, QFeP minimizes memory overhead, increasing memory usage by less than 0.05% in LLaMA-2-7B for 2 K tokens, while efficiently managing problematic activations through KV caching mechanisms. We expect that our work sheds light on the potential challenges in future studies on quantization and facilitates the development of efficient LLM systems.