Parallel Accelerating Number Theoretic Transform for Bootstrapping on a Graphics Processing Unit

Abstract

The bootstrapping procedure has become the main bottleneck affecting the efficiency of all known fully homomorphic encryption (FHE) schemes. The state-of-the-art scheme for efficient bootstrapping, which is called fully homomorphic encryption over the torus (TFHE), accelerates polynomial multiplication by leveraging number theoretic transform (NTT) and implementing NTT in parallel on a GPU. Unfortunately, almost none of the recent advancements in NTT take full advantage of a GPU, leading to the need for more time. With this in mind, in this work, a novel faster number theoretic transform based on a GPU is proposed, in which matrix multiplication is used to implement a decomposed small-point NTT. When implementing matrix multiplication, we introduce a merging preprocessing method to merge multiple inputs of the small-point NTT, aiming to effectively minimize the count of modulo operations. Subsequently, when the merged result is multiplied by rotation factors, we use logical left shift rather than arithmetic multiplication to improve the computational efficiency. Our scheme can easily be used to realize a 1024-point NTT and the results of the experiments show that the speedup ratio of our method over the butterfly algorithm is about 2.49.

1. Introduction

1.1. Motivation

Fully homomorphic encryption (FHE) provides an amazing ability to calculate ciphertext data without decryption, which makes it possible to process data without destroying sensitive source data. On the premise of security, the ciphertext can be used for all kinds of data processing (addition, subtraction, multiplication, division, square, polynomial, etc.) as if it were plaintext. So, FHE perfectly realizes the secure separation of data ownership and data access rights. The “computable but invisible” feature of fully homomorphic encryption has removed the biggest obstacle—privacy protection from the development of big data, cloud computing, intelligent manufacturing, the industrial internet, etc., which will directly promote the leapfrog development of the above industries.

Since Gentry [1] proposed the first fully homomorphic encryption scheme by using bootstrapping technology in 2009, many FHEs have been proposed, such as BGV [2], BFV [3] and CKKS [4]. Despite the significant application value of FHE, its high computation complexity has been a longstanding obstacle to its practical implementation. Among all the time-consuming components, the technical process of ciphertext bootstrapping proposed by Gentry is the most time-consuming one, as it involves a large amount of computation. Consequently, enhancing the efficiency of bootstrapping has become the central focus of FHE research.

Now, the gate bootstrapping of the TFHE scheme [5] is the fastest bootstrapping algorithm, which was proposed by Chillotti et al. in 2016. During the gate bootstrapping process of the TFHE scheme, a significant number of polynomial multiplications occur, and these multiplications over polynomial rings are computationally expensive. Bianchi T et al. [6] first utilized fast Fourier transform (FFT) to improve the speed of polynomial multiplications in homomorphic encryption. However, there is also the problem of rounding error in their method because FFT is a floating-point number operation. To avoid this error, number theoretic transformation (NTT) [7] was introduced, which uses another “primitive root” instead of the unit root to solve polynomial multiplication problems. In fact, NTT is a specialized form of FFT in the finite field of integers. NTT is more suitable for accelerating large polynomial multiplication on the ring, because NTT’s inherently parallel structure makes it a very attractive candidate for implementation on parallel architectures. Recently, many efficient implementations of NTT and NTT-based polynomial multiplication have been proposed in the literature, which makes NTT-based polynomial multiplication the core arithmetic operation in the majority of homomorphic encryption implementations. In addition, there have been some recent studies that improve the efficiency of bootstrapping by optimizing the blind rotation algorithm [8,9].

1.2. Prior Work

Theoretically, NTT accelerates polynomial multiplication to a large extent; however, its efficiency is still not high enough for practical applications. Therefore, various NTT-based polynomial multiplication optimized implementations aiming at accelerating FHE have been studied. At present, there are different platforms available for NTT-based polynomial multiplication: hardware architecture, CPU implementation, and GPU implementation.

The hardware architecture of NTT is mainly implemented on an FPGA (field programmable gate array). The main implementation of the current design is an NTT hardware multiplier [10], which uses a parallel architecture to implement NTT operations and basically uses addition and shift operations to ensure a large quantity of parallel processing. The test results show that the efficiency on an FPGA is 60 times that on a CPU. In addition, to improve efficient accelerators in lattice-based cryptosystems, Mert et al. [11] designed and implemented an NTT-based polynomial multiplication architecture on an FPGA, which can be used to execute NTT and INTT operations, and achieve a nearly 7 times speedup for the BFV scheme.

On a CPU, we can reduce the size of parameters or speed up the operation of NTT by improving the algorithm of NTT. For example, the Kyber–NTT algorithm [12] and PTNTT algorithm [13] are used to weaken the constraint of modulus q on ring polynomials, and then accelerate the NTT process by combining Karatsuba multiplication [14] and NTT multiplication.

In addition, due to the inherent parallel structure, NTT can be implemented on GPU platforms, which also makes it a research hotspot for accelerating NTT based on a GPU. Wang et al. [15] first explored the feasibility of implementing FHE on a GPU. In their scheme, the polynomial multiplication is implemented in parallel by the Schönhage–Strassen algorithm [16], and a new pre-computation technique is introduced to eliminate the expensive back-and-forth conversion in the Schönhage–Strassen algorithm, which enables an efficient acceleration of polynomial multiplication on a GPU. However, as the dimensions and the number of operands grow, the size of the pre-computation makes the GPU memory become a bottleneck, which results in less speedup. Later, Dai et al. [17] proposed the cuHE library, and summarized various optimizations and alternative ways to take advantage of the memory organization and bandwidth of CUDA GPU, such as algebraic optimization for efficient computation, memory minimization optimization, memory and thread scheduling optimization, etc. Dai et al. resolved the GPU memory problem in Wang et al.’s scheme. However, in their scheme, shared memory can run into serious bank conflict issues if there is a large amount of data to be processed at the same time. Goey et al. [18] introduced an efficient GPU accelerator for NTT in FHE, which has been demonstrated to be faster than the implementation presented by Dai et al. To take full advantage of the memory hierarchy in a GPU, they used faster memory types to reduce communication between registers and global memory. In addition, they used warp shuffle to allow threads to access registers, which enables faster data access compared to global and shared memory. Based on the previous technique, Lee et al. [19] designed a new scheme to improve the performance of the NTT algorithm. In this scheme, the Nussbaumer algorithm [20] was implemented on a GPU, where the recursive part is eliminated, and then the NTT algorithm is accelerated by optimizing the non-merged memory access pattern. Later, Kim et al. [21] introduced the on-the-fly twiddle factor into NTT and then analyzed and improved the performance of the NTT operation in the FHE scheme. Recently, Özgün Özerk et al. [22] provided a high-performance and efficient GPU implementation for NTT, INTT, and NTT-based polynomial multiplication. They significantly modified the NTT and INTT algorithms for te threading used on a GPU and proposed single-core and multi-core methods that perform better on small-degree and large-degree polynomials, respectively.

1.3. Our Contributions

Currently, the bootstrapping schemes based on a GPU primarily employ the Cooley–Tukey algorithm [23], also known as the butterfly algorithm, to execute the NTT operation. The optimized implementation of this algorithm is the cuFHE library [24]. When using the butterfly algorithm, the NTT operation needs to temporarily cache the calculation data, and reuse the intermediate data, which makes it impossible to calculate on multiple threads. In this case, to realize NTT based on the butterfly algorithm cannot give full play to the advantages of the GPU parallel architecture. In addition, because the calculation is carried out in a finite field, the implementation of the butterfly algorithm requires multiple modulo operations, which also consumes a lot of time.

Motivated by these concerns, we proposed a new NTT accelerating method on a GPU, which uses matrix multiplication to compute a small-point NTT. By employing a merging preprocessing technique, we can execute the same operation on multiple operands simultaneously, thus reducing the number of modulo operations. Additionally, logical left shift instead of multiplication operation is used to improve efficiency when calculating the NTT of a small point. It is shown that this new method is more suitable for polynomial multiplication in the process of gate bootstrapping in the TFHE scheme. The main process that increases the computational cost in TFHE is bootstrapping. Our acceleration scheme can significantly reduce the running time of bootstrapping, thereby improving the efficiency of TFHE, which is particularly important for handling large-scale data and complex computations.

Roadmap. Section 2 introduces the notations and the proposed accelerating NTT implementation on a GPU. Section 3 introduces the results of comparison and Section 4 makes the discussion. Appendix A defines multiple abbreviations that appear in this paper.

2. Materials and Methods

2.1. Preliminary

2.1.1. Notations

First, we present the definitions of symbols used in this paper which follows the conventions set by TFHE [5]. Let $\mathbb{B}=\left\{0, 1\right\}$ denote the set of 1 and 0, $\mathbb{R}$ denote the set of real numbers, $\mathbb{N}$ denote the set of natural numbers, and $\mathbb{Z}$ denote the set of integers. The real Torus $\mathbb{T}=\mathbb{R}/\mathbb{Z}=\mathbb{R}\mathrm{mod}1$ is the set of real number modulo 1. The ring ${\mathbb{Z}}_q$ consists of the set of integers $\left\{0,1, 2, \dots , q-1\right\}$, in which q is a prime number. The polynomial ring ${\mathbb{P}}_q={\mathbb{Z}}_q\left[x\right]/Q\left[x\right]$ represents all of the polynomials divided with the polynomial $Q\left[x\right]$, and all the coefficients are in ${\mathbb{Z}}_q$. $Q\left[x\right]$ is defined as $\left(x^N+1\right)$, and the degree of the polynomial ring is at most $\left(N-1\right)$. ${\mathbb{Z}}_N\left[x\right]$ denotes the ring of polynomials $\mathbb{Z}\left[x\right]/\left(x^N+1\right)$, ${\mathbb{T}}_N\left[x\right]$ denotes $\mathbb{T}\left[x\right]/\left(x^N+1\right)$, and ${\mathbb{B}}_N\left[x\right]$ denotes the polynomials with binary coefficients.

2.1.2. Number Theoretic Transform

NTT is used for calculating the multiplication of the two polynomials $f\left(x\right)=a_0+a_1\cdot x+\dots +a_{N-1}\cdot x^{N-1}$ and $g\left(x\right)$$=b_0+b_1\cdot x+\dots +b_{N-1}\cdot x^{N-1}$. If we utilize the traditional method, the multiplication of polynomials $f\left(x\right)$ and $g\left(x\right)$ can be carried out by using Equation (1). Obviously, the time complexity of this method is $O\left(n^2\right)$.

$$h\left(x\right)=f\left(x\right)\cdot g\left(x\right)={\displaystyle \sum _{i=0}^{N-1}{\displaystyle \sum _{j=0}^{N-1}{a}_i\cdot b_j}}\cdot x^{i+j}.$$

(1)

In fact, we can describe the degree of polynomial N−1 by using N points. In this way, $f\left(x\right)$ and $g\left(x\right)$ can be described as follows:

$$\begin{array}{l}f(x)=\{(x_0, f(x_0)), (x_1, f(x_1)), (x_2, f(x_2)), \dots , (x_{N-1}, f(x_{N-1}))\},\\ g(x)=\{(x_0, g(x_0)), (x_1, g(x_1)),(x_2, g(x_2)), \dots , (x_{N-1}, g(x_{N-1}))\}.\end{array}$$

Then, the multiplication of polynomials of $f\left(x\right)$ and $g\left(x\right)$ can be represented as $h\left(x\right)=\{\left(x_0, f\left(x_0\right)\cdot g\left(x_0\right)\right), \left(x_1, f\left(x_1\right)\cdot g\left(x_1\right)\right), \dots , \left(x_{N-1}, f\left(x_{N-1}\right)\cdot g\left(x_{N-1}\right)\right)\}.$

We usually use discrete Fourier transform (DFT) to calculate this form of polynomial, and NTT is defined as DFT over the ring ${\mathbb{Z}}_q$. An N-point NTT operation transforms a vector x with N elements to another vector X with N elements, which is described by Equation (2):

$$X_k={\displaystyle \sum _{i=0}^{N-1}{x}_i}\cdot W_N^{i\cdot k}\left(\mathrm{mod}q\right), k=0,1,\dots ,N-1.$$

(2)

The constant $W_N\in {\mathbb{Z}}_q$ is called the twiddle factor, which satisfies the condition of $W_N^N\equiv 1(\mathrm{mod}q)$, $W_N^{\frac{N}{2}}\equiv -1(\mathrm{mod}q)$ and $W_N^{2k+N}\equiv W_N^{2k}\equiv W_{N/2}^k(\mathrm{mod}q)$. Similarly, the INTT (inverse number theoretic transform) operation uses almost the same formula as the NTT operation as shown in Equation (3):

$$x_k=\frac{1}{N}{\displaystyle \sum _{i=0}^{N-1}{X}_i}\cdot W_N^{-i\cdot k}\left(\mathrm{mod}q\right), k=0,1,\dots ,N-1.$$

(3)

Here, $W_N^{-1}\in {\mathbb{Z}}_q$ is the modulo inverse of $W_n$ in ${\mathbb{Z}}_q$.

The multiplication of polynomials $f\left(x\right)$ and $g\left(x\right)$ can be simplified by using NTT and INTT. However, for two polynomials $f\left(x\right)$ and $g\left(x\right)$ of degree (N − 1), we cannot use the N-point NTT because the multiplication result has 2N terms. In this case, we should use the 2N-point NTT, which is described as Equation (4):

$$h\left(x\right)={\mathrm{INTT}}_{2N}\left({\mathrm{NTT}}_{2N}\left(f\left(x\right)\right)\cdot {\mathrm{NTT}}_{2N}\left(g\left(x\right)\right)\right)\left(\mathrm{mod}Q\left(x\right)\right).$$

(4)

In fact, $Q\left(x\right)$ is usually defined as $\left(x^N+1\right)$. In this case, we can use the Formulas (5) and (6) to represent NTT and INTT. In this method, we can use an N-point NTT to calculate the multiplication of two polynomials of the degree (N − 1) rather than a 2N-point NTT, and eliminate the operation of module $Q\left(x\right)$.

$$X_k={\displaystyle \sum _{i=0}^{N-1}{x}_i}\cdot {\omega }^i{W}_n^{i\cdot k}\left(\mathrm{mod}q\right), k=0,1,\dots ,N-1_{,}$$

(5)

$$x_k=\frac{1}{N}{\displaystyle \sum _{i=0}^{N-1}{X}_i}\cdot {\omega }^{-i}{W}_n^{-i\cdot k}\left(\mathrm{mod}q\right), k=0,1,\dots ,N-1.$$

(6)

It should be noted that $W={\omega }^2$ in the above Formulas (5) and (6). If we take $x_i{\omega }^i$ as the input, then the Formula (5) is the same as an N-point NTT and the Formula (6) is the same as an N-point INTT in value.

Obviously, when using this method, the time of multiplication of the two polynomials is reduced by using NTT. However, the time complexity of the NTT algorithm is still $O\left(n^2\right)$. To further reduce the complexity, the use of Cooley–Tukey NTT, also named the butterfly algorithm, is proposed. The time complexity of NTT is reduced to $O\left(n{\log }_{2}n\right)$ by the butterfly algorithm. The pseudo-code of the Cooley–Tukey NTT algorithm [23] is described as Algorithm 1.

Algorithm 1. Cooley–Tukey NTT

In the above algorithm, the bit-reverse operation means to array the bits of an integer with the binary form in reverse order and obtain the corresponding decimal integer. And the butterfly operation is given in Algorithm 2.

Algorithm 2. Butterfly operation

Input: A, B, W, module q Output: A, B B′ = B · W (mod q) B = A − B′ A = A + B′

2.1.3. Ciphertexts in TFHE

Gate bootstrapping was proposed by Chillotti et al. in 2016 [5]. In this section, we will introduce the ciphertexts used in TFHE, which includes TLWE, TRLWE, TGSW, and TRGSW.

Let $k\ge 1$ be an integer, $N$ be a power of 2, and $\alpha \in {\mathbb{R}}^{+}$ be a standard deviation, where ${\mathbb{R}}^{+}$ denotes the set of positive real numbers. Let $\mu \in {\mathbb{T}}_N\left[x\right]$ be the plaintext and $s\in {\mathbb{B}}_N{\left[x\right]}^k$ be the TLWE secret key. The error is expressed as $e\in D_{{\mathbb{T}}_N\left[x\right],\alpha }$, where $D_{{\mathbb{T}}_N\left[x\right],\alpha }$ is the modulo Gaussian distribution of standard deviation $\alpha$ over ${\mathbb{T}}_N\left[x\right]$. The TRLWE ciphertext is defined as $c=\left(a, s\cdot a+e+\mu \right)$, where a is uniformly selected from ${\mathbb{T}}_N{\left[x\right]}^k$, and $\mu$ is the plaintext of c. When $N=1$, the elements of c are numbers rather than polynomials, and in the rest of the paper we call it TLWE, which means the scalar form of TRLWE.

We first introduce the gadget before we give TRGSW. A gadget H is a block diagonal matrix which has $\left(K+1\right)\cdot L$ rows. Let K, L and $B_g$ be three positive integers, and $M\in {\mathbb{T}}_N{\left[x\right]}^{K+1}$, then the gadget H is described as the Formula (7):

$$H=\left[\begin{array}{ccc}1/B_g& \dots & 0\\ \dots & \dots & \dots \\ 1/B_g^L& \dots & 0\\ \dots & \dots & \dots \\ 0& \dots & 1/B_g\\ \dots & \dots & \dots \\ 0& \dots & 1/B_g^L\end{array}\right]\in {\mathbb{T}}_N{\left(X\right)}^{\left(K+1\right)\cdot L,\left(K+1\right)}.$$

(7)

C is a TRGSW sample if $C=Z+\mu \cdot H$ holds. Each element of $Z\in M^{\left(K+1\right)\cdot L}$ is a TRLWE ciphertext obtained by encrypting the plaintext 0. When $N=1$, the scalar form TRGSW is called TGSW.

Now we introduce the product of the two ciphertexts. First we should introduce the gadget decomposition of TRLWE samples. For a TRLWE sample $c\in {\mathbb{T}}_N{\left[x\right]}^{K+1}$ and a gadget H, the result of the gadget decomposition of c based on H is called $v=de{c}_H\left(c\right)\in {\mathbb{Z}}_N{[X]}^{(K+1)\cdot L}$, which is a vector and satisfies $c=v\cdot H$. Then, we can obtain the definition of the external product of the two kinds of ciphertexts through formula 8, where the external product $\otimes$ of a TRGSW ciphertext $A=\mathrm{TRGSW}\left({\mu }_A\right)$ and a TRLWE ciphertext $b=\mathrm{TRLWE}\left({\mu }_b\right)$ is defined. The result of $A\otimes b$ is a TRLWE ciphertext, which corresponds to the plaintext ${\mu }_A\cdot {\mu }_b$.

$$A\otimes b=de{c}_H\left(b\right)\cdot A.$$

(8)

Next, we will introduce the important structure of the gate bootstrapping in TFHE, the CMux gate. There is one control input and two data inputs in one CMux gate. The control input is a TRGSW sample $C=\mathrm{TRGSW}\left({\mu }_C\right)$, and the data inputs are two TRLWE samples: $d_0=\mathrm{TRLWE}\left({\mu }_0\right)$ and $d_1=\mathrm{TRLWE}\left({\mu }_1\right)$. The output is a TRLWE ciphertext. When ${\mu }_C=0$, the plaintext corresponding to the output ciphertext is $\mu ={\mu }_0$, and when ${\mu }_C=1$, the plaintext corresponding to the output ciphertext is $\mu ={\mu }_1$, as shown in Figure 1.

2.1.4. Graphical Processing Unit (GPU)

A GPU (graphics processing unit) is a microprocessor designed for data processing on computers [25]. In this section, we will introduce the GPU from three aspects: hardware level, thread organization level, and memory structure.

On the hardware level, a GPU consists of multiple streaming multiprocessors and each streaming multiprocessor contains multiple streaming processors.

• Streaming Multiprocessor (SM): It is the core component of the GPU hardware and serves as the basic control instruction unit with an independent instruction scheduling circuit. Multiple streaming processors exist within a single streaming multiprocessor, and all streaming processors share the same set of control instructions;
• Streaming Processor (SP): It is the fundamental processing unit where specific instructions and tasks are processed. A GPU performs parallel computation, which means numerous streaming processors are processing simultaneously.

On the thread organization level, we will introduce it from the perspective of CUDA [26]. CUDA is a programming interface provided by NVIDIA for writing programs that run on its GPUs, utilizing multiple threads for parallel execution.

• Kernel: In CUDA, you will execute the calculations you want to run on a GPU in a form similar to C/C++ functions, which are called kernels;
• Grid, Block, Thread: This is a hierarchical structure for thread allocation. When programming with CUDA, a grid is divided into multiple blocks, while a block is divided into multiple threads. The division is based on the btask characteristics and the hardware features of the GPU.

On the memory storage level, the main types of memory include registers, constant memory, shared memory, and global memory.

• Registers: They are the smallest storage units in a GPU, but they offer the fastest execution speed. Each thread is allocated private registers during execution, and other threads cannot read or write to these registers;
• Constant Memory: it is used to cache constant data utilized in the code executed on the streaming multiprocessor. We need to explicitly declare objects as constants in the code so that a GPU can cache and store them in the constant cache;
• Shared Memory: Each streaming multiprocessor also has a block of shared memory. It is a small, fast and low-latency on-chip programmable static random access memory (SRAM) used for sharing data among the thread blocks running on the SM;
• Global Memory: A GPU also has an off-chip global memory, which is a large-capacity and high-bandwidth dynamic random access memory (DRAM).

2.2. Parallel Accelerating NTT for Bootstrapping on a GPU

In this section, we will introduce the proposed parallel accelerating NTT on a GPU. For the sake of completeness, we will describe the accelerating NTT on GPU in a whole gate bootstrapping process. The entire gate bootstrapping process consists of three parts: blind rotation, ciphertext extraction, and key switching. The most time-consuming part is the blind-rotation process, which is the focus of our optimization. The essence of the blind-rotation process is the multiplication of two polynomials with large coefficients and degrees, and NTT can be used to accelerate this process. Our scheme is primarily designed to further speed up the NTT process. To fully leverage the parallel characteristics of a GPU, we implement the multiplication of polynomial coefficients and rotation factors using matrix multiplication. Additionally, due to the large degrees of the polynomial, we can decompose this large matrix into multiple smaller matrices. For each small matrix, we use logical left shift instead of multiplication operation to improve the running speed. The bootstrapping process with parallel accelerating NTT is shown in Figure 2 below.

Next, we will describe the gate bootstrapping process based on the proposed parallel accelerating NTT in detail. Given a TLWE sample $c={\mathrm{TLWE}}_s\left(\mu \right)$, the gate bootstrapping process constructs an encryption of $\mu$ under the same key s with a fixed amount of noise.

2.2.1. Initialization

Randomly choose two fixed messages ${\mu }_0,{\mu }_1\in \mathbb{T}$, such that ${\mu }_0\ne {\mu }_1$, and set the message space to be $\text{ℳ}=\left\{{\mu }_0, {\mu }_1\right\}$. The input of gate bootstrapping is a TLWE ciphertext $c={\mathrm{TLWE}}_s\left(\mu \right)=\left(a, b\right)$, where $a=\left(a_1, a_2, \dots , a_p\right)\in {\mathbb{T}}^p$, $b={\displaystyle \sum _{i=1}^p{s}_i}\cdot a_i+e+\mu$ and $\mu \in \mathbb{T}$. The secret key is a vector of p bits, i.e., $s=\left(s_1, s_2, \dots , s_p\right)\in {\mathbb{B}}^p$. The bootstrapping key is $\mathrm{BK}=\left({\mathrm{BK}}_1, {\mathrm{BK}}_2, \dots , {\mathrm{BK}}_p\right)$, where ${\mathrm{BK}}_i={\mathrm{TRGSW}}_{s^{″}}\left(s_i\right)$. The key-switching key is ${\mathrm{KSK}}_{s\to s^{\prime }}=\left\{{\mathrm{KSK}}_{i,j}\in {\mathrm{TLWE}}_s\left(s_i\cdot 2^{-j}\right)\right\}$ $\left(i=1, 2, \dots , p, j=1, 2, \dots , t\right)$, where t represents the precision of the key-switching algorithm. For all $i\left(i=1, 2, \dots , p\right)$, let $\overline{a_i}$ be the nearest integer to $2N\cdot a_i$, and let $\overline{b}$ be the nearest integer to $2N\cdot b$. Let ${\mu }^{\prime }=\frac{1}{2}{\mu }_1\in \mathbb{T}$, and compute $v=\left(1+x+\dots +x^{N-1}\right)\cdot x^{-N/2}\cdot {\mu }^{\prime }$. At last, construct the vector $\left(0, v\right)$, where 0 is a p-dimension zero vector.

2.2.2. Gadget Decomposition

In the first CMux gate, the inputs are ${\mathrm{ACC}}_0$ and ${\mathrm{BK}}_1$, and the output is ${\mathrm{ACC}}_1$. Set ${\mathrm{ACC}}_{tmp}=\left(x^{\overline{a_1}}-1\right)\cdot {\mathrm{ACC}}_0$, then decompose ${\mathrm{ACC}}_{tmp}$ by the basis $H\in {\mathbb{T}}_N{\left(x\right)}^{\left(K+1\right)\cdot L,K+1}$. For all $A_j=c_{0,j}+c_{1,j}x+\dots +c_{N-1,j}{x}^{N-1}$ in ${\mathrm{ACC}}_{tmp}=\left(A_1, A_2, \dots , A_K, A_{K+1}\right)$ $\in {\mathbb{T}}_N{\left(x\right)}^{K+1}$, compute the coefficient $c_{i,j}={\displaystyle \sum _{q=1}^L{c}_{i,j,q}}\frac{1}{B_g^q}$. Set $A_{j,q}={\displaystyle \sum _{i=1}^{N-1}{c}_{i,j,q}}{x}^i$, and next we can obtain the result of gadget decomposition $de{c}_H\left({\mathrm{ACC}}_{tmp}\right)=$$\left(A_{1,1}, \dots , A_{1,L}, \dots , A_{K+1,1}, \dots , A_{K+1,L}\right)$. At this time, we will calculate the multiplication of $de{c}_H\left({\mathrm{ACC}}_{tmp}\right)$ and ${\mathrm{BK}}_1$, which are both matrices constructed by polynomials. The multiplication of polynomials will be calculated by NTT and then we can obtain ${\mathrm{ACC}}_1$, which is the result of multiplication.

2.2.3. Decomposition of an N-Point NTT

When calculating an N-point NTT, we should firstly choose the prime $q=2^{64}-2^{32}+1$ [27,28], which satisfies $2^{192}\mathrm{mod}q=1$ and $2^{96}\mathrm{mod}q=-1$. Additionally, the twiddle factors of small-point NTTs are the power of 2, for example $W_4=2^{48}$, $W_8=2^{24}$, $W_{16}=2^{12}$, and $W_8=2^6$, so we can use logical left shift to replace the multiplication with twiddle factors to simplify the calculations.

The calculation of an N-point NTT usually can be decomposed into many calculations of small-point NTTs. Let $k=n\cdot k_1+k_2$ and $i=t\cdot i_2+i_1$, where $k_1=0,1,\dots ,t-1$, $k_2=0,1,\dots ,n-1$, $i_1=0,1,\dots ,t-1$ and $i_2=0,1,\dots ,n-1$, respectively represent the groups and offsets of t n-point NTTs and n t-point NTTs, then the twiddle factor can be described as $W_N^{i\cdot k}=W_N^{\left(t\cdot i_2+i_1\right)\cdot \left(n\cdot k_1+k_2\right)}=W_t^{i_1\cdot k_1}\cdot W_n^{i_2\cdot k_2}\cdot W_N^{i_1\cdot k_2}$. Obviously, $W_t$ is the twiddle factor of the t-point NTT and $W_n$ is the twiddle factor of the n-point NTT. At this time, the N-point NTT can be defined in Equation (9):

$$X_{n\cdot k_1+k_2}={\displaystyle \sum _{i_1=0}^{t-1}{W}_t^{i_1\cdot k_1}}\left({\displaystyle \sum _{i_2=0}^{n-1}{W}_n^{i_2\cdot k_2}\cdot x_{t\cdot i_2+i_1}}\right)W_N^{i_1\cdot k_2}\left(\mathrm{mod} q\right).$$

(9)

From Equation (9), it can be seen that the calculation of an N-point NTT can be decomposed into the computation of t n-point NTTs and n t-point NTTs. Therefore, the computation of an N-point NTT can be divided into two steps. Firstly, compute the t n-point NTTs by calculating ${\displaystyle \sum _{i_2=0}^{n-1}{W}_n^{i_2\cdot k_2}\cdot x_{t\cdot i_2+i_1}}$ with $x_{t\cdot i_2+i_1},i_2=0,1,\dots ,n-1$ as inputs. Then, the result of these t n-point NTTs is multiplied by the rotation factor $W_N^{i_1\cdot k_2}$ of the N-point NTT to obtain $x_{t\cdot k_2+i_1}=\left({\displaystyle \sum _{t_2=0}^{n-1}{W}_n^{i_2\cdot k_2}\cdot x_{t\cdot i_2+i_1}}\right)\cdot W_N^{i_1\cdot k_2}$, which will serve as the input of t-point NTTs. Secondly, compute the n t-point NTTs by ${\displaystyle \sum _{i_1=0}^{t-1}{W}_t^{i_1\cdot k_1}}\cdot x_{t\cdot k_2+i_1}$ with $x_{t\cdot k_2+i_1},i_1=0,1,\dots ,t-1$ as inputs. The results obtained from these n t-point NTTs form the final result $X_{n\cdot k_1+k_2}$ of the N-point NTT.

2.2.4. Thread Assignment

For an N-point NTT, we need N threads in one block. When calculating the N-point NTT on GPU, we should divide the N threads into t groups, with n threads in each group. $i_1$ in each group represents the identification number of that group. The input of each group is $x_{t\cdot i_2+i_1}\left(i_2=0, 1, \dots , n-1\right)$, and the output is $X_k^{\prime }=$${\displaystyle \sum _{i_2=0}^{n-1}{W}_n^{i_2\cdot k_2}\cdot x_{t\cdot i_2+i_1}}$. At this time, the n-point NTT can be calculated using Equation (10):

$$X_k^{\prime }={\displaystyle \sum _{i=0}^{n-1}{x}_i^{\prime }}\cdot W_n^{i\cdot k}\left(\mathrm{mod}q\right), k=0,1,\dots ,n-1.$$

(10)

And we transform them into matrix form as shown in Equation (11):

$$\left[\begin{array}{c}{X}_0^{\prime }\\ X_1^{\prime }\\ X_2^{\prime }\\ \dots \\ X_{n-1}^{\prime }\end{array}\right]=\left[\begin{array}{ccccc}1& 1& 1& \dots & 1\\ 1& W_n^1& W_n^2& \dots & W_n^{n-1}\\ 1& W_n^2& W_n^4& \dots & W_n^{2\cdot \left(n-1\right)}\\ \dots & \dots & \dots & \dots & \dots \\ 1& W_n^{n-1}& W_n^{2\cdot \left(n-1\right)}& \dots & W_n^{\left(n-1\right)\cdot \left(n-1\right)}\end{array}\right]\cdot {\left[\begin{array}{c}{x}_0^{\prime }\\ x_1^{\prime }\\ x_2^{\prime }\\ \dots \\ x_{n-1}^{\prime }\end{array}\right]}_{.}$$

(11)

2.2.5. n-Point NTT

1

Merging

After thread assignment in Section 2.2.4, there are n inputs in each thread group. We further categorize the n inputs into two groups based on the parity of k. Next, we take one operand from each group and describe how to perform merging preprocessing on these two operands in this section because of the nature of prime q, $2^{96}\mathrm{mod}q=-1$. When k is odd, $x_m^{\prime }\left(m=0, 1, \dots , n/2-1\right)$ and $x_{m+n/2}^{\prime }\left(m=0, 1, \dots , n/2-1\right)$ can be merged before they are logically left-shifted.$x_m^{\prime }\left(m=0, 1, \dots , n/2-1\right)$ should be shifted left by $l=j\cdot m\cdot k\left(\mathrm{mod}192\right)$ bits, and $x_{m+n/2}^{\prime }\left(m=0, 1, \dots , n/2-1\right)$ should be shifted left by $\left(l+96\right)$ bits. Because of the nature of prime q, $2^{96}\mathrm{mod}q=-1$, we can obtain $x_{m+n/2}^{\prime }\cdot 2^{96}\mathrm{mod}q=-x_{m+n/2}^{\prime }$. Therefore, we perform modulo subtraction firstly on $x_m^{\prime }\left(m=0, 1, \dots , n/2-1\right)$ and $x_{m+n/2}^{\prime }\left(m=0,1, \dots , n/2-1\right)$, and secondly the result of modulo subtraction is shifted left by l bits. For the same reason, when k is even, we perform modulo subtraction firstly on $x_m^{\prime }(m=0, 1, \dots , n/4-1, n/2, \dots , 3\cdot n/4-1)$ and $x_{m+n/4}^{\prime }(m=0, 1, \dots , n/4-1,n/2, \dots , 3\cdot n/4-1)$, and secondly the result is shifted left by l bits.

That is to say, when k is an odd number, we calculate $x_m^{″}$ using Equation (12):

$$x_m^{″}={\left\{\begin{array}{c}\mathrm{mod}sub\left(x_m^{\prime },x_{m+n/2}^{\prime }\right), m\cdot k\left(\mathrm{mod}n\right)<\frac{n}{2} \\ \mathrm{mod}sub\left(x_{m+n/2}^{\prime },x_m^{\prime }\right), \left(m+n/2\right)\cdot k\left(\mathrm{mod}n\right)<\frac{n}{2}\end{array}\right.}_{.}$$

(12)

When k is an even number, we calculate $x_m^{″}$ using Equation (13):

$$x_m^{″}=\left\{\begin{array}{c}\mathrm{mod}sub\left(x_m^{\prime },x_{m+n/4}^{\prime }\right), m\cdot k\left(\mathrm{mod}n\right)<\frac{n}{2} \\ \mathrm{mod}sub\left(x_{m+n/4}^{\prime },x_m^{\prime }\right), \left(m+n/4\right)\cdot k\left(\mathrm{mod}n\right)<\frac{n}{2}\end{array}\right..$$

(13)

2

Expansion and logical left shift

First, expand the operand $x_m^{″}$ to 192 bits, which are described as three 64-bit unsigned numbers $a_m$, $b_m$ and $c_m$. Set $a_m=0$, $b_m=0$ and $c_m={x_m}^{″}$. Then, we use logical left shift to replace the multiplication of operands with twiddle factor. When $m\cdot k\left(\mathrm{mod}n\right)<n/2$, the operand $x_m^{″}$ is described as 192 bits and it should be logically left-shifted by $l=j\cdot m\cdot k\left(\mathrm{mod}192\right)$ bits. When $\left(m+n/2\right)\cdot k\left(\mathrm{mod}n\right)<n/2$, the operand $x_m^{″}$ should be logically left-shifted by $l=j\cdot \left(m+n/2\right)\cdot k\left(\mathrm{mod}192\right)$ bits, where $0\le l\le 96$. When $0<l<64$, the specific logical left shift is described as Equation (14):

$$\begin{array}{l}{a}_m^{\prime }=0,\\ b_m^{\prime }=c_m>>\left(64-l\right),\\ c_m^{\prime }=c_m<<l.\end{array}$$

(14)

when $64\le l<128$, the specific logical left shift is described as Equation (15):

$$\begin{array}{l}{a}_m^{\prime }=c_m>>\left(128-l\right),\\ b_m^{\prime }=c_m<<\left(l-64\right),\\ c_m^{\prime }=0.\end{array}$$

(15)

We can obtain a 192-bit operand $x_m^{″}$, which is composed of three 64-bit unsigned numbers $a_m^{\prime }$, $b_m^{\prime }$, and $c_m^{\prime }$. Due to the fact $0\le l\le 96$, logical left shift does not exceed the limit of 192 bits, so there is no need for a cyclic shift. Each thread can obtain n/2 192-bit operands. Then, we will sum them up. The addition process is described as Equation (16):

$$\begin{array}{l}w=\mathrm{trunc}\left({\displaystyle \sum _{m=0}^h{a}_m^{\prime }}\right), \\ y=\mathrm{trunc}\left({\displaystyle \sum _{m=0}^h{b}_m^{\prime }}\right), \\ z=\mathrm{trunc}\left({\displaystyle \sum _{m=0}^h{c}_m^{\prime }}\right).\end{array}$$

(16)

The function $\mathrm{trunc}\left(\right)$ means that we obtain the lower 64 bits of the 192-bit input. The value of h in Equation (16) varies depending on the parity of k. When k is odd, h is $n/2-1$ and $m=0, 1, \dots , n/2-1$. When k is even, h is $3\cdot n/4-1$ and $m=0, 1, \dots , n/4-1,n/2, \dots , 3\cdot n/4-1$. The maximum value of l is 96, so each $a_m^{\prime }$ will not exceed 32 bits and the summation of $a_m^{\prime }$ will not exceed 64 bits. Therefore, there is no carry generated when we calculate $w$. When calculating $y$, we use a variable u to represent the carry. When calculating $z$, we use a variable v to represent the carry. Then, the 192-bit version s of the result of the k-th thread $X_k^{\prime }$ can be described as Equation (17):

$$s=\left(w+u\right)\cdot 2^{128}+\left(y+v\right)\cdot 2^{64}+z.$$

(17)

3

Turn back to 64-bit version

Let $y^{\prime }=\mathrm{trunc}\left(y+v\right)$ and $w^{\prime }=\mathrm{trunc}\left(w+u\right)$. We can obtain s as described in Equation (18):

$$s=w^{\prime }\cdot 2^{128}+y^{\prime }\cdot 2^{64}+z\left(\mathrm{mod}q\right).$$

(18)

Because of the nature of NTT’s primitive roots, we can simplify $s$ as described in Equation (19):

$$s=2^{32}\left(y^{\prime }-w^{\prime }\right)+z-y^{\prime }\left(\mathrm{mod}q\right).$$

(19)

Let $\mathrm{modadd}\left(a,b\right)=\mathrm{mod}q\left(a+b\right)$ express the modulo addition of two 64-bit unsigned numbers a and b, then the result of the k-th thread $X_k^{\prime }$ can be described in Equation (20):

$$X_k^{\prime }=\mathrm{modadd}(\mathrm{left}32(\mathrm{modsub}(y^{\prime },w^{\prime })),\mathrm{modsub}(z-y^{\prime })).$$

(20)

The $\mathrm{left}32\left(a\right)$ means to shift the 64-bit unsigned number a to the left by 32 bits and returns the result. This process can be described as Equation (21):

$$\mathrm{left}32(a)=\mathrm{modp}((a>>32)<<32-(a>>32)+(a<<32)).$$

(21)

The result of the n-point NTT is composed by the result of all the n threads in one group.

2.2.6. N-Point NTT

We can obtain the result of the n-point NTT through Section 2.2.5. Then, we compute the result ${\displaystyle \sum _{i_2=0}^{n-1}{W}_n^{i_2\cdot k_2}\cdot x_{t\cdot i_2+i_1}}$ of each thread multiplied by $W_N^{i_1\cdot k_2}$. After this, we store it as $x_{t\cdot k_2+i_1}=\left({\displaystyle \sum _{i_2=0}^{n-1}{W}_n^{i_2\cdot k_2}\cdot x_{t\cdot i_2+i_1}}\right)W_N^{i_1\cdot k_2}$. Next, we calculate the result of the N-point NTT as the formula Equation (22). Actually, we can see that it is the calculation of a t-point NTT.

$$X_{n\cdot k_1+k_2}={\displaystyle \sum _{i_1=0}^{t-1}{W}_t^{i_1\cdot k_1}}\cdot x_{t\cdot k_2+i_1}\left(\mathrm{mod} q\right).$$

(22)

If t is small enough, such as 4, 8, 16, or 32, we can obtain the result of the N-point NTT through the calculation of n t-point NTTs. The calculation of the t-point NTT is the same as the n-point NTT.

If t is bigger than 32, the calculation process of the t-point NTT is similar to the process of the N-point NTT, in which the t-point NTT can be decomposed into t/n n-point NTTs and n t/n-point NTTs. If t/n is still a big number, the t/n-point NTT can be further decomposed by using the same method.

2.2.7. Multiplication of Polynomials

First, obtain $\mathrm{NTT}\left(A_{j,q}\right)$ for each $A_{j,q}={\displaystyle \sum _{i=0}^{N-1}{c}_{i,j,q}{x}^i}$ by using the previous N-point NTT method. And then, obtain $\mathrm{NTT}\left({\mathrm{BK}}_{0,j,q}\right)$ for each ${\mathrm{BK}}_{0,j,q}$ by using the previous N-point NTT method. Next, we should calculate the multiplication of $\mathrm{NTT}\left(A_{j,q}\right)$ and $\mathrm{NTT}\left({\mathrm{BK}}_{0,j,q}\right)$. At last, we can calculate the INTT on all multiplications using Equation (23):

$$x_i=\mathrm{INTT}(X_k)=\frac{1}{N}{\displaystyle \sum _{k=0}^{N-1}{X}_k\cdot \left(W_{2N}\right)\cdot \left(W_N^{-i\cdot k}\right)}.$$

(23)

According to the formula shown in Equation (24), we can add all the results of INTT, for $j=1, 2, \dots , K+1$ and $q=1, 2, \dots ,L$, and then continue to add ${\mathrm{ACC}}_{tmp}$ to obtain ${\mathrm{ACC}}_1$.

$${\mathrm{ACC}}_1={\displaystyle \sum _{j=1}^{K+1}{\displaystyle \sum _{q=1}^L\mathrm{INTT}\left(\mathrm{NTT}\left(A_{j,q}\right)\cdot \mathrm{NTT}\left({\mathrm{BK}}_{0,j,q}\right)\right)}}+{\mathrm{ACC}}_{tmp}.$$

(24)

2.2.8. The Result of Bootstrapping

First, store the value ${\mathrm{ACC}}_1$ obtained above on the shared memory. Then, take ${\mathrm{ACC}}_d$ and ${\mathrm{BK}}_{d+1}$, $d=1,2,\dots ,p-1$ as the blind rotation input to repeat Section 2.2.2 to Section 2.2.7 and we can obtain the blind rotation output ${\mathrm{ACC}}_{d+1}$, $d=1,2,\dots ,p-1$. Next, for all polynomials $A_j^{\prime }\left(j=1, 2, \dots , K+1\right)$ in ${\mathrm{ACC}}_p=\left(A_1^{\prime }, A_2^{\prime }, \dots , A_{K+1}^{\prime }\right)$, we execute the sample extraction in bootstrapping to extract the coefficient to form a new TLWE ciphertext $c^{\prime }=(c_0,c_1,\dots ,c_{K\cdot N}={\mu }^{\prime })$. Finally, we perform key switching in bootstrapping. Let ${c_j}^{\prime }$ be the closest multiple of $\frac{1}{2^{t^{\prime }}}$ to $c_j$ and set $c_j^{\prime }={\displaystyle \sum _{r=1}^{t^{\prime }}{c}_{j,r}^{\prime }}\cdot 2^{-r}{c^{\prime }}_j$, where $t^{\prime }\in \mathbb{N}$ is a precision parameter and $c_{j,r}^{\prime }\in \left\{0, 1\right\}$. At last, the result of bootstrapping is shown in Equation (25):

$$c^{″}=\left(0, c_K+{\mu }^{\prime }\right)-{{\displaystyle \sum _{j=1}^K{\displaystyle \sum _{r=1}^{t^{\prime }}{c}_{j,r}^{\prime }\cdot {\mathrm{KSK}}_{j,r}}}}_{.}$$

(25)

3. Results

As we know, we can improve the efficiency of bootstrapping from different aspects, such as optimizing blind rotation and accelerating NTT. Our solution primarily aims to accelerate NTT on a GPU. In this direction, the current implementation of NTT based on the butterfly algorithm in cuFHE [24] is the best. Therefore, we chose NTT based on the butterfly algorithm for comparison. In our scheme, we adopt the original matrix structure that is more suitable for a GPU to calculate number theoretic transform and add a merging preprocessing operation to parallelize the multiplication process. It should be noted that we have implemented two versions of the proposed scheme, denoted as our scheme and our scheme (no merging), in order to demonstrate the effect of the merging preprocessing operation. In our scheme, we adopt the multiplication of the matrix with the merging operation, while in our scheme (no merging), we adopt the multiplication of the matrix without the merging operation. All the schemes are implemented in C++ 17, and the main functions run on Intel R Core (TM) i9-9900k at 3.60 GHz. The NTT algorithms were run on a TU102 [GeForce] RTX 2080 Ti graphics card. Because the running time is very short, the NTT algorithms were run 10,000 times on a GPU and we took the average time as the result. The performance comparison is shown in

Table 1. The running time of small-point NTT by three different methods.

	4-Point NTT	8-Point NTT	16-Point NTT	32-Point NTT
Our scheme	0.72	1.37	2.93	5.28
Our scheme (no merging)	1.18	2.33	4.78	9.42
Butterfly algorithm [27]	0.98	2.49	7.11	19.52

and the time unit is milliseconds (ms).

In order to have a clearer view of the time consumption trends for each scheme, we provide the corresponding curve of Table 1 in Figure 3. From Figure 3, it can be observed that the running time of both our schemes increases approximately linearly as the scale of NTT expands. In comparison to our solutions, the butterfly algorithm exhibits a longer running time, and as the number of points N increases, its running time grows more quickly. The running time of our scheme (no merging) is longer than that of our scheme, which shows the importance of the merging operation.

Furthermore, we designed experiments to compare the performance of a 1024-point NTT realized by different decomposition methods using our scheme and the responding results are shown in

Table 2. The running time of a 1024-point NTT achieved using different decomposition methods.

Decomposition Method	4 × 4 × 4 × 4 × 4	8 × 8 × 16	16 × 16 × 4	32 × 32
Running time (ms)	7.60	6.03	7.24	11.45

. To intuitively demonstrate the running time of different decomposition methods, we converted Table 2 into Figure 4. As before, each of our solutions was run 10,000 times on a GPU and we took the average time as the result. From Figure 4, it can be seen that it is wrong to say that the more times a decomposition is run, the shorter the running time. Because we should calculate a multiplication between any two small-point NTTs, which cannot be replaced by using the logical left-shift operation. That is to say, the more decomposition time, the longer the multiplication time. At the same time, we can see that the decomposition of 8 × 8 × 16 is a suitable method. In this situation, the running time of a 1024-point NTT based on our scheme is only 6.03 ms while that based on the butterfly algorithm is 15.01 ms. The speedup ratio of our method over the butterfly algorithm is about 2.49.

4. Discussion

In this paper, we presented an accelerated NTT algorithm based on GPU, which is suitable for polynomial multiplication in the process of gate bootstrapping of the TFHE scheme. In our scheme, we adopt the original matrix structure that is more suitable for a GPU to calculate number theoretic transform and add a merging preprocessing operation to parallelize the multiplication process, aiming to improve the running speed of number theoretic transform. Furthermore, we improved computational efficiency by logical left shifts instead of multiplication when computing the NTT value of a small point. Compared with the current mainstream Cooley–Turkey NTT algorithm, our scheme greatly reduces the use of a GPU’s shared memory and avoids the inherent sequential bottleneck between the front and back layers of the butterfly algorithm. In summary, when using our scheme instead of the butterfly algorithm to realize a 1024-point NTT, we can obtain a speedup ratio as large as 2.49 or so. Of course, our scheme is not perfect either. For example, fully exploring the allocation of multiple threads on GPU to adapt to our scheme is left for future research.