Application of Tensor Decomposition to Reduce the Complexity of Neural Min-Sum Channel Decoding Algorithm

Abstract

Channel neural decoding is very promising as it outperforms the traditional channel decoding algorithms. Unfortunately, it still faces the disadvantage of high computational complexity and storage complexity compared with the traditional decoding algorithms. In this paper, we propose that low rank decomposition techniques based on tensor train decomposition and tensor ring decomposition can be utilized in neural offset min-sum (NOMS) and neural scale min-sim (NSMS) decoding algorithms. The experiment results show that the proposed two algorithms achieve near state-of-the-art performance with low complexity.

1. Introduction

Driven by breakthroughs of deep learning (DL) in image processing, natural language processing and other fields, researchers have proposed channel decoding algorithms based on DL in recent years. DL-based decoding has now become an active area of research. In [1], neural belief propagation (BP) for linear codes was proposed and then the authors applied recurrent neural network (RNN) to linear block codes [2]. Improving decoding performance by adding appropriate weight parameters and loss functions to the decoding algorithm, where the loss function is used to train these weight parameters, is called the model-driven decoding algorithm. The offset min-sum (OMS) decoding algorithm is an excellent decoding method with low complexity and high performance [3], and is widely used in Hamming codes, BCH and low-density parity check (LDPC) codes. However, there is still room for performance improvement in these schemes. Lugosch and W. J. G proposed a neural offset min-sum (NOMS) decoding algorithm in 2017. The authors in [4] proposed the model-driven deep learning neural OMS decoding algorithm for LDPC code, and proposed the neural share offset min-sum (NSOMS) decoding algorithm with lower complexity. In [5,6,7], both multiplicative weights and offset weights are used in neural scale min-sim (NSMS), which needs more storage space and computational complexity.

The model-driven BP decoding algorithm achieves state-of-the-art performance and outperforms traditional BP decoder algorithm by adding weights at appropriate locations, which also causes an increase in storage space and computational complexity. However, model-driven BP makes practical implementation challenging. These issues can be solved by model compression techniques. Low rank decomposition is one such model compression technique. Fully connected deep neural network (DNN) parameters are converted into tensor train format through tensor train (TT) decomposition [8]. In [9], it achieves seven times the compression while obtaining a state-of-the-art performance, which converts the fully connected layer to the tensorizing layer. In [10,11,12,13], they converted the dense weight matrices of the fully connected layers and convolutional layers to the tensorizing layer. Recurrent layers in [14,15] are converted to TT-Recurrent layers. Since tensor decomposition is highly dependent on rank, it is difficult to find the optimal rank that minimizes the error of the product of the decomposed tensors and the original tensor. However, the restricted setting of ranks and restricted ordering of core tensors make TT-based compression models sensitive to parameter selection.

The authors of [16] proposed tensor ring (TR) decomposition, which satisfies the circular dimensional permutation invariance. The authors of [17] proposed a TR-based long short-term memory (TR-LSTM) model to reconstruct the input-to-hidden transformation by exploiting tensor ring decomposition, which achieved higher accuracy and higher compression rate than the TT-based long short-term memory (TT-LSTM). TR networks are proposed in [18]. The authors of [19] proposed nonlinear tensor ring networks (NTRN) to compress fully linked neural networks and convolutional neural networks (CNN), which adds a nonlinear activation function after the convolution operation inside the compression layer. TT-based neural normalized min-sum (TT-NNMS) was proposed in [20], which reduces the number of parameters and multiplications. However, ref. [20] only performed TT decomposition for the NNMS decoding algorithm, and did not perform TT decomposition for NOMS. The NOMS algorithm is different from the NNMS algorithm, and the offset addition replaces the factor multiplication in the neural normalized min-sum (NNMS) algorithm. In addition, ref. [20] only applied TT decomposition to the decoding algorithm of LDPC codes, and does not apply it to BCH and Hamming codes.

In this work, methods to reduce the complexity of NOMS and the NSMS decoding algorithm are proposed, which involve using TT decomposition or TR decomposition, and apply it to the decoding of Hamming code, BCH codes and LDPC code. The novelty of this paper is summarized as follows:

• As a continuation of the work in paper [20], we apply the TT decomposition to the NOMS and NSMS decoding algorithms, called the TT-NOMS and TT-NSMS algorithms, which can reduce the complexity of the algorithm.
• We apply the TR decomposition to the NOMS and NSMS decoding algorithms, and achieve the same compression effect as TT, but TR decomposition has more flexible choices in rank.
• In addition, we apply the TT-NOMS, TT-NSMS, TR-NSMS, and TR-NOMS algorithms to the decoding of Hamming code, BCH codes and LDPC code, which reduces the complexity of these decoding algorithms, while the performance is almost not degraded.

The remainder of this paper is organized as follows: OMS and SMS, NOMS and NSMS, TT decomposition and TR decomposition are described briefly in Section 2. In Section 3, we propose the TT-NOMS and TT-NSMS decoding algorithms. TR-NOMS and TR-NSMS decoding algorithms are proposed in Section 4. Simulations results and the complexity comparison are shown in Section 5. Section 6 shows the performance evaluation with NOMS, NSMS, TT-NOMS, TT-MSMS, TR-NOMS and TR-NSMS algorithm. Section 7 concludes the paper.

2. Preliminary

2.1. Offset Min Sum Decoder and Scale Min Sum Decoder

We consider (N,K) LDPC, BCH and Hamming code where N denotes the length of codeword, K denotes the number of information bits in the message. We use binary phase shift keying (BPSK) modulation, where the check matrix is H. The message X is encoded by generator matrix G. The message obtained is $Y=X+Z$ over the additive white gaussian noise (AWGN) channel, where $Z\in {\mathbb{R}}^n$ is the channel noise. The log-likelihood ratio (LLR) of the v-th bit can be computed as

$$l_v=log(\frac{P(Y_v=y_v|X_v=-1)}{P(Y_v=y_v|X_v=+1)})=\frac{2y_v}{{\sigma }^2}$$

(1)

where ${\sigma }^2=N_0/2$ is the variance of the channel noise, $N_0$ is noise spectral density, and we use $E_b/N_0=1/\left(2(K/N){\sigma }^2\right)$ as the signal-to-noise ratio, where $E_b$ is the energy per information bit, $y_v$ is the v-th value of the received message.

The decoding graph is a bipartite graph, which consists of variable nodes (VNs) and check nodes (CNs). LLRs $l_v$ are transmitted from the VNs $u_{v,c}^t$ to the CNs $u_{c,v}^t$, and then return back. Each iteration requires the following steps:

Step 1: From the VNs to CNs the message $u_{v,c}^t$ is passed in the graph as.

$$u_{v,c}^t=l_v+\sum _{c^{{}^{\prime }}\in N\left(v\right)\backslash c}{u}_{c^{{}^{\prime }},v}^{t-1}.$$

(2)

where the neighboring node set v consisting of CNs is denoted as $N\left(v\right)$, the symbol $N\left(v\right)\backslash c$ denotes the set $N\left(v\right)$ except the check node c, $u_{v,c}^0$ is set to 0, add t denotes the number of iteration.

Step 2: From the CNs to VNs the message $u_{v,c}^t$ is passed in the graph as.

$$u_{c,v}^t=max(\underset{v^{{}^{\prime }}\in M\left(c\right)\backslash v}{min}|u_{v{}^{\prime },c}^t|-\beta ,0)•\prod _{v^{{}^{\prime }}\in M\left(c\right)\backslash v}sign(u_{v^{{}^{\prime }},c}^t).$$

(3)

where the neighboring node set c consisting of VNs is denoted as $M\left(c\right)$, the symbol $M\left(c\right)\backslash v$ denotes the set $M\left(c\right)$ except the variable node v.

Step 3: The soft estimation message $s_v^t$ after t iterations is written as

$$s_v^t=l_v+\sum _{c^{{}^{\prime }}\in N\left(v\right)}{u}_{c^{{}^{\prime }},v}^t.$$

(4)

Repeat Step 1, 2, 3 until conditions of iterations are reached. The recovered binary vector ${\widehat{x}}_v^t$ is computed by $s_v^t$ as

$${\widehat{x}}_v^t=\left\{\begin{array}{cc}1,\hfill & if{s}_v^t>0\\ 0,\hfill & otherwise,\end{array}\right.$$

(5)

In the iterative process, we use the Equation (6) below to replace the Equation (3) in step 2 of CNs layers, and the decoding process is collectively called scale min-sim (SMS) decoder.

$$u_{c,v}^t=(\alpha \times \underset{v^{{}^{\prime }}\in M\left(c\right)\backslash v}{min}|u_{v{}^{\prime },c}^t|-\beta )•\prod _{v^{{}^{\prime }}\in M\left(c\right)\backslash v}sign(u_{v^{{}^{\prime }},c}^t).$$

(6)

2.2. Neural Offset Min Sum Network and Neural Scale Min Sum Network

The NOMS algorithm takes the messages transmitted by each edge and subtracts a number of learnable parameters. The task of the network decoder is to use the received vector y to recover the messages ${\widehat{x}}_v$. Input layer, output layer and hidden layers make up the network, as shown in Figure 1. Each of the hidden layer has one CNs layer and one VNs layer. The LLR of the received message is computed by Equation (1).

The CNs layer is given as [21]

$$u_{c,v}^t=max(\underset{v^{{}^{\prime }}\in M(c\backslash v)}{min}|u_{v^{{}^{\prime }},c}^t|-{\beta }_{v^{{}^{\prime }},c}^t,0)•\prod _{v^{{}^{\prime }}\in M(c\backslash v)}sign(u_{v^{{}^{\prime }},c}^t).$$

(7)

where the learning parameter is ${\beta }_{v^{{}^{\prime }},c}^t$.

The VNs layer is given as [21,22]

$$u_{v,c}^t=w_{v,c}^t(l_v+\sum _{c^{{}^{\prime }}\in N(v\backslash c)}{u}_{c^{{}^{\prime }},v}^{t-1}).$$

(8)

where the learning parameter is $w_{v,c}^t$.

Finally, the output of each iteration can be expressed as

$$s_v^t=w_v^t(l_v+\sum _{c^{{}^{\prime }}\in N\left(v\right)}{u}_{c^{{}^{\prime }},v}^t),$$

(9)

$$o_i=\sigma \left(s_v^t\right)$$

(10)

where $\sigma \left(x\right)={(1+e^{-x})}^{-1}$ is the activation function. The cross-entropy loss can be expressed as

$$L\left(x,o\right)=-\frac{1}{N}\sum _{i=1}^N{x}_{i}log\left(o_i\right)+(1-x_i)log(1-o_i)$$

(11)

where $x_i$ is the information bit transmitted, and $o_i$ is the estimated information bit.

$$\begin{array}{c}\hfill u_{c,v}^t=max({\alpha }_{v^{{}^{\prime }},c}^t\times \underset{v^{{}^{\prime }}\in M(c\backslash v)}{min}|u_{v^{{}^{\prime }},c}^t|-{\beta }_{v^{{}^{\prime }},c}^t,0)\\ \hfill •\prod _{v^{{}^{\prime }}\in M(c\backslash v)}sign(u_{v^{{}^{\prime }},c}^t).\end{array}$$

(12)

In the iterative layer, if we replace Equation (7) with Equation (12) in CNs layers, then it becomes the NSMS decoding algorithm.

2.3. Tensor-Train Decomposition

Assuming that there is a high-order tensor $\mathcal{X}$, a sequence of third-order core tensors ${\mathcal{G}}^{\left(1\right)},{\mathcal{G}}^{\left(2\right)},\dots ,{\mathcal{G}}^{\left(N\right)}$ are used to express $\mathcal{X}$, which is called TT decomposition. $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_n}$ is given by

$$\mathcal{X}=\ll {\mathcal{G}}^{\left(1\right)},{\mathcal{G}}^{\left(2\right)},\dots ,{\mathcal{G}}^{\left(N\right)}\gg$$

(13)

where $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_n}$, $\mathcal{G}\in {\mathbb{R}}^{R_{n-1}\times I_n\times R_n}$, $R_0=R_N=1$. The notation $\ll \gg$ represents the approximation of the core tensor sequence to the tensor $\mathcal{X}$. $I_0,I_1,\dots ,I_N$ are used to define the dimension of $\mathcal{X}$. $R_0,R_1,\dots ,R_N$ determine the size of each core tensor, which is called TT rank. Smaller TT rank yields a smaller overall dimension of the series $\mathcal{G}$, which will greatly reduce the complexity. In addition, the $(i_1,i_2,\dots ,i_N)$-th element of the tensor $\mathcal{X}$ can be expressed as

$$x_{i_1,i_2,\dots ,i_N}=\prod _{n=1}^N{G}_{i_n}^{\left(n\right)}.$$

(14)

where $G_{i_1}^{\left(1\right)},G_{i_2}^{\left(2\right)},\dots ,G_{i_N}^{\left(N\right)}$ is the sequence of slices obtained from each core tensor. There is $\mathcal{O}\left(dI{R}^2\right)$ spatial complexity of TT format, where R is the maximum value of the TT ranks, I is the maximum value of modes in $I_1,I_2,\dots ,I_n$, and d is the order of the tensor.

For completeness, the background on TT decomposition is explained in the following. For further studies, readers can refer to [8,23]. TT decomposition employs a sequence of singular value decomposition (SVD) or the cross approximation algorithm. Assume that we want to approximate a given tensor $\mathcal{A}$ by a tensor $\mathcal{X}$, that is $\mathcal{X}\approx \mathcal{A}$, with a prescribed accuracy $ϵ$. We then need to find the optimal rank and the optimal series of core tensors, ${\mathcal{G}}^{\left(1\right)},{\mathcal{G}}^{\left(2\right)},\dots ,{\mathcal{G}}^{\left(N\right)}$ under the condition ${\Vert \mathcal{X}-\mathcal{A}\Vert }_F\le ϵ$. It follows that N consecutive shaping and SVD calculations constitute the computation of the TT decomposition. The detailed steps of TT decomposition via recursive SVD are as follows. First, we validate the TT rank, $R_0,R_1,\dots ,R_N$, with below rules: (1) the length of rank should match with the dimension of $\mathcal{A}$, (2) consecutive factors should have matching ranks, (3) boundary conditions dictate that the head and tail rank set to 1. Second, we set ${\mathcal{X}}_1=\mathcal{X}$, and we will find the optimal ${\mathcal{G}}^{\left(1\right)},{\mathcal{G}}^{\left(2\right)},\dots ,{\mathcal{G}}^{(N-1)}$ one by one. We flatten the unfolding tensor ${\mathcal{X}}_1$ of the remaining factors, where ${\mathcal{X}}_1\in {\mathbb{R}}^{I_1\times I},I=I_2\times \dots \times I_n$. We get 1-st TT factor U in ${\mathcal{X}}_1=US{V}^T$ with U as a $I_1\times I_1$ matrix and V as an $I\times I_1$ matrix. The head TT core ${\mathcal{G}}^{\left(1\right)}=U$ is given by the U indexed by $R_0,I_1,R_1$. SVD is chosen for its numerical stability in the TT algorithm for binary analysis in this step. Then, the new unfolding tensor for the remaining factors is ${\mathcal{X}}_1=S{V}^T$. Repeat the second step $N-2$ times, and reshape ${\mathcal{X}}_1$ such that its rows are indexed by $R_{n-1},I_n,\forall n\in [N-1]$ and its columns by remaining portion. Then we will get each core, ${\mathcal{G}}^{\left(1\right)},{\mathcal{G}}^{\left(2\right)},\dots ,{\mathcal{G}}^{(N-1)}$ which are reshaped into ${\mathcal{G}}^{\left(n\right)}\in {\mathbb{R}}^{R_{n-1}\times I_n\times R_n}$ tensors, $\forall n\in [N-1]$. Finally, the last factor ${\mathcal{G}}^{\left(N\right)}\in {\mathbb{R}}^{R_{N-1}\times I_N\times R_N}$ is obtained by ${\mathcal{X}}_1$. Suppose there is a three-dimensional tensor $\mathcal{X}\in {\mathbb{R}}^{3\times 4\times 2}$; we set rank $R_0=1,R_1=3,R_2=2,R_3=1$, and the detailed TT decomposition process is shown in Figure 2. By reshaping tensor $\mathcal{X}$ into a $3\times 8$ matrix ${\mathcal{X}}_1$, SVD of the matrix is then ${\mathcal{X}}_1=US{V}^T$. The head core $3\times 3$ tensors is ${\mathcal{G}}^{\left(1\right)}=U$. Set ${\mathcal{X}}_1=S{V}^T$, ${\mathcal{X}}_1$ is reshaped into a $12\times 2$ tensor. Again, SVD decomposition was carried out on ${\mathcal{X}}_1$. The middle and tail core tensors are ${\mathcal{G}}^{\left(2\right)}=U,{\mathcal{G}}^{\left(3\right)}=S{V}^T$ after reshaping into $3\times 4\times 2$ and $2\times 2$. The head and tail core tensors ${\mathcal{G}}^{\left(1\right)},{\mathcal{G}}^{\left(3\right)}$ are in fact matrices.

2.4. Tensor-Ring Decomposition

Tensor ring decomposition is proposed on the basis of tensor train decomposition [16]. The main advantage of the tensor ring decomposition is that the tensor is used in the form of a ring. With the first rank equal to the last rank, so that there are more optimal rank choices. Assuming that $\mathcal{T}$ is a d-dimensional tensor, denoted by $\mathcal{T}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_d}$. After TR decomposition, $\mathcal{T}$ can be expressed as a series of low rank tensors $\mathcal{Z}\in {\mathbb{R}}^{r_k\times I_k\times r_{k+1}}$, $k=1,2,\dots ,d$, which can be expressed as, according to [16],

$$\mathcal{T}(i_1,i_2,\dots ,i_d)=Tr\left\{Z_1\left(i_1\right)Z_2\left(i_2\right)\dots Z_d\left(i_d\right)\right\},$$

(15)

where $\mathcal{T}(i_1,i_2,\dots ,i_d)$ represents the $(i_1,i_2,\dots ,i_d)$th element of the original tensor $\mathcal{T}$. $Z_k\left(i_k\right)$ represents the $i_k$th slice matrix of the original tensor $\mathcal{T}$. Tensor ring decomposition can also be expressed in the index form:

$$\mathcal{T}(i_1,i_2,\dots ,i_d)=\sum _{a_1,\dots ,a_d=1}^{r_1,\dots ,r_d}\prod _{k=1}^d{Z}_k(a_k,i_k,a_{k+1}).$$

(16)

where k is the index of the tensor dimension, $a_k$ is the k-th rank, and $i_k$ is the index of data dimensions. We can also write Equation (16) in tensor format, given by

$$\mathcal{T}=\sum _{a_1,\dots ,a_d=1}^{r_1,\dots ,r_d}{z}_1(a_1,a_2)\circ z_1(a_2,a_3)\circ \dots \circ z_d(a_d,a_1).$$

(17)

where the mathematical notation ∘ indicates the outer product of vectors; $z_k(a_k,a_{k+1})\in {\mathbb{R}}^{I_k}$ indicates the $(a_k,a_{k+1})$th mode-2 fiber of tensor ${\mathcal{Z}}_k$. There is $\mathcal{O}\left(dI{R}^2\right)$ spatial complexity of TR format, which is linear to the order of the tensor d. TR satisfies the property of circular dimensional permutation invariance, which is the property of TR decomposition different from that of TT decomposition.

The steps to decompose the TR tensor in detail are the same as those of TT discussed in Section 2.3. The only difference is that when verifying rank, $R_0=R_N$ in TR decomposition, So, these tensor cores form a ring. $R_0=R_N=1$ in TT decomposition is a special case of TR decomposition, so that TR decomposition subsumes TT decomposition, and its rank selection is more than that of TT decomposition. Thus, there are more chances of obtaining the optimal core sequence tensors $\mathcal{Z}\in {\mathbb{R}}^{r_k\times I_k\times r_{k+1}}$, $k=1,2,\dots ,d$ of optimal rank in TR decomposition.

3. TT-NOMS and TT-NSMS Channel Decoding Algorithm

The NMS algorithm adds a multiplicative factor to the MS algorithm, and then expands it into an NNMS network decoding algorithm through the unfold method. Different from the NMS algorithm, the OMS algorithm adds an additive factor to the MS algorithm, and there is no multiplicative factor. Whether the TT and related tensor decomposition algorithms are suitable for the NOMS algorithm needs further verification. Parameters matrices are decomposed by TT in NOMS algorithm, called TT-NOMS. In Equation (7), the scalar $\beta$ in the traditional OMS algorithm becomes the tensor ${\beta }_{v^{{}^{\prime }},c}^t$ in each iteration for each edge of NOMS algorithm, which results in additional complexity in the calculation of information transfer for CNs layers. In Equations (8) and (9), tensors $w_{v,c}^t$ and $w_v^t$ are additionally added as training parameters in the neural decoding algorithm. First, we transform the three-dimensional parameter tensors ${\beta }_{v^{{}^{\prime }},c}^t$, $w_{v,c}^t$, $w_v^t$ into four-dimensional tensors ${\mathit{\beta }}_{{\mathit{v}}^{{}^{\prime }},\mathit{c}}^{\mathit{t}}$, ${\mathit{w}}_{\mathit{v},\mathit{c}}^{\mathit{t}}$, ${\mathit{w}}_{\mathit{v}}^{\mathit{t}}$ respectively. Then we use TT decomposition on the resulting three four-dimensional tensors as shown below,

$${\mathit{\beta }}_{v^{{}^{\prime }},c}^t={\beta }_{v^{{}^{\prime }},c}^{1t}{\beta }_{v^{{}^{\prime }},c}^{2t}{\beta }_{v^{{}^{\prime }},c}^{3t}{\beta }_{v^{{}^{\prime }},c}^{4t}.$$

(18)

$${\mathit{w}}_{v,c}^t=w_{v,c}^{1t}{w}_{v,c}^{2t}{w}_{v,c}^{3t}{w}_{v,c}^{4t}.$$

(19)

$${\mathit{w}}_v^t=w_v^{1t}{w}_v^{2t}{w}_v^{3t}{w}_v^{4t}.$$

(20)

Substituting Equations (18)–(20) into Equations (7)–(9), we have

$$\begin{array}{c}\hfill u_{c,v}^t=max(\underset{v^{{}^{\prime }}\in M(c\backslash v)}{min}|u_{v^{{}^{\prime }},c}^t|-{\beta }_{v^{{}^{\prime }},c}^{1t}{\beta }_{v^{{}^{\prime }},c}^{2t}{\beta }_{v^{{}^{\prime }},c}^{3t}{\beta }_{v^{{}^{\prime }},c}^{4t},0)\\ \hfill •\prod _{v^{{}^{\prime }}\in M(c\backslash v)}sign(u_{v^{{}^{\prime }},c}^t),\end{array}$$

(21)

$$\begin{array}{c}\hfill u_{v,c}^t=w_{v,c}^{1t}{w}_{v,c}^{2t}{w}_{v,c}^{3t}{w}_{v,c}^{4t}(l_v+\sum _{c^{{}^{\prime }}\in N(v\backslash c)}{u}_{c^{{}^{\prime }},v}^{t-1}),\end{array}$$

(22)

$$\begin{array}{c}\hfill s_v^t=w_v^{1t}{w}_v^{2t}{w}_v^{3t}{w}_v^{4t}(l_v+\sum _{c^{{}^{\prime }}\in N\left(v\right)}{u}_{c^{{}^{\prime }},v}^t).\end{array}$$

(23)

In the TT-NOMS algorithm, we use Equations (21)–(23) to calculate CNs layers’ information, VNs layers’ information, and the soft output information, respectively, instead of Equations (7)–(9) in the NOMS algorithm. The loss function is then expressed as

$$L\left(s^t\right)=\frac{1}{N-k}\sum _{i=1}^{N-k}max(1-softsynd{\left(s\right)}_c^t,0).$$

(24)

Syndrome loss function [24] in TT-NOMS is used in Equation (24), where $softsynd{\left(s\right)}_c^t$ denotes the syndrome, t denotes the number of iteration and $s_v^t$ denotes the estimated message. The syndrome loss calculates errors through estimated information $s_v^t$ without the need of labels, so NOMS is called unsupervised neural network. The syndrome can be expressed as

$$softsynd{\left(s\right)}_c^t=\underset{v\in M\left(c\right)}{min}{s}_v^t\prod _{v\in M\left(c\right)}sign\left(s_v^t\right)$$

(25)

In TT-NSMS algorithm, Equation (12) is replaced by Equation (26), two weights ${\alpha }_{v^{{}^{\prime }},c}^t$, ${\beta }_{v^{{}^{\prime }},c}^t$ and messages ${min}_{v^{{}^{\prime }}\in M(c\backslash v)}|u_{v^{{}^{\prime }},c}^t|$ are decomposed by TT into four low-rank weights after reshaping in high order, where ${\alpha }_{v^{{}^{\prime }},c}^t={\alpha }_{v^{{}^{\prime }},c}^{1t}{\alpha }_{v^{{}^{\prime }},c}^{2t}{\alpha }_{v^{{}^{\prime }},c}^{3t}{\alpha }_{v^{{}^{\prime }},c}^{4t}$, ${\beta }_{v^{{}^{\prime }},c}^t={\beta }_{v^{{}^{\prime }},c}^{1t}{\beta }_{v^{{}^{\prime }},c}^{2t}{\beta }_{v^{{}^{\prime }},c}^{3t}{\beta }_{v^{{}^{\prime }},c}^{4t}$, ${min}_{v^{{}^{\prime }}\in M(c\backslash v)}|u_{v^{{}^{\prime }},c}^t|=U_{v^{{}^{\prime }},c}^{1t}{U}_{v^{{}^{\prime }},c}^{2t}{U}_{v^{{}^{\prime }},c}^{3t}{U}_{v^{{}^{\prime }},c}^{4t}$. That is,

$$\begin{array}{c}\hfill u_{c,v}^t=max({\alpha }_{v^{{}^{\prime }},c}^{1t}{\alpha }_{v^{{}^{\prime }},c}^{2t}{\alpha }_{v^{{}^{\prime }},c}^{3t}{\alpha }_{v^{{}^{\prime }},c}^{4t}\times U_{v^{{}^{\prime }},c}^{1t}{U}_{v^{{}^{\prime }},c}^{2t}{U}_{v^{{}^{\prime }},c}^{3t}{U}_{v^{{}^{\prime }},c}^{4t}\\ \hfill -{\beta }_{v^{{}^{\prime }},c}^{1t}{\beta }_{v^{{}^{\prime }},c}^{2t}{\beta }_{v^{{}^{\prime }},c}^{3t}{\beta }_{v^{{}^{\prime }},c}^{4t},0)•\prod _{v^{{}^{\prime }}\in M(c\backslash v)}sign(u_{v^{{}^{\prime }},c}^t).\end{array}$$

(26)

4. TR-NOMS and TR-NSMS Channel Decoding Algorithm

The TR decomposition has circular dimensional permutation invariance property and is more expressive than the TT decomposition. Better compression ratio and lower storage consumption are obtained in practice. Therefore, we also apply the TR decomposition to the NOMS algorithm, which is called TR-NOMS algorithm. We also transform the three-dimensional parameter tensors ${\beta }_{v^{{}^{\prime }},c}^t$, $w_{v,c}^t$, $w_v^t$ into four-dimensional tensors ${\mathit{\beta }}_{{\mathit{v}}^{{}^{\prime }},\mathit{c}}^{\mathit{t}}$, ${\mathit{w}}_{\mathit{v},\mathit{c}}^{\mathit{t}}$, ${\mathit{w}}_{\mathit{v}}^{\mathit{t}}$ respectively. Then we use TR decomposition on the resulting three four-dimensional tensors as shown below,

$${\mathit{\beta }}_{v^{{}^{\prime }},c}^{r\left(t\right)}={\beta }_{v^{{}^{\prime }},c}^{r\left(1t\right)}{\beta }_{v^{{}^{\prime }},c}^{r\left(2t\right)}{\beta }_{v^{{}^{\prime }},c}^{r\left(3t\right)}{\beta }_{v^{{}^{\prime }},c}^{r\left(4t\right)}.$$

(27)

$${\mathit{w}}_{v,c}^{r\left(t\right)}=w_{v,c}^{r\left(1t\right)}{w}_{v,c}^{r\left(2t\right)}{w}_{v,c}^{r\left(3t\right)}{w}_{v,c}^{r\left(4t\right)}.$$

(28)

$${\mathit{w}}_v^{r\left(t\right)}=w_v^{r\left(1t\right)}{w}_v^{r\left(2t\right)}{w}_v^{r\left(3t\right)}{w}_v^{r\left(4t\right)}.$$

(29)

Substituting Equations (27)–(29) into Equations (7)–(9), we have

$$\begin{array}{c}\hfill u_{c,v}^t=max(\underset{v^{{}^{\prime }}\in M(c\backslash v)}{min}|u_{v^{{}^{\prime }},c}^t|-{\beta }_{v^{{}^{\prime }},c}^{r\left(1t\right)}{\beta }_{v^{{}^{\prime }},c}^{r\left(2t\right)}{\beta }_{v^{{}^{\prime }},c}^{r\left(3t\right)}{\beta }_{v^{{}^{\prime }},c}^{r\left(4t\right)},0)\\ \hfill •\prod _{v^{{}^{\prime }}\in M(c\backslash v)}sign\left(u_{v^{{}^{\prime }},c}^t\right),\end{array}$$

(30)

$$\begin{array}{c}\hfill u_{v,c}^t=w_{v,c}^{r\left(1t\right)}{w}_{v,c}^{r\left(2t\right)}{w}_{v,c}^{r\left(3t\right)}{w}_{v,c}^{r\left(4t\right)}(l_v+\sum _{c^{{}^{\prime }}\in N(v\backslash c)}{u}_{c^{{}^{\prime }},v}^{t-1}),\end{array}$$

(31)

$$\begin{array}{c}\hfill s_v^t=w_v^{r\left(1t\right)}{w}_v^{r\left(2t\right)}{w}_v^{r\left(3t\right)}{w}_v^{r\left(4t\right)}(l_v+\sum _{c^{{}^{\prime }}\in N\left(v\right)}{u}_{c^{{}^{\prime }},v}^t).\end{array}$$

(32)

In the TR-NOMS algorithm, we use Equations (30)–(32) to calculate CNs layers’ information, VNs layers’ information, and the soft output information, respectively, instead of Equations (7)–(9) in the NOMS algorithm. The product of four low-rank weights replaces the decomposed original weights.

In the TR-NSMS algorithm, the CNs layer is similar to Equation (26), except that the original weight decomposition algorithm is changed to TR, in which we have options for more ranks.

5. Complexity Analysis

We compare the complexity of NOMS, TT-NOMS, TT-NSMS, TR-NOMS and TR-NSMS in storage complexity and computational complexity. We only analyze the parameters and information from the CNs to the VNs in Tanner graph with other parameters being set to 1.

For all of the parameters with T iterations, that is, ${\beta }_{v^{{}^{\prime }},c}^1,{\beta }_{v^{{}^{\prime }},c}^t,\dots ,{\beta }_{v^{{}^{\prime }},c}^T\in {\mathbb{R}}^{E\times B}$ in Equation (7), the number of parameters is $EBT$. Denote E as the edge in graph, T as the number of iterations and B as the size of batch. In order to minimize the parameters and multiplication, we reshape the original tensor shape into high order $x_1,\dots ,x_D$, so $EBT={\prod }_{d=1}^D{x}_d$. Then, we use the product of a series of tensors in TT format $\mathcal{B}(1,2,\dots ,D)$ to replace the original tensor $[{\beta }_{v^{{}^{\prime }},c}^1,{\beta }_{v^{{}^{\prime }},c}^t,\dots ,{\beta }_{v^{{}^{\prime }},c}^T]$, where $\mathcal{B}\left(d\right)\in {\mathbb{R}}^{R_d-1\times x_d\times R_d}$ and rank set is $R_0,R_1,\dots ,R_D$, $\forall d\in \left[D\right]$. So, $[{\beta }_{v^{{}^{\prime }},c}^1,{\beta }_{v^{{}^{\prime }},c}^t,\dots ,{\beta }_{v^{{}^{\prime }},c}^T]={\prod }_{d=1}^D{\mathcal{B}}_d$. The TT-NOMS algorithm reduces the number of parameters of $[{\beta }_{v^{{}^{\prime }},c}^1,{\beta }_{v^{{}^{\prime }},c}^t,\dots ,{\beta }_{v^{{}^{\prime }},c}^T]$ from $EBT$ to ${\sum }_{d=1}^D{x}_d{R}_{d-1}{R}_d$, where $x_d$, $R_{d-1}$ and $R_d$ are smaller than $E,B$ and T. The few multiplications in TT-NOMS algorithm come from the multiplication of several tensors $\mathcal{B}(1,2,\dots ,D)$. TR-NOMS algorithm and TT-NOMS algorithm have the same multiplication and parameters, but with more choices of rank set. In NSMS algorithm, there are two kinds of parameters, multiplicative parameters ${\alpha }_{v^{{}^{\prime }},c}^1,{\alpha }_{v^{{}^{\prime }},c}^t,\dots ,{\alpha }_{v^{{}^{\prime }},c}^T\in {\mathbb{R}}^{E\times B}$ and offset parameters ${\beta }_{v^{{}^{\prime }},c}^1,{\beta }_{v^{{}^{\prime }},c}^t,\dots ,{\beta }_{v^{{}^{\prime }},c}^T\in {\mathbb{R}}^{E\times B}$ in Equation (12). The number of parameters is $2EBT$. We reshape two kinds of tensor shape into high-order $x_1,\dots ,x_D$, so $EBT={\prod }_{d=1}^D{x}_d$, and use the product of a series of tensors in TT format $\mathcal{A}(1,2,\dots ,D)$, $\mathcal{B}(1,2,\dots ,D)$ to replace original tensor $[{\alpha }_{v^{{}^{\prime }},c}^1,{\alpha }_{v^{{}^{\prime }},c}^t,\dots ,{\alpha }_{v^{{}^{\prime }},c}^T]$, $[{\beta }_{v^{{}^{\prime }},c}^1,{\beta }_{v^{{}^{\prime }},c}^t,\dots ,{\beta }_{v^{{}^{\prime }},c}^T]$, where $\mathcal{A}\left(d\right)\in {\mathbb{R}}^{R_d-1\times x_d\times R_d}$, $\mathcal{B}\left(d\right)\in {\mathbb{R}}^{R_d-1\times x_d\times R_d}$ and rank set is $R_0,R_1,\dots ,R_D$, $\forall d\in \left[D\right]$. So, $[{\alpha }_{v^{{}^{\prime }},c}^1,{\alpha }_{v^{{}^{\prime }},c}^t,\dots ,{\alpha }_{v^{{}^{\prime }},c}^T]={\prod }_{d=1}^D{\mathcal{A}}_d$, $[{\beta }_{v^{{}^{\prime }},c}^1,{\beta }_{v^{{}^{\prime }},c}^t,\dots ,{\beta }_{v^{{}^{\prime }},c}^T]={\prod }_{d=1}^D{\mathcal{B}}_d$. The TT-NSMS algorithm reduces the number of parameters of $[{\alpha }_{v^{{}^{\prime }},c}^1,{\alpha }_{v^{{}^{\prime }},c}^t,\dots ,{\alpha }_{v^{{}^{\prime }},c}^T]$ and $[{\beta }_{v^{{}^{\prime }},c}^1,{\beta }_{v^{{}^{\prime }},c}^t,\dots ,{\beta }_{v^{{}^{\prime }},c}^T]$ from $2EBT$ to $2{\sum }_{d=1}^D{x}_d{R}_{d-1}{R}_d$, where $x_d$, $R_{d-1}$ and $R_d$ are smaller than E, B and T. In order to reduce more multiplication, we also need to decompose the information in TT-NSMS algorithm. We reshape message tensor shape into high order $y_1,\dots ,y_D$. We use the product of a series of tensors in TT format $U(1,2,\dots ,D)$ to replace original message, so ${min}_{v^{{}^{\prime }}\in M(c\backslash v)}\left|u_{v^{{}^{\prime }},c}^t\right|={\prod }_{d=1}^D{U}_d$, where $U\left(d\right)\in {\mathbb{R}}^{R_d-1\times y_d\times R_d}$ with rank set $R_0,R_1,\dots ,R_D$. The multiplication in TT-NSMS algorithm comes from the multiplication of several tensors $\mathcal{B}(1,2,\dots ,D)$ and multiplication of $\mathcal{A}(1,2,\dots ,D)$ and $U(1,2,\dots ,D)$. The total number of multiplication is ${\sum }_{d=1}^{D-1}{x}_d{y}_d{x}_{d+1}{y}_{d+1}{R}_{d-1}{R}_d{R}_{d+1}T+{\sum }_{d=1}^{D-1}{x}_d{x}_{d+1}{R}_{d-1}{R}_d{R}_{d+1}$. The TR-NSMS algorithm and TT-NSMS algorithm have the same multiplication and parameters, but with more choices of rank set.

6. Simulation Results and Discussions

We investigated the decoding performance of various TT-NOMS, TT-NSMS, TR-NOMS and TR-NSMS for Hamming code, BCH codes and LDPC code. The parameters of the simulation are shown in

Table 1. Parameters of simulation.

Parameters	Content
Code	$LDPC,BCH,Hamming$
Min frame error	100
Max frame	100,000
Learning rate	$0.01$
Batch size	120
Optimizer	$Adam$
$E_b/N_0$	1–8 (dB)
Number of iterations	$5,15$

. Denote E as the edge in graph, T as the number of iterations and B as the size of batch. $E_b/N_0$ represents the signal-to-noise ratio. The optimal multiplicative coefficient is set to 0.8 in the normalized min-sum (NMS) algorithm, and the optimal offset coefficient is set to 0.15 in the OMS algorithm.

Because the short loops in the tanner graph affect the performance of decoding, the information transmitted by the CNs layer are unreliable, and the VNs layer transmit reliable information. The performance of these layers does not improve due to the increase in weight. Therefore, the weight $w_{v,c}^t$, $w_v^t$ are set to 1 in Equations (8) and (9), which is similarily done in [3]. $w_{v,c}^t$, $w_v^t$ are not decomposed by TT or TR. All of w in Equations (22), (23), (31) and (32) are set to 1. Tensorly [25] of low rank decomposition is used in our codes.

6.1. TT-NOMS and TT-NSMS

First, the performance of the proposed TT-NOMS, TT-NSMS algorithms with the NOMS [22] and NSMS algorithms are obtained, in which we use five iterations. As shown in Figure 3, when the decoding network has 12 layers in five iterations, the bit error rate (BER) and frame error rate (FER) performances of TT-NOMS, TT-NSMS algorithms and NOMS, NSMS algorithms are basically the same for the Hamming(7,3) code. However, TT-NOMS and TT-NSMS algorithms only use $1.5\%$ the number of parameters in NOMS and NSMS algorithm while the multiplication of TT-NSMS is reduced 222 times, as listed in

Table 2. Comparison of parameters and multiplication in NOMS [22], NSMS, TR-NOMS and TR-NSMS algorithms.

Code	Algorithm	Number of Parameters	Parameter Reduction Percentage	Number of Multiplications
LDPC(576,432)	NOMS	1,224,000	100%	0
	NSMS	2,448,000	100%	146,880,000
	TT-NOMS	474	0.03%	25,780
	TR-NOMS	474	0.03%	25,780
	TT-NSMS	948	0.03%	88,700
	TR-NSMS	948	0.03%	88,700
BCH(127,106)	NOMS	604,800	100%	0
	NSMS	1,209,600	100%	72,576,000
	TT-NOMS	430	0.70%	19,692
	TR-NOMS	430	0.70%	19,692
	TT-NSMS	860	0.70%	67,812
	TR-NSMS	860	0.70%	67,812
BCH(63,45)	NOMS	259,200	100%	0
	NSMS	518,400	100%	31,104,000
	TT-NOMS	316	0.12%	7584
	TR-NOMS	316	0.12%	7584
	TT-NSMS	632	0.12%	21,264
	TR-NSMS	632	0.12%	21,264
Hamming(7,3)	NOMS	7200	100%	0
	NSMS	14,400	100%	864,000
	TT-NOMS	108	1.5%	1008
	TR-NOMS	108	1.5%	1008
	TT-NSMS	216	1.5%	3888
	TR-NSMS	216	1.5%	3888

. TT-NOMS adds low-rank multiplication to reduce the number of parameters.

As shown in Figure 4 and Figure 5, when the decoding network has 12 layers, there is not much difference in BER performance between TT-NOMS, TT-NSMS algorithm, NOMS and NSMS algorithms for BCH(63,45) and BCH(127,106) codes. However, TT-NOMS algorithm only uses $0.12\%$ and $0.7\%$ number of parameters in NOMS algorithm, while at the same time, the multiplication of TT-NSMS is reduced rapidly, as listed in Table 2.

As shown in Figure 6, the BER performances of the TT-NOMS algorithm and NOMS algorithm are basically the same for LDPC(576,432) code. However, TT-NOMS algorithm only uses $0.03\%$ the number of parameters in NOMS algorithm while the multiplication of TT-NSMS is reduced from 14,688 W to 8.9 W as listed in Table 2.

6.2. TR-NOMS and TR-NSMS

Figure 7, Figure 8 and Figure 9 show the performance of the NOMS and TR-NOMS algorithms for hamming and BCH codes. There is not much difference in BER performance between TR-NOMS algorithm, TT-NOMS algorithm, TR-NSMS algorithm and TT-NSMS algorithm. When the two algorithms have the same rank, the reduction of parameters and multiplication are the same.

As shown in Figure 10, performance in TR-NOMS algorithm and NOMS algorithm are basically the same for LDPC(576,432) code. However, TR-NOMS algorithm only uses $0.03\%$ the number of parameters in NOMS algorithm and the same saving is achieved by TT-NOMS for the two algorithms have the same rank while the multiplication of TR-NSMS is reduced rapidly, as listed in Table 2. Figure 11 shows algorithms with 5 and 15 iterations, in which the decomposed network has continuous stability. Figure 12 shows TR-NOMS algorithm for LDPC with different ranks. As the rank increases, the performance improves to a certain extent but the number of parameters and multiplication increase. We can choose different ranks according to our needs. The choice of the optimal rank is another researchable issue.

6.3. Complexity Analysis

The number of parameters and multiplications in NOMS, NSMS, TT-NOMS, TT-NSMS, TR-NOMS and TR-NSMS algorithms are compared in

Table 3. Comparison of parameters and multiplication in NOMS [22], NSMS, TT-NOMS, TT-NSMS, TR-NOMS and TT-NSMS algorithms.

Algorithm	Number of Parameters	Number of Multiplications
NOMS	$EBT$	0
NSMS	$2EBT$	$E{B}^{2}T$
TT-NOMS or TR-NOMS	${\sum }_{d=1}^D{x}_d{R}_{d-1}{R}_d$	${\sum }_{d=1}^{D-1}{x}_d{x}_{d+1}{R}_{d-1}{R}_d{R}_{d+1}$
TT-NSMS or TR-NSMS	$2{\sum }_{d=1}^D{x}_d{R}_{d-1}{R}_d$	$\begin{array}{c}{\sum }_{d=1}^{D-1}{x}_d{y}_d{x}_{d+1}{y}_{d+1}{R}_{d-1}{R}_d{R}_{d+1}T+\\ {\sum }_{d=1}^{D-1}{x}_d{x}_{d+1}{R}_{d-1}{R}_d{R}_{d+1}\end{array}$

, where E is the number of edges in Tanner graph, B is the number of batch size, T is the number of iterations. The number of multiplications in Table 2 and Table 3 include multiplications in CNs layers.

The same rank is used in TT and TR decomposition. We only use 108 parameters in the TT-NOMS and TR-NOMS algorithms and 216 parameters in TT-NSMS and TR-NSMS algorithms for Hamming(7,3) code, and the percentage reduction of parameters is $1.5\%$. The number of multiplication has been reduced to $0.45\%$ in TT-NSMS and TR-NSMS algorithms. We only use 316 parameters in the TT-NOMS and TR-NOMS algorithms and 632 parameters in TT-NSMS and TR-NSMS algorithms for BCH(63,45) code, and the percentage reduction of parameters is $0.12\%$. The number of multiplication has been reduced to $0.0068\%$ in TT-NSMS and TR-NSMS algorithms.We only use 430 parameters in the TT-NOMS and TR-NOMS algorithms and 860 parameters in TT-NSMS and TR-NSMS algorithms in BCH(127,106) code, and the percentage reduction of parameters is $0.70\%$. The number of multiplication has been reduced to $0.0093\%$ in TT-NSMS and TR-NSMS algorithms. We use 474 parameters in the TT-NOMS and TR-NOMS algorithms and 984 parameters in TT-NSMS and TR-NSMS algorithms for LDPC(576,432) code, the percentage reduction of parameters is $0.03\%$. The number of multiplication has been reduced to $0.006\%$ in TT-NSMS and TR-NSMS algorithms. TT-NOMS, TR-NOMS, TT-NSMS, TR-NSMS algorithms can reduce the storage space and computational complexity during deployment.

7. Conclusions

This paper investigates low-complexity NOMS and NSMS decoding algorithms using TT decomposition and TR decomposition, which have almost the same performance, but lower computational complexity and fewer parameters in comparison with the the NOMS and NSMS algorithms. We have also provided an analytical framework on the complexity of the TT-NOMS, TT-NSMS, TR-NOMS and TT-NSMS algorithms. We conducted experiments on the proposed TT-NOMS, TT-NSMS, TR-NOMS and TT-NSMS algorithms for BCH codes, Hamming code and LDPC code, and the experimental results show that among these codes, the TT-NOMS, TT-NSMS, TR-NOMS and TT-NSMS decoding algorithms have comparable decoding performances, while the proposed algorithms have fewer parameters and reduced multiplications.