Application of Tensor Decomposition to Reduce the Complexity of Neural Min-Sum Channel Decoding Algorithm

Round 1

Reviewer 1 Report

The paper proposes applying Tensor Decomposition to reduce the Complexity of Neural Min-Sum Channel Decoding Algorithm. Overall, the paper is interesting and relevant.

I have a few concerns about the paper in its current state.

First, my main concern is that sections 5 and 6.3 present the main contribution of the paper in an unfitting, too concise and somewhat superficial manner. The paper does not outperform the existing decoders’ performance, and its main validity lies in the low number of parameters and number of multiplications (compared to NSMS). I would expect this section to be explained and developed more carefully and more rigorously. The current state of those sections is not enough. Moreover, I would ask the authors to check their calculations. It is unclear if the number of parameters/multiplications consider the neglection of some weights (were set to 1 at the beginning of section 4), and if it is the case in TT-NOMS/TT-NSMS/TR-NOMS/TR-NSMS as well as NOMS/NSMS which also neglect some of the weights in some situations without any performance loss (as specified in some of the references).

Second, I believe that some of the sentences and how things are presented in the paper should be carefully reviewed, as there are some unclear statements. Some examples:

• “dimensions less” – dimensionless (Page 4) • “decoding algorithms are proposed and simulation shows that its complexity is compared in Section 4 and Section 5.” – what about the complexity? (Page 2) • “As an example.” – a complete sentence (Page 7)

These sort of sentences and the overall English makes the paper in its current state hard to read.

Third, the necessity of transforming the three-dimensional tensors into dour-dimensional tensors (section 3) is not supported by the background in section 2. Please explain it in more detail (I would appreciate expanding the background regarding Tensors Decomposition as well as it is not intuitive). Also, I would like to understand the choice (which is unexplained) using the “Syndrome loss” for the training. Even the cited paper [22] uses a mix of its proposed“Syndrome loss” and the usual cross entropy loss to achieve better performance. I have not seen any proposition using this loss alone. Please state if it has been proposed or is it a newproposition.

Furthermore, I believe the figures are not as clear as they should. I believe it is confusing showing both FER and BER result on the same figure. The figures are also blurred anddistinguishing between the plots is difficult. Please sharpen the plots and separate BER and FER.

Finally, some smaller concerns that I would appreciate being addresses:

• “the sequence of slice from each core tensor” –further explain what you mean by that. • – Page 4, d is left unexplained at that point.  • “n is the maximum value of modes.” – what modes? • “which can be expressed as [17]” – I do not understand what that means. Did you say can be expressed as shown in [17]? • “Bas” – Page 7 – please correct to “B as”. • Is the SNR region presented is also the SNR region in training? • “As listed in Table 3” instead of “As list in Table 3”

Author Response

Response to Reviewer 1 Comments

 

Point 1: my main concern is that sections 5 and 6.3 present the main contribution of the paper in an unfitting, too concise and somewhat superficial manner. The paper does not outperform the existing decoders’ performance, and its main validity lies in the low number of parameters and number of multiplications (compared to NSMS). I would expect this section to be explained and developed more carefully and more rigorously. The current state of those sections is not enough. Moreover, I would ask the authors to check their calculations. It is unclear if the number of parameters/multiplications consider the neglection of some weights (were set to 1 at the beginning of section 4), and if it is the case in TT-NOMS/TT-NSMS/TR-NOMS/TR-NSMS as well as NOMS/NSMS which also neglect some of the weights in some situations without any performance loss (as specified in some of the references).

 

Response 1: Thanks for your comments. In sections 5 and 6.3, we explain in detail the calculation methods of the number of weights and the number of multiplications in the TT-NOMS, TR-NOMS, TT-NSMS and TR-NSMS algorithms, and compare them with the original NOMS, NSMS , OMS, NMS algorithms. We only analyze the parameters and information from the check nodes (CNs) to the variable nodes (VNs) in Tanner graph with other parameters are set to 1. “since training these parameter did not yield additional improvements. Note that these weights multiply the self message from the channel, which is a reliable message, unlike the messages received from CNs, which may be unreliable due to the presence of short cycles in the Tanner graph. Hence, this self message from the channel can be taken as is”, which is from [1]. So, message from CNs layer is unreliable,  message from VNs is reliable in tanner graph. Therefore,  there are only parameters of the CNs layer in the simulation experiment, and the parameter is decomposed by tensor train  (TT) and tensor ring (TR), and its multiplication and storage complexity are analyzed. The TT-NSMS algorithm and TR-NSMS algorithm have the same multiplication and parameters, but with more choice of rank set. TT-NOMS, TR-NOMS, TT-NSMS and TR-NSMS algorithms achieve basically the same performance as the original algorithm, but at the same time have lower complexity in storage and multiplication calculations.

 

Please refer to sections 5 and 6.3 in the update manuscript on pages 7-8 and 12-13. 

 

 

Point 2: I believe that some of the sentences and how things are presented in the paper should be carefully reviewed, as there are some unclear statements. Some examples:

• “dimensions less” – dimensionless (Page 4) • “decoding algorithms are proposed and simulation shows that its complexity is compared in Section 4 and Section 5.” – what about the complexity? (Page 2) • “As an example.” – a complete sentence (Page 7)

These sort of sentences and the overall English makes the paper in its current state hard to read.

 

Response 2: Thanks for your comments, we change the sentence where "dimensionless" is located to "Smaller TT rank yields a smaller overall dimension of the series G". “As an example.”is deleted. We have also made a lot of changes to improve the readability.

At present, we have modified the bad sentences and expressions in the paper.

 

Point 3: the necessity of transforming the three-dimensional tensors into dour-dimensional tensors (section 3) is not supported by the background in section 2. Please explain it in more detail (I would appreciate expanding the background regarding Tensors Decomposition as well as it is not intuitive). Also, I would like to understand the choice (which is unexplained) using the “Syndrome loss” for the training. Even the cited paper [22] uses a mix of its proposed“Syndrome loss” and the usual cross entropy loss to achieve better performance. I have not seen any proposition using this loss alone. Please state if it has been proposed or is it a new proposition.

 

Response 3: Thanks for your comments. Tensors of third order and above can be decomposed into products of a series of small tensors through TT/TR. The purpose of converting the original tensor into higher-order tensor is to minimize the number of multiplications and weights as much as possible. Three-dimensional tensors can be decomposed into the product of small tensors by TT or TR, and higher-order tensors can also be decomposed in this way.

Training was conducted using stochastic gradient descent with mini-batches. The training data is created by transmitting the zero codeword through a binary input additive white Gaussian noise channel (AWGN) with varying SNRs ranging from 1 dB to 8 dB.  Mini-batch is used, and its size that depended on the code used and on its parity check matrix. Our weights are three-dimensional, , which three dimensions represent the number of iterations, the number of edges. We use [iterations, edge_number, batch_size] to represent the dimension size of tensor. The number of edges in the tanner graph increases rapidly with the increase of code length. In particular, the number of edges is 2040 in medium length LDPC (576, 432) code. If we take the three-dimensional weight reshaped as a higher-order tensor,  and the values of each dimension are close, then the dimensions of the decomposed tensors are less than the original  tensors dimensions after decomposed three-dimensional weight. For example, we use LDPC (576, 432) and its weights’ size is given by [5,2040,120], we reshape weights to four-dimensional tensor [51,20,30 40], the size of decomposed tensors are [R0,51,R1],  [R1,20,R2],  [R2,30,R3], [R3,40,R4], which rank is [R0, R1, R2, R3, R4]. Set rank is [1,2,2,2,1], the storage needed is 1*51*2+2*20*2+2*30*2+2*40*1=382. However, the tensors dimensional size after three-dimensional weight decomposition are [R0,5,R1],  [R1,2040,R2],  [R2,120,R3], Set rank is [1,2,2,1], the storage needed is 1*5*2+2*240*2+2*120*2=8410. The number of multiplication will be greatly reduced.

 

  Thank you for your comments for sydrome loss. As you said, [22] uses the combination of syndrome loss and cross-entropy loss as the loss function.  The syndrome loss can help FER but may hurt BER, Most of the simulation experiments also use this loss function. However, when simulating BCH (63,36) in [22], there is a situation where  is equal to 0, that is, only using syndrome loss, suggesting it is effective for BCH codes. In addition, the works in [2] also use syndrome loss alone.

 

 

Point 4: I believe the figures are not as clear as they should. I believe it is confusing showing both FER and BER result on the same figure. The figures are also blurred and distinguishing between the plots is difficult. Please sharpen the plots and separate BER and FER.

Response 4: Thank you for your comments. Currently, we have revised the figure in the update manuscript. For detail, please refer to Section 6 of our updated manuscript

 

 Point 5:  some smaller concerns that I would appreciate being addresses:

• “the sequence of slice from each core tensor” –further explain what you mean by that. • – Page 4, d is left unexplained at that point.  • “n is the maximum value of modes.” – what modes? • “which can be expressed as [17]” – I do not understand what that means. Did you say can be expressed as shown in [17]? • “Bas” – Page 7 – please correct to “B as”. • Is the SNR region presented is also the SNR region in training? • “As listed in Table 3” instead of “As list in Table 3”

Response 5: At present, we have modified the sentences and expressions in the paper as much as possible.

• “the sequence of slice from each core tensor”, It means that the slice at the corresponding position in each dimension plane constitutes an element x in Tensors X.
• d is left unexplained at that point.We delete this paragraph.
• n is the maximum value of modes.It means the largest value in the dimension of Tensor X.
• “which can be expressed as shown in [17]”instead of“which can be expressed as, according to [17],”
• “Bas” to “B as”
• Is the SNR region presented is also the SNR region in training?Yes, we use SNR region in training and test. SNR=1-8dB in Hamming, BCH codes, SNR=0-5dB in LDPC code.
• “As listed in Table 3”instead of “As list in Table 3”

 

 

Response 6  Confused concepts

• Complexity in TT and TR

The original tensor is converted into a d-order tensor. The d-order tensor is decomposed into the product of d small tensors through tensor-train decomposition. Each small tensor is (, , ). Therefore, d small tensors need  storage space, n is the max value of . is the max value of . Please refer to sections 2.3 in the update manuscript on pages 4.

 

l  Fiber and Slice

Fibers are operations that extract vectors from tensors. Fix one of the dimensions in the matrix to get rows or columns. Similar to matrix operations, retaining one dimension change and fixing other dimensions, called fiber. We use the third-rank tensor as an example: the dimensional size of tensor is [i, j, k] .When we expand a tensor along its k-th dimension, we obtain mode-k fiber. The mode-1 fiber of the third-order tensor X is: X_{: jk}, which meanings string each small square in the k direction into a vector. In other words, the index of all dimensions remains unchanged, except that the k-th dimension is expanded. 

                           

       Fig 1 Original tensor                                 Fig 2 mode−1 fiber  X_{:jk}

• In the same way, we get slice when we keep all indexes unchanged except for the expansion of two dimensions.A slice operation refers to the operation of decimating a matrix in a tensor. In the tensor, if the two dimensions are kept and the other dimensions are fixed, a matrix can be obtained, which is the slice of the tensor.For example, the horizontal slice of the third-order tensor X is: X_{i::}.

 

         Fig 3 slice X_{i::}

Please refer to sections 2.4 in the update manuscript on pages 4-5.

 

 

 

Reference

[1] Nachmani E, Marciano E, Lugosch L, et al. Deep learning methods for improved decoding of linear codes[J]. IEEE Journal of Selected Topics in Signal Processing, 2018, 12(1): 119-131.

[2] L. Yuanhui, C.-T. Lam, and B. K. Ng, “A low-complexity neural normalized min-sum ldpc decoding algorithm using tensor-train decomposition,” IEEE Communication Letters,doi:10.1109/LCOMM.2022.3207506.

 

 

Author Response File: Author Response.docx

Reviewer 2 Report

This article optimizes the low-complexity NOMS and NSMS decoding algorithm using tensor train decomposition (TT) and tensor ring decomposition (TR), where four kinds of algorithms are proposed including TT-NOMS, TR-NOMS, TT-NSMS and TR-NSMS algorithms. This article applies these four algorithms to the decoding of BCH codes and LDPC codes, which have fewer parameters and reduced multiplications.

 1.      In the figures of BER and FER performance, only the conventional sum-product algorithm and the min-sum algorithm are compared with the various neural network (NN) decoders. The near optimal normalized/offset min-sum (NMS/OMS) algorithm should be considered, which can achieve noticeable performance gain over Min-Sum algorithm with almost negligible increase of decoding complexity. Furthermore, the classic BM algorithm and the chase decoding also should be compared with these NN decoders.

 

2.      Why the decoding performance of NSMS algorithm is much worse than that of NOMS algorithm? According to Eq. (7) and Eq. (12) and reference [6], this decoding performance may not be very reasonable.

3.      Table 2 shows the parameters and multiplications in different NN decoders. I think that these comparisons are all about the offline training stage which are not critical for the online implementations of these decoders.

4.      There may be some errors in Eq.(22) and Eq.(31), two $w^{4t}_{v, c}$ ? and $4t$ or $4^t$?

Author Response

Response to Reviewer 2 Comments

 

Point 1: In the figures of BER and FER performance, only the conventional sum-product algorithm and the min-sum algorithm are compared with the various neural network (NN) decoders. The near optimal normalized/offset min-sum (NMS/OMS) algorithm should be considered, which can achieve noticeable performance gain over Min-Sum algorithm with almost negligible increase of decoding complexity. Furthermore, the classic BM algorithm and the chase decoding also should be compared with these NN decoders.

 

Response 1: Thanks for your comments. Yes, we add normalized min-sum (NMS) and offset min-sum (OMS) algorithms. In particular, if the decoder is constrained to use a single offset for all edges, then NOMS reverts to OMS. Indeed, constraining the decoder to learning a single offset yields a new way to choose the offset for OMS, which is called neural shared offset min-sum(NSOMS) we proposed in [1], in addition to the existing methods of simulation and density evolution. The same single offset coefficient in neural normalized min-sum(NNMS) is converted to NMS. According to the referenced literature, the optimal multiplicative coefficient is set to 0.8 in the NMS algorithm, the optimal offset coefficient is set to 0.15 in the OMS algorithm. The bit error ratio (BER) of NMS is better than OMS. We have added NMS and OMS algorithms in Fig 2-Fig 9 in the manuscript for comparison. The BER performance of NOMS algorithm is better than the conventional OMS, 0. 5dB better than the conventional NMS with high storage complexity and computational complexity. TT-NOMS and TR-NOMS algorithms we proposed are used to reduce the complexity using low rank decomposition techniques.

Due to time limitation in the revision period, we only reproduce the BM algorithm in RS code, and do not realize the decoding of BCH code. We currently do not consider the chase algorithm but we will consider it in future work.

 

Point 2:  Why the decoding performance of NSMS algorithm is much worse than that of NOMS algorithm? According to Eq. (7) and Eq. (12) and reference [6], this decoding performance may not be very reasonable.

 

Response 2: Thanks for your comments. For the sake of fairness, we keep the simulation conditions and training times of NSMS and NOMS the same, and the experimental results are shown in the updated manuscript. Your comment is very reasonable. We guess that NSMS may need more training times than NOMS to be able to decode SMS better than OMS like non-neural decoding[6]. The performance of TR-NSMS is comparable to that of TR-NOMS, and sometimes better.

 

Point 3:   Table 2 shows the parameters and multiplications in different NN decoders. I think that these comparisons are all about the offline training stage which are not critical for the online implementations of these decoders.

Response 3: Thank you for your comments, the NSMS algorithm and NOMS algorithm involved in this article are deployed after training. After deployment, the parameters cannot be updated at will, so they belong to offline deployment. At present, our research has not involved online deployment. Thank you for your suggestion. We look forward to future research involving online deployment. TT-NOMS, TR-NOMS, TT-NSMS, TR-NSMS algorithms can reduce the storage space and computational complexity during deployment. Please refer to sections 6 in the update manuscript on pages 13.

 

 

Point 4:   There may be some errors in Eq.(22) and Eq.(31), two $w^{4t}_{v, c}$ ? and $4t$ or $4^t$?

Response 4: Thank you for your comments, we removed the redundant one $w^{4t}_{v, c}$, referring to pages 6 and 7 of the manuscript. We use four TT format tensors to replace the original tensor, so here should be $4t$ not  $4^t$.  We also modified all occurrences $1t$, $2t$, $3t$, $4t$ of the formulas.

 

 

 

 

Reference

[1] Qingle W U, Su-Kit T, Liang Y, et al. A Low Complexity Model-Driven Deep Learning LDPC Decoding Algorithm[C]//2021 IEEE 6th International Conference on Computer and Communication Systems (ICCCS). IEEE, 2021: 558-563.

 

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Overall, the revised version readability is of much improvement.

I feel that most of my concerns and questions were addressed.

I do recommend the authors to further expand the background in 2.3 and 2.4, as Tensor Decomposition is not intuitive to all reader, and some additional background (that was part of the authors’ response) may be welcomed.

Author Response

Response to Reviewer 1 Comments

 

Point 1: Overall, the revised version readability is of much improvement.

I feel that most of my concerns and questions were addressed.

I do recommend the authors to further expand the background in 2.3 and 2.4, as Tensor Decomposition is not intuitive to all reader, and some additional background (that was part of the authors’ response) may be welcomed.

 

Response 1: Thanks for your comments. In sections 2.3 and 2.4, we explain in detail the TT and TR decomposition via recursive SVD, and a simple example is introduced graphically.

Please refer to sections 2.3 and 2.4 in the update manuscript on pages 5-7.

 

Author Response File: Author Response.docx

Reviewer 2 Report

No further comments.  In this revised version, authors have already addressed all my comments. Thanks for the authors' effort and time. 

Author Response

Response to Reviewer 2 Comments

 

Point 1: No further comments.  In this revised version, authors have already addressed all my comments. Thanks for the authors' effort and time. 

Response 1: Thanks for your comments.

 

Author Response File: Author Response.docx