Machine Learning-Based Dimension Optimization for Two-Stage Precoder in Massive MIMO Systems with Limited Feedback

Abstract

A two-stage precoder is widely considered in frequency division duplex massive multiple-input and multiple-output (MIMO) systems to resolve the channel feedback overhead problem. In massive MIMO systems, users on a network can be divided into several user groups of similar spatial antenna correlations. Using the two-stage precoder, the outer precoder reduces the channel dimensions mitigating inter-group interferences at the first stage, while the inner precoder eliminates the smaller dimensions of intra-group interferences at the second stage. In this case, the dimension of effective channel reduced by outer precoder is important as it leverages the inter-group interference, the intra-group interference, and the performance loss from the quantized channel feedback. In this paper, we propose the machine learning framework to find the optimal dimensions reduced by the outer precoder that maximizes the average sum rate, where the original problem is an NP-hard problem. Our machine learning framework considers the deep neural network, where the inputs are channel statistics, and the outputs are the effective channel dimensions after outer precoding. The numerical result shows that our proposed machine learning-based dimension optimization achieves the average sum rate comparable to the optimal performance using brute-forcing searching, which is not feasible in practice.

1. Introduction

Massive multiple-input and multiple-output (MIMO) is one of the most promising technologies for the next-generation wireless mobile communication systems [1,2,3,4]. The large-scale antennas equipped at a base station (BS) can considerably improve the data rate, energy efficiency, and reliability; the theoretical results show that it perfectly cancels out inter-user interference with simple linear transceivers. In this case, the accurate knowledge of channel state information (CSI) is crucial at the BS to achieve the potential gain from large-scale antennas [1,2,3,4]. In time division duplex (TDD) systems, the BS can acquire CSI during uplink training thanks to channel reciprocity [4,5], and, hence, many existing works on massive MIMO systems consider TDD systems. However, many of current wireless mobile communication systems are frequency division duplexing (FDD) systems [6,7], whose uplink and downlink channels are independent from each other, which motivates research to achieve the potential gains also in FDD systems [8,9,10,11,12,13].

In FDD systems, a BS obtains the CSI from users’ channel feedback due to the lack of reciprocity between the uplink and the downlink channels [6,7,8,9,10,11,12,13]. In massive MIMO systems, the CSI feedback overhead problem becomes more severe because of the large number of antennas; it was already shown that the feedback size should linearly scale with the number of antennas to fully obtain the multiplexing gain [6,7]. To resolve the feedback overhead problem, a two-stage precoder was widely used for FDD massive MIMO systems, which consists of the outer and the inner precoders [10,11,12,13]. In the two-stage precoder, the outer precoder projects the original channel space of large dimensions onto a smaller dimensional subspace, and then the inner precoder controls the inter-user interference as multiuser MIMO precoding. There are various types of two-stage precoder designs in massive MIMO systems, and, among them, the hybrid architecture is widely considered for mmWave bands due to the limitation of hardware implementation [14,15,16,17]. As a full digital architecture has many benefits in sub-6 GHz bands, there are also many papers that consider the full digital architecture in massive MIMO systems [10,11,12,13]. Thus, we mainly consider a joint spatial division and multiplexing (JSDM) [10] with the full digital architecture in massive MIMO systems with a correlated channel environment.

The key idea of JSDM is to divide whole users into multiple user groups according to the channel covariance matrices, and then the outer precoder and the inner precoder sequentially mitigate inter-group and inter-user interference, respectively [10,18]. At the first stage, the outer precoder mitigates the inter-group interference (IGI) by projecting the original channel into a smaller dimensional subspace. Then, the inner precoder cancels out the same-group interference (SGI) with the dimension-reduced effective channels produced by the outer precoder; in this case, the BS exploits the quantized versions of dimension-reduced effective channels obtained via limited feedback. Therefore, the outer precoder design is important as it leverages many performance achieving factors such as the inter-group interference, the intra-group interference, and the channel quantization error. This motivates the sophisticated outer precoder design taking into account all of these factors. In this context, we optimized the dimension of outer precoder in [19] for a downlink massive MIMO system with limited feedback based on the lower bound-based analysis.

Meanwhile, machine learning recently has attracted a lot of attention in wireless communication systems [20,21]. The machine learning technique has shown good performance in many applications from image processing to economics [20,21,22]. In addition, machine learning is applied to the physical layer processing such as antenna selection and beamforming design in MIMO systems [23,24], and channel estimation and hybrid precoding for massive MIMO systems [25,26]. In particular, deep learning [27], which is one of key machine learning techniques, is tackling complicated nonlinear problems and high-computation issues in many areas [22] and overwhelms many existing schemes [22,24,25,26].

In this paper, we develop our initial work [19] on the dimension optimization for the outer precoder design with the machine learning framework. Our contributions can be summarized as follows:

• We introduce our two-stage precoder design with limited feedback, where the quantized CDI of the dimension-reduced effective channel is only fed back to the BS. We first derive a lower bound of the average sum rate and then optimize the dimension of the outer precoder to maximize average sum rate.
• We propose the machine learning framework for the dimension optimization based on a deep neural network (DNN); we determine the DNN architecture of the input, the hidden, and the output layers as well as training procedure. Our DNN architecture takes eigenvalues of covariance matrices of user groups as inputs and returns the structure of outer precoder, which represents the allocated dimensions for all user groups.
• We evaluate our DNN model and show that our proposed machine learning based outer precoder dimension optimization improves the average sum-rate and achieves near-optimal performance.

The rest of this paper is organized as follows. We introduces our system model in Section 2 and describe our problem in Section 3. We introduce our previous work on the lower bound of the achievable sum rate in Section 4 and propose a machine learning framework for the dimension optimization in Section 5. We evaluate our proposed DNN model in Section 6 and conclude our paper in Section 7.

Notations: We use upper and lower case boldfaces to denote matrices and vectors, respectively. The notations ${(·)}^T$ and ${(·)}^{†}$ represent the transpose and complex conjugate transpose, respectively. In addition, the notations $\mathbb{E}[·]$ and $Pr[·]$ denote the expectation and the probability, respectively.

2. System Model

Our system model is illustrated in Figure 1. We consider a single-cell multi-user massive MIMO downlink system with limited feedback, where a BS with M transmit antennas serves K single-antenna users simultaneously. Let $\mathbf{F}(\triangleq [{\mathbf{f}}_1,\cdots ,{\mathbf{f}}_K])\in {\mathbb{C}}^{M\times K}$ be a linear precoding matrix and $\mathit{d}(\triangleq [d_1,\dots ,d_K])\in {\mathbb{C}}^{K\times 1}$ be a data symbol vector for all $k\in \{1,\dots ,K\}$. Then, the transmit signal vector at the BS denoted by $\mathbf{x}\in {\mathbb{C}}^{M\times 1}$ is obtained from $\mathbf{x}=\mathbf{F}\mathit{d}$. The received signal denoted by $\mathbf{y}\in {\mathbb{C}}^{K\times 1}$ becomes

$$\mathbf{y}={\mathbf{H}}^{†}\mathbf{x}+\mathbf{n}={\mathbf{H}}^{†}\mathbf{F}\mathit{d}+\mathbf{n},$$

(1)

where $\mathbf{H}\in {\mathbb{C}}^{M\times K}$ is a concatenated channel matrix such that $\mathbf{H}\triangleq [{\mathbf{h}}_1,\dots ,{\mathbf{h}}_K]$, where ${\mathbf{h}}_k\in {\mathbb{C}}^{M\times 1}$ is the user k’s channel vector, and $\mathbf{n}(\triangleq [n_1,\dots ,n_K])\in {\mathbb{C}}^{K\times 1}$ is additive white Gaussian noise.

In this paper, we assume that the BS only utilizes the channel direction information (CDI) for beamforming vector design to save the additional feedback overhead required for power allocation [6,7]. Therefore, when the total transmit signal power is P, the BS allocates equal powers to each user, i.e., $\mathbb{E}\left[{\left|d_k\right|}^2\right]=P/K$. Meanwhile, the beamforming vector for user k should satisfy ${\mathbf{f}}_k^{†}{\mathbf{f}}_k=1$.

In our channel model, we consider the Rayleigh correlated channel such that ${\mathbf{h}}_k\sim \mathcal{C}\mathcal{N}(\mathbf{0},{\mathbf{R}}_k)$, where ${\mathbf{R}}_k\in {\mathbb{C}}^{M\times M}$ is a positive semi-definite channel covariance matrix represented by ${\mathbf{R}}_k\triangleq \left({\mathbf{U}}_k^{\prime }\right)\left({\mathbf{\Lambda }}_k^{\prime }\right){\left({\mathbf{U}}_k^{\prime }\right)}^{†}$ with the singular value decomposition (SVD). We denote by $r_k$ the number of non-zero singular values in ${\mathbf{\Lambda }}_k^{\prime }$. Then, with the Karhunen–Loeve representation, ${\mathbf{h}}_k$ can be represented by [9,10,19]

$${\mathbf{h}}_k\triangleq {\mathbf{U}}_k{\mathbf{\Lambda }}_k^{1/2}{\mathbf{e}}_k,$$

(2)

where ${\mathbf{U}}_k\in {\mathbb{C}}^{M\times r_k}$ is a tall unitary matrix comprised of $r_k$ column vectors in ${\mathbf{U}}_k^{\prime }$ corresponding to non-zero singular values, ${\mathbf{\Lambda }}_k\in {\mathbb{R}}^{r_k\times r_k}$ is a diagonal matrix with $r_k$ non-zero positive eigenvalues, and ${\mathbf{e}}_k\in {\mathbb{C}}^{r_k\times 1}\sim \mathcal{C}\mathcal{N}(\mathbf{0},{\mathbf{I}}_{r_k})$.

Assuming the one-ring scattering model along with spatial correlation of the transmitter’s uniform linear array (ULA) antennas (as shown in Figure 1), the channel covariance matrix of the g-th group is given by [9,10,19]

$${\left[{\mathbf{R}}_g\right]}_{p,q}={\displaystyle \frac{1}{2{\mathbf{\Delta }}_g}}{\int }_{{\theta }_g-{\mathbf{\Delta }}_g}^{{\theta }_g+{\mathbf{\Delta }}_g}{e}^{-j2\pi \frac{{\delta }_a}{{\lambda }_c}(p-q)sin\theta }d\theta ,$$

(3)

where p and q denote $(p,q)$-th element of covariance matrix ${\mathbf{R}}_g$, ${\lambda }_c$ is the carrier wavelength, ${\delta }_a$ is the space between adjacent antennas, and ${\theta }_g$ and ${\mathbf{\Delta }}_g$ represent the azimuth center angle and angular spread, respectively. With this model, total K users are divided into G groups according to the channel covariance matrices ${\left\{{\mathbf{R}}_g\right\}}_{g=1}^G$. We denote by $K_g$ the number of users in the g-th group, so we have $K={\sum }_{g=1}^G{K}_g$. We assume that all users in the same group have the same channel covariance matrix, and the BS perfectly knows the covariance matrices of all user groups, i.e., ${\mathbf{R}}_1,\dots ,{\mathbf{R}}_G$.

3. Limited Feedback with Two-Stage Precoder

In this section, we first overview of the structure of the two-stage precoder and briefly explain the limited feedback method for the two-stage precoder. Then, we formulate our problem.

3.1. Two-Stage Precoder

We adopt the two-stage precoder to reduce complexity and feedback overhead induced by large number of antennas [10,19], which is illustrated in Figure 2. In this case, the linear precoder becomes $\mathbf{F}\triangleq \mathbf{TV}$, where $\mathbf{T}\in {\mathbb{C}}^{M\times S}$ is the outer precoder, which is for spatial division, while $\mathbf{V}\in {\mathbb{C}}^{S\times K}$ is the inner precoder, which is for spatial multiplexing in each group. The outer and the inner precoders are given by $\mathbf{T}\triangleq [{\mathbf{T}}_1,\dots ,{\mathbf{T}}_G]$ and $\mathbf{V}\triangleq \mathrm{bldiag}\{{\mathbf{V}}_1,\dots ,{\mathbf{V}}_G\}$, respectively, where ${\mathbf{T}}_g\in {\mathbb{C}}^{M\times s_g}$ and ${\mathbf{V}}_g\in {\mathbb{C}}^{s_g\times K_g}$ are the outer and inner precoder of the g-th group, respectively, where “bldiag” denotes block-diagonalized matrix. Here, the dimension of effective channels, i.e., $S\triangleq {\sum }_{g=1}^G{s}_g$, is a design parameter, where $s_g$ is the reduced-dimension of the effective channel for the g-th group to be optimized.

We denote by ${\mathbf{H}}_{\mathbf{g}}\triangleq [{\mathbf{h}}_{g,1},\dots ,{\mathbf{h}}_{g,K_g}]$ the channel matrix of the g-th group. Then, the concatenated channel matrix is represented by $\mathbf{H}=[{\mathbf{H}}_1,\dots ,{\mathbf{H}}_G]$. For given channel covariance matrices, the dimension-reduced effective channel after outer precoding is represented as follows:

$$\begin{array}{c}\hfill {\mathbf{H}}^{\mathrm{eff}}\triangleq {\mathbf{T}}^{†}\mathbf{H}\triangleq \left[\begin{array}{cccc}{\mathbf{T}}_1^{†}{\mathbf{H}}_1& {\mathbf{T}}_1^{†}{\mathbf{H}}_2& \cdots & {\mathbf{T}}_1^{†}{\mathbf{H}}_G\\ {\mathbf{T}}_2^{†}{\mathbf{H}}_1& {\mathbf{T}}_2^{†}{\mathbf{H}}_2& \cdots & {\mathbf{T}}_2^{†}{\mathbf{H}}_G\\ ⋮& ⋮& \ddots & ⋮\\ {\mathbf{T}}_G^{†}{\mathbf{H}}_1& {\mathbf{T}}_G^{†}{\mathbf{H}}_2& \cdots & {\mathbf{T}}_G^{†}{\mathbf{H}}_G\end{array}\right].\end{array}$$

(4)

Note that it is difficult to acquire the CDI of the whole effective channel ${\mathbf{H}}^{\mathrm{eff}}$ at the BS due to the heavy channel feedback overhead. Thus, we focus on a more practical approach with low computational complexity, which quantizes and feeds back only the dimension-reduced effective channel of each group, i.e., ${\mathbf{H}}_g^{\mathrm{eff}}(\triangleq {\mathbf{T}}_g^{†}{\mathbf{H}}_g)\in {\mathbb{C}}^{s_g\times K_g}$, whose dimension is determined by $s_g$. Consequently, the received signal at user k in the g-th group is given by

$$\begin{array}{c}\hfill y_{g,k}=\sqrt{{\displaystyle \frac{P}{K}}}{\mathbf{h}}_{g,k}^{†}{\mathbf{T}}_g{\mathbf{v}}_{g,k}{d}_{g,k}+\underset{\mathrm{same}\mathrm{group}\mathrm{interference}\left(\mathrm{SGI}\right)}{\underbrace{\sqrt{{\displaystyle \frac{P}{K}}}{\mathbf{h}}_{g,k}^{†}{\displaystyle \sum _{i\ne k}^{K_g}}{\mathbf{T}}_g{\mathbf{v}}_{g,i}{d}_{g,i}}}+\underset{\mathrm{inter-group}\mathrm{interference}\left(\mathrm{IGI}\right)}{\underbrace{\sqrt{{\displaystyle \frac{P}{K}}}{\mathbf{h}}_{g,k}^{†}{\displaystyle \sum _{c\ne g}^G}{\displaystyle \sum _{j=1}^{K_c}}{\mathbf{T}}_c{\mathbf{v}}_{c,j}{d}_{c,j}}}+n_{g,k},\end{array}$$

(5)

where ${\mathbf{v}}_{g,k}$ and ${\mathbf{v}}_{g,i}$ are the beamforming vectors of user k and user $j(\ne k)$ in the g-th group, respectively, which consist of the inner precoder for the g-th group represented by ${\mathbf{V}}_g=[{\mathbf{v}}_{g,1},\dots ,{\mathbf{v}}_{g,K_g}]$; ${\mathbf{v}}_{c,j}$ is the beamforming vector of the j-th user in group $c(\ne g)$, i.e., ${\mathbf{V}}_c=[{\mathbf{v}}_{c,1},\dots ,{\mathbf{v}}_{c,K_c}]$; $n_{g,k}$ is a complex Gaussian noise with zero mean and variance ${\sigma }^2$, i.e., $n_{g,k}\sim \mathcal{C}\mathcal{N}(0,{\sigma }^2)$. Note that the second and third terms on the right-hand side of (5) are the same-group interference (SGI) and inter-group interference (IGI), respectively.

Treating each group separately and exploiting only the CDI of the dimension-reduced effective channel of each group, the IGI should be cancelled out by the outer precoder only on the channel covariance matrices i.e., ${\mathbf{R}}_1,\dots ,{\mathbf{R}}_G$. Hence, the outer precoder for the g-th group is designed with the criterion given by

$${\mathbf{H}}_c^{†}{\mathbf{T}}_g\approx 0\mathrm{for}\mathrm{all}c\ne g.$$

(6)

According to the approximation (6), we adopt the block diagonalization (BD) method for the design structure of the outer precoder proposed in [10] in order to cancel out the IGI among the other groups, which constructs the precoder by the null-space of the channels of the other groups. Note that the BD method is a generalization of the zero-forcing channel-inversion for multi-user MIMO channels with linear processing [10,28]. With a similar approach used in [10], the outer precoder for the g-th group is designed as follows. We define the matrix

$${\mathbf{Y}}_g\triangleq [{\mathbf{U}}_1^{★},\dots ,{\mathbf{U}}_{g-1}^{★},{\mathbf{U}}_{g+1}^{★},\dots ,{\mathbf{U}}_G^{★}],$$

(7)

where ${\mathbf{U}}_c^{★}$ is an $M\times r_c^{★}$ matrix comprised of the dominant eigenvectors of ${\mathbf{U}}_c$, which is the eigenmatrix of the cth group covariance matrix such that ${\mathbf{R}}_c={\mathbf{U}}_c{\mathbf{\Lambda }}_c{\mathbf{U}}_c^{†}$, i.e., $r_c^{★}\le r_c$. The dimension of ${\mathbf{Y}}_g$ is $M\times {\sum }_{c\ne g}^G{r}_c^{★}$ with satisfying ${\sum }_{c\ne g}^G{r}_c^{★}\le M$. Note that, if ${\sum }_{g=1}^G{r}_g\le M$, we can choose $r_c^{★}=r_c$ to reflect the eigenvectors of the other groups exactly. Thus, we assume $r_c^{★}=r_c,$ ($\forall c\ne g$) in the definition (7) to construct ${\mathbf{Y}}_g$ for group $g\in \{1,2,\dots ,G\}$. Using the SVD, ${\mathbf{Y}}_g$ can be expressed as

$${\mathbf{Y}}_g={\mathbf{\Psi }}_g{\mathbf{\Sigma }}_g{[{\mathbf{\Phi }}_g,{\mathbf{\Phi }}_g^{\perp }]}^{†},$$

(8)

where ${\mathbf{\Phi }}_g^{\perp }$ is an $M\times (M-{\sum }_{c\ne g}^G{r}_c)$ sub-unitary matrix ( ${\left({\mathbf{\Phi }}_g^{\perp }\right)}^{†}{\mathbf{\Phi }}_g^{\perp }={\mathbf{I}}_{M-{\sum }_{c\ne g}^G{r}_c}$ ) comprised of the orthogonal bases of the null-space of ${\mathbf{Y}}_g$. After projecting onto the null-space, i.e., $\mathrm{Span}\left({\mathbf{\Phi }}_g^{\perp }\right)$, the covariance matrix of the projected channel is obtained by

$$\begin{array}{cc}\hfill {\overline{\mathbf{R}}}_g& ={\left({\mathbf{\Phi }}_g^{\perp }\right)}^{†}{\mathbf{R}}_g{\mathbf{\Phi }}_g^{\perp }\hfill \\ & ={\overline{\mathbf{U}}}_g{\overline{\mathbf{\Lambda }}}_g{\overline{\mathbf{U}}}_g^{†},\hfill \end{array}$$

(9)

where (9) is obtained from the SVD. Selecting the dominant $s_g$ ($\le r_g$) eigenmodes of ${\overline{\mathbf{R}}}_g$, we can construct the dimension-reduced effective channel for group g according to the BD method. Consequently, the outer precoder of the g-th group, i.e., ${\mathbf{T}}_g\in {\mathbb{C}}^{M\times s_g}$, is given by

$${\mathbf{T}}_g={\mathbf{\Phi }}_g^{\perp }{\overline{\mathbf{U}}}_g^{★},$$

(10)

where ${\overline{\mathbf{U}}}_g^{★}$ contains the dominant $s_g$ eigenvectors of ${\overline{\mathbf{U}}}_g$, where $s_g$ is the design parameter to be optimized. Note that it is satisfied that ${\mathbf{T}}_g^{†}{\mathbf{T}}_g={\left({\overline{\mathbf{U}}}_g^{★}\right)}^{†}{\left({\mathbf{\Phi }}_g^{\perp }\right)}^{†}{\mathbf{\Phi }}_g^{\perp }{\overline{\mathbf{U}}}_g^{★}={\left({\overline{\mathbf{U}}}_g^{★}\right)}^{†}{\overline{\mathbf{U}}}_g^{★}={\mathbf{I}}_{s_g}$.

For the inner precoder, we adopt the zero-forcing (ZF) beamforming to mitigate multiuser interference among the users in each group (i.e., SGI). The beamforming vector of user k in the g-th group is constructed by null-space of the effective channel vectors of all the other users in the g-th group. It is obvious that the minimum mean squared error (MMSE) type of precoder can achieve better performance than the ZF precoder. However, with the MMSE precoder, the optimal regularization parameter for the inner precoder design is dependent on the outer precoder design, so it is not easy to find. Moreover, we consider limited feedback environment, so the effects of channel quantization errors make the optimal regularization factor much more difficult to find. Thus, we consider the ZF precoder for inner precoder design thanks to its simplicity and analytical tractability. Note that the ZF beamforming is asymptotically optimal among all downlink beamforming strategies in a high SNR region [6,29], and it guarantees high spectral efficiency for large-scale antennas with low-complexity linear processing [2,9]. Then, the inner precoder for the g-th group is given by

$$\begin{array}{c}\hfill {\mathbf{V}}_g=[{\mathbf{v}}_{g,1},\dots ,{\mathbf{v}}_{g,K_g}]\in {\mathbb{C}}^{s_g\times K_g},\end{array}$$

(11)

where ${\mathbf{v}}_{g,k}$ is a ZF beamforming vector of user k in the g-th group. Note that a beamforming vector ${\mathbf{v}}_{g,k}$ is normalized such that ${\parallel {\mathbf{v}}_{g,k}\parallel }^2=1$, since the two-stage beamforming vector for user k in the g-th group with the BD-based outer precoder, i.e., ${\mathbf{f}}_k={\mathbf{T}}_g{\mathbf{v}}_{g,k}$, should satisfy ${\mathbf{f}}_k^{†}{\mathbf{f}}_k=1$, which is given by

$$\begin{array}{c}\hfill {\mathbf{f}}_k^{†}{\mathbf{f}}_k={\mathbf{v}}_{g,k}^{†}{\mathbf{T}}_g^{†}{\mathbf{T}}_g{\mathbf{v}}_{g,k}={\mathbf{v}}_{g,k}^{†}{\mathbf{I}}_{s_g}{\mathbf{v}}_{g,k}={\mathbf{v}}_{g,k}^{†}{\mathbf{v}}_{g,k}={\parallel {\mathbf{v}}_{g,k}\parallel }^2=1,\end{array}$$

(12)

for equal power allocation at the BS in (1). The details of the construction of a ZF beamforming vector of user k in the g-th group is characterized in the following subsection.

3.2. Limited Feedback Method with a Two-Stage Precoder

In the previous section, the dimension of the effective channel of each user in group g is reduced to $s_g$ by the outer precoder. Hence, we focus on the the limited feedback system to acquire the CDI of the dimension-reduced effective channel of each group, i.e.,  ${\mathbf{H}}_g^{\mathrm{eff}}$. We define ${\mathbf{h}}_{g,k}^{\mathrm{eff}}\in {\mathbb{C}}^{s_g\times 1}$ the dimension-reduced effective channel of user k in the g-th group as

$$\begin{array}{c}\hfill {\mathbf{h}}_{g,k}^{\mathrm{eff}}\triangleq {\mathbf{T}}_g^{†}{\mathbf{h}}_{g,k}.\end{array}$$

(13)

Given the covariance matrix ${\mathbf{R}}_g$ and the outer precoder ${\mathbf{T}}_g$, each user only needs to quantize its effective channel (${\mathbf{h}}_{g,k}^{\mathrm{eff}}$) and feeds the quantized channel back to the BS.

For quantization of the effective channel, we adopt the random vector quantizer (RVQ), which is widely used to analyze the effects of quantization error and performs close to optimal quantization with a Rayleigh fading channel environment [6,7,9]. Using the RVQ, the Rayleigh component of the effective channel should be quantized according to (13). By the Karhunen–Loeve representation, the effective channel in (13) can be decomposed by the SVD [13], given as

$$\begin{array}{c}\hfill {\mathbf{h}}_{g,k}^{\mathrm{eff}}={\mathbf{T}}_g^{†}{\mathbf{U}}_g{\mathbf{\Lambda }}_g^{1/2}{\mathbf{e}}_{g,k}={\mathbf{\Omega }}_g{\overline{\mathbf{\Sigma }}}_g{\mathbf{\Gamma }}_g^{†}{\mathbf{e}}_{g,k}={\mathbf{\Omega }}_g{\mathbf{\Sigma }}_g{\mathbf{w}}_{g,k},\end{array}$$

(14)

where ${\mathbf{\Omega }}_g{\overline{\mathbf{\Sigma }}}_g{\mathbf{\Gamma }}_g^{†}$ (${\mathbf{\Omega }}_g\in {\mathbb{C}}^{s_g\times s_g},{\overline{\mathbf{\Sigma }}}_g\in {\mathbb{C}}^{s_g\times r_g},{\mathbf{\Gamma }}_g\in {\mathbb{C}}^{r_g\times r_g}$) is SVD of ${\mathbf{T}}_g^{†}{\mathbf{U}}_g{\mathbf{\Lambda }}_g^{1/2}$, and ${\mathbf{\Sigma }}_g\in {\mathbb{C}}^{s_g\times s_g}$ is the matrix comprised of the first $s_g$ columns of ${\overline{\mathbf{\Sigma }}}_g$, i.e., the diagonal matrix with $s_g$ non-zero positive eigenvalues, and ${\mathbf{w}}_{g,k}\in {\mathbb{C}}^{s_g\times 1}$ is the vector with the first $s_g$ elements of ${\mathbf{Y}}_g^{†}{\mathbf{e}}_{g,k}$. Note that ${\mathbf{w}}_{g,k}$ follows the distribution of $\mathcal{C}\mathcal{N}(\mathbf{0},{\mathbf{I}}_{s_g})$, and  we have

$$\begin{array}{c}\hfill {\mathbf{T}}_g^{†}{\mathbf{R}}_g{\mathbf{T}}_g=\left({\mathbf{\Omega }}_g{\mathbf{\Sigma }}_g\right){\left({\mathbf{\Omega }}_g{\mathbf{\Sigma }}_g\right)}^{†}\end{array}$$

(15)

due to the fact that $\mathbb{E}\left[{\mathbf{h}}_{g,k}^{\mathrm{eff}}{\left({\mathbf{h}}_{g,k}^{\mathrm{eff}}\right)}^{†}\right]={\mathbf{T}}_g^{†}\mathbb{E}\left[{\mathbf{h}}_{g,k}{\mathbf{h}}_{g,k}^{†}\right]{\mathbf{T}}_g={\mathbf{T}}_g^{†}{\mathbf{R}}_g{\mathbf{T}}_g$ from (13), and $\mathbb{E}\left[{\mathbf{h}}_{g,k}^{\mathrm{eff}}{\left({\mathbf{h}}_{g,k}^{\mathrm{eff}}\right)}^{†}\right]=\left({\mathbf{\Omega }}_g{\mathbf{\Sigma }}_g\right)\mathbb{E}\left[{\mathbf{w}}_{g,k}{\mathbf{w}}_{g,k}^{†}\right]{\left({\mathbf{\Omega }}_g{\mathbf{\Sigma }}_g\right)}^{†}=\left({\mathbf{\Omega }}_g{\mathbf{\Sigma }}_g\right){\left({\mathbf{\Omega }}_g{\mathbf{\Sigma }}_g\right)}^{†}$ from (14) and ${\mathbf{w}}_{g,k}\sim \mathcal{C}\mathcal{N}(\mathbf{0},{\mathbf{I}}_{s_g})$, respectively.

Allocating B-bits to each user’s feedback size and, using the RVQ, a quantization codebook for user k in the g-th group, i.e., ${\mathcal{C}}_{g,k}=\{{\mathbf{c}}_{g,k,1},\dots ,{\mathbf{c}}_{g,k,2^B}\}$, consists of $2^B$ randomly chosen isotropic $s_g$-dimensional unit norm vectors. In this case, the quantized CDI of ${\mathbf{w}}_{g,k}$ in (14), i.e., ${\widehat{\mathbf{w}}}_{g,k}$, is obtained by

$$\begin{array}{cc}\hfill {\widehat{\mathbf{w}}}_{g,k}& =\underset{\mathbf{c}\in {\mathcal{C}}_{g,k}}{arg~max}{cos}^2\left(\angle ({\tilde{\mathbf{w}}}_{g,k},\mathbf{c})\right)=\underset{\mathbf{c}\in {\mathcal{C}}_{g,k}}{arg~max}{\left|{\tilde{\mathbf{w}}}_{g,k}^{†}\mathbf{c}\right|}^2,\hfill \end{array}$$

(16)

where ${\tilde{\mathbf{w}}}_{g,k}=\frac{{\mathbf{w}}_{g,k}}{\parallel {\mathbf{w}}_{g,k}\parallel }$. Then, the quantization error denoted by $Z_{g,k}^{\widehat{\mathbf{w}}}\in [0,1]$ is defined as

$$\begin{array}{c}\hfill Z_{g,k}^{\widehat{\mathbf{w}}}\triangleq 1-{\left|{\tilde{\mathbf{w}}}_{g,k}^{†}{\widehat{\mathbf{w}}}_{g,k}\right|}^2.\end{array}$$

(17)

For an arbitrary codeword $\mathbf{c}\in {\mathcal{C}}_{g,k}$ in (16), the value ${\left|{\tilde{\mathbf{w}}}_{g,k}^{†}\mathbf{c}\right|}^2$ follows the beta distribution with parameters $(s_g-1,1)$ because it is the square of the inner product of two independent and isotropic unit-norm random vectors in ${\mathbb{C}}^{s_g}$ [6,7]. Consequently, a quantization error using B-bits RVQ, i.e., $Z_{g,k}^{\widehat{\mathbf{w}}}$ in (17), is the minimum of $2^{s_g}$ independent beta distributed random variables, of which the complementary cumulative density function (CDF) is given by $\mathrm{Pr}\left[Z_{g,k}^{\widehat{\mathbf{w}}}>z\right]={\left(1-z^{s_g-1}\right)}^{2^B}$, and the expectation is bounded as [6,7]

$$\begin{array}{c}\hfill \mathbb{E}\left[Z_{g,k}^{\widehat{\mathbf{w}}}\right]<2^{-\frac{B}{s_g-1}}.\end{array}$$

(18)

After receiving the feedback of ${\widehat{\mathbf{w}}}_{g,k}$, the BS obtains the quantized CDI of the dimension-reduced effective channel according to (14) given by

$$\begin{array}{c}\hfill {\widehat{\mathbf{h}}}_{g,k}^{\mathrm{eff}}=\frac{{\mathbf{\Omega }}_g{\mathbf{\Sigma }}_g{\widehat{\mathbf{w}}}_{g,k}}{∥{\mathbf{\Omega }}_g{\mathbf{\Sigma }}_g{\widehat{\mathbf{w}}}_{g,k}∥},\end{array}$$

(19)

and the quantization error of the dimension-reduced effective channel ${\widehat{\mathbf{h}}}_{g,k}^{\mathrm{eff}}$ denoted by $Z_{g,k}^{{\widehat{\mathbf{h}}}^{\mathrm{eff}}}\in [0,1]$ is defined as

$$\begin{array}{c}\hfill Z_{g,k}^{{\widehat{\mathbf{h}}}^{\mathrm{eff}}}\triangleq 1-{\left|{\left({\tilde{\mathbf{h}}}_{g,k}^{\mathrm{eff}}\right)}^{†}{\widehat{\mathbf{h}}}_{g,k}^{\mathrm{eff}}\right|}^2.\end{array}$$

(20)

Note that the distribution of $Z_{g,k}^{{\widehat{\mathbf{h}}}^{\mathrm{eff}}}$ in (20) can be different from the distribution of $Z_{g,k}^{\widehat{\mathbf{w}}}$ in (17) (i.e., beta distribution) since the quantized CDI of the dimension-reduced effective channel is projected by the covariance matrix and the outer precoder.

Based on the quantized CDI of the dimension-reduced effective channel of the g-th group, i.e., ${\widehat{\mathbf{H}}}_g^{\mathrm{eff}}=\left[{\widehat{\mathbf{h}}}_{g,1}^{\mathrm{eff}},\dots ,{\widehat{\mathbf{h}}}_{g,K_g}^{\mathrm{eff}}\right]\in {\mathbb{C}}^{s_g\times K_g}$, the BS constructs the inner precoder of the g-th group, i.e., ${\mathbf{V}}_g=\left[{\mathbf{v}}_{g,1},\dots ,{\mathbf{v}}_{g,K_g}\right]$, where a ZF beamforming vector of user k in the g-th group, i.e., ${\mathbf{v}}_{g,k}$ is obtained as

$$\begin{array}{c}\hfill {\mathbf{v}}_{g,k}=\frac{{\mathcal{A}}_{g,k}{\widehat{\mathbf{h}}}_{g,k}^{\mathrm{eff}}}{∥{\mathcal{A}}_{g,k}{\widehat{\mathbf{h}}}_{g,k}^{\mathrm{eff}}∥},\end{array}$$

(21)

where ${\mathcal{A}}_{g,k}\triangleq {\mathbf{A}}_{g,k}^{\perp }{\left({\mathbf{A}}_{g,k}^{\perp }\right)}^{†}\in {\mathbb{C}}^{s_g\times s_g}$ is the null-space of the quantized CDI of the effective channels of all other users in group g, where ${\mathbf{A}}_{g,k}^{\perp }$ is a $s_g\times (s_g-K_g+1)$ submatrix consisted by orthonormal column vectors, which is obtained from the SVD of ${\widehat{\mathbf{H}}}_{g,-k}^{\mathrm{eff}}$ such as

$$\begin{array}{c}\hfill {\widehat{\mathbf{H}}}_{g,-k}^{\mathrm{eff}}=\left[{\mathbf{A}}_{g,k},{\mathbf{A}}_{g,k}^{\perp }\right]{\mathbf{\Xi }}_{g,k}{\mathbf{L}}_{g,k}^{†},\end{array}$$

(22)

where ${\widehat{\mathbf{H}}}_{g,-k}^{\mathrm{eff}}$ is the quantized CDI of the effective channels of all other users in group g, i.e., ${\widehat{\mathbf{H}}}_{g,-k}^{\mathrm{eff}}=\left[{\widehat{\mathbf{h}}}_{g,1}^{\mathrm{eff}},\dots ,{\widehat{\mathbf{h}}}_{g,k-1}^{\mathrm{eff}},{\widehat{\mathbf{h}}}_{g,k+1}^{\mathrm{eff}},\dots ,{\widehat{\mathbf{h}}}_{g,K_g}^{\mathrm{eff}}\right]$ [11,29].

3.3. Problem Formulation

With the two-stage precoder and the quantized CDI of the effective channel, the signal-to-interference plus noise ratio (SINR) of user k in the g-th group is obtained from (5) as follows:

$$\begin{array}{cc}\hfill {\mathrm{SIN}\mathrm{R}}_{g,k}& =\frac{\gamma {\left|{\left({\mathbf{h}}_{g,k}^{\mathrm{eff}}\right)}^{†}{\mathbf{v}}_{g,k}\right|}^2}{\underset{\mathrm{SGI}}{\underbrace{\gamma {\displaystyle \sum _{i\ne k}^{K_g}}{\left|{\left({\mathbf{h}}_{g,k}^{\mathrm{eff}}\right)}^{†}{\mathbf{v}}_{g,i}\right|}^2}}+\underset{\mathrm{IGI}}{\underbrace{\gamma {\displaystyle \sum _{c\ne g}^G}{\displaystyle \sum _{j=1}^{K_c}}{\left|{\mathbf{h}}_{g,k}^{†}{\mathbf{T}}_c{\mathbf{v}}_{c,j}\right|}^2}}+1},\hfill \end{array}$$

(23)

where $\gamma \triangleq \frac{P}{K{\sigma }^2}$ is the signal-to-noise ratio (SNR) at each user. Then, the average sum rate denoted by ${\mathcal{R}}_{{}_{\mathrm{sum}}}$ is given by

$$\begin{array}{c}\hfill {\mathcal{R}}_{{}_{\mathrm{sum}}}={\displaystyle \sum _{g=1}^G}{\displaystyle \sum _{k=1}^{K_g}}\mathbb{E}\left[{\log }_2\left(1+{\mathrm{SIN}\mathrm{R}}_{g,k}\right)\right].\end{array}$$

(24)

Analyzing (23) and (24), the quantized CDI of the effective channel, i.e., ${\mathbf{h}}_{g,k}^{\mathrm{eff}}$, is only fed back to the BS, so the IGI term in ${\mathrm{SIN}\mathrm{R}}_{g,k}$ is not affected by the quantized CDI. In this case, the IGI term is determined by the outer precoders for the other groups, i.e., ${\left\{{\mathbf{T}}_c\right\}}_{c\ne g}^G$. With exploiting the BD method for the outer precoder design as in (10), the performance of IGI cancellation, i.e., ${\sum }_{c\ne g}^G{\sum }_{j=1}^{K_c}{\left|{\mathbf{h}}_{g,k}^{†}{\mathbf{T}}_c{\mathbf{v}}_{c,j}\right|}^2$, depends on the reduced dimension numbers of the effective channels of the other groups, i.e., ${\left\{s_c\right\}}_{c\ne g}^G$. On the other hand, the magnitude of the desired signal term, i.e., ${\left|{\left({\mathbf{h}}_{g,k}^{\mathrm{eff}}\right)}^{†}{\mathbf{v}}_{g,k}\right|}^2$, and the performance of SGI cancellation, i.e., ${\sum }_{i\ne k}^{K_g}{\left|{\left({\mathbf{h}}_{g,k}^{\mathrm{eff}}\right)}^{†}{\mathbf{v}}_{g,i}\right|}^2$, are restricted by the dimension of the outer precoder of g-th group, i.e., $s_g$ [19]. Note that the quantization error of the effective channel, i.e., ${\widehat{\mathbf{h}}}_{g,k}^{\mathrm{eff}}$ in (20), is more limited by $s_g$ given feedback bit allocation, i.e., B-bits. Therefore, the reduced dimensions of the effective channels for all groups (i.e., $s_1,\dots ,s_G$) should be jointly optimized considering all the interactions among them in order to maximize the average sum rate in (24). Thus, we formulate the optimization problem to find the optimal $s_1,\dots ,s_G$ as follows [19]:

$$\begin{array}{cc}\hfill \mathit{P}\mathbf{1}:\underset{s_1,\dots ,s_G}{\mathrm{maximize}}& {\displaystyle \sum _{g=1}^G}{\displaystyle \sum _{k=1}^{K_g}}\mathbb{E}\left[{\log }_2\left(1+{\mathrm{SIN}\mathrm{R}}_{g,k}\right)\right],\hfill \\ \hfill \mathrm{subject}\mathrm{to}& s_g\in {\mathbb{Z}}^{+},\forall g=1,\dots ,G,\hfill \\ & K_g\le s_g\le r_g.\hfill \end{array}$$

(25)

The optimization problem $\mathit{P}\mathbf{1}$ is difficult to solve directly because it is a mixed-integer problem, which is generally known to be NP-hard. Moreover, the effect of the dimensions is implicit in the objective function, and the optimal solution can be obtained only by numerical search for every channel realization, which is almost impossible in practice. Note that, once the reduced dimensions ${\left\{s_g\right\}}_{g=1}^G$ are determined, the BS informs $s_g$ to the users in the g-th group so that they can quantize $s_g$-dimensional effective channels with the corresponding codebook.

4. Our Previous Work on Dimension Optimization

As we explained in the previous section, the problem $\mathit{P}\mathbf{1}$ is hard to solve because it is an NP-hard problem and the effect of the dimensions is implicit in the objective function. In this section, we briefly explain how we solved the problem $\mathit{P}\mathbf{1}$ in our previous work [19].

In [19], we showed that the objective function of the problem $\mathit{P}\mathbf{1}$ can be approximated [3] and then lower bounded as follows:

$$\begin{array}{cc}{\displaystyle \sum _{g=1}^G}{\displaystyle \sum _{k=1}^{K_g}}\mathbb{E}\left[{\log }_2\left(1+{\mathrm{SIN}\mathrm{R}}_{g,k}\right)\right]\hfill & \\ \approx {\displaystyle \sum _{g=1}^G}{\displaystyle \sum _{k=1}^{K_g}}{\log }_2\left(1+\frac{\mathbb{E}\left[\gamma {\left|{\left({\mathbf{h}}_{g,k}^{\mathrm{eff}}\right)}^{†}{\mathbf{v}}_{g,k}\right|}^2\right]}{\mathbb{E}\left[\gamma {\sum }_{i\ne k}^{K_g}{\left|{\left({\mathbf{h}}_{g,k}^{\mathrm{eff}}\right)}^{†}{\mathbf{v}}_{g,i}\right|}^2+\gamma {\sum }_{c\ne g}^G{\sum }_{j=1}^{K_c}{\left|{\mathbf{h}}_{g,k}^{†}{\mathbf{T}}_c{\mathbf{v}}_{c,j}\right|}^2+1\right]}\right)\hfill & \hfill \left(26\right)\\ >{\displaystyle \sum _{g=1}^G}{\mathcal{R}}_g^{\prime }\left(s_1,\dots ,s_G\right),\hfill & \hfill \left(27\right)\end{array}$$

where

$$\begin{array}{cc}\hfill {\mathcal{R}}_g^{\prime }& \left(s_1,\dots ,s_G\right)\triangleq {\log }_2\left(1+\frac{\gamma \left\{\mathrm{Tr}\left({\mathbf{T}}_g^{†}{\mathbf{R}}_g{\mathbf{T}}_g\right)-{\sum }_{j=1}^{K_g-1}{\lambda }_j\left({\mathbf{R}}_g\right)\right\}}{\gamma \frac{K_g-1}{s_g-1}{2}^{-\frac{B}{s_g-1}}\mathrm{Tr}\left({\mathbf{T}}_g^{†}{\mathbf{R}}_g{\mathbf{T}}_g\right)+\gamma {\sum }_{c\ne g}^G{K}_c\mathrm{Tr}\left({\mathbf{T}}_c^{†}{\mathbf{R}}_g{\mathbf{T}}_c\right)+1}\right),\hfill \end{array}$$

(28)

with ${\lambda }_j\left({\mathbf{R}}_g\right)$ the j-th largest eigenvalue of ${\mathbf{R}}_g$. Thus, the effect of dimensions becomes explicit in (27). Note that, given ${\left\{s_g\right\}}_{g=1}^G$, the BS obtains the lower bound of (27) based only on the covariance matrices ${\left\{{\mathbf{R}}_g\right\}}_{g=1}^G$ and the outer precoders ${\left\{{\mathbf{T}}_g\right\}}_{g=1}^G$ without the CDI of $s_g$-dimensional effective channels (i.e., ${\mathbf{h}}_{g,k}^{\mathrm{eff}}$ for all $g\in \{1,\dots ,G\}$ and $k\in \{1,\dots ,K_g\}$).

For a practical design, we can establish using the lower bound (28) as follows:

$$\begin{array}{cc}\hfill \mathit{P}\mathbf{2}:\underset{s_1,\dots ,s_G}{\mathrm{maximize}}& {\displaystyle \sum _{g=1}^G}{\mathcal{R}}_g^{\prime }\left(s_1,\dots ,s_G\right),\hfill \\ \hfill \mathrm{subject}\mathrm{to}& s_g\in {\mathbb{Z}}^{+},\forall g=1,\dots ,G,\hfill \\ & K_g\le s_g\le r_g.\hfill \end{array}$$

(29)

The problem $\mathit{P}\mathbf{2}$ is also a mixed-integer problem, and obtaining the objective function of (29) in a closed-form is not available because the outer precoders ${\left\{{\mathbf{T}}_g\right\}}_{g=1}^G$ are constructed by the procedure in (7)–(10) with respect to ${\left\{s_g\right\}}_{g=1}^G$. Thus, the optimal solution to problem $\mathit{P}\mathbf{2}$ is obtained by a combinatorial joint optimization of $s_1,\dots ,s_G$, which requires the G-dimensional numerical search, so it is still complex.

To reduce complexity of the G-dimensional numerical search in problem $\mathit{P}\mathbf{2}$, we can assume that the dimensions of the effective channels among all groups are the same as s, i.e., $s_1=\cdots =s_G=s$. Then, the problem $\mathit{P}\mathbf{2}$ is reduced to

$$\begin{array}{cc}\hfill \mathit{P}\mathbf{3}:\underset{s}{\mathrm{maximize}}& {\displaystyle \sum _{g=1}^G}{\mathcal{R}}_g^{\prime }\left(s\right),\hfill \\ \hfill \mathrm{subject}\mathrm{to}& s\in {\mathbb{Z}}^{+}\hfill \\ & max\{K_1,\dots ,K_g\}\le s\le min\{r_1,\dots ,r_G\},\hfill \end{array}$$

(30)

and the optimal solution of the problem $\mathit{P}\mathbf{3}$ can be obtained from one-dimensional numerical search [19]. We describe the detailed procedure to optimal the optimal solution of the problem $\mathit{P}\mathbf{3}$ in Algorithm 1.

Algorithm 1 Finding the optimal solution of the problem $\mathit{P}\mathbf{3}$

1: Input: Channel covariance matrices ${\left\{{\mathbf{R}}_g\right\}}_{g=1}^G$, number of users in the groups ${\left\{K_g\right\}}_{g=1}^G$, SNR $\gamma$, and feedback bit B   2: Obtain ${\mathbf{U}}_g$ and $r_g$ for all $g\in \{1,\dots ,G\}$ by SVD   3: Define $s_{min}\triangleq max\{K_1,\dots ,K_g\}$ and $s_{max}\triangleq min\{r_1,\dots ,r_G\}$   4: Set $s\in {\mathbb{Z}}^{+}$ and $s_{min}\le s\le s_{max}$   5: for$s=s_{min},\cdots ,s_{max}$do   6:     Construct outer precoder ${\mathbf{T}}_g\in {\mathbb{C}}^{M\times s}$ for all $g\in \{1,\dots ,G\}$ (referring to Section 3.1)   7:     Compute ${\mathcal{R}}_g^{\prime }\left(s\right)$ based on (28) for all $g\in \{1,\dots ,G\}$   8:     Compute ${\sum }_{g=1}^G{\mathcal{R}}_g^{\prime }\left(s\right)$   9: end for 10: Obtain $s^{★}=\arg {\displaystyle \underset{s_{min}\le s\le s_{max}}{max}}{\mathcal{R}}_g^{\prime }\left(s\right)$ 11: Output: The optimal dimension $s^{★}$

5. Machine Learning Framework for Dimension Optimization

In this section, we propose the machine learning framework for dimension optimization (i.e., the problem $\mathit{P}\mathbf{1}$). Note that our machine learning framework is based on a deep neural network (DNN) and tackles problem $\mathit{P}\mathbf{1}$ directly.

5.1. Preliminary: The General DNN Architecture

Our proposed machine learning-based dimension optimization utilizes a DNN [27], so, in this subsection, we briefly explain a general DNN model. A DNN is one of the most popular algorithms in machine learning, which is considered as a multiple layer perceptron (MLP) [20,21,22]. The DNN is comprised of one input layer, $L-2$ hidden layers, and one output layer. The DNN with many hidden layers enhances its learning and mapping abilities, so it is capable of handling complicated nonlinear problems [20,21,22,25,26].

Let $\mathbf{w}\in {\mathbb{R}}^{N_0}$ be the input vector of the input layer and $\mathbf{o}\in {\mathbb{R}}^{N_{(L-1)}}$ be the output vector of the output layer. Then, the mapping between them can be mathematically represented as follows:

$$\begin{array}{c}\hfill \mathbf{o}=f\left(\mathbf{w};\mathit{\theta }\right)=f^{(L-1)}\left(f^{(L-2)}(\cdots f^{\left(2\right)}\left(f^{\left(1\right)}(\mathbf{w};{\mathit{\theta }}_l)\right))\right),\end{array}$$

(31)

where $f^{\left(l\right)}(\mathbf{w};{\mathit{\theta }}_l)$ for $l\in \{1,2,\dots ,L-1\}$ is an activation function of the l-th layer, which maps the input and output vectors of the l-th layer, and $\mathit{\theta }=\{{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_{(L-1)}\}$ is a set of parameters such as weights and biases to calculate the weighted sum of nodes, which can be adjusted during the training procedure [20,21,22]. Activation functions in (31) are prerequisite to tackle nonlinear problems in the DNN model. In particular, multiple nodes in each layer operate with the activate functions to map its input vector to the output vector with nonlinear operations. Various activation functions are listed in

Table 1. Mathmatical expressions for activation functions.

Name	Sigmoid	tanh	ReLU	SSaLU	Softmax
$f\left(w\right)$	$\frac{1}{1+e^{-w}}$	$\frac{1-e^{-w}}{1+e^{-w}}$	$max(0,w)$	$\left\{\begin{array}{cc}max(-1,w)\hfill & w<0\hfill \\ min(1,w)\hfill & w\ge 0\hfill \end{array}\right.$	$\frac{e^w}{{\sum }_j{e}^{w_j}}$

and illustrated in Figure 3.

A set of parameters in (31), i.e., $\mathit{\theta }$, is adjusted in the training procedure to minimize the loss function [20,21,22]. In supervised learning, each training sample is labeled by the desired output $\overline{\mathbf{o}}$ (i.e., the correct answer), and hence the loss function between the DNN-based model output $\mathbf{o}$ and the desired output $\overline{\mathbf{o}}$ becomes

$$\begin{array}{c}\hfill loss\left(\mathit{\theta }\right)=\frac{1}{\overline{n}}{\displaystyle \sum _{j=1}^{\overline{n}}}{L}_f\left({\overline{\mathbf{o}}}_j,{\mathbf{o}}_j\right),\end{array}$$

(32)

where ${\overline{\mathbf{o}}}_j$ and ${\mathbf{o}}_j$ are the desired output and the predicted output from the j-th training sample, respectively. In addition, $\overline{n}$ is the batch-size of training samples, where the batch-size means the total number of training samples in a mini batch.

There are mainly two types of loss functions for supervised learning in a DNN [20,21,22] as shown in

Table 2. Mathematical expressions for loss functions (i denotes i-th element of a vector).

Name	Mean Square Error	Cross Entropy
$L_f\left(\overline{\mathbf{o}},\mathbf{o}\right)$	${\parallel \overline{\mathbf{o}}-\mathbf{o}\parallel }^2$	$-{\sum }_i\left({\overline{o}}_{i}log{o}_i+(1-{\overline{o}}_i)log(1-o_i)\right)$

.

For supervised learning, the ultimate goal of DNN design is to map $\mathbf{w}$ to $\overline{\mathbf{o}}$ from (31) based on training samples. Thus, training is the process of adjusting $\mathit{\theta }$ to minimize the loss function in (32). The widely used algorithm for training is a stochastic gradient descent (SGD) method [20,21,22,24], which updates its gradient for each layer to minimize the loss function at each step, i.e., mini batch. In addition, there are many modified ones such as SGD with momentum (SGDM), adaptive gradient (AdaGrad), and root mean square propagation (RMSProp) [20,21,22,30,31,32]. Adaptive moment estimation (Adam) is also widely used for DNN training.

5.2. The Proposed DNN Framework

The proposed DNN architecture is illustrated in Figure 4, where it is comprised of one input layer of $\left(\right(M+1)G+2)$ input nodes, one output layer of M output nodes, and $\overline{L}$ hidden layers of $(N_1,\dots ,N_{\overline{L}})$ nodes. The input layer of the proposed DNN model takes each user’s SNR, feedback sizes, the number of users, and eigenvalues of each user group’s covariance matrix. In addition, the output layer returns the dimension for each group. Although the number of input nodes depends on the number of user groups, they are fixed in our design because the number of groups is given for the system model according to channel statistics. Meanwhile, the number of hidden layers and nodes are mediatable variables according to the DNN architecture.

5.2.1. Input Layer

To establish the DNN-based supervised learning framework, the input data of a learning system, i.e., $\mathbf{w}$ in (31), should be determined considering the system model and the problem $\mathit{P}\mathbf{1}$. Owing to the insight from the lower bound ${\sum }_{g=1}^G{\mathcal{R}}_g^{\prime }\left(s_1,\dots ,s_G\right)$ in Section 4, our proposed DNN model takes the input of SNR at each user ($\gamma$), feedback bit allocation (B), and the number of users ($K_g$) and eigenvalues of covariance matrix for all groups as follows:

$$\begin{array}{c}\hfill \mathbf{w}={\left[B,\gamma ,\underset{\mathrm{Group}1}{\underbrace{\{K_1,{\lambda }_{1,1},\dots ,{\lambda }_{1,M}\}}},\dots ,\underset{\mathrm{Group}\mathrm{G}}{\underbrace{\{K_G,{\lambda }_{G,1},\dots ,{\lambda }_{G,M}\}}}\right]}^T\in {\mathbb{R}}^{(M+1)G+2},\end{array}$$

(33)

where $\{{\lambda }_{g,1},\dots ,{\lambda }_{g,M}\}\in {\mathbb{R}}^M$ for the g-th group can be divided by the $r_g$ non-zero eigenvalues of the covariance matrix of the g-th group and $(M-r_g)$ zeros. For the feature information of the input, the eigenvalues characterize the optimal dimensions because the covariance matrices affect the objective function of problem $\mathit{P}\mathbf{1}$, which is statistically determined by the effective channels and quantization codebooks. Note that we adopt the eigenvalues rather than the covariance matrices itself in the view of principal component analysis (PCA) for the machine learning technique [22]. It reduces the complexity of the proposed DNN model from large-scale antennas because taking covariance matrices (${\left\{{\mathbf{R}}_g\right\}}_{g=1}^G$) as the input requires $((M^2+1)G+2)$ input nodes.

5.2.2. Output Layer

The proposed DNN framework aims at obtaining the solution of problem $\mathit{P}\mathbf{1}$, i.e., $\{s_1^{★},\dots ,s_G^{★}\}$. The number of eigenvalues for the g-th group is variable dependent on the channel statistics, i.e., covariance matrices, whose range is $r_g\le M$. Thus, the maximum number of classes for the optimal dimensions considering all groups becomes $M^G$, which is almost impossible to establish the DNN model in practice for massive MIMO systems. We adopt the parallel DNN framework for each group as shown in Figure 4. Then, the desired output vector ${\overline{\mathbf{o}}}_g$ for the g-th group is expressed by

$$\begin{array}{c}\hfill {\overline{\mathbf{o}}}_g={\left[0,\dots ,0,\underset{\left(s_g^{★}\text{-}\mathrm{th}\mathrm{component}\right)}{1},0,\dots ,0\right]}^T\in {\mathbb{R}}^M,\end{array}$$

(34)

where (34) is comprised of the $s_g^{★}$-th component one and the other components zero. For the output layer, the Softmax activation function is used for outputs in the interval $(0,1)$. Thus, the position of 1 in the g-th output vector becomes the dimension for the g-th group.

5.2.3. Hidden Layer

The hidden layer is designed to learn the features of the training samples. They are redefined according to the system model to characterize the relationship between the input and the output layers. We adopt the tanh function as the activation function and determine the number of hidden layers and the number of nodes to increase the performance in the training procedure.

5.2.4. DNN Training

For the precise classification in the training to map the input of (33) into the desired output of (34), we need enough training samples that reflects many situations. Thus, we generate the training samples varying SNR, the feedback size of each user, and eigenvalues of the covariance matrices. In particular, we focus on randomly generating covariance matrices varying angular spread in (3) to learn the feature information of eigenvalues, i.e., channel statistics of the system model. With the fixed azimuth center angle in (3), the number of eigenvalues of covariance matrix increases as the angular spread increases. It also leads to variation of magnitude of the eigenvalues. Note that the number of groups and the number of users in each group are the system model parameters, so they are fixed in the DNN training.

Then, we obtain the optimal dimensions of the problem $\mathit{P}\mathbf{1}$ by numerical search for every channel realization and label the samples according to (34) with perspective of the given SNR, feedback size, and covariance matrices of the groups. For the proposed DNN training, we adopt cross entropy as the loss function in (32), which is widely used for multi-class classification of DNN architecture [20,21,22]. Thus, the loss function of the g-th group DNN model is given by

$$\begin{array}{cc}\hfill loss\left({\mathit{\theta }}_g\right)& =\frac{1}{\overline{n}}{\displaystyle \sum _{j=1}^{\overline{n}}}{L}_f\left({\overline{\mathbf{o}}}_j^g,{\mathbf{o}}_j^g\right)\hfill \\ & =-\frac{1}{\overline{n}}{\displaystyle \sum _{j=1}^{\overline{n}}}{\displaystyle \sum _{i=1}^M}\left({\overline{o}}_{j,i}^{g}log{o}_{j,i}^g+(1-{\overline{o}}_{j,i}^g)log(1-o_{j,i}^g)\right),\hfill \end{array}$$

(35)

where $\overline{n}$ is the batch-size of the training samples, and j denotes the j-th training sample; ${\overline{\mathbf{o}}}_j^g$ and ${\mathbf{o}}_j^g$ are the desired output vector determined by $s_g^{★}$ and (34), and the predicted output in the output layer obtained by the Softmax activation function of the g-th group; ${\overline{o}}_{j,i}^g$ and $o_{j,i}^g$ denote the i-th elements of ${\overline{\mathbf{o}}}_j^g$ and ${\mathbf{o}}_j^g$, respectively.

For our DNN model training, we divided the samples into the training and the validation sets with the ratio of $0.85$ and $0.15$, respectively. We use the Adam algorithm [32,33] for training, which is the combined and updated version of AdaGrad [30] and RMSProp [31]. To prevent under-fitting and over-fitting, we set the maximum number of epochs to 3000, the batch-size to 1000, and initial learning rate to $0.001$. The learning rate is adjusted by the drop factor of $0.5$ for every 1000 epoch during the training. In addition, we used the $L_2$-regularization factor of $0.01$, gradient decay factor of $0.98$, and squared gradient decay factor of $0.99$. To prevent the over-fitting, we also adopt an early stopping strategy, which terminates the training when the maximum number of epochs or validation patience criterion is reached. The validation patience is the number of times that the loss on the validation set is larger than or equal to the previously obtained smallest loss [33]. We adopt the validation check with every five epochs and validation patience criterion of eight.

6. DNN Performance and Numerical Results

In this section, we evaluate our proposed DNN framework. We consider a single-cell massive MIMO systems with limited feedback, where the BS is equipped with 64 ULA antennas (i.e., $M=64$). The BS serves 15 single-antenna users (i.e., $K=15$) that can be divided into three groups (i.e., $G=3$) with 5 users in each group (i.e., $K_g=5$ for $g\in \{1,2,3\})$. For the channel model, we consider the Ralyeigh correlated channel with ${\theta }_g=-\frac{\pi }{4}+\frac{\pi }{4}(g-1)$ and $\frac{{\delta }_a}{{\lambda }_c}=0.5$ in (3). As mentioned in Section 5.2.4, we generate covariance matrices varying angular spread, where ${\mathbf{\Delta }}_g$ is randomly generated in the range of ${\mathbf{\Delta }}_g\in \left[\frac{2\pi }{45},\frac{\pi }{9}\right]$. We denote the number of eigenvalues of the g-th group covariance matrix by $r_g$ (i.e., rank). With this setting, the ranges of each group’s rank are given by $r_1\in \{10,\dots ,20\}$, $r_2\in \{13,\dots ,27\}$, and $r_3\in \{10,\dots ,20\}$, respectively. To obtain the average sum rate of ${\mathcal{R}}_{{}_{\mathrm{sum}}}$ in (24), we average out channel realizations based on RVQ codebooks varying small-scale Rayleigh channel fading ${\mathbf{e}}_{g,k}$ in (2) with the fixed covariance matrices ${\left\{{\mathbf{R}}_g\right\}}_{g=1}^3$.

First of all, we establish our proposed DNN framework and train our machine learning model for our system model setting. Our DNN model operates in parallel and thus is trained for each group as explained in Section 5.2. Thus, there are a total of three DNN models of the same structure at the BS. The DNN model of each group is comprised of the input layer of 197 nodes, the output layer of 64 nodes, and six hidden layers of 200 nodes each. For the DNN training, we generate $4\times {10}^5$ training samples varying each user’s SNR, feedback sizes, and covariance matrices as the optimal dimensions (i.e., ${\left\{s_g^{★}\right\}}_{g=1}^3$) are obtained by every training sample according to the problem $\mathit{P}\mathbf{1}$ by numerical search. To train our DNN model for each group, each sample for the g-th group DNN model is labeled by its own desired output, i.e., ${\overline{\mathbf{o}}}_g$ in (34) according to $s_g^{★}$. To improve the training performance, the samples are are divided into the training set and the validation set, and we adopt the Adam algorithm with the parameter setting as explained in Section 5.2.4.

In Figure 5, we show the training state and performance of the proposed DNN model for every group. Figure 5a shows the cross entropy loss of the training and validation set with respect to the number of iterations. The cross entropy losses for all groups decrease as the number of iterations increases, but the gap between the training and validation sets also increases as the number of iterations increases. Thus, the stopping strategy is required to terminate the training to avoid over-fitting as we explained in Section 5.2.4. The losses of group 1 and group 3 are similar, while the loss of group 2 is larger than them. However, the gap between them decreases as the number of iterations increases.

Figure 5b shows the accuracy of our machine learning model for the training and the validation sets with respect to the number of iterations. The accuracy is measured with the ratio of the numbers of correct answers (i.e., $\mathbf{o}=\overline{\mathbf{o}}$) to wrong answers (i.e., $\mathbf{o}\ne \overline{\mathbf{o}}$). The accuracy increases as the number of iterations increases as we can expect from Figure 5a. The final accuracies from the training set and the validation set become $(84.7,82.0,85.7)\%$, and $(83.2,78.4,84.3)\%$, respectively. Figure 5c shows the accuracy for all samples in the validation set with respect to the number of ranks for the covariance matrix of each group. Thus, we can conclude that the our proposed DNN model is well trained and learn the features from channel statistics information.

Figure 6 compares the average sum rate of the proposed DNN model-based scheme and other reference schemes, where the feedback size are 6 bits and 10 bits in Figure 6a,b, respectively. For comparison, we consider the following five reference schemes:

• Optimal scheme: The optimal dimensions of the outer precoder are obtained via brute-force numerical search.
• Full rank-based scheme: The dimensions of the outer precoder are the number of ranks of covariance matrices.
• Lower bound-based scheme: The dimensions of the outer precoder are obtained by the lower bound-based analysis and the solution of problem $\mathit{P}\mathbf{3}$.
• Fixed dimension scheme 1: The outer precoder dimensions are fixed to five, i.e., $s_1=\cdots =s_G=5$.
• Fixed dimension scheme 2: The outer precoder dimensions are fixed to eight, i.e., $s_1=\cdots =s_G=8$.

For performance comparison, we average out 100 covariance realizations varying the angular spread. In Figure 6a, the proposed scheme outperforms the reference schemes and is comparable to the optimal scheme. The reason is that the dimensions obtained by the proposed DNN framework are closer to the optimal dimensions. Note that the lower bound-based scheme increases the average sum rate compared to the other reference schemes, (i.e., full rank-based scheme, and fixed dimension scheme 1 and 2), and it is lower bounded on both the optimal and the proposed schemes. In Figure 6b, as expected, the proposed scheme outperforms the reference schemes and achieves near-optimal performance. The gap between the proposed scheme and other reference schemes is different from those from the gap in Figure 6a. This is because the optimal dimensions are affected by feedback bit allocations. Consequently, we conclude that the dimension of the outer precoder is optimized to maximize the average sum rate and the proposed DNN model-based scheme performs well and is comparable to the optimal scheme.

7. Conclusions

In this paper, we optimized the dimension of the outer precoder in the two-stage precoder to maximize the average sum rate in massive MIMO systems when feedback size is limited. We proposed the DNN framework to find the optimal dimensions in order to maximize the average sum rate, where the original problem is an NP-hard problem. We established the DNN-based supervised learning framework, which takes the SNR at each user, feedback bit allocation, and eigenvalues of covariances of user groups as inputs and returns the optimal dimensions allocating for all user groups. The numerical results showed that our proposed machine learning based outer precoder dimension optimization improves the average sum-rate and achieves near-optimal performance using brute-forcing searching, which is not feasible in practice. Although we consider single-antennas users in our system model, our proposed scheme can be extended for the multi-antenna users. In this case, for inner precoder design, we can use block diagonalization (BD) to cancel the inter-user interference instead of the ZF precoder. However, it is not easy to design the matrix codebooks, which are shared among the transmitter and the users for limited feedback. The extensions of our proposed scheme for more general system models are our on-going research topics.