Performance Analysis of Generalized Quadrature Spatial Modulation with Quasi-Orthogonal Space-Time Block Codes Under Nakagami m-Fading Channels

Abstract

Spatial Modulation (SM) is a promising technique for future wireless communication systems, as it reduces hardware cost and complexity while maintaining good bit error rate (BER) performance in MIMO systems. However, in real-world scenarios, systems often face challenges like antenna correlation and partial knowledge of the channel at the receiver (CSIR). This paper examines the performance of a new communication method called Generalized Quadrature Spatial Modulation (GQSM), combined with Quasi-Orthogonal Space-Time Block Codes (QOSTBC), under realistic fading conditions using Nakagami-m channels. To address the impact of imperfect CSIR, the paper introduces three new QR decomposition-based detection techniques. These methods are specifically designed to reduce errors and enhance reliability in conditions where traditional maximum likelihood (ML) detection performs poorly. A detailed theoretical analysis of all three detection schemes is provided to explain their performance and advantages. Among them, Technique III yields the best results in extensive Monte Carlo simulations, demonstrating improved error performance with significantly lower computational complexity than ML detection. Overall, the proposed detection methods not only overcome the limitations of ML detection but also provide a practical and scalable solution for challenging wireless environments by effectively leveraging the numerical stability of QR decomposition.

1. Introduction

Sixth-generation (6G) wireless communication systems are expected to support a wide range of demanding applications such as ultra-reliable low-latency communications (URLLC), enhanced mobile broadband (eMBB), and massive machine-type communications (mMTC) [1,2,3,4,5]. To meet these requirements, ultra-massive multiple-input multiple-output (UM-MIMO) technology has emerged as a crucial catalyst, using a very large number of antennas to achieve high data rates, energy efficiency and link reliability [6,7,8]. Despite its advantages, UM-MIMO increases system complexity and power consumption due to an extensive array of radio frequency (RF) chains that are required and accurate channel state information at the receiver (CSIR) [9,10].

Spatial modulation (SM) and its variants, such as space shift keying (SSK) and generalized spatial modulation (GSM), offer an elegant trade-off by conveying information using antenna indices, thereby reducing hardware complexity and energy consumption [11,12]. Recently, generalized quadrature spatial modulation (GQSM) [13,14] has been introduced, which further improves spectral efficiency by transmitting independent data symbols on the in-phase and quadrature branches using multiple antennas. However, most studies on GQSM have been conducted under ideal conditions, assuming uncorrelated MIMO channels and perfect CSIR, conditions that are rarely encountered in practical systems.

In realistic wireless environments, two major factors can severely degrade the performance of spatial modulation schemes: transmit antenna correlation and imperfect Channel State Information at the Receiver (CSIR). Due to space limitations in UM-MIMO systems, transmit antennas are often placed close together, resulting in spatial correlation that reduces the orthogonality of channel paths and complicates detection [15,16,17,18,19,20]. Furthermore, CSIR is rarely perfect in real-world deployments due to factors such as estimation errors, quantization, and feedback delays, which lead to residual interference and mismatch in the detection process [21]. Bit error rate (BER) performance frequently suffers as a result, particularly for maximum likelihood (ML) detection techniques [12].

Although several previous studies have examined spatial modulation under Rayleigh or Rician fading and partial CSIR [22,23,24], few have addressed the combined effects of transmit antenna correlation and imperfect CSIR in the context of GQSM. Moreover, most analyses are limited to simplified fading models. The Nakagami-m fading model offers a more general and flexible alternative, able to simulate a variety of fading scenarios by adjusting the shape parameter m. For example, $m=1$ represents Rayleigh fading typical in urban environments, $m>1$ captures less severe fading as in Rician environments, and $m<1$ reflects more severe fading due to shadowing and heavy multipath [15,16,25,26]. Given its applicability to scenarios such as IoT, satellite, and cellular networks, Nakagami-m fading provides a realistic channel model for 6G systems [27,28,29].

Motivation: GQSM has demonstrated promising performance in ideal settings, but its robustness in practical conditions involving correlated antennas and partial CSIR remains largely unexplored. As 6G networks are deployed in dynamic and interference-prone environments, there is a critical need to develop reliable and low-complexity detection schemes that maintain performance despite these impairments.

Research Gap and Objectives: There is a lack of comprehensive analysis and robust detection techniques for GQSM under the combined influence of transmit antenna correlation, partial CSIR, and Nakagami-m fading channel. This paper aims to close this gap by (i) modeling and analyzing GQSM performance under realistic channel impairments, (ii) proposing three QR-based detection techniques that mitigate BER degradation and error floor effects, and (iii) evaluating their performance and complexity against the conventional ML detection method.

Core Contributions: This work proposes and evaluates three QR-based detection methods for GQSM over Nakagami-m fading channels with transmit antenna correlation and partial CSIR. Unlike ML detection, the proposed schemes are shown to be more resilient to estimation errors and significantly decrease the error floor. The simulation results demonstrate that Technique III offers the best BER performance while maintaining a lower computational complexity compared to ML detection, making it more suitable for practical 6G scenarios.

Paper Organization: This is how the next part of the paper is structured. The system model, spatial correlation, and estimate error models under Nakagami-m fading are presented in Section 2. Section 3 introduces the proposed detection techniques using QR decomposition. Section 4 introduces the mathematical analysis of the theoretical analysis of the average bit error rate for the proposed detection techniques. Performance comparisons and simulation results are shown and discussed in Section 5. The paper is finally concluded with significant conclusions and suggestions for further research in Section 6. The future research direction has been presented in Section 7. The potential implementation constraints are displayed in Section 8. The research work limitation is represented in Section 9.

2. System Model

We consider an MISO system consisting of $N_t$ sending antennas and a single receiving antenna in a slightly static spatially correlated Nakagami m-fading channel scenario. We do this by implementing a new transmitter scheme that combines quasi-orthogonal space-time block coding (QOSTBC) with recently proposed generalized quadrature spatial modulation (GQSM) [14]. In this system, out of all the $N_t$ antennas, a variable number, p, of the transmitting antennas are turned on. Through the use of antenna activation permutation (AAP), the actual component of the modulated signal vector ($I^I$) can be sent through activated antennas. Similarly, an alternate AAP is used for activated antennas to transmit the imaginary component ($I^Q$) of the modulated signal vector. The total AAPs for the real and imaginary components are denoted by $C(N_t,p)$. Specifically, $L=2^{⌊lo{g}_{2}C(N_t,p)⌋}$ is the AAP of the modulated signal vector. In this case, $C(N_t,p)$ represents the process of identifying p antennas from collection of $N_t$ antennas. We then describe the transmission and detection of GQSM.

The entire set of given input bits ($\mathit{m}$) was divided after the data was separated into ${\mathit{m}}_1$, ${\mathit{m}}_2$, and ${\mathit{m}}_3$. The binary components that modulate M-ary are described by ${\mathit{m}}_1$, which results in a signal-modulated vector $\mathbf{s}$ with elements $s_1,s_2,\dots ,s_p$. With ${\mathit{m}}_1=p·\log 2(M)$ bits, the starting vector $\mathbf{s}$ is created. For $1\le \tau \le p$, each of the components $s_{\tau }$ in the vector is represented as $s_{\tau }=s_{\tau }^I+j{s}_{\tau }^Q$. $s_{\tau }$ is a member of the set $\chi$, which is made up of M -ary constellation points. Importantly, $s_{\tau }^Q$ with $s_{\tau }^I$ are taken from ${\chi }^Q$ and ${\chi }^I$, which represent both the imaginary and the real components of $\chi$, respectively. The bits ${\mathit{m}}_2$ and ${\mathit{m}}_3$ represent the index bits of the real and imaginary components, respectively. The bits of ${\mathit{m}}_2=⌊lo{g}_{2}C(N_t,p)⌋$ are used to calculate the AAPs of $I^I$ and ${\mathit{m}}_3=⌊lo{g}_{2}C(N_t,p)⌋$. ${\left\{s_{\tau }^I\right\}}_{\tau =1}^p$ and ${\left\{s_{\tau }^Q\right\}}_{\tau =1}^p$ are signals that use bits to represent the AAPs of $I^Q$ to transmit their real and imaginary parts.

Example: Four $N_t$ = four p = two and 4-QAM $(M=4)$: As per the equation ${\mathit{m}}_1=plo{g}_{2}M$, there are four 4-QAM modulation bits. There are two AAP real bits in ${\mathit{m}}_2$ and two AAP imaginary bits in ${\mathit{m}}_3$ = $⌊lo{g}_{2}C(N_t,p)⌋$. Consequently, there are eight bits in all. The data input bits for $\mathit{m}$ are now [0 1 1 0 1 1 0 0]. The system model states that the first 4 $({\mathit{m}}_1=4)$ bits [0 1 1 0] are turned into 2 $(p=2)$ modulated characters $\mathbf{s}=[-1+j,1-j]$ with ${\mathbf{s}}^I=[-1,1]$ and ${\mathbf{s}}^Q=[1,-1]$. The middle or second two $({\mathit{m}}_2=2)$ bits [1 1] are used to opt the AAP $I_4^I=\{2,3\}$ (refer to

Table 1. GQSM mapping table with $N_t=4$, $m=4-QAM$ and $p=2$.

Index Bit	AAP	Index Bit	AAP	Index Bit	AAP	Index Bit	AAP
00	$(1,2)$	10	$(1,4)$	01	$(1,3)$	11	$(2,3)$

) to transmit $s_1^I=-1$ and $s_2^I=1$ via the second $(i_4^I(1)=2)$ and third $(i_4^I(2)=3)$ antennas for the real part of s, leading to ${\mathit{x}}^I={[0,1,1,0]}^T$. The third or last two $({\mathit{m}}_3=2)$ bits $[0,0]$ are used to select the AAP $I_1^Q=\{1,2\}$ (refer to Table 1) to transmit $s_1^Q=1$ and $s_2^Q=-1$ through the first $(i_1^Q(1)=1)$ and second $(i_1^Q(2)=2)$ antennas, resulting in ${\mathit{x}}^Q={[1,1,0,0]}^T$. Thus, $\mathit{x}={\mathit{x}}^I+{\mathit{x}}^Q={[j,-1-j,1,0]}^T$ yields the transmitted signal vector.

Let ${\mathbf{s}}^I$ denote the real components of the modulated signal vector $\mathbf{s}$, which is denoted as $[s_1^I,s_2^I,\dots ,s_p^I]$. Likewise, ${\mathbf{s}}^Q$ denotes the imaginary elements in the modulated signal vector $\mathbf{s}$, which in turn can be written as $[s_1^Q,s_2^Q,\dots ,s_p^Q]$.

For $I^Q$ and $I^I$, respectively, assume that the βth legal AAPs are $I_{\beta }^I$ and $I_{\alpha }^I$, and may be stated as

$$I_{\alpha }^I=\{i_{\alpha }^I(1),i_{\alpha }^I(2),\dots ,i_{\alpha }^I(p)\},$$

(1)

and

$$I_{\beta }^Q=\{i_{\beta }^Q(1),i_{\beta }^Q(2),\dots ,i_{\beta }^Q(p)\},$$

(2)

Therefore, for the numbers $\eta$ that lie between one and p, $i_{\alpha }^I(\eta )$ and $i_{\beta }^Q(\eta )$ are part of the collection of integers ranging from 1 to $N_t$. The $N_t\times 1$ signal vectors for the real and imaginary sections are as follows:

$${\mathit{x}}^I=\{s_1^I,0,s_2^I,\dots 0,s_p^I\},$$

(3)

and

$${\mathit{x}}^Q=\{0,s_1^Q,s_2^Q,0,\dots 0,s_p^Q\},$$

(4)

Thus, Keep in mind that AAPs $I^I$ and $I^Q$ decide the non-zero elements in ${\mathit{x}}^I$ and ${\mathit{x}}^Q$. The transmission is the last step in creating the signal vector, which has a dimension of $N_t\times 1$.

$$\mathit{x}={\mathit{x}}^I+j{\mathit{x}}^Q$$

(5)

According to [14], the GQSM symbol, which comprises real, imaginary, along with values of zero for antenna indices, is represented by the symbol x in (5).

For $N_t\times 1$, the transmit GQSM symbol is $\mathit{x}$; it also has a covariance matrix and a zero mean. ${\mathit{R}}_{\mathit{x}}=\left[\mathbf{E}\mathit{x}{\mathit{x}}^H\right]={\sigma }_{\mathit{x}}^2\mathbf{I}$

Using a covariance matrix and a zero average ${\mathbf{R}}_n=\mathbb{E}\left[\mathbf{n}{\mathbf{n}}^H\right]={\sigma }_n^2\mathbf{I}$, we have the complex Gaussian noise vector $\mathbf{n}$.

The structure of the matrix represents the MIMO channel

$$\mathbf{H}={[h_{j,k}\triangleq {\alpha }_{j,k}{e}^{i{\varphi }_{j,k}}]}_{j,k=1}^{N_r,N_t}$$

(6)

where the potential mutually correlated channel coefficients are denoted by $h_{j,k}$ and by $i^2=-1$. The path gains ${\alpha }_{j,k}$ of $\kappa \triangleq n_T{n}_R$ are Nakagami distributed, as per the probability density function (PDF) provided by [30].

$$p_{{\alpha }_{j,k}}(\alpha )=2{\sigma }_{j,k}^{-2m}{\displaystyle \frac{{\alpha }^{2m-1}}{\mathrm{\Gamma }(m)}}\exp \left(-{\displaystyle \frac{{\alpha }^2}{{\sigma }_{j,k}^2}}\right),\alpha \ge 0.$$

(7)

The function $\mathrm{\Gamma }(.)$ represents the Gamma function [31]. The parameter m, with m being greater than or equal to 1/2, captures the severity of fading in the Nakagami model. The term ${\sigma }_{j,k}^2$ is defined as the expectation of ${\alpha }_{j,k}^2$ divided by m, in the same $E[.]$ represent the expectation operator. The phase ${\varphi }_{j,k}$ associated with the channel coefficient $h_{j,k}$ follows a uniform distribution over the interval $[0,2\pi ]$. The reason we chose the Nakagami fading model is that it reflects a wide range of multipath channels using the parameter m [30]. The PDF Equation (7) represents the one-sided Gaussian distribution at m = 0.5, but at m = 1, it represents the Rayleigh distribution. The Nakagami distribution provides a good approximation of the Rice distribution for $m>1$ because the Nakagami fading parameter m and the Rician factor K are mapped one to one.

$\mathit{h}={[h_1,h_2,h_3,h_4]}^T$ is the matrix where the partially static spatially correlated (associated) Nakagami m-fading channel coefficient of the j-th sending out (transmitting) antenna to the receiving antenna is denoted by $h_k$, for $j=1,\dots ,4$. The associated channel matrix $\mathit{H}$ is indicated by the matrix.

The received signal vector is given by ref. [32]:

$$re\left[\begin{array}{c}{\mathit{y}}_1\\ {\mathit{y}}_2^{*}\\ {\mathit{y}}_3\\ {\mathit{y}}_4^{*}\end{array}\right]=\left[\begin{array}{cccc}{\mathit{h}}_1& {\mathit{h}}_2& {\mathit{h}}_3& {\mathit{h}}_4\\ {\mathit{h}}_2^{*}& -{\mathit{h}}_1^{*}& {\mathit{h}}_4^{*}& -{\mathit{h}}_3^{*}\\ {\mathit{h}}_3& {\mathit{h}}_4& {\mathit{h}}_1& {\mathit{h}}_2\\ {\mathit{h}}_4^{*}& -{\mathit{h}}_3^{*}& {\mathit{h}}_2^{*}& -{\mathit{h}}_1^{*}\end{array}\right]\left[\begin{array}{c}{\mathit{x}}_1\\ {\mathit{x}}_2\\ {\mathit{x}}_3\\ {\mathit{x}}_4\end{array}\right]+\left[\begin{array}{c}{\mathit{n}}_1\\ {\mathit{n}}_2^{*}\\ {\mathit{n}}_3\\ {\mathit{n}}_4^{*}\end{array}\right]$$

(8)

That is equivalent to

$$\mathit{y}=\mathit{H}\mathit{x}+\mathit{n}=\mathit{H}({\mathit{x}}^I+j{\mathit{x}}^Q)+\mathit{n}.$$

(9)

In this case, H must be built using the procedures in Algorithm 1.

Algorithm 1 Transmit Receive antennas correlation Algorithm

Each H components should be layered in only one column called $vec(\mathit{H})$. $vec(\mathit{H})=\left[\begin{array}{c}{h}_{1,1}\\ h_{2,1}\\ h_{3,1}\\ h_{4,1}\end{array}\right]$ The following is one way to describe the matrices for the transmitter ${\mathbf{\ae }}_{\mathbf{t}}$ and the receiver ${\mathbf{\ae }}_{\mathbf{r}}$: ${\mathbf{\ae }}_{\mathbf{t}}=\left[\begin{array}{cccc}E\left[h_{1,1}{h_{1,1}}^{*}\right]& E\left[h_{1,1}{h_{2,1}}^{*}\right]& E\left[h_{1,1}{h_{3,1}}^{*}\right]& E\left[h_{1,1}{h_{4,1}}^{*}\right]\\ E\left[h_{2,1}{h_{1,1}}^{*}\right]& E\left[h_{2,1}{h_{2,1}}^{*}\right]& E\left[h_{2,1}{h_{3,1}}^{*}\right]& E\left[h_{2,1}{h_{4,1}}^{*}\right]\\ E\left[h_{3,1}{h_{1,1}}^{*}\right]& E\left[h_{3,1}{h_{2,1}}^{*}\right]& E\left[h_{3,1}{h_{3,1}}^{*}\right]& E\left[h_{3,1}{h_{4,1}}^{*}\right]\\ E\left[h_{4,1}{h_{1,1}}^{*}\right]& E\left[h_{4,1}{h_{2,1}}^{*}\right]& E\left[h_{4,1}{h_{3,1}}^{*}\right]& E\left[h_{4,1}{h_{4,1}}^{*}\right]\end{array}\right]$ ${\mathbf{\ae }}_{\mathbf{r}}=\left[\begin{array}{c}E\left[h_{1,1}{h_{1,1}}^{*}\right]\end{array}\right.$ The following is a display of the channel correlation R matrix R= ${\mathbf{\ae }}_{\mathbf{t}}\otimes {\mathbf{\ae }}_{\mathbf{r}}$ In this case, the Kronecker product is represented by “⊗”. $\mathbf{R}={\mathbf{VDV}}^{*}$ Each component in r is divided and dispersed separately as a Gamma distribution with squared mean $\mathrm{\Omega }=\mathbf{E}[|h_{j,k}{|}^2]$ in the vector r $16\times 1$ formation, where the expectation operator is indicated by E[·] The following can now be said regarding $vec(\mathit{H})$ $\mathrm{vec}(\mathbf{H})={\mathbf{VD}}^{\mathbf{1}/\mathbf{2}}\mathbf{r}$

Additionally, we assume that the receiver has access to a portion of the channel state information only. As an expression, the ideal maximum likelihood (ML) detection is

$$\widehat{\mathit{x}}=\arg \underset{\mathit{x}}{\min }{\parallel \mathit{y}-\widehat{\mathit{H}}\mathit{x}\parallel }^2$$

(10)

where imperfect or partial channel state information available at the receiver is expressed as $\widehat{\mathit{H}}$ in Equation (10).

$$\widehat{\mathit{H}}=\mathit{H}+\mathit{e}$$

(11)

The channel estimation vector is $\mathit{H}$, whereas the channel estimation error vector is $\mathit{e}$. We suppose that the Nakagami distributions apply to all parts of the channel estimation vector. On the other hand, the complex Gaussian random variables that make up the channel estimation error vector have zero means and variances of ${\sigma }_{\mathit{e}}^2$ [33].

$$\widehat{\mathit{x}}=\arg \underset{\mathit{x}}{\min }{\parallel \mathit{y}-\widehat{\mathit{H}}({\mathit{x}}^I+j{\mathit{x}}^Q)\parallel }^2$$

(12)

In Section 3, we provide the three different QOSTBC detection methods used in [14] for clarity.

3. Detection Techniques

In this part, we present three techniques for QOSTBC detection [14].

3.1. Detection Technique I

The received vector for detection is articulated in Equation (13), incorporating the multiplication of the channel matrix between the transmitting and receiving antennas, the signal vector, and the received noise.

$$\mathit{y}=\widehat{\mathit{H}}\mathit{x}+\mathit{n},$$

(13)

where, $\mathit{x}={\mathit{x}}^I+j{\mathit{x}}^Q$.

One way to factorize the relevant channel matrix is as follows.

$$\widehat{\mathit{H}}={\mathit{Q}}_1{\mathit{R}}_1={\mathit{Q}}_1\left[\begin{array}{cccc}\mathit{a}& 0& \mathit{b}& 0\\ 0& \mathit{a}& 0& \mathit{b}\\ 0& 0& \mathit{c}& 0\\ 0& 0& 0& \mathit{c}\end{array}\right]$$

(14)

The upper diagonal matrix is represented by ${\mathit{R}}_1$, whereas the unitary matrix is shown by ${\mathit{Q}}_1$. On each side of (13), ${\mathit{Q}}_1^H$ gets multiplied to the left. The matrix that is produced is,

$$\left[\begin{array}{c}{\tilde{\mathit{y}}}_1\\ {\tilde{\mathit{y}}}_2\\ {\tilde{\mathit{y}}}_3\\ {\tilde{\mathit{y}}}_4\end{array}\right]=\left[\begin{array}{cccc}\mathit{a}& 0& \mathit{b}& 0\\ 0& \mathit{a}& 0& \mathit{b}\\ 0& 0& \mathit{c}& 0\\ 0& 0& 0& \mathit{c}\end{array}\right]\left[\begin{array}{c}{\mathit{x}}_1\\ {\mathit{x}}_2\\ {\mathit{x}}_3\\ {\mathit{x}}_4\end{array}\right]+\left[\begin{array}{c}{\tilde{\mathit{n}}}_1\\ {\tilde{\mathit{n}}}_2\\ {\tilde{\mathit{n}}}_3\\ {\tilde{\mathit{n}}}_4\end{array}\right]$$

(15)

Using (11), the $\widehat{\mathit{H}}\mathit{x}$ can be expressed as

$$\left[\begin{array}{cccc}{h}_4& h_3& h_2& h_1\\ -h_3^{*}& h_4^{*}& -h_1^{*}& h_2^{*}\\ h_2& h_1& h_4& h_3\\ -h_1^{*}& h_2^{*}& -h_3^{*}& h_4^{*}\end{array}\right]\left[\begin{array}{c}{x}_4\\ x_3\\ x_2\\ x_1\end{array}\right]$$

(16)

Consequently, we obtain

$$\mathit{y}=\widehat{\mathit{H}}\mathit{x}+\mathit{n}={\widehat{\mathit{H}}}^{\prime }{\mathit{x}}^{\prime }+\mathit{n}$$

(17)

To find the ${\widehat{\mathit{H}}}^{\prime }$ QR decomposition,

$${\widehat{\mathit{H}}}^{\prime }={\mathit{Q}}_2{\mathit{R}}_2={\mathit{Q}}_2\left[\begin{array}{cccc}\mathit{m}& 0& \mathit{n}& 0\\ 0& \mathit{m}& 0& \mathit{n}\\ 0& 0& \mathit{k}& 0\\ 0& 0& 0& \mathit{k}\end{array}\right]$$

(18)

$Q_2^H$ multiplied by the sides to the left and right of (17) produces

$$\left[\begin{array}{c}{\tilde{\mathit{y}}}_1^{\prime }\\ {\tilde{\mathit{y}}}_2^{\prime }\\ {\tilde{\mathit{y}}}_3^{\prime }\\ {\tilde{\mathit{y}}}_4^{\prime }\end{array}\right]=\left[\begin{array}{cccc}\mathit{m}& 0& \mathit{n}& 0\\ 0& \mathit{m}& 0& \mathit{n}\\ 0& 0& \mathit{k}& 0\\ 0& 0& 0& \mathit{k}\end{array}\right]\left[\begin{array}{c}{\mathit{x}}_4\\ {\mathit{x}}_3\\ {\mathit{x}}_2\\ {\mathit{x}}_1\end{array}\right]+\left[\begin{array}{c}{\tilde{\mathit{n}}}_1^{\prime }\\ {\tilde{\mathit{n}}}_2^{\prime }\\ {\tilde{\mathit{n}}}_3^{\prime }\\ {\tilde{\mathit{n}}}_4^{\prime }\end{array}\right],$$

(19)

which can be stated as

$${\tilde{\mathit{y}}}^{\prime }={\mathit{R}}_2{\mathit{x}}^{\prime }+{\tilde{\mathit{n}}}^{\prime }$$

(20)

The 3rd along with 4th row of the (19) is combined with the 3rd and 4th of the (15) to create a new matrix, which brings about

$$\left[\begin{array}{c}{\tilde{\mathit{y}}}_4^{\prime }\\ {\tilde{\mathit{y}}}_3^{\prime }\\ {\tilde{\mathit{y}}}_3\\ {\tilde{\mathit{y}}}_4\end{array}\right]=\left[\begin{array}{cccc}\mathit{k}& 0& 0& 0\\ 0& \mathit{k}& 0& 0\\ 0& 0& \mathit{c}& 0\\ 0& 0& 0& \mathit{c}\end{array}\right]\left[\begin{array}{c}{\mathit{x}}_1\\ {\mathit{x}}_2\\ {\mathit{x}}_3\\ {\mathit{x}}_4\end{array}\right]+\left[\begin{array}{c}{\tilde{\mathit{n}}}_1^{\prime }\\ {\tilde{\mathit{n}}}_2^{\prime }\\ {\tilde{\mathit{n}}}_3\\ {\tilde{\mathit{n}}}_4\end{array}\right]$$

(21)

The four symbols are now separated as shown in (19) and (15). Consequently, parallel detection of ${\mathit{x}}_1,{\mathit{x}}_2$$,{\mathit{x}}_3$, and ${\mathit{x}}_4$ is possible.

$$\widehat{{\mathit{x}}_1}=\arg \underset{{\mathit{x}}_1\in \xi }{\min }\parallel {\widetilde{{\mathit{y}}_4}}^{\prime }-\mathit{k}{\mathit{x}}_1{\parallel }^2\widehat{{\mathit{x}}_2}=\arg \underset{{\mathit{x}}_2\in \xi }{\min }{\parallel {\widetilde{{\mathit{y}}_3}}^{\prime }-\mathit{k}{\mathit{x}}_2\parallel }^2$$

(22)

$$\widehat{{\mathit{x}}_3}=\arg \underset{{\mathit{x}}_3\in \xi }{\min }\parallel \widetilde{{\mathit{y}}_3}-\mathit{c}{\mathit{x}}_3{\parallel }^2\widehat{{\mathit{x}}_4}=\arg \underset{{\mathit{x}}_4\in \xi }{\min }{\parallel \widetilde{{\mathit{y}}_4}-\mathit{c}{\mathit{x}}_4\parallel }^2$$

(23)

$\widehat{{\mathit{x}}_1}$, $\widehat{{\mathit{x}}_2}$, $\widehat{{\mathit{x}}_3}$ and $\widehat{{\mathit{x}}_4}$ are employed to identify the GQSM symbol.

3.2. Detection Technique II

Detection Technique I simplifies the process but comes with a slight increase in bit error rates. To tackle this challenge, we use Detection Technique II, also known as the Low Complexity Maximum Likelihood (LC-ML) decoder. This method strikes a balance between reducing complexity and managing bit error rates effectively.

Therefore, this technique serves as an interim solution that successfully reduces both the bit error rate and the complexity of detection. Continuing with this, the Equation (15) is shown as

$$\tilde{\mathit{y}}={\mathit{R}}_1\mathit{x}+\tilde{\mathit{n}}$$

(24)

The LC-ML decoder [14] then selects a maximum likelihood (ML) result. Four components make up the vector $\widehat{\mathit{x}}$: ${\widehat{\mathit{x}}}_1$, ${\widehat{\mathit{x}}}_2$, ${\widehat{\mathit{x}}}_3$, along with ${\widehat{\mathit{x}}}_4$. The value of $\xi$ from the sample space satisfies the following criterion:

$$\widehat{\mathit{x}}=\arg \min {\parallel \tilde{\mathit{y}}-{\mathit{R}}_1\mathit{x}\parallel }^2=\arg \underset{\mathit{x}\in \xi }{\min }\sum _{\mathit{k}=1}^4{\mathit{d}}_k$$

(25)

In the same ${\mathit{d}}_k$ for $\mathit{k}=1,\dots ,4$ are given by

$${\mathit{d}}_1=|{\tilde{\mathit{y}}}_1-\mathit{b}{\mathit{x}}_3-\mathit{a}{\mathit{x}}_1{|}^2{\mathit{d}}_2={|{\tilde{\mathit{y}}}_2-\mathit{b}{\mathit{x}}_4-\mathit{a}{\mathit{x}}_2|}^2$$

(26)

$${\mathit{d}}_3=|{\tilde{\mathit{y}}}_3-\mathit{c}{\mathit{x}}_3{|}^2{\mathit{d}}_4={|{\tilde{\mathit{y}}}_4-\mathit{c}{\mathit{x}}_4|}^2$$

(27)

Hence, (25) can be expressed as

$$({\widehat{\mathit{x}}}_1.{\widehat{\mathit{x}}}_3)=\underset{{\mathit{x}}_1,{\mathit{x}}_3\in \xi }{\min }({\mathit{d}}_1+{\mathit{d}}_3)({\mathit{x}}_2.{\mathit{x}}_4)=\underset{{\mathit{x}}_2,{\mathit{x}}_4\in \xi }{\min }({\mathit{d}}_2+{\mathit{d}}_4)$$

(28)

The $\widehat{{\mathit{x}}_1}$, $\widehat{{\mathit{x}}_2}$, $\widehat{{\mathit{x}}_3}$ and $\widehat{{\mathit{x}}_4}$ are used to recognize the GQSM symbol.

3.3. Detection Technique III

Detection Technique I prioritizes complexity over bit error rate, while Detection Technique II aims to balance both bit error rate and detection complexity. However, the bit error rate of Detection Technique II does not approach the ideal performance of maximum likelihood detection. Therefore, it is essential to identify a detection technique that enhances the bit error rate compared to both Techniques I and II, with only a slight increase in complexity. Accordingly, the output symbol can be decoded from Equations (26) and (27) as follows:

$$\widehat{{\mathit{x}}_3}=\arg \underset{{\mathit{x}}_3\in \xi }{\min }\parallel \widetilde{{\mathit{y}}_3}-\mathit{c}{\mathit{x}}_3{\parallel }^2\widehat{{\mathit{x}}_4}=\arg \underset{{\mathit{x}}_4\in \xi }{\min }{\parallel \widetilde{{\mathit{y}}_4}-\mathit{c}{\mathit{x}}_4\parallel }^2$$

(29)

$$\widehat{{\mathit{x}}_1}=\arg \underset{{\mathit{x}}_1\in \xi }{\min }\parallel {\tilde{\mathit{y}}}_1-\mathit{b}{\widehat{\mathit{x}}}_3-\mathit{a}{\mathit{x}}_1{\parallel }^2\widehat{{\mathit{x}}_2}=\arg \underset{{\mathit{x}}_2\in \xi }{\min }{\parallel {\tilde{\mathit{y}}}_2-\mathit{b}{\widehat{\mathit{x}}}_4-\mathit{a}{\mathit{x}}_2\parallel }^2$$

(30)

In this method, rather than using Equation (26) to jointly identify two pairs of transmitted symbols in the LC-ML decoder, we first use Equation (27) to calculate the values of ${\widehat{\mathit{x}}}_3$ and ${\widehat{\mathit{x}}}_4$. Then, by replacing ${\mathit{x}}_3$ and ${\mathit{x}}_4$ into (29), we can determine ${\mathit{x}}_1$ and ${\mathit{x}}_2$. Equation (19) yields the following when the first and second rows are eliminated:

$$\left[\begin{array}{c}{\tilde{\mathit{y}}}_1^{\prime }\\ {\tilde{\mathit{y}}}_2^{\prime }\end{array}\right]=\left[\begin{array}{cccc}\mathit{m}& 0& \mathit{n}& 0\\ 0& \mathit{m}& 0& \mathit{n}\end{array}\right.\left[\begin{array}{c}{\mathit{x}}_4\\ {\mathit{x}}_3\\ {\mathit{x}}_2\\ {\mathit{x}}_1\end{array}\right]+\left[\begin{array}{c}{\tilde{\mathit{n}}}_1^{\prime }\\ {\tilde{\mathit{n}}}_2^{\prime }\end{array}\right]$$

(31)

Then, ${\mathit{x}}_3$ and ${\mathit{x}}_4$ can be detected as

$$\widehat{{\mathit{x}}_3}=\arg \underset{{\mathit{x}}_3\in \xi }{\min }\parallel {\tilde{\mathit{y}}}_2^{\prime }-\mathit{n}{\widehat{\mathit{x}}}_1-\mathit{m}{\mathit{x}}_3{\parallel }^2\widehat{{\mathit{x}}_4}=\arg \underset{{\mathit{x}}_4\in \xi }{\min }{\parallel {\tilde{\mathit{y}}}_1^{\prime }-\mathit{n}{\widehat{\mathit{x}}}_2-\mathit{m}{\mathit{x}}_4\parallel }^2$$

(32)

The D-QR-IC can be summed up using the steps below:

• Find ${\mathit{x}}_3$ and ${\mathit{x}}_4$ from (27) first. After substituting ${\mathit{x}}_3$ and ${\mathit{x}}_4$ into (26), ${\mathit{x}}_1$ and ${\mathit{x}}_2$ should be discovered.
• To get the final values of ${\mathit{x}}_3$ and ${\mathit{x}}_4$, substitute ${\mathit{x}}_1$ and ${\mathit{x}}_2$ in Equation (32).

Utilizing the $\widehat{{\mathit{x}}_1}$, $\widehat{{\mathit{x}}_2}$, $\widehat{{\mathit{x}}_3}$, and $\widehat{{\mathit{x}}_4}$, GQSM symbol detection is accomplished.

4. Theoretical Analysis of Different Detection Schemes

Consider a MIMO system employing Generalized Quadrature Spatial Modulation (GQSM) combined with Quasi-Orthogonal Space-Time Block Codes (QOSTBC), operating under Nakagami-m fading conditions. The received signal can be modeled as:

$$\mathbf{y}=\mathbf{H}\mathit{x}+\mathbf{n}$$

(33)

where:

• $\mathbf{y}\in {\mathbb{C}}^{N_r\times 1}$: received signal vector,
• $\mathbf{H}\in {\mathbb{C}}^{N_r\times N_t}$: MIMO channel matrix,
• $\mathbf{x}\in {\mathbb{C}}^{N_t\times 1}$: GQSM-QOSTBC encoded transmit signal,
• $\mathbf{n}\sim \mathcal{CN}(0,{\sigma }^2\mathbf{I})$: additive white Gaussian noise.

The entries of $\mathbf{H}$ follow Nakagami-m fading with PDF:

$$f_{|h|}(x)={\displaystyle \frac{2m^m}{\mathrm{\Gamma }(m){\mathrm{\Omega }}^m}}{x}^{2m-1}\exp \left(-{\displaystyle \frac{m{x}^2}{\mathrm{\Omega }}}\right),x\ge 0$$

(34)

4.1. BER Analysis of D-QR Detection

Applying QR decomposition:

$$\mathbf{H}=\mathbf{Q}\mathbf{R}$$

(35)

Multiplying both sides by ${\mathbf{Q}}^H$:

$$\tilde{\mathbf{y}}={\mathbf{Q}}^H\mathbf{y}=\mathbf{R}\mathit{x}+\tilde{\mathbf{n}},\tilde{\mathbf{n}}={\mathbf{Q}}^H\mathbf{n}$$

(36)

The pairwise error probability (PEP) for transmitted ${\mathbf{x}}_i$ and erroneous ${\mathbf{x}}_j$ is:

$$P({\mathbf{x}}_i\to {\mathbf{x}}_j)={\mathbb{E}}_{\mathbf{H}}\left[Q\left(\sqrt{{\displaystyle \frac{\parallel \mathbf{R}({\mathbf{x}}_i-{\mathbf{x}}_j){\parallel }^2}{2{\sigma }^2}}}\right)\right]$$

(37)

Under Nakagami-m fading, using Craig’s formula:

$$Q(\sqrt{a})={\displaystyle \frac{1}{\pi }}{\int }_0^{\pi /2}\exp \left(-{\displaystyle \frac{a}{2{\sin }^2\theta }}\right)d\theta$$

(38)

$$P({\mathbf{x}}_i\to {\mathbf{x}}_j)={\displaystyle \frac{1}{\pi }}{\int }_0^{\pi /2}{\left({\displaystyle \frac{m}{m+\frac{\parallel \mathbf{R}({\mathbf{x}}_i-{\mathbf{x}}_j){\parallel }^2}{4{\sigma }^2{\sin }^2\theta }}}\right)}^{m}d\theta$$

(39)

The average bit error rate is bounded as:

$$P_b^{\text{D-QR}}\le {\displaystyle \frac{1}{{\log }_{2}M}}\sum _{{\mathbf{x}}_i}\sum _{{\mathbf{x}}_j\ne {\mathbf{x}}_i}{d}_H({\mathbf{x}}_i,{\mathbf{x}}_j)·P({\mathbf{x}}_i\to {\mathbf{x}}_j)$$

(40)

4.2. BER Analysis of LCML Detection

The average BER over Nakagami-m fading is given by:

$${\overline{P}}_b={\int }_0^{\infty }Q\left(\sqrt{2\kappa \gamma }\right)f_{\gamma }(\gamma )d\gamma$$

(41)

Using the identity:

$$Q(x)={\displaystyle \frac{1}{\pi }}{\int }_0^{\pi /2}\exp \left(-{\displaystyle \frac{x^2}{2{\sin }^2\theta }}\right)d\theta$$

(42)

We substitute into the BER integral:

$${\overline{P}}_b={\displaystyle \frac{1}{\pi }}{\int }_0^{\pi /2}\left[{\int }_0^{\infty }\exp \left(-{\displaystyle \frac{\kappa \gamma }{{\sin }^2\theta }}\right)f_{\gamma }(\gamma )d\gamma \right]d\theta$$

(43)

Substituting the Nakagami-m PDF:

$${\overline{P}}_b={\displaystyle \frac{m^m}{\pi \mathrm{\Gamma }(m){\overline{\gamma }}^m}}{\int }_0^{\pi /2}\left[{\int }_0^{\infty }{\gamma }^{m-1}\exp \left(-\gamma \left({\displaystyle \frac{\kappa }{{\sin }^2\theta }}+{\displaystyle \frac{m}{\overline{\gamma }}}\right)\right)d\gamma \right]d\theta$$

(44)

The inner integral is a gamma integral:

$${\int }_0^{\infty }{x}^{\nu -1}{e}^{-ax}dx={\displaystyle \frac{\mathrm{\Gamma }(\nu )}{a^{\nu }}}$$

(45)

Letting $A(\theta )={\displaystyle \frac{\kappa }{{\sin }^2\theta }}+{\displaystyle \frac{m}{\overline{\gamma }}}$, we get:

$${\overline{P}}_b={\displaystyle \frac{m^m}{\pi {\overline{\gamma }}^m}}{\int }_0^{\pi /2}{\left({\displaystyle \frac{1}{A(\theta )}}\right)}^{m}d\theta$$

(46)

Thus, the average BER under Nakagami-m fading is:

$${\overline{P}}_b\approx {\displaystyle \frac{m^m}{\pi {\overline{\gamma }}^m}}{\int }_0^{\pi /2}{\left({\displaystyle \frac{1}{\frac{\kappa }{{\sin }^2\theta }+\frac{m}{\overline{\gamma }}}}\right)}^{m}d\theta$$

(47)

4.3. BER Analysis of D-QR-IC Detection

D-QR-IC detection uses interference cancellation (SIC) after QR decomposition. The SNR at the k-th detection layer is:

$${\gamma }_k={\displaystyle \frac{|{\mathbf{r}}_{k,k}{|}^2}{{\sigma }^2+{\sum }_{l>k}{|{\mathbf{r}}_{k,l}|}^2{P}_l}}$$

(48)

The BER at each layer is:

$$P_{b,k}^{\text{D-QR-IC}}\approx Q\left(\sqrt{{\displaystyle \frac{{\gamma }_k}{2}}}\right)$$

(49)

Thus, the average BER becomes:

$$P_b^{\text{D-QR-IC}}\approx {\displaystyle \frac{1}{K}}\sum _{k=1}^{K}Q\left(\sqrt{{\displaystyle \frac{|{\mathbf{r}}_{k,k}{|}^2}{2\left({\sigma }^2+{\sum }_{l>k}{|{\mathbf{r}}_{k,l}|}^2{P}_l\right)}}}\right)$$

(50)

Under Nakagami-m fading, this can be expressed as:

$$P_b^{\text{D-QR-TC}}\approx {\displaystyle \frac{1}{K\pi }}\sum _{k=1}^K{\int }_0^{\pi /2}{\left({\displaystyle \frac{m}{m+\frac{{\gamma }_k}{{\sin }^2\theta }}}\right)}^{m}d\theta$$

(51)

5. Result and Discussion

Figure 1 illustrates the relationship between BER and SNR for different values of $\eta$, including $0.98$, $0.99$, and 1, for a $4\times 1$ system with a transmit antenna correlation $\rho$ = 0.9. The error floor for ML is established between 10 and 12 SNR for $\eta$ = 0.99 and 0.98, as shown in

Table 2. Lowest Estimated SNR for $(4,6;2)$ MIMO with various Transmit Antenna Selection (TAS) schemes under imperfect channel state information available at receiver ($\rho =0.96$)—for video (${10}^{-3}$ SER) application.

TAS Scheme	SNR Required
No TAS	Not achievable
Conventional TAS	1.85 dB
Best II Random II TAS	2.9 dB
Near conventional TAS	1.85 dB

. In contrast, techniques I, II, and III do not have an error floor because they employ QR decomposition. No further modifications are made to the BER for SNR of 9 dB. Technique III outperforms Technique II, and the simulation results shows improved performance across all three partial CSIRs ($\eta$ = 0.98, 0.99, and 1) due to the utilization of QR decomposition. The QR decomposition procedure reduces the complexity of data detection systems by simplifying the received signal. As a result, symbol or bit detection is made easier, particularly in situations involving partial CSI where the channel information is limited. QR decomposition also helps mitigate interference caused by channel effects through the decomposition of the channel matrix, enhancing the accuracy of symbol detection even when the receiver has access to only a portion of the CSI.

This section compares the MIMO BER simulation outcomes of three detection methodologies (technique I, technique II, and technique III) by utilizing different parameter values for channel state information at the receiver (1, 0.99, and 0.98) and a transmit antenna correlation parameter of 0.9. The simulation is conducted using $N_t=4,N_r=1,p=2$, and $m=8$ with 4-QAM modulation. The outcomes are illustrated in Figure 1.

From

Table 3. Bit Error Rate (BER) for a particular Average Signal to Noise Ratio (SNR) of GQSM−QOSTBC when three distinct low detection techniques are employed, each with partial channel state information accessible to the receiver ($\eta$ = 0.98, 0.99, and 1) and a transmit antenna correlation of $\rho$ = 0.9.

Partial CSIR (η)	ML	Technique I	Technique II	Technique III
0.98 (SNR = 9 dB)	0.0112	0.01399125	0.01107	0.0092475
0.98 (SNR = 18 dB)	0.0037	0.000399	0.0002837	0.000246
0.99 (SNR = 9 dB)	0.008016	0.01386125	0.01161	0.0096875
0.99 (SNR = 18 dB)	0.001515	0.0004	0.000275	0.00027
1 (SNR = 9 dB)	0.005304	0.01464375	0.01139	0.0096425
1 (SNR = 18 dB)	0.0001875	0.00032875	0.0003275	0.0002625

, it has been observed that ML ($\eta$ = 0.98, SNR = 18 dB) gives 0.0037 BER and Technique III ($\eta$ = 0.98, SNR = 18 dB) gives 0.000246 BER, whereas ML ($\eta$ = 1, SNR = 18 dB) gives 0.0001875 BER and Technique III ($\eta$ = 1, SNR = 18 dB) gives 0.0002525 BER. It has been proved that the performance of ML degraded with partial CSIR ($\eta$ = 0.98) from full CSIR ($\eta$ = 1), which has been improved with techniques I, II, and III using QR decomposition.

Figure 2 shows how the BER performance of the proposed GQSM-QOSTBC system changes with average SNR under different Nakagami-m fading conditions ($m=0.5$, $m=1$, and $m=1.5$), which represent one-sided Gaussian, Rayleigh, and Rician fading channels, respectively. The analysis of three detection schemes—DQR, LCML, and QRIC—uses a fixed transmit antenna correlation of $\rho =0.9$ and full channel state information at the receiver ($\eta =1$). As anticipated, BER increases as m increases, demonstrating the system’s increased resilience under less severe fading situations. DQR has the largest BER across all channel conditions, whereas MLD performs the best among the detection methods, closely followed by QRIC and LCML. The BER quickly drops, and QRIC’s performance gets close to MLD’s, especially in Rician fading ($m=1.5$), demonstrating its effectiveness with less complexity. Interestingly, up to 30 dB SNR, no BER floor is seen, suggesting consistent detection accuracy across all channels and schemes. These findings show that the suggested system successfully adjusts to various fading conditions and that performance-complexity trade-offs are significantly impacted by the detection method selection.

In the context of Nakagami-m fading channels, higher m values, which indicate less severe fading, are expected to further enhance BER performance due to improved signal reliability. Similarly, introducing the transmit antenna correlation ($\rho$ > 0) is likely to affect the antenna index bits and reduce spatial diversity, thereby increasing detection ambiguity in GQSM and deteriorating system performance. Although the current simulations do not explicitly alter these parameters, their influence mechanisms align with established MIMO communication theory and highlight insightful areas for further research.

In terms of search times,

Table 4. Complexity of detection for various detection methods.

Parameter	Technique I	Technique II	Technique III	ML
Search Times	4 × ${(\xi ) }^{†}$	2 × ${(\xi +{(\xi )}^2) }^{†}$	6 × ${(\xi ) }^{†}$	2m

^† $\xi$ show the sample space.

shows the intricacy of the system under examination using the three detection methods in addition to ML. detection scheme 3 is more complex than detection schemes 1 and 2, according to [14]. Compared to all three detection methods, machine learning is much more difficult and grows exponentially.

6. Conclusions

In summary, this study integrates Generalized Quadrature Spatial Modulation with Quasi-Orthogonal Space-Time Block Codes in Nakagami-m fading conditions and real-world system impairments, providing a reliable and effective solution for practical MIMO systems. Notably, Technique III, based on QR decomposition, demonstrates significant advantages in mitigating the effects of transmit antenna correlation and imperfect CSIR. For next-generation wireless communications operating in challenging propagation environments, the proposed design represents a viable and scalable option, achieving superior error performance with reduced computational complexity compared to traditional ML detection.

7. Potential Future Research Direction

In the future, this work shall extend to multiple receiving antennas, as it can increase receive diversity, which will eventually enhance the performance of data rate and coverage, and reduce noise, interference, and call dropout. Additionally, optimizing detection, such as utilizing maximum likelihood with necessary modifications, could enhance the receiver detection process. This will add some computational complexity compared to the proposed techniques. GQSM-QOSTBC methods can be adapted for terahertz or millimeter-wave (mmWave) bands by using hybrid beamforming, which is effective for sparse channel characteristics and antenna arrays that are suitable for spatial modulation. To better represent real-life wireless situations, it can study and simulate GQSM-QOSTBC systems using different fading models (like Nakagami-m, $\kappa$-$\mu$, or $\eta$-$\mu$), which affect how reliable and diverse the system is. To scale the system to 6G network scenarios with high user density, it can investigate multi-user MIMO or massive MIMO variants of GQSM-QOSTBC and create user scheduling and precoding algorithms. AI can assist in overcoming analytical issues in complicated or time-varying channels by using deep learning to dynamically adjust GQSM-QOSTBC parameters (such as antenna activation patterns and coding rate) under changing channel conditions.

8. Potential Implementation Constraints

Although QR decomposition-based detection improves the MIMO system’s accuracy, there are several challenges that make it difficult to use in practice, especially in large-scale MIMO. High processing and memory requirements are imposed by the computationally demanding matrix operations used in the decomposition process, such as Givens rotations, Householder reflections, and Gram-Schmidt orthogonalization. Due to these criteria, real-time QR decomposition is challenging to do in high-throughput environments where latency must be kept to a minimum, such as massive MIMO. Additionally, QR-based techniques are frequently inappropriate for systems with limited power or resources, like Internet of Things devices, where memory availability and computational overhead are constrained. Furthermore, QR decomposition presents more hardware implementation complexity than more straightforward detection techniques like Minimum Mean Square Error (MMSE) or Zero-Forcing (ZF). As a result, it is less suitable for deployment on FPGA or ASIC platforms where timing, power, and area constraints are crucial design factors.

9. Research Work Limitation

There is a lack of research and a lack of analytical understanding in mathematical form of the combined effect of correlation on GQSM and QOSTBC. One major obstacle is the absence of near-optimal low-complexity detectors for GQSM-QOSTBC systems. When antenna correlation and channel estimation errors are taken into account, it might be challenging to obtain precise BER or SER expressions for GQSM-QOSTBC systems over conventional fading channels.