AIDETECT2: A Novel AI-Driven Signal Detection Approach for beyond 5G and 6G Wireless Networks

Abstract

Artificial intelligence (AI) is revolutionizing multiple-input-multiple-output (MIMO) technology, making it a promising contender for the coming sixth-generation (6G) and beyond-fifth-generation (B5G) networks. However, the detection process in MIMO systems is highly complex and computationally demanding. To address this challenge, this paper presents an optimized AI-based signal detection method known as AIDETECT-2 which is based on feed forward neural network (FFNN) for MIMO systems. The proposed AIDETECT-2 network model demonstrates superior efficiency in signal detection in comparison with conventional and AI-based MIMO detection methods, particularly in terms of symbol error rate (SER) at various signal-to-noise ratios (SNR). This paper thoroughly explores various signal detection aspects using FFNN, including the design of system architecture, preparation of data, training processes of the network model, and performance evaluation. Simulation results show that the proposed model demonstrates a significant performance improvement ranging between 13.75% to 99.995% better SER compared to the best conventional method and also achieved between 56.52% to 97.69 better SER compared to benchmark AI-based MIMO detectors at 20 dB SNR for given MIMO scenarios respectively. It also presented the computational complexity analysis of different conventional and AI-based MIMO detectors. We believe that this optimized AI-based network model can serve as a comprehensive guide for deploying deep-learning (DL) neural networks for signal detection in the forthcoming 6G wireless networks.

1. Introduction

We are now in the beyond-5G (B5G) era, also called sixth-generation (6G) networks, where millions of internet-connected devices compete for connectivity every day, and this number keeps increasing. This surge puts immense pressure on network resources, especially regarding energy efficiency, power consumption, reliability, and latency. Meeting these ever-growing network demands has become a significant challenge [1]. To address these needs, multiple-input-multiple-output (MIMO) technology, in combination with artificial intelligence (AI), plays a vital role in the development of 6G and B5G networks. MIMO technologies have brought remarkable improvements in wireless communication, especially when compared to older generation technologies such as single-input multiple-output (SIMO), multiple-input single-output (MISO), and single-input single-output (SISO) in terms of throughput, channel selection, spatial diversity, and spectral efficiency [2]. When the number of transmitting and receiving antennas is large, typically from tens to hundreds, these systems are referred to as massive multiple-input-multiple-output (Ma-MIMO) systems. Integrating AI with traditional networking techniques significantly transforms how we manage and enhance networks. AI can swiftly process large amounts of data, anticipate network performance, and adjust settings accordingly. This holds great potential for improving network reliability and speed. AI techniques are increasingly valued for tackling intricate and difficult problems with large search spaces [3,4]. With AI, network operators can detect and resolve issues before they impact users. Additionally, AI can optimize network resource utilization based on real-time usage, enhancing efficiency and reducing congestion. AI systems continuously learn and adjust their parameters to meet evolving network demands.

1.1. Motivation

The conventional wireless network technology is segmented into various working blocks. However, the delay between these blocks has made it challenging to achieve highly reliable and low-latency networks. Hence, nowadays, researchers are trying to integrate AI with each block of the conventional network technology, so that the working blocks will get replaced by AI or machine learning (ML) blocks as shown in Figure 1. Our study focuses on the red hierarchical part. In the next phase, multiple working blocks will be replaced by a single AI block. In the vision the future 6G network is expected to have end-to-end learning, i.e., both transmitter and receiver side will get replaced by a single block of AI as shown in Figure 1. This paper explores the development of AI-based signal detection for MIMO systems. To address the growing needs of wireless networks and the computational demands of signal detection in MIMO systems, AI integration is a promising solution. In our exploration of different models, we implemented AIDETECT with PatternNet, as well as Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNNs). While these models were evaluated, we encountered difficulties in achieving the desired balance between output performance and computational complexity. LSTMs and CNNs, despite their advanced capabilities, proved to be computationally intensive. As a result, we developed a customized feed-forward neural network (FFNN) that strikes a more effective balance between computational efficiency and performance. Also, given the significant data load in our MIMO system, FFNN has been chosen for its ability to capture long-term dependencies and handle sequential data input [5].

1.2. Contributions

• In this paper, we present an optimized novel AI-enhanced signal detection model known as AIDETECT2. The proposed model employs an FFNN to enhance reliability and reduce the latency of wireless communication systems.
• The new AIDETECT2 model surpasses the performance of our previously proposed AIDETECT model, as well as the benchmarked AI models and traditional detectors.
• The performance evaluation of the proposed model is conducted in terms of both, symbol error rate (SER) and computational complexity.

By incorporating cutting-edge neural network techniques and refining the signal detection process, our model significantly boosts detection accuracy and computational efficiency. This advancement not only establishes a new benchmark in AI-powered signal detection but also paves the way for future developments in wireless communication, showcasing the transformative potential of AI in this domain. The paper is organized as follows: The system model is explained in Section 2. Section 3 reviews related work on signal detection in MIMO systems, focusing on the core aspects of MIMO systems and exploring various conventional methods and existing AI models. The problem formulation and data preparation are outlined in Section 4. This section details the framework and challenges of the system model used in the study, including assumptions, parameters, and configurations for MIMO systems. Section 5 introduces the proposed deep-learning (DL) model for signal detection in MIMO systems, specifically the novel AIDETECT2 scheme. This section describes the architecture and innovative aspects of AIDETECT2, highlighting how it builds upon previous models to improve performance. Theoretical underpinnings and algorithmic steps are elaborated to provide a comprehensive understanding of the scheme. In Section 6, we discuss the simulation setup used to evaluate the proposed models and the simulation results. This section details the simulation environment, including software tools, hardware specifications, and the parameters set for simulations. Also it presents the findings from the simulations and experiments, including a comparative analysis of the proposed models against existing techniques, highlighting performance improvements and any observed limitations. Following this, Section 6 also presents the optimization of the proposed model, and addresses the computational complexities of the signal detection models. The computational complexity analysis breaks down the complexity of each component, including the LSTM model, AIDETECT, AIDETECT2, and conventional methods. Finally, conclusions are drawn in Section 7. This section summarizes the key findings and contributions of the paper, reflecting on the effectiveness of the proposed models and their potential impact on MIMO signal detection. Future research directions are suggested, including possible enhancements and new areas of application.

2. System Model

We examine a system having N transmitting antennas and M receiving antennas, in which every antenna on both sides can communicate with each antenna on the other side. This results in N × M communication channels. Noise is considered for each receiving antenna. The transmitting symbols are generated using quadrature amplitude modulation of constellation size 16 (16-QAM) [6,7]. System performance is evaluated within a signal-to-noise ratio (SNR) range of 0 to 20 dB, with 5 dB step size. Figure 2 illustrates the general model for a MIMO system containing M receiving antennas and N transmitting antennas, which includes transmitted signal vector (x), received signal vector (y), channel matrix vector (H), and noise vector (n) [8]. The proposed system’s development is based on the following equation:

$$\mathit{y}=\mathit{Hx}+\mathit{n}$$

(1)

In this equation, y represents the signal that is received, H denotes the channel coefficient matrix vector, x signifies the signal vector which is transmitted, and n is the noise vector [9]. The model that was developed estimates the transmitted signal as $\widehat{\mathit{x}}$ using Equation (1).

3. Signal Detection in MIMO Systems

Multiple-input multiple-output, abbreviated as MIMO, is a wireless communication technology that employs numerous antenna elements at base stations to improve both spectral efficiency and energy [10]. This technology increases coverage at cell edges by spatially targeting transmissions toward users, resulting in enhanced signal strength as users distance themselves from the base station. Spatial multiplexing allows MIMO to communicate with several user devices simultaneously, enhancing overall spectral efficiency and throughput. Furthermore, when combined with millimeter-wave frequencies, MIMO boosts signal power [11]. During the signal detection phase in wireless networks, MIMO sustains a high SNR without altering bandwidth, unlike conventional approaches. Although MIMO has been successful in addressing these challenges, the growing number of connected devices introduces significant issues, such as fading, signal loss, and interference within networks. Various traditional signal detection techniques are available, including the commonly used minimum mean square error (MMSE) detector, maximum likelihood detector (MLD) and the zero-forcing (ZF) detectors [12].

3.1. Conventional MIMO Detectors

3.1.1. Zero-Forcing Detector

The zero-forcing detector effectively minimizes interference in the signal received by the receiver. It accomplishes this by designing a linear filter, which nullifies other user’s interference within the network, effectively removing the presence of interference from the received signal at the same frequencies. This process allows the receiver to distinguish the target signal from the interference, thereby enhancing the accuracy of signal detection [12]. The ZF algorithm employs following equation to detect the transmitted signal $\widehat{\mathit{x}}$.

$$\widehat{\mathit{x}}={\mathit{H}}^{†}\mathit{y}={\mathit{H}}^{\mathrm{H}}{\left(\mathit{H}{\mathit{H}}^{\mathrm{H}}\right)}^{-1}{\mathit{H}}^{\mathrm{H}}\mathit{y}$$

(2)

In this equation, H represents the channel matrix vector and y denotes the received signal matrix vector [13,14,15]. The pseudo-inverse of H is represented by H^†.

3.1.2. Minimum Mean Square Error Algorithm

The minimum mean square error detector is utilized in wireless communication systems for signal detection as it aims to reduce the mean square error between the actual signal detection response and the estimated signal response, thus reducing the effects of interference and noise on the signal received. In contrast to the ZF algorithm, which targets interference at specific frequencies, the MMSE detector employs a statistical method for signal detection, accounting for system interference and noise [16]. The detector employs particular equations to detect $\widehat{\mathit{x}}$.

$$\widehat{\mathit{x}}={\mathit{H}}^{\mathrm{H}}\mathit{H}+{\sigma }^2{E}_s^{-1}{\mathit{H}}^{\mathrm{H}}\mathit{y}$$

(3)

where H represents the channel matrix vector and y signifies the matrix of the received signal vector [13,14,15]. E_s stands for the mean symbol energy.

3.1.3. Maximum Likelihood Detector

The maximum likelihood detector is utilized in wireless communication systems to detect signals by identifying values that increase the likelihood of receiving the given signal, given the parameters that are estimated. This method aims to pinpoint the most probable parameter set that could have generated the data that is observed. Specifically, channel estimation seeks to determine the channel characteristics that best explain the received signal. Unlike the MMSE and ZF detectors, which focus on reducing noise and interference, the MLD aims to identify the most likely parameters that describe the signal based on data that is observed [17].

$$\widehat{\mathit{x}}=\underset{\mathit{x}\in A^P}{\mathrm{argmin}}\left|\right|\mathit{y}-{\mathit{H}\mathit{x}\left|\right|}^2$$

(4)

Here, $\mathit{H}$ represents the channel matrix vector, x denotes the transmitted signal matrix vector, and y indicates the received signal matrix vector. $A^P$ is the finite set of all possible sequences of 16-QAM, where each sequence forms a vector with P components [12]. ${\mathit{z}}_i$ is the ith component of estimated signal.

3.2. AI Based MIMO Detectors

3.2.1. LSTM Model

The LSTM model is a recently developed DL-based signal detection model using LSTM neural network. It has demonstrated superior performance compared to traditional methods [8]. However, the increased computational complexity posed challenges when scaling to higher models. Mathematical representation is

$$\begin{array}{cc}\hfill \widehat{\mathit{x}}=argmax(& \mathrm{softmax}(W_4·{\mathrm{Dropout}}_{0.2}({\mathrm{LSTM}}_{16}\{W_3,b_3,\hfill \\ \hfill & {\mathrm{LSTM}}_{32}[W_2,b_2,{\mathrm{LSTM}}_{64}(W_1,b_1,F)]\left\}\right)+b_4\left)\right)\hfill \end{array}$$

(5)

where F is $arg{min}_{x\in X^{N_t}}f\left(\mathbf{x},\left({\mathbf{y}}_{Re},{\mathbf{H}}_{\mathrm{Re}}\right),\left({\mathbf{y}}_{\mathrm{Im}},{\mathbf{H}}_{\mathrm{Im}}\right)\right)$, which is a cost function, $\widehat{\mathit{x}}$ denotes the estimated signal, $W_i$, $b_i$ are weights bias of $i^{th}$ hidden layer, and H, y and x represent the coefficients of channel vector, received signal vector, and transmitted signal vector respectively. ${\mathrm{Dropout}}_{0.2}$ is the dropout layer having 0.2 dropout rate and ${\mathrm{LSTM}}_n$ is the LSTM layer having n number of neurons. softmax denotes the softmax function. The softmax function is a neural function that transforms a vector of raw scores into a probability distribution over potential classes. The argmax operation is then applied to choose the index of the highest value in the softmax output, indicating the predicted class label [18].

3.2.2. AIDETECT

AIDETECT is a recently introduced DL based model for signal detection within wireless networks [19]. The exceptional performance of this model serves as a benchmark for our proposed model. This model employs a 3-layer optimized pattern recognition DL network as shown in Figure 3. Mathematical representation of the model is given by the following equation:

$$\widehat{\mathit{x}}=argmax\left(\mathrm{softmax}(W_4\times {\sigma }_3\{W_3\times {\sigma }_2[W_2\times {\sigma }_1(W_1\times F+b_1)+b_2]+b_3\}+b_4)\right)$$

(6)

Here $\widehat{x}$ denotes the estimated signal, $W_i$, $b_i$, ${\sigma }_i$ are weight, bias, clipped ReLU activation function of $i^{th}$ hidden layer, and F is $arg{min}_{x\in X^{N_t}}f\left(\mathbf{x},\left({\mathbf{y}}_{Re},{\mathbf{H}}_{\mathrm{Re}}\right),\left({\mathbf{y}}_{\mathrm{Im}},{\mathbf{H}}_{\mathrm{Im}}\right)\right)$, which is a cost function. H, y and x represents the coefficients of channel vector, received signal vector and transmitted signal vector respectively. The softmax activation function maps a vector of raw scores to a normalized probability distribution, enabling classification. The argmax function then identifies the class label with the highest predicted probability [19].

4. Problem Formulation and Data Preparation

This part provides a discussion on the formulation of the problem and preparation of data in the MIMO systems.

4.1. Problem Formulation

Algorithms based on mathematical models encounter several limitations, such as limited scalability and high complexity [20]. To avoid these issues and enhance the utilization of MIMO systems present in wireless communications, incorporating AI-based methods, such as ML, offers a promising solution. ML algorithms can indeed be beneficial in the development of ultra-reliable and low-latency communication (URLLC) networks [21]. By optimizing channel allocation based on traffic patterns, AI can significantly improve overall spectral efficiency and network performance. The primary goal here is to develop an AI-enhanced signal detection model for MIMO systems in wireless communication. Instead of traditional detection methods, we designed a neural network method to detect the transmitted signal. Following the development of the neural network, we trained it utilizing multiple transmitted symbols and subsequently evaluated the system’s performance with various symbols. The efficiency of the developed system needs to be contrasted with that of conventional methods and other AI models.

4.2. Data Preparation

Initially, symbols are produced based on the modulation scheme and constellation size. Random indices for the ntrain and ntest number of symbols are generated, which are then used to form the transmitted symbol vector x for training and testing. ntrain and ntest are the size of training testing dataset. The channel vector H is also generated and normalized so that the magnitude of its elements follows a Rayleigh distribution having a scaling parameter of 1. Rayleigh distribution is a mathematical model representing the probability of various measurement values influenced by several random factors [22]. The vector y for the received signal is created from the previously generated vectors for a selected SNR range, which is multiple times the size of vector x. Feature extraction involves obtaining the real and imaginary parts from vectors y and H. The targets correlating to the feature data developed are generated accordingly.

5. Proposed AIDETECT2 Scheme for MIMO Detection

The proposed model begins by generating transmitted signals, channel coefficients, and noise. Using these inputs, the received signal vector y is computed for various signal-to-noise ratios. Importantly, the system generates all necessary data internally, without relying on any pre-trained data from external sources. This internally generated data is then used to create training and testing datasets for the neural network, as mentioned above in the data preparation section. After the neural network is implemented and integrated into the system, it undergoes training and testing to ensure that the proposed system can efficiently detect signals. We have developed an AI model for signal detection using an optimized FFNN model, which is then integrated with the proposed system to form AIDETECT2. After the training and testing of the neural network, the network can detect the signals efficiently from y and H, as shown in Figure 4.

Previously, a recently developed AI-based model AIDETECT fell short of performance expectations. To address these shortcomings, we developed a tailored FFNN model AIDETECT2. Drawing inspiration from the AIDETECT, our customized lightweight FFNN aims to reduce complexity while enhancing overall performance. AIDETECT has utilized patter-net neural network, which has three hidden layers consisting of 64, 32 and 16 neurons. The customized FFNN used in AIDETECT2 is employed for a sequence of regression tasks. Initially, the feature data is transformed into a sequential format and passed to a sequence input layer sized according to the number of rows in the feature data. This input then flows through three hidden layers. Each hidden layer includes a clipped ReLU layer for introducing non-linearity, aiding the neural network in pattern recognition. In the fully connected layers, each neuron connects to the previous layer, incorporating weights and biases through an activation function [23]. Following these three hidden layers, the data enters the output layer for regression tasks. Training employs the Adam optimizer with an initial learning rate set to 0.001 and a mini-batch size of 10 [24]. The following Figure 5 shows the hidden layer architecture of the AIDETECT-2 network model.

The mathematical representation of the proposed system is given by:

$$\widehat{\mathit{x}}=argmax\left(W_4\times {\sigma }_3[W_3\times {\sigma }_2\{W_2\times {\sigma }_1(W_1\times F+b_1)+b_2\}+b_3]+b_4\right)$$

(7)

where $\widehat{x}$ denotes the estimated signal vector, $W_i$, $b_i$, and ${\sigma }_i$ are weight, bias, and the clipped ReLU activation function of $i^{th}$ hidden layer, and F is $arg{min}_{x\in X^{N_t}}f\left(\mathbf{x},\left({\mathbf{y}}_{Re},{\mathbf{H}}_{\mathrm{Re}}\right),\left({\mathbf{y}}_{\mathrm{Im}},{\mathbf{H}}_{\mathrm{Im}}\right)\right)$, which is a cost function. H, y and x represent the coefficients of channel vector, received signal vector, and transmitted signal vector, respectively.

The following Algorithm 1 shows the framework of the proposed AIDETECT2 MIMO detector. The output of the proposed system model is a predicted class with the highest probability. The model predicts the probability distribution across classes, and the class with the highest probability is selected as the output. This predicted output is then compared with the actual signal to evaluate the efficiency of the model, typically measured using metrics such as SER.

Algorithm 1: AIDETECT2: AI-based MIMO detector

AIDETECT2 represents a significant enhancement over its predecessor, AIDETECT, primarily due to a novel network model architecture. Notably, AIDETECT2 replaces the original network model with a customized FFNN design. This deliberate modification aims to reduce computational complexity and enhance the network’s efficiency, particularly in terms of its ability to maintain robust performance across various SNRs. By leveraging this lightweight FFNN, AIDETECT2 aims for improved adaptability and accuracy in handling diverse signal conditions, which will make it a more effective solution. The comparison of different network architectures is provided in

Table 1. Comparison of Neural Network Architectures.

Feature	LSTM	PatternNet-AIDETECT	AIDETECT2
Model Type	Recurrent	Feedforward	Feedforward
Layers	LSTM + Fully Connected	Custom Layers	Fully Connected + ClippedRelu
Activation Functions	Tanh, ReLU, Softmax	Sigmoid, Softmax	ReLU, Sigmoid
Input Type	Sequential	Pattern-based input	Sequential
Output Type	Sequence output	Classification output	Fixed-size output vector
Output Operation	Regression	Classification	Regression
Number of hidden layers	3	3	3
Number of neurons	64, 32, 16	64, 32, 16	128, 128, 128

.

6. Simulation Setup and Result Discussion

To replicate the proposed method, a custom simulator based on MATLAB was developed. This simulator runs on a stand-alone Intel i9-10900K CPU, @ 3.70 GHz with 128 GB RAM, and includes a 32 GB GPU for improved computational efficiency.

Table 2. Simulation Parameters.

Parameters	Parameter Settings
Number of transmitters	2, 4, 8
Number of receivers	2, 4, 8
Modulation	QAM
Constellation size	16
Test data size ($ntest$)	1 Million
Train data size ($ntrain$)	1 Million
SNR Range	0:5:20 dB
Training Function	Adam
Number of neurons	128
Number of hidden layers	3
Maximum epochs	2000
Mini Batch Size	10
Initial learning rate	0.001

outlines the parameters used in the simulation setup.

To train and evaluate the performance of the proposed deep learning model, simulations were conducted for both training data and testing data having a data size of 1 million (in terms of several symbols). QAM modulation with a constellation size of 16 was applied across different SNR points.

6.1. Result Discussion

To analyze the performance of the proposed AIDETECT-2 scheme is benchmarked against conventional (MMSE, ZF, and MLD) and AI-based detectors such as Variational Autoencoder-Enhanced Deep Neural Network-Based Detection (VAE-DNN-Det), LSTM, and AIDETECT MIMO detectors for different MIMO scenarios. The performance of the proposed AIDETECT-2 scheme is measured by conducting a simulation study based on 3 different MIMO simulating scenarios including 2 × 2, 4 × 4, and 8 × 8 MIMO systems with an equal number of transmitting and receiving antennas. Figure 6 presents the SER measurement of the 2 × 2 MIMO system for the proposed AIDETECT-2 MIMO detectors against benchmark conventional and AI-based MIMO detectors. Figure 6 simulation results show that the proposed AIDETECT-2 outperforms the conventional MIMO detectors and achieves between 98.31% to 95.56% better SER performance at 20 dB SNR. Similarly, simulation results also show that not only against conventional MIMO detectors but also, against AI-based MIMO detectors, the proposed AIDETECT-2 also achieves better SER performance ranging between 56.52% to 92.92% at 20 dB SNR respectively.

We further extended the number of transmitting and receiving antennas and conducted a study for 4 × 4 MIMO scenarios as presented in Figure 7. Figure 8 simulation results show that the proposed AIDETECT-2 achieves excellent SER performance than conventional MIMO detectors and beat them with 99.914% to 99.995% better SER performance at 20 dB SNR. Compared with the AI-based MIMO detectors, the proposed AIDETECT-2 beat LSTM and VAE-DNN-Det MIMO detectors by a big margin and achieved 70% to 97.69% better SER performance than both AI-based MIMO detectors at 20 dB SNR. However, previously developed AIDETECT holds almost similar performance as AIDETECT-2 from 0-dB to 10-dB but after 10-dB the proposed AIDETECT-2 surpase the benchmark AIDETECT MIMO detector and achieves 88% better SER performance at 20 dB SNR.

Finally, in our last simulation scenario, we considered a bigger scenario of an 8 × 8 MIMO system with 8 transmitting and 8 receiving antennas as shown in Figure 8. Similar to Figure 6, Figure 7 and Figure 8 also follows a similar trend showing the superiority of the proposed AIDETECT-2 against the benchmark conventional and AI-based MIMO detectors. Figure 8 results show that at 20 dB SNR, the proposed AIDETECT-2 achieves 99.4% to 99.7% better SER performance than conventional ZF and MMSE MIMO detectors. However, for a bit large 8 × 8 MIMO scenario, the optimal MLD MIMO detector achieves better SER performance than other AI-based MIMO detectors and holds very close SER performance (13.75% lower SER) than AIDETECT-2 at 20 dB SNR, but is still unable to beat the proposed AIDETECT-2 MIMO detection method. Figure 8 simulation results also show that the proposed AIDETECT-2 outperforms the benchmark AI-based MIMO detectors including AIDETECT and achieves between 78.5% to 89.2% better SER performance than all AI-based MIMO detectors at 20 dB SNR respectively.

We also analyze the training and validation graph based on root mean square error (RMSE) and loss for 8 × 8 MIMO system mode. Figure 9 shows the plotting of both loss and RMSE against the number of iterations in the model training and validation process. Figure 9 demonstrates how, as a result of the initial randomness of the model, RMSE begins high and rapidly drops in the first 200–300 iterations as the model learns, then steadily drops throughout fine-tuning. With a somewhat larger, steady value, the validation RMSE closely tracks the training RMSE, suggesting little overfitting and strong generalization.

Furthermore, in the case of loss scenario, Figure 9 shows how the loss increases in the first few hundred iterations as the model learns, but then rapidly drops as a result of the model’s initial ignorance. Following this sharp reduction, the loss decreases more gradually when the model is fine-tuned. The validation loss closely follows the training loss, suggesting a similar trend in the form of small overfitting and good generalization.

In conclusion, the proposed model consistently outperformed the benchmark MIMO detectors including both conventional (MMSE, ZF, and MLD) and AI-based (LSTM, VAE-DNN-Det, and AIDETECT) MIMO detectors, and demonstrated higher noise resilience. These results suggest that AIDETECT-2 provides more reliable and accurate estimations, even under high interference conditions.

Furthermore, we also conducted an optimization study aimed at improving the performance of the proposed AIDETECT-2 network model based on the number of hidden layers for a 4 × 4 MIMO system focusing specifically on the proposed model’s neural network architecture. The study involved evaluating the model’s performance with varying numbers of hidden layers, specifically 2, 3, and 4 hidden layers as shown in Figure 10. Through experimentation and analysis, it was observed that AIDETECT-2 with 3 hidden layer model consistently outperformed both AIDETECT-2 with 2 hidden layer and AIDETECT-2 with 4 hidden layer configurations across key performance metrics. Consequently, the AIDETECT-2 network model based on FFNN with 3 hidden layers was selected as the optimal architecture for the proposed signal detection model. This decision was based on its superior ability to handle the complexity of MIMO systems, achieve higher detection accuracy, and maintain computational efficiency. The findings underscore the critical role of architectural design in optimizing neural network models for practical applications in communication systems.

6.2. Computational Complexity Analysis

Computational complexity is an important parameter to measure the effectiveness of any MIMO detector. The complexity analysis is required to comprehend standard MIMO detectors’ computational computing needs. In this section, we further analyze the computational complexity of different conventional (MMSE, ZF, and MLD) and AI-based (LSTM, VAE-DNN Det, AIDETECT, and AIDETECT-2) MIMO detectors respectively as presented in

Table 3. Big-O Notation Comparison Table.

Model	Computational Complexity
ZF	$O(N^3+N^{2}M)$
MMSE	$O(N^3+N^{2}M)$
MLD	$O\left(M^N\right)$
VAE-DNN-Det	Varies widely
AIDETECT	$O\left(L^4\right)$
LSTM Model	O(T(4a · 64 + 4 · 642 + 3 · 64 + 4a · 32 + 4 · 322 + 3 · 32 + 4a · 16 + 4 · 162 + 3 · 16))
AIDETECT2	$O(128a+{128}^2+128m)$

. The computational complexity of the proposed model on comparison with other model is plotted in Figure 11 in the form of number flops. Flop is the measure of number of floating point operations per second [25,26]. Where M is the number of receiving antennas and N is the transmitting antennas.

6.2.1. Conventional Methods

Conventional signal detection algorithms such as ZF, MMSE, and MLD have distinct computational complexities. The ZF and MMSE algorithms involve matrix inversions, which have a complexity of $O\left(N^3\right)$, where N is the number of transmitting antennas. They also require matrix multiplications, adding $O\left(N^{2}M\right)$, where M is the number of receiving antennas. The ML algorithm has an exponential complexity $O\left(M^N\right)$ due to its exhaustive search through all possible transmitted vectors, making it impractical for large systems despite its optimal performance [27]. The LSTM model, with its multi-layer structure and sequence processing capabilities, exhibits a higher computational complexity compared to the FFNN model, which benefits from a simpler feed-forward architecture. The neural network models show significantly improved performance in signal detection over conventional algorithms such as ZF, MMSE, and MLD [28]. Despite their higher complexity, the adaptability and accuracy of AI-driven models make them suitable for advanced wireless communication systems like B5G and 6G networks.

6.2.2. VAE-DNN-DET Model

The computational complexity of the VAE-DNN-Det network model varies widely as does not provide its computational complexity exactly as the computational needs for training the VAE are determined by the design and size of the encoder and decoder networks, which include elements like layer count, neuron amount, and layer types. In addition, the sampling method for the latent variable space adds extra complexity. The size of the latent space and the architecture of the encoder network affect the difficulty of VAE inference.

6.2.3. LSTM Model

The computational complexity of the LSTM for the forward pass is determined by the operations in each LSTM cell, which include matrix-vector multiplications and element-wise operations. For an input dimension a and hidden dimensions of 64, 32, and 16, the complexity per time step per layer is $O(4ad+4d^2+3d)$, where d represents the hidden dimension [29]. Summing up the layers and considering the sequence length T, the overall forward pass complexity is

$$O\left(T(4a·64+4·{64}^2+3·64+4a·32+4·{32}^2+3·32+4a·16+4·{16}^2+3·16)\right)$$

(8)

The backward pass doubles this complexity. Given the high number of epochs and the sequence data involved, the computational burden of the LSTM model is substantial but justified by its improved performance in signal detection.

6.2.4. AIDETECT

The patternet feed-forward network used in the benchmark model AIDETECT has three hidden layers with 64, 32, and 16 neurons. The computational complexity is largely influenced by the number of neurons in the layer. The total complexity of the model is given by

$$O\left(L^4\right)$$

(9)

where L is the number of hidden layers in the network [30].

6.2.5. AIDETECT2

The customized FFNN model used in AIDETECT2 has 3 layers with 128 units and uses the Clipped ReLU activation function. The complexity of the FFNN is largely influenced by the fully connected layers, each layer has a complexity of $O(n·m)$, where n and m are the sizes of the input and output layers, respectively. For the developed architecture, each layer performs $128·n+128·128$ operations, summed across the layers. Therefore, the total complexity for a forward pass through the network is

$$O(128a+{128}^2+128m)$$

(10)

where a is the input feature dimension. The regression layer at the end contributes less to the overall complexity. Given the fixed input size and number of layers, the computational complexity of the FFNN is generally lower than that of the LSTM and patternet.

7. Conclusions and Future Discussion

This study underscores the pivotal role of AI-driven methodologies in advancing beyond 5G wireless networks, with a focus on signal detection in MIMO systems. More specifically, this study presents AIDETECT-2, an AI-based signal detection technique designed for MIMO systems based on the FFNN network. Compared to conventional and other AI-based techniques, AIDETECT-2 greatly increases the efficiency of signal detection, especially in lowering the SER at various SNR points. The developed AI model AIDETECT2 for signal detection demonstrates superior performance compared to conventional methods and AI methods. Specifically, our proposed AIDETECT2 model surpasses the benchmarks set by traditional approaches, AIDETCT, LSTM Model, and VAE-DNN-Det. The simulation results indicate that the proposed model achieves a significant performance improvement, with SER improvements ranging from 13.75% to 99.995% better than the best conventional method and from 56.52% to 97.69 better than benchmark AI-based MIMO detectors at 20 dB SNR for given MIMO scenarios, respectively. This excellent performance with low complexity makes AIDETECT2 a more effective solution for signal detection in challenging environments. Despite these promising results, integrating multiple neural network architectures into a hybrid model could yield even better outcomes. Additionally, developing an AI-enhanced channel estimation model and integrating it with our current system could significantly impact the industry. This single, unified model would have the capability to estimate channels and detect signals more effectively.