A Wireless Channel Equalization Method Based on Hybrid Whale Optimization: For Constant Modulus Blind Equalization System

Abstract

This paper proposes a wireless channel equalization method applied to the constant modulus blind equalization system, which addresses the slow convergence and strong randomness in the initialization of equalizer weights in the constant modulus blind equalization algorithm (CMA) by introducing a hybrid arithmetic whale optimization algorithm (HAWOA). The mean square error in the CMA is utilized as the cost function for the HAWOA to obtain a more effective initial weights for the equalizer. To validate the superiority of the hybrid arithmetic whale constant modulus blind equalization algorithm, tests are conducted on the equalization system using 16QAM and 64QAM signals. The simulation results demonstrate that the proposed algorithm achieved better initial weights compared to the CMA and the constant modulus blind equalization algorithm based on the whale optimization algorithm. It can obtain the desired mean square error with a lower symbol error rate in fewer iterations. Furthermore, the hybrid arithmetic whale constant modulus blind equalization algorithm exhibited faster convergence in optimizing initial weights, effectively enhancing the equalization performance of the CMA in wireless channels while ensuring timeliness.

1. Introduction

In wireless communication, the transmitted signal has linear distortion because of multipath decline and limited bandwidth [1], which leads to Inter-symbol interference (ISI) at the receiving terminal. The ISI will lead to waveform distortion and broadening of front and rear symbols as well as affect the error performance of the communication system [2,3].

In consequence, it is essential to study how to reduce ISI. To recover distorted signals, equalization techniques are typically employed at the receiver of a communication system. Traditional equalization techniques can be categorized into adaptive equalization, semi-blind equalization and blind equalization. The key of adaptive equalization is to use training sequences or pilot symbols to assist in signal equalization and processing [4]. In communication systems, having sufficient available bandwidth ensures high-speed data transmission. The performance of the system is somewhat compromised by the use of training sequences in adaptive equalization techniques, as mentioned in reference [5]. In some application scenarios, such as unmanned aerial vehicle (UAV) communication, it is difficult to allocate sufficient bandwidth for transmitting training sequences due to bandwidth limitations, rendering adaptive equalization unsuitable for all communication scenarios. Furthermore, due to the time-varying nature of wireless fading channels, it is necessary for the equalizer to track the time-varying characteristics of the channel in real-time. Remarkably, adaptive equalization techniques often exhibit poor timeliness, making it challenging to cope with fast channel variations and communication demands in modern applications. This limitation imposes certain constraints on their practical applicability. The semi-blind equalization technique combines non-blind and blind methods by incorporating a small amount of training sequences onto the basis of blind equalization, thereby achieving improved channel tracking capability and equalization performance to some extent [6]. Although semi-blind equalization combines certain performance advantages of blind equalization and adaptive equalization, it also inherits the drawbacks of adaptive equalization, such as the inability to handle completely unknown channel conditions compared to blind equalization. In contrast to the other two equalization techniques, blind equalization primarily relies on prior information of the received signal, exhibiting strong robustness and applicability [7,8]. The advantages of blind equalization lie in its minimal bandwidth requirement and strong timeliness, enabling signal recovery and communication to be achieved in the shortest possible time. However, blind equalization also suffers from high computational complexity and potential degradation of convergence accuracy. Therefore, choosing the appropriate equalization approach based on specific communication requirements is essential for signal recovery and high-quality communication.

Based on the analysis above, both adaptive equalization and semi-blind channel equalization require the transmission of training sequences for parameter updating of the equalizer. These two methods need to continuously send training sequences to adapt to the variations in the channel in the presence of time-varying channels, which leads to increased bandwidth consumption and reduced channel utilization [9]. For applications that require high communication efficiency, such as wireless sensor networks, mobile communication systems and radar systems [10,11,12], the transmission of training sequences may introduce delays in information delivery and affect the control from the transmitter to the receiver. Due to the strong bandwidth constraints and strict requirements for real-time control of components in such applications, relying on training-based equalization techniques cannot guarantee real-time information transmission at the receiver in high-speed mobile scenarios, and it may result in decreased communication efficiency due to limited available bandwidth. Adaptive equalization and semi-blind channel equalization fail to demonstrate significant superiority in these scenarios. In certain applications involving mobile communication and real-time communication, employing blind equalization methods that do not require training becomes a reliable approach [13].

Blind equalization indicates that the prior information equalization channel characteristics brought by the receiving sequence are only used in the equalization process without the help of the training sequence so that the output sequence of the equalizer is constantly close to the sending sequence [14]. Blind equalization is divided into Bussgang blind equalization algorithm, algorithms based on higher order spectrum estimation and algorithms based on neural networks, fuzzy theory, wavelet theory and other theories [13]. Meanwhile, the CMA [15] and Multi-modulus algorithm [16] (MMA) in the Bussgang blind equalization algorithm have been widely studied and applied because of their simple structure and strong stability. As a classic blind equalization algorithm, the constant modulus algorithm has advantages, such as low operational complexity, but disadvantages, such as slow convergence speed and large mean square error [17], which are not conducive to information transmission.

To get the preferable performance of blind equalization, some extensive studies were conducted on the slow convergence speed and other characteristics of blind equalization. The traditional blind equalization algorithm usually adopts a gradient descent for weight iteration. Although this method has a simple structure and low computational complexity, it converges slowly. This method is challenging to ensure efficient information transmission in higher communication requirements. Some researchers put forward various improved ways for weight iteration of blind equalization algorithm to obtain the fast equalizer convergence. To enhance the convergence of traditional blind equalization, Ma et al. [18] proposed a novel fast algorithm under non-Gaussian noise. The quasi-Newton method is used in blind equalization to accelerate the convergence rate. It is used to replace the original gradient descent in the updating process of the weight of the constant modulus blind equalization. It could take advantage of the quasi-Newton convergence rate to obtain better performance. Li et al. [19] proposed a second-order Gauss-Newton algorithm based on a multi-modulus algorithm to improve the speed of solving the optimal coefficient of the equalizer. The purpose of fast convergence is realized by using the second-order Gauss-Newton algorithm to search the optimal coefficient of the multi-modulus algorithm. Compared with other Newtonian methods, the second-order Gauss-Newton has a more stable convergence performance and effectively reduces the computational complexity. In the above two studies, the Newton-type method replaces or improves the original weight iteration of blind equalization. Still, Newton’s method is mostly local convergence, and it may not converge to the optimal value. Meanwhile, Newton’s method will increase the computation and communication costs.

To solve the problems of large mean square error problems, Sun et al. [20] introduced a novel enhanced step function that can be variable in weight updating and accelerate the convergence speed into CMA. Although this method has preferable balance and anti-noise performance, it still relies on the CMA and the original iterative process of LMS. It is still affected by the shortcomings of classical methods. Zhang et al. [21] proposed a MMA based on the maximum correlation criterion. It proposed a method to calculate the weight coefficient, which reduces the computational complexity while maintaining the equalization effect. This algorithm has a preferable convergence performance when processing high-order QAM signals and can show excellent balance performance in Gaussian and pulse noise environments. The above research optimizes and improves blind equilibrium through various methods and strategies. Still, most studies use classical methods such as gradient descent and its improvement method for iterative updating. The classical method has poor robustness, and the solution result depends on the initial value, so the initial conditions strongly constrain it. In addition, partial blind equalization, such as CMA, has multiple local optima in the iterative process, so the traditional method may fall into local optima and cannot get out. It will result in a poor equalization effect. The research in the literature did not consider the above problems, so the use of more effective methods to solve the drawbacks of traditional methods has become the goal of this study.

In the process of equalization, the main task of the adaptive algorithm is to determine the optimal coefficient setting of the equalization system, so the adaptive problem can also be viewed as an optimization problem [22]. Meanwhile, the coefficient setting of traditional methods is realized by gradient descent, which has problems such as slow convergence speed, poor robustness and fall into local optima. It still has drawbacks in the scenario with high communication demand. In reference [23], artificial intelligence algorithms have gradually been employed in the next-generation wireless communication models. Their excellent computational capabilities can optimize the parameters of the communication model to improve system performance. Inspired by the AI-assisted channel estimation techniques, leveraging the technical characteristics of artificial intelligence, swarm intelligence algorithms from intelligent computing are utilized to solve the channel equalization problem. The algorithmic features can compensate for the drawbacks of convergence in channel equalization. While maintaining a high convergence rate, the swarm intelligence algorithm has strong robustness, which can bypass local optimization due to its unique population diversity and update iteration [24]. Some scholars began to seek swarm intelligence algorithms with higher convergence speeds to solve the blind equalization problem.

Xu et al. [25] proposed a modified CMA based on the bat algorithm. They took the error function of the CMA as the algorithm’s cost function and then used the bat algorithm to calculate the weight vector’s initial value. According to the analysis, this method had a strong applicability. Zhang et al. [26] proposed a CMA based on chaotic chickens. The simulation showed the algorithm has a fast convergence speed and robust global convergence ability. As mentioned in the above literature, some research has used swarm intelligence algorithms to optimize blind equalization problems and has a compelling performance. Some research and optimization algorithms have been of general performance. Compared with the swarm intelligence algorithms proposed in recent years, it performs imperfectly, and the solution vector with less fitness cannot be obtained in a short time. The whale optimization algorithm [27] (WOA), which had strong optimization performance in recent years, is selected as the optimization method of the improved CMA in this paper.

As a bionic optimization algorithm, the swarm intelligence algorithm is inspired by the simple behavior or survival instinct among organisms [28]. It can establish mathematical models by imitating the activities of all kinds of organisms and effectively solving various complex optimization problems. Standard swarm intelligence algorithms include particle swarm optimization [29] (PSO), artificial colony algorithm [30] (ABC), whale optimization algorithm (WOA), Harris hawks optimization algorithm [31] (HHO), Aquila optimizer [32] (AO). The WOA was proposed by Australian scholar Seyedali Mirjalili et al. [27] in 2016. This algorithm established a mathematical model based on the group hunting behavior of humpback whales, and it is very competitive compared to the traditional optimization methods.

Remarkably, WOA also has the disadvantage of low search accuracy and suffers from entrapment in local optima [33,34]. Consequently, to improve the optimization performance of the WOA and further improve the solving efficiency, corresponding improvements of the WOA are made before the actual optimization problem. In Ref. studies [35,36,37], nonlinear control parameters, opposition learning and other methods were used to improve the convergence accuracy and search speed of WOA and to enhance the global search capability by introducing effective search strategies. The simulation proved that it has an excellent performance in micro-grid. The above studies indicated that WOA and its improved algorithm had been widely used in various optimization problems. It can be viewed as an effective optimization method to face practical optimization problems, such as blind equalization, because of its excellent performance.

It can be seen from the above literature that adaptive parameters or partial strategies are introduced to help search for the WOA. There are fewer methods to achieve improvement through hybridization between algorithms. Therefore, the method of algorithm hybridization is used in this study to give play to the unique excellent performance of different algorithms. Through hybridization, the search mechanism of other algorithms is introduced without changing the search mechanism of the WOA. It will make up some defects of the WOA and enhance the search performance.

The innovation of this article is as follows:

• The arithmetic optimization algorithm with fast optimization speed was hybridized with the whale optimization algorithm, and the honey badger honey factor was introduced to obtain a hybrid arithmetic whale optimization algorithm.
• A constant modulus blind equalization algorithm based on hybrid arithmetic whale optimization is designed to enhance the equalization effect of the system. The hybrid arithmetic whale optimization algorithm (HAWOA) with the idea of algorithmic hybridization is applied to the weight updating of CMA. It fully plays the hybrid arithmetic whale optimization algorithm’s robustness and convergence speed advantages. It makes up for the shortcomings of CMA in the channel equalization process, such as slow convergence speed and inability to quickly determine the optimal coefficient setting. This study fills the research gap of the blind equalization problem by using the new meta-heuristic algorithm.
• This study considers the computational problem caused by the swarm intelligence algorithm, which was not considered in previous studies. It can effectively reduce the system model’s computational time and computation amount while ensuring the equalization effect.

The rest of the paper is structured as follows. Section 2 indicates the optimization methods used in this study and their complexity analysis. Section 3 focuses on the CMA and its improvements. Section 4 describes the system simulation experiments and validity analysis. The conclusion of the paper is summarized in Section 5.

2. Method

This section mainly introduces the WOA and its improved algorithm. After describing the improvement principle of the hybrid arithmetic whale optimization algorithm, the effectiveness of the algorithm is verified by reference function test and complexity analysis.

2.1. Basic Principle of WOA

The WOA is based on the mathematical modeling of the unique foraging behavior of the whales. Its mathematical model is divided into three parts: encircling prey, bubble-net attacking and searching for prey [27].

2.1.1. Encircling Prey

In hunting, whale groups constantly communicate among individuals and gradually surround prey through cooperation. Each individual in the population will continue to approach prey, and the overall situation is shrinking and encircling. This method is mathematically formulated as follows.

$$X(t+1)=X^{*}(t)-A\cdot D$$

(1)

$$D=\left|C\cdot X^{*}(t)-X(t)\right|$$

(2)

$$A=2a\left(t\right)\times r_1-a\left(t\right)$$

(3)

$$C=2r_2$$

(4)

$X(t)$ is the current position vector of the whale; $X^{*}(t)$ is the current optimal position vector of the iteration; $t$ is the number of iterations; $D$ is the distance between the whale individual and the current optimal position of the whale; $\cdot$ is the of elements by element multiplication; $A$ and $C$ are the coefficient vector; $r_1,r_2$ is the random vector between [0,1]; and $a(t)$ decreases linearly from 2 to 0 from the increase of $t$.

2.1.2. Bubble-Net Attacking

Bubble-net attacking indicates that the whales constantly produce bubbles to attack and encircle the prey. This stage is divided into shrinking, encircling and spiral updating positions. The formula for bubble-net attacking is as follows.

$$X\left(t+1\right)=D^{\prime }\cdot e^{bl}\cdot \cos \left(2\pi l\right)+X^{\ast }\left(t\right)$$

(5)

$$D^{\prime }=\left|X^{\ast }\left(t\right)-X\left(t\right)\right|$$

(6)

where $b$ is a constant, the usual value is 1. $l$ is a random number between [−1,1], and $D^{\prime }$ is the absolute distance between the searching individual and the current optimal individual.

In bubble-net attacking stage, the probability parameter $P\left(P\in \left[0,1\right]\right)$ selects the shrinking, encircling and spiral updating positions strategies. The formula is shown in Equation (7).

$$X(t+1)=\left\{\begin{array}{l}{X}^{*}(t)-A\cdot D\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} P<0.5\\ D^{\prime }\cdot e^{bl}\cdot \cos (2\pi l)+X^{*}(t)\hspace{1em}P\ge 0.5\end{array}\right.$$

(7)

2.1.3. Searching for Prey

When the WOA is in the shrinking encircling phase, parameter $\left|A\right|$ is usually used to decide whether the algorithm enters the searching for prey or the encircling prey. When $\left|A\right|\ge 1$, the algorithm enters the stage of searching for prey, the algorithm will randomly select position points in an extensive range and then shrink encircling according to the selected position. When $\left|A\right|<1$, the algorithm enters the encircling prey phase. The formula for search for prey is as follows.

$$X(t+1)=X_{rand}(t)-A\cdot D$$

(8)

$$D=\left|C\cdot X_{rand}(t)-X(t)\right|$$

(9)

where $X_{rand}(t)$ is the whale position vector randomly selected by the current whale population.

2.2. Hybrid Arithmetic Whale Optimization Algorithm

As for the shortcomings of WOA, such as low search accuracy, slow convergence rate and easily falling into local optimal [38], the hybrid arithmetic whale optimization algorithm was obtained by introducing multiplication, division, addition and subtraction search strategies [39] and honey badger honey factor [40] into WOA. The principle is mathematically formulated as follows.

2.2.1. Introduced Arithmetic Optimization Algorithm

Because of the large randomness in the initial search stage of WOA, the convergence performs imperfectly. The mathematical calculation performed by the subtraction and addition operators in the arithmetic optimization algorithm can obtain a lower distribution of values, reducing the randomness of the search for prey. This proposal lets the individual get closer to the target quickly and accelerates algorithm convergence. Introducing an arithmetic optimization algorithm into WOA can effectively enhance the early search performance of the algorithm.

$$X(t)=\left\{\begin{array}{l}{X}^{*}(t)-MOP\times ((UB-LB)\times \mu +LB)\hspace{1em}\hspace{1em} r_3<0.5\\ X^{*}(t)+MOP\times ((UB-LB)\times \mu +LB)\hspace{1em}\hspace{1em}otherwise\end{array}\right.$$

(10)

$$MOP(t)=1-\frac{t^{\frac{1}{a}}}{T^{\frac{1}{a}}}$$

(11)

It should be noted that $UB,LB$ is the upper bound and lower bound of the constraint range respectively. $\mu$ is the control parameter, which set to 0.499 according to the experience [39]. $MOP$ is the probability coefficient of the mathematical optimizer, and $r_3$ is a random number from 0 to 1. $t,T$ is the current number of iterations and the maximum number of iterations. $a$ is the coefficient factor, and the value is 5 according to experience.

In bubble-net attacking stage, the arithmetic optimization algorithm introduces division and multiplication operators aiming at the low search accuracy of spiral updating position in a short time, which can improve population diversity. It can improve the convergence speed of WOA and search accuracy.

$$Z(t+1)=\left\{\begin{array}{l}{X}^{*}(t)÷(MOP+\epsilon )\times ((UB-LB)\times \mu +LB)\hspace{1em}\hspace{1em} r_4<0.5\\ X^{*}(t)\times MOP\times ((UB-LB)\times \mu +LB)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} otherwise\end{array}\right.$$

(12)

$$q=1-\frac{t}{T}$$

(13)

$$X(t+1)=\left\{\begin{array}{l}Z(t+1)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hspace{1em}\hspace{1em}q>0.5\\ D^{\prime }\cdot e^{bl}\cdot \cos \left(2\pi l\right)+X^{\ast }\left(t\right)\hspace{1em}q\le 0.5\end{array}\right.$$

(14)

$\epsilon$ is a small integer; r₄ is a random number between [0,1]; and $q$ is a linear function. According to Equation (14), in the bubble-net attacking stage, the value of $q$ should be used to judge the multiplication and division search strategy or spiral updating strategy of the arithmetic optimization algorithm. When, q > 0.5, the multiplication and division search strategy of the arithmetic optimization algorithm is implemented to accelerate the convergence rate. When, q ≤ 0.5, the spiral updating strategy is implemented to enhance the search accuracy.

2.2.2. Introduced Honey Badger Honey Factor

Honey Badger algorithm can effectively jump out of local optimization and obtain effective target value when solving the multi-extreme point problem [40]. Meanwhile, the honey stage of the honey badger algorithm plays a leading role in the whole optimization. The unique parameter $F$ of this stage can change the search direction of the population and provide a scanning search function for searching individuals, which is conducive to the algorithm jumping out of the local optimal. Inspired by the honey stage, the honey badger algorithm was introduced into the exploitation strategies of WOA to jump out of the local optimal. This method is mathematically formulated as follows.

$$F=\left\{\begin{array}{l}1\hspace{1em}\hspace{1em} r_5\le 0.5\\ -1\hspace{1em}\hspace{1em}else\end{array}\right.$$

(15)

$$w=F\times r_6\times 2\times e^{-\frac{t}{T}}$$

(16)

$$X(t+1)=\left\{\begin{array}{l}w\cdot X^{*}(t)-A\cdot D\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} P<0.5\\ D^{\prime }\cdot e^{bl}\cdot \cos (2\pi l)+w\cdot X^{*}(t)\hspace{1em}P\ge 0.5\end{array}\right.$$

(17)

where $w$ is the honey factor, $r_5,r_6$ is the random number between 0 and 1.

2.3. Complexity Analysis

2.3.1. Time Complexity Analysis

On the basis of WOA, the HAWOA incorporates the multiplication, division, addition and subtraction search strategies in arithmetic optimization algorithm, as well as the honey badger honey factor. The time complexity of HAWOA primarily consists of several parts, including the initialization of search individuals, fitness calculation and individual position updates. The parameters used in the complexity analysis include the population size $N$, dimensionality $D$ and the number of iterations $T$.

In the initialization phase, the time complexity of population initialization in HAWOA is $O\left(N\times D\right)$; the time complexity of fitness calculation is $O\left(N\right)$. The searching for prey is a triple cycle when the position is updating. The time complexity of the searching for prey is $O\left(N\times D\times T\right)$, and the computational complexity of the shrinking, encircling and bubble net attack phase is $O\left(N\times D\times T\right)$. For the stopping decision, the time complexity is $O\left(1\right)$, so the time complexity of HAWOA is $O\left(N\times D\times T\right)$. In HAWOA, although the number of cycles increases, the main cycle structure of the algorithm does not change. Since the time complexity of WOA is $O\left(N\times D\times T\right)$, which is the same as that of HAWOA, it can be inferred that the improvement of the algorithm does not increase its time complexity.

2.3.2. Space Complexity Analysis

The storage space consumed by an algorithm is usually defined as space complexity. Where the population number is $N$ and the dimension is $D$, the total space complexity of HAWOA is $O_1\left(N\times D\right)$ while the total space complexity of WOA is $O_2\left(N\times D\right)$, which is the same. Therefore, the improvement of HAWOA does not increase the spatial complexity.

3. The Constant Modulus Blind Equalization Model

3.1. Basic Principles of Blind Equalization

The blind equalization system model is shown in Figure 1.

Where $s(n)$ is the sending sequence, $h(n)$ is the impulse response of the discrete time transmission channel, $n(n)$ is the superimposed noise on the channel, such as Gaussian white noise, and $y(n)$ is the receiving sequence. The specific formula is shown in Equation (18). The impulse response of the blind equalizer is $w(n)$; $L$ is the length of the equalizer; and $\tilde{s}(n)$ is the recovery sequence of the blind equalization system.

$$y(n)=h(n)\ast s(n)+n(n)={\displaystyle \sum _{i}h(i)s(n-i)+n(n)}$$

(18)

where $\ast$ represents convolution, $i\in [1,n]$.

3.2. Basic Principle of CMA

The CMA was one of the most widely studied algorithms in blind equalization because of its simple structure, strong practicability and excellent performance [41]. However, in the process of equalization, the constant modulus algorithm also has the disadvantages of falling into local optimal and being unable to jump out, having a slow convergence rate and requiring more signal points for iteration [42]. The whole system structure is formed $y(n)$ by adding Gaussian white noise $n(n)$ to the input sequence $s(n)$ after passing through the channel $h(n)$. The $y(n)$ is an input sequence in equalizer to get the estimated sequence $\tilde{s}(n)$. Finally, the decision sequence $\widehat{s}(n)$ is generated by the decision device. The principal diagram of the CMA is shown in Figure 2.

Where $\psi (\cdot )$ is the error generating function, $e(n)$ is the error function of constant modulus algorithm, $\widehat{s}(n)$ is the decision value of recovery sequence $\tilde{s}(n)$ obtained by the decision device.

$$S(n)={[s(n),s(n-1),\dots ,s(n-N+1)]}^T$$

(19)

$$Y(n)={[y(n),y(n-1),\dots ,y(n-L+1)]}^T$$

(20)

where $L$ is the length of a finite-length transverse filter, $N$ is the length of channel. $T$ represents the transpose of the matrix.

$$y(n)={\displaystyle \sum _{i=0}^{N-1}h(i)s(n-i)}+n(n)=H^T(n)S(n)+n(n)$$

(21)

$$\tilde{s}(n)={\displaystyle \sum _{i=0}^{L-1}w(i)y(n-i)=W^T(n)}Y(n)$$

(22)

The error function of CMA is as follows:

$$e(n)=|\tilde{s}(n){|}^2-R_2$$

(23)

$$R_2=\frac{E(|\tilde{s}(n){|}^4)}{E(|\tilde{s}(n){|}^2)}$$

(24)

The cost function of CMA is as follows:

$$J_{CMA}=E(e^2(n))=E({(|\tilde{s}(n){|}^2-R_2)}^2)$$

(25)

The updated formula of weight of CMA algorithm is obtained by using the fastest descent method:

$$W(n+1)=W(n)-\mu \nabla J_{CMA}$$

(26)

where $\mu$ is the step of the iteration, $\nabla J_{CMA}$ is the instantaneous value after taking the partial derivative. The specific formula is as follows.

$$\nabla J_{CMA}=\frac{\partial J_{CMA}}{\partial W(n)}$$

(27)

$$\begin{array}{l}E(2(|\tilde{s}(n){|}^2-R_2)\frac{\partial (\tilde{s}(n){\tilde{s}}^{\ast }(n))}{\partial W(n)})\\ =E(4(|\tilde{s}(n){|}^2-R_2)Y^{\ast }(n)Y^T(n)W(n))\\ =E(4(|\tilde{s}(n){|}^2-R_2)Y^{\ast }(n)\tilde{s}(n))\end{array}$$

(28)

$$\nabla J_{CMA}=4(|\tilde{s}(n){|}^2-R_2)Y^{\ast }(n)\tilde{s}(n)$$

(29)

$$\begin{array}{l}W(n+1)=W(n)-4\mu (|\tilde{s}(n){|}^2-R_2)Y^{\ast }(n)\tilde{s}(n)\\ =W(n)-4\mu e(n)Y^{\ast }(n)\tilde{s}(n)\end{array}$$

(30)

where ${\tilde{s}}^{*}(n)$ is conjugate of the recovery sequence, $Y^{\ast }(n)$ is the conjugate of the input signal of the equalizer.

3.3. The Constant Modulus Blind Equalization Algorithm Based on Hybrid Arithmetic Whale Optimization

Because of the slow convergence speed and strong randomness of initial weights of the CMA, a hybrid arithmetic whale constant modulus blind equalization algorithm (HAWOA-CMA) is designed in this section. The schematic diagram of the HAWOA-CMA system is shown in Figure 3. The received sequence is taken as the input signal of the HAWOA algorithm, and the mean square error of the whole equalization system is set as the cost function of HAWOA. The formula is shown in Equations (31) and (32).

$$MSE=\frac{1}{N}{\displaystyle \sum _{n=1}^N{\left|\widehat{s}(n)-\tilde{s}(n)\right|}^2}$$

(31)

$$fitnes{s}_{HAWOA}=MSE$$

(32)

where $N$ represent the number of iteration points, $MSE$ is the mean square error, $fitnes{s}_{HAWOA}$ is the fitness of HAWOA.

In HAWOA, the individual whale corresponds to the weight vector of the equalizer, and the MSE is taken as the cost function. In the iteration process, the minimum value of the cost function is found by updating the position of each whale. Therefore, a group of optimal whale positions under the condition of minimum fitness value is obtained. Then the optimal whale position is regarded as the initial weight value assigned to the optimal weight vector in the CMA equalizer. Finally, the optimal estimation sequence is obtained by iteration and decision. The specific process of HAWOA-CMA is shown in Algorithm 1. The schematic flowchart of HAWOA-CMA is shown in Figure 4.

Algorithm 1. The process of HAWOA-CMA algorithm

1: Initializing the step size ${\mu }_{HAWOA-CMA}$

2: Initializing the whale position and using the MSE of CMA to initialize the cost function of HAWOA

3: Obtaining the weight vector with minimum fitness by taking the iteration calculating of HAWOA

4: Defining the weight vector obtained from HAWOA as the initial weight vector $w_{HAWOA-CMA}$ of HAWOA-CMA

5: Using the Equation (21) to calculate the output sequence $y(n)$ of equalizer

6: Using the Equation (23) to calculate the error

7: Using the Equation (26) to update the weight $w_{HAWOA-CMA}$

8: Calculating the MSE

9: Determine whether the end condition has been met, if so, proceed to step 10, if not, return to step 4

10: End

4. Experiment and Discussion

4.1. Validity Analysis

To demonstrate the suitability of the HAWOA for addressing the constant modulus blind equalization problem, this section analyzes the advantages of HAWOA compared to other algorithms from the perspective of test functions. The effectiveness of the algorithm is further analyzed through test data. Therefore, an analysis of the cost function to CMA is conducted, and experiments are designed to verify the applicability and validity of the cost function. In this section, the popular optimization algorithms were selected for comparison, including the Harris hawks optimization algorithm (HHO), butterfly optimization algorithm [43] (BOA), seagull optimization algorithm [44] (SOA), Aquila optimizer (AO), arithmetic optimization algorithm (AOA) and WOA. The minimum distribution of the CMA cost function is shown in Figure 5.

Based on the cost function and the distribution of global minima in the CMA, it can be observed that the distribution of global minima is similar to the bowl-shaped, plate-shaped and valley-shaped functions. Therefore, this subset of test functions are employed to conduct experiments on HAWOA, aiming to validate its effectiveness and applicability further. Detailed information on the test functions is presented in

Table 1. The bowl-shaped, plate-shaped and valley-shaped test functions.

No.	Functions	Function Name	Range/Dim	Fmin
$F_1$	$F(x)=x_1{}^2+2x_2{}^2-0.3\cos (3\pi x_1)-0.4\cos (4\pi x_2)+0.7$	Bohachevsky	[−100,100]/2	0
$F_2$	$F(x)={\displaystyle \sum _{i=1}^{\dim }{\left({\displaystyle \sum _{j=1}^{\dim }(j+\beta )(x_j{}^2-\frac{1}{j^i})}\right)}^2}$	Perm	[−dim,dim]/30	0
$F_3$	$F(x)={\displaystyle \sum _{i=1}^{\dim }{\displaystyle \sum _{j=1}^i{x}_j{}^2}}$	Rotated Hyper-Ellipsoid	[−65.536,65.536]/30	0
$F_4$	$F(x)={\displaystyle \sum _{i=1}^{\dim }{\left|x_i\right|}^{i+1}}$	Sum of Different Powers	[−1,1]/3	0
$F_5$	$F(x)={\displaystyle \sum _{i=1}^{\dim }i{x}_i{}^2}$	Sum Squares	[−10,10]/30	0
$F_6$	$F(x)={\displaystyle \sum _{i=1}^{\dim }{\left(x_i-1\right)}^2-{\displaystyle \sum _{i=2}^{\dim }{x}_i{x}_{i-1}}}$	Trid	[−dim2,dim2]/30	−dim(dim + 4)(dim − 1)/6
$F_7$	$F(x)={\left(x_1+2x_2-7\right)}^2+{\left(2x_1+x_2-5\right)}^2$	Booth	[−10,10]/2	0
$F_8$	$F(x)=0.26\left(x_1{}^2+x_2{}^2\right)-0.48x_1{x}_2$	Malyas	[−10,10]/2	0
$F_9$	$F(x)=\sin \left(x_1+x_2\right)+{\left(x_1-x_2\right)}^2-1.5x_1+2.5x_2+1$	McCormick	[−1.5,4]/2	−1.9133
$F_{10}$	$F(x)={\displaystyle \sum _{i=1}^{\dim }{\left[\left({\displaystyle \sum _{j=1}^{\dim }{x}_j{}^i}\right)-b_i\right]}^2},b=\left(8,18,44,114\right)$	Power sum	[0,dim]/4	0
$F_{11}$	$F(x)={\displaystyle \sum _{i=1}^{\dim }{x}_i{}^2}+{\left({\displaystyle \sum _{i=1}^{\dim }0.5i{x}_i}\right)}^2+{\left({\displaystyle \sum _{i=1}^{\dim }0.5i{x}_i}\right)}^4$	Zakharov	[−5,10]/30	0
$F_{12}$	$F(x)=2x_1{}^2-1.05x_1{}^4+\frac{x_1{}^6}{6}+x_1{x}_2+x_2{}^2$	Three-hump camel	[−5,5]/2	0
$F_{13}$	$F(x)=\left(4-2.1x_1{}^2+\frac{x_1{}^4}{3}\right)x_1{}^2+x_1{x}_2+\left(-4+4x_2{}^2\right)x_2{}^2$	Six-hump camel	[−3,3]/2	−1.0316
$F_{14}$	$F(x)={\left(x_1-1\right)}^2+{\displaystyle \sum _{i=2}^{\dim }i{\left(2x_i{}^2-x_{i-1}\right)}^2}$	Dixon-price	[−10,10]/30	0
$F_{15}$	$F(x)={\displaystyle \sum _{i=1}^{\dim -1}\left[100{\left(x_{i+1}-x_i{}^2\right)}^2+{\left(x_i-1\right)}^2\right]}$	Rosenbrock	[−5,10]/30	0

.

Where the Range and dim respectively represent the definition domain and dimension. The shape of function is shown in Figure 6.

Where $F_1~F_6$ are the bowl-shaped functions, $F_7~F_{11}$ are the plate-shaped functions and $F_{12}~F_{15}$ are the valley-shaped functions. The results of HAWOA and other comparison algorithms on $F_1~F_{15}$ are shown in

Table 2. The results on bowl-shaped, plate-shaped and valley-shaped functions.

Function	Measure	Algorithms
		HAWOA	WOA	AOA	AO	BOA	HHO	SOA
F1	Mean	0	0	0	0	1.24 × 10−8	0	0
	Std	0	0	0	0	6.78 × 10−8	0	0
	Best	0	0	0	0	3.24 × 10−14	0	0
	Worst	0	0	0	0	3.71 × 10−7	0	0
	Rank	1	1	1	1	2	1	1
F2	Mean	9.05 × 102	1.03 × 103	1.23 × 103	1.49 × 103	1.04 × 1089	2.48 × 103	5.17 × 102
	Std	5.99 × 102	9.82 × 102	5.74 × 102	7.06 × 102	1.70 × 1089	9.11 × 102	4.28 × 102
	Best	1.57 × 10	6.11 × 10	2.06 × 102	4.36 × 102	7.71 × 1085	7.64 × 102	5.66 × 10
	Worst	1.87 × 103	3.33 × 103	1.92 × 103	2.95 × 103	7.80 × 1089	3.66 × 103	1.57 × 103
	Rank	2	3	4	5	7	6	1
F3	Mean	0	3.02 × 10−73	1.26 × 10−20	4.32 × 10−100	1.35 × 10−11	2.90 × 10−96	4.67 × 10−11
	Std	0	1.26 × 10−72	6.88 × 10−20	2.22 × 10−99	1.01 × 10−12	1.41 × 10−95	1.60 × 10−10
	Best	0	4.99 × 10−90	5.36 × 10−252	1.87 × 10−162	1.13 × 10−11	7.89 × 10−111	2.36 × 10−13
	Worst	0	6.76 × 10−72	3.77 × 10−19	1.21 × 10−98	1.50 × 10−11	7.71 × 10−95	8.81 × 10−10
	Rank	1	4	5	2	6	3	7
F4	Mean	0	1.75 × 10−106	0	3.47 × 10−26	9.41 × 10−14	2.92 × 10−117	2.51 × 10−43
	Std	0	9.54 × 10−106	0	1.90 × 10−25	1.06 × 10−13	1.60 × 10−116	9.71 × 10−43
	Best	0	1.21 × 10−135	0	8.19 × 10−107	3.44 × 10−15	1.08 × 10−145	5.18 × 10−54
	Worst	0	5.22 × 10−105	0	1.04 × 10−24	3.51 × 10−13	8.77 × 10−116	4.53 × 10−42
	Rank	1	3	1	5	6	2	4
F5	Mean	0	5.11 × 10−74	0	3.49 × 10−106	1.18 × 10−11	6.09 × 10−100	7.07 × 10−13
	Std	0	1.57 × 10−73	0	1.91 × 10−105	9.33 × 10−13	2.45 × 10−99	1.31 × 10−12
	Best	0	3.99 × 10−86	0	1.86 × 10−164	1.02 × 10−11	2.88 × 10−117	7.58 × 10−16
	Worst	0	6.40 × 10−73	0	1.05 × 10−104	1.43 × 10−11	1.28 × 10−98	4.20 × 10−12
	Rank	1	4	1	2	6	3	5
F6	Mean	−2.99 × 103	−2.46 × 103	2.01 × 106	−2.95 × 103	1.92 × 10	−3.55 × 103	−2.22 × 10
	Std	5.69 × 102	4.16 × 102	8.34 × 105	2.20 × 102	6.67	1.39 × 102	1.92 × 10
	Best	−4.22 × 103	−3.10 × 103	1.07 × 105	−3.42 × 103	−2.46	−3.83 × 103	−8.03 × 10
	Worst	−1.97 × 103	−1.12 × 103	3.02 × 106	−2.59 × 103	2.77 × 10	−3.28 × 103	5.93
	Rank	2	4	7	3	6	1	5
F7	Mean	2.14 × 10−1	1.40 × 10−3	5.46 × 10−7	2.80 × 10−4	3.17 × 10−1	4.52 × 10−5	4.98 × 10−5
	Std	5.78 × 10−1	1.70 × 10−3	3.86 × 10−7	3.13 × 10−4	5.73 × 10−1	6.14 × 10−5	5.47 × 10−5
	Best	1.32 × 10−7	1.66 × 10−5	7.87 × 10−9	2.37 × 10−6	1.70 × 10−3	1.00 × 10−8	2.44 × 10−7
	Worst	2.54	6.80 × 10−3	1.59 × 10−6	1.30 × 10−3	2.64	2.04 × 10−4	2.50 × 10−4
	Rank	6	5	1	4	7	2	3
F8	Mean	0	1.77 × 10−191	0	7.23 × 10−106	9.49 × 10−13	8.07 × 10−124	3.74 × 10−76
	Std	0	0	0	3.96 × 10−105	2.72 × 10−13	4.33 × 10−123	1.26 × 10−75
	Best	0	3.75 × 10−222	0	8.54 × 10−162	5.18 × 10−13	8.17 × 10−162	2.14 × 10−97
	Worst	0	5.30 × 10−190	0	2.17 × 10−104	1.58 × 10−12	2.37 × 10−122	5.05 × 10−75
	Rank	1	2	1	4	6	3	5
F9	Mean	−1.91	−1.91	−1.91	−1.91	−1.70	−1.91	−1.91
	Std	5.56 × 10−6	1.30 × 10−15	1.10 × 10−5	1.17 × 10−5	1.97 × 10−1	0	9.59 × 10−9
	Best	−1.91	−1.91	−1.91	−1.91	−1.22	−1.91	−1.91
	Worst	−1.91	−1.91	−1.91	−1.91	−1.91	−1.91	−1.91
	Rank	1	1	1	1	2	1	1
F10	Mean	1.13 × 10	5.38 × 10	8.31	2.70 × 10−1	2.09	1.33 × 10−1	2.68 × 103
	Std	1.05 × 10	6.21 × 10	7.98	2.24 × 10−1	1.85	1.36 × 10−1	5.75 × 103
	Best	3.82 × 10−1	4.55 × 10−1	1.04 × 10−1	1.00 × 10−2	3.47 × 10−2	1.70 × 10−3	1.53 × 102
	Worst	3.60 × 10	1.53 × 102	2.51 × 10	9.74 × 10−1	8.79	5.80 × 10−1	1.53 × 104
	Rank	5	6	4	2	3	1	7
F11	Mean	2.24 × 10−264	5.41 × 102	3.04 × 102	2.68 × 10−75	1.10 × 10−11	2.66 × 10−45	8.85 × 10−6
	Std	0	1.18 × 102	6.65 × 10	1.42 × 10−74	8.60 × 10−13	1.46 × 10−44	2.40 × 10−5
	Best	0	3.12 × 102	1.72 × 102	2.38 × 10−129	8.78 × 10−12	7.36 × 10−87	1.27 × 10−8
	Worst	6.41 × 10−245	8.15 × 102	4.42 × 102	7.79 × 10−74	1.27 × 10−11	7.98 × 10−44	1.07 × 10−4
	Rank	1	7	6	2	4	3	5
F12	Mean	0	1.40 × 10−77	0	3.96 × 10−109	7.38 × 10−14	1.01 × 10−104	1.96 × 10−99
	Std	0	7.33 × 10−77	0	2.17 × 10−108	3.47 × 10−14	5.50 × 10−104	9.74 × 10−99
	Best	0	4.66 × 10−106	0	2.22 × 10−161	9.10 × 10−15	4.89 × 10−123	8.57 × 10−125
	Worst	0	4.02 × 10−76	0	1.19 × 10−107	1.69 × 10−13	3.01 × 10−103	5.33 × 10−98
	Rank	1	5	1	2	6	3	4
F13	Mean	−1.03	−1.03	−1.03	−1.03	−1.09	−1.03	−1.03
	Std	8.48 × 10−7	1.51 × 10−9	4.51 × 10−8	4.06 × 10−4	2.19 × 10−1	1.87 × 10−9	4.16 × 10−6
	Best	−1.03	−1.03	−1.03	−1.03	−1.05	−1.03	−1.03
	Worst	−1.03	−1.03	−1.03	−1.03	−1.15	−1.03	−1.03
	Rank	1	1	1	2	3	1	1
F14	Mean	6.72 × 10−1	6.67 × 10−1	6.67 × 10−1	2.50 × 10−1	9.84 × 10−1	2.49 × 10−1	6.67 × 10−1
	Std	2.00 × 10−3	1.50 × 10−3	3.31 × 10−8	4.39 × 10−4	6.80 × 10−3	4.00 × 10−3	1.83 × 10−5
	Best	6.69 × 10−1	6.67 × 10−1	6.67 × 10−1	2.49 × 10−1	9.65 × 10−1	2.28 × 10−1	6.67 × 10−1
	Worst	6.76 × 10−1	6.75 × 10−1	6.67 × 10−1	2.51 × 10−1	9.95 × 10−1	2.52 × 10−1	6.67 × 10−1
	Rank	5	4	3	2	6	1	3
F15	Mean	7.00	2.79 × 10	3.52 × 104	5.90 × 10−3	2.88 × 10	9.10 × 10−3	2.83 × 10
	Std	8.90	4.76 × 10−1	2.52 × 104	1.43 × 10−2	2.58 × 10−2	1.21 × 10−2	5.14 × 10−1
	Best	3.50 × 10−3	2.70 × 10	7.40 × 103	2.24 × 10−6	2.87 × 10	1.51 × 10−4	2.72 × 10
	Worst	2.87 × 10	2.87 × 10	1.37 × 105	6.66 × 10−2	2.88 × 10	4.21 × 10−2	2.89 × 10
	Rank	3	4	7	1	6	2	5
Mean	Rank	2.13	3.6	2.93	2.53	5.07	2.2	3.8
Final	Ranking	1	5	4	3	7	2	6

Where the bold black represents the optimal average value, and Rank represents the ranking of the average value from the smallest to the largest. In the lower part of the table, the Mean indicates the average ranking of the algorithms among the 15 test functions, while the Final represents the final ranking of the algorithms. The convergence curves are shown in Figure 7.

.

As listed in Table 2, it can be observed that HAWOA has the highest overall ranking, while WOA has the fifth overall ranking. In the case of bowl-shaped, plate-shaped and valley-shaped test functions, HAWOA shows a certain superiority over other comparative algorithms. It has been verified experimentally that the improvement strategies employed in HAWOA effectively enhance the search performance of the WOA.

Figure 7 depicts the convergence curves of HAWOA and other comparative algorithms when tested on $F_1~F_{15}$. According to the comparison, it is evident that the curve of HAWOA exhibits a lower slope, allowing it to achieve smaller optimization values within a shorter number of iterations. In function F1, HAWOA demonstrates significantly faster convergence speed than other algorithms, proving its excellent optimization performance in dealing with bowl-shaped cost functions, which are characterized by a distribution of local minima in a bowl shape. Furthermore, HAWOA maintains its leading position in terms of convergence speed when tested on plate-shaped and valley-shaped functions. Combining the convergence ranking results in Table 2, HAWOA outperforms the comparative algorithms regarding overall optimization performance. This experiment tests the optimization performance of HAWOA using bowl-shaped, plate-shaped and valley-shaped functions, which bear similarities to the cost function in constant modulus blind equalization problems. These results indirectly demonstrate the promising performance of HAWOA in solving blind equalization problems.

4.2. Test and Analysis of HAWOA-CMA

In this section, 16QAM and 64QAM were used to test the HAWOA-CMA system. Five hundred Monte Carlo experiments were conducted, respectively. CMA and WOA-CMA were used as comparison objects for simulation. The channel parameters are h = [0.9656, −0.0906, 0.0578, 0.2368]; the signal-to-noise ratio was 20 dB; and the length of the equalizer weight was 16. The initial weight of the CMA is set by experience, with the eighth tap coefficient set to 1 and the remaining tap coefficient set to 0. The input weights of WOA-CMA and HAWOA-CMA are set as random values between [−1,1].

4.2.1. The Sending Sequence Is 16QAM

When the HAWOA-CMA system sends 16QAM signals with a sequence of 10,000 points, the step size is ${\mu }_{CMA}=0.000005$, ${\mu }_{WOA}=0.000001$ and ${\mu }_{HAWOA-CMA}=0.000001$, respectively. WOA and HAWOA populations were 50, and the maximum number of iterations was 50. The experimental results are shown in Figure 8.

In Figure 8, Figure 8a is the input sequence constellation diagram of the equalizer. The red point is the phase point of the 16QAM signal. Due to the influence of channel and noise, the input signal becomes disordered and divergent, which does not conform to the optimal constellation structure of the 16QAM signal. Figure 8b is the signal constellation diagram after equalization by the constant modulus algorithm. The balanced points can be close to the optimal constellation structure of 16QAM on a macro level. However, many divergence points have a larger amplitude, and its equilibrium performs imperfectly. Figure 8c,d are signal constellation figures after WOA-CMA and HAWOA-CMA equalization. It should be noted that two balanced signal points converge to the optimal constellation structure of 16QAM, and there are fewer divergence points and no points with larger amplitude compared to Figure 8b. This difference is because the CMA has a slow convergence in the equalization process, and it is difficult to obtain the equalizer weight suitable for the unknown channel in a short iteration. Before getting the suitable equalizer weight, the equalization performs poorly, so some diverging points with large amplitude will be generated. In other words, WOA-CMA and HAWOA-CMA can quickly obtain better equalizer weights, and the equalization effect is better than CMA. The MSE is shown in Figure 9.

Where the abscissa represents the number of iteration points of the blind equalization algorithm, and the ordinate is the size of MSE. The initial MSE of WOA-CMA and HAWOA-CMA is smaller than CMA, and HAWOA-CMA has a smaller number of iteration points when reaching the same convergence value. Compared with the CMA and WOA-CMA, it can reduce nearly 1000 and 300 iterations under the same mean square error condition. It can be proven that HAWOA-CMA has a faster convergence rate and can get a convergence value in the shortest time.

WOA-CMA and HAWOA-CMA have substantial advantages over CMA in equalization effect. However, using a swarm intelligence algorithm to optimize blind equalization will bring some problems, such as increased computation and longer computation time. Therefore, reducing the number of iterations of the swarm intelligence algorithm to achieve the same equalization effect becomes the key to effectively reducing the computation amount and time. As mentioned in Section 4.1, HAWOA has a faster convergence speed and better accuracy than WOA. To reflect the superiority of HAWOA in blind equalization, the number of iterations obtained by the algorithm in the optimization process of WOA-CMA and HAWOA-CMA initialization weights was recorded. Figure 10 shows the boxplot of the number of iterations when WOA and HAWOA find the optimal solution in 500 Monte Carlo experiments.

According to Figure 10, the ordinate is the iterations of the algorithm. According to the analysis of boxplot properties [45], the iterations of HAWOA-CMA are generally small, and the upper limit and lower limit are lower than WOA-CMA. The extreme outliers are less than WOA-CMA, indicating that HAWOA-CMA has more substantial stability and better performance than WOA-CMA. The average iterations of WOA-CMA and HAWOA-CMA were 20.57 and 11.12, respectively. Therefore, it can be concluded that HAWOA-CMA uses fewer iterations, which can effectively reduce the calculation amount and time compared with WOA-CMA.

The Figure 11 below illustrates the comparison of the symbol error rate (SER) of the model under different signal-to-noise ratio (SNR) conditions. It can be observed that both WOA-CMA and HAWOA-CMA have lower BER than CMA under the same SNR. When the SNR is 20 dB, the SER of the WOA-CMA system is approximately −35 dB, while the SER of the HAWOA-CMA system is around −40 dB. The difference between the two is approximately 5 dB, indicating that the blind equalization system incorporating HAWOA has a lower SER compared to the system incorporating WOA. Furthermore, when the SNR is 25 dB and 30 dB, both WOA-CMA and HAWOA-CMA systems exhibit a linear SER value of 0 while the SER of the CMA system remains between −20 dB and −25 dB. This further confirms that the model effectively reduces the SER of the traditional CMA system in the channel equalization process. Additionally, the HAWOA-CMA model requires a lower SNR to achieve the same SER, making it more suitable for wireless communication environments with higher noise power.

4.2.2. The Sending Sequence Is 64QAM

When the HAWOA-CMA system sends 64QAM signals with a sequence of 10,000 points, the step size is ${\mu }_{CMA}=0.0000001$, ${\mu }_{WOA}=0.0000001$ and ${\mu }_{HAWOA-CMA}=0.0000001$, respectively. WOA and HAWOA populations were 50, and the maximum number of iterations was 50.

In Figure 12, Figure 12a is the input sequence constellation diagram of the equalizer. The red point is the phase point of the 64QAM signal. Figure 12b is the signal constellation diagram after equalization by the constant modulus algorithm. It is worth mentioning here that the balanced points can be close to the optimal constellation structure of 64QAM on a macro level. However, there are still more divergence points and some points have larger amplitude. Figure 12c,d are signal constellation figures after WOA-CMA and HAWOA-CMA equalization. Meanwhile, the two balanced signal points converge to the optimal constellation structure of 64QAM, and there are fewer divergence points and no points with larger amplitude compared to Figure 12b. It is further proven that WOA-CMA and HAWOA-CMA can quickly obtain appropriate equalizer weights, and the equalization effect is very competitive compared to CMA. Its mean square error is shown in Figure 13.

It can be seen from Figure 13 that HAWOA-CMA has a smaller number of iteration points when reaching the same convergence value, maintaining the fastest convergence rate. Compared with the CMA and WOA-CMA, it can reduce nearly 1500 and 300 iterations under the same mean square error condition. It is proven that HAWOA-CMA still has an excellent balancing performance when the input signal is 64QAM. Figure 14 shows the boxplot of the iterations of WOA and HAWOA in the 64QAM signal.

As depicted in Figure 14, the iterations of HAWOA-CMA are relatively small, and the upper limit and lower limit are lower than WOA-CMA. The extreme outliers were far less than WOA-CMA. The average number of iterations was 20.21 for WOA-CMA and 10.38 for HAWOA-CMA. The figure proves that HAWOA-CMA has a faster optimization speed and notably outperforms WOA-CMA.

In Figure 15, similarly, when the input signal of the equalization system is 64QAM, both WOA-CMA and HAWOA-CMA exhibit lower SER than CMA under different SNR conditions, with HAWOA-CMA demonstrating superior SER performance compared to CMA. When the SNR is 25 dB, the SER of HAWOA-CMA is lower than that of WOA-CMA, with a difference of approximately 3 dB. Combining the analysis of MSE, HAWOA-CMA achieves faster convergence speed while maintaining preferable SER performance. In contrast, the traditional CMA exhibits poor equalization performance and higher SER when the input is a higher-order QAM signal. Compared to CMA, the proposed model effectively reduces the SER of the equalization system in wireless communication, mitigating the impact of channel and noise to improve communication quality.

4.2.3. Result Analysis and Contribution of HAWOA-CMA

The QAM signal is commonly used in wireless communication systems due to its high data transmission rate, spectral efficiency and good noise resistance. To evaluate the performance of HAWOA-CMA in wireless communication systems, this study conducted simulation tests using both 16QAM and 64QAM signals. Based on the simulation data, it was observed that HAWOA-CMA demonstrated faster convergence, minor mean square error and lower bit error rate compared to the original method for both 16QAM and 64QAM input signals. These improvements indicate that HAWOA-CMA enhances the performance of the equalizer to a certain extent. In wireless communication systems, faster convergence improves real-time capability and resistance to interference while a lower bit error rate enhances communication quality and system reliability. The performance above enhancements signifies that the HAWOA-CMA model can improve the equalization effectiveness and elevate data transmission quality and system stability in wireless communication systems.

Additionally, this study combines channel equalization with metaheuristic algorithms from artificial intelligence, offering valuable insights into integrating wireless communication systems with artificial intelligence.

5. Conclusions

In this study, based on the idea of AI-assisted channel estimation, a hybrid arithmetic whale constant modulus blind equalization algorithm is proposed as an effective channel equalization method. By improving the blind equalization algorithm, the advantages in fast time-varying channels are given full play. The novelty of this research lies in using swarm intelligence algorithms with substantial population diversity to address practical channel equalization problems. By incorporating a hybrid approach, the deficiencies of the whale optimization algorithm are addressed, providing a viable approach to enhance the blind equalization systems in wireless channels through the integration of artificial intelligence. In the process of experiment, firstly, the new swarm intelligence algorithms in recent years are compared to illustrate the effectiveness of the HAWOA, and the applicability of the algorithm is verified according to the characteristics of the CMA cost function. Secondly, the simulation is carried out with the CMA and the WOA-CMA under the input signal of 16QAM and 64QAM, respectively. The results show that the HAWOA has a preferable optimization performance and obvious advantages in comparison. It can be regarded as an efficient optimization method for solving bowl-shaped optimization problems. In the channel equalization process, HAWOA-CMA has a faster convergence speed and better convergence accuracy than CMA and WOA-CMA. In the initial weight optimization process, HAWOA-CMA can obtain the weight with a smaller fitness value and fewer iterations. Compared with the classical method, HAWOA-CMA can converge quickly and effectively reduce the mean square error. The computation amount and time of the equilibrium system are reduced while keeping the time and space complexity unchanged. This method can perfectly improve communication efficiency under the time-varying channel and ensure a low SER, reduce bandwidth consumption and enhance the equalization effect.

This finding will facilitate further investigation into applying the swarm intelligence algorithm in blind equalization. The convergence speed of this model still needs to be improved, and the optimization method still has some stability problems. Future research can be aimed at using a more stable optimization method to solve the blind equalization problem. At the same time, applying the popular deep learning method to the optimization of the constant modulus blind equalization system also has a certain research value.