Multi-AUV Dynamic Maneuver Countermeasure Algorithm Based on Interval Information Game and Fractional-Order DE

Abstract

The instability of the underwater environment and underwater communication brings great challenges to the coordination and cooperation of the multi-Autonomous Underwater Vehicle (AUV). In this paper, a multi-AUV dynamic maneuver countermeasure algorithm is proposed based on the interval information game theory and fractional-order Differential Evolution (DE), in order to highlight the features of the underwater countermeasure. Firstly, an advantage function comprising the situation and energy efficiency advantages is proposed on account of the multi-AUV maneuver strategies. Then, the payoff matrix with interval information is established and the payment interval ranking is achieved based on relative entropy. Subsequently, the maneuver countermeasure model is presented along with the Nash equilibrium condition satisfying the interval information game. The fractional-order DE algorithm is applied for solving the established problem to determine the optimal strategy. Finally, the superiority of the proposed multi-AUV maneuver countermeasure algorithm is verified through an example.

1. Introduction

Autonomous underwater vehicles (AUVs) are being widely used for ocean observation, sea rescue, minefield search, enemy reconnaissance, and in other related fields [1,2]. Compared to manned equipment, AUVs present the advantages of low cost, zero casualty count, good concealment, strong mobility, low energy consumption, recyclability, etc. [3,4,5]. However, in the face of high efficiency and large-scale task requirements, a single AUV hardly fulfills all these demands. A multi-AUV system is a new alternative for complex ocean tasks due to its high efficiency and reliability, achieved through its time-space distribution and redundant configuration. In recent years, studies of multi-AUV cooperative formation and navigation research has made significant headway [6,7,8], but the multi-AUV cooperative counter-games are still limited. The multi-AUV cooperative countermeasure can be used in marine research and military tasks, including underwater multi-target tracking, monitoring, and detection, which effectively expands the underwater combat radius and reduces underwater equipment loss and casualties.

The maneuver countermeasure is a special case of maneuver decision-making, which can be roughly divided into two categories. One is the ‘one-sided’ optimization algorithm, which only considers one’s self strategy optimization. The other is the ‘two-sided’ game algorithm, which fully analyzes the influence of both sides and emphasizes the conflict and confrontation of the situation. The one-sided optimization algorithm always includes an intelligent algorithm, guidance law algorithm, and expert system [9]. The expert system-based maneuver decision-making algorithm presented in Ref. [10] is one of the earliest works. However, this method relies heavily on experience, the chosen strategy is not guaranteed to be optimal, and adaptability is not reliable. An air combat maneuver decision-making algorithm based on heuristic reinforcement learning is proposed in Ref. [11], which realizes the online task allocation of actual confrontation. In Ref. [12], a machine learning algorithm is used to train a large-scale cluster UAV, showcasing its high intelligence decision-making ability. This kind of one-sided optimization algorithm may achieve the desired results to a certain extent, but the optimization process is not completely objective because the influence of the opponent strategy is not considered, and the interaction and conflict of the game are ignored.

The matrix game and differential game are the most widely studied aspects regarding the two-sided counter-game algorithm [13]. In Ref. [14], the matrix game is first used to study the two UAVs counter-game, but its model and strategy are relatively simple and cannot be applied to actual confrontation. The integration of a game strategy solution to this model, based on intelligent algorithms, is proposed in Ref. [15], which improves the applicability of the former model. Ref. [16] discusses the maneuver counter-game of UAV clusters based on the intuitionistic fuzzy game, mapping the uncertainty of information such as a game environment to fuzzy membership, which is more aligned with the reality of a counter-game scenario. Furthermore, Ref. [17] examines the problem of an active defense cooperative differential game with three players, and Ref. [18] discusses the problem of a qualitative differential game in which multiple pursuers hunt for an advantage runner. Overall, the two-sided game algorithm could highlight the complex conflict of an unmanned system cluster countermeasure, which may realize the maneuver strategy in a more scientific and accurate way. However, owing to the complex marine environment [19], it is necessary to further explore the multi-stage dynamic countermeasure algorithm of multi-AUV systems with regard to underwater characteristics.

Classical game theory only discusses games with clear payment information [20]. However, most of the information in the marine environment is uncertain, which is the main problem in the counter-game process. If the uncertain information is directly converted to a certain value, the maneuver countermeasure algorithm may lose its credibility in the strategy selection. In this paper, this problem is solved by introducing the interval information into the cooperative dynamic maneuver countermeasure algorithm of the multi-AUV. The uncertainty in the multi-AUV counter-game, confrontation effectiveness, and the marine environment is regarded as an interval information series. Meanwhile, an interval payment multi-attribute evaluation of the multi-AUV maneuver strategy is carried out, and the interval payment matrix is obtained. Therefore, the uncertain information is well considered and settled in the proposed counter-game algorithm, which make this algorithm suitable to application in a marine environment. Moreover, the Nash equilibrium condition satisfying the interval payment is then proposed, and the countermeasure model of a Nash equilibrium strategy in a dynamic marine environment is established. To determine the optimal strategy, an improved fractional-order Differential Evolution (DE) algorithm is presented by a combination of DE algorithm and fractional calculus, which could lead to a raise of the convergence rate in the optimization process. Fractional calculus owns a long-memory ability and does not rely on current gradients [21]. Thus, the fractional-order DE contains great potential in avoiding slow convergence and local extremum.

Overall, the main contributions of the proposed dynamic maneuver countermeasure algorithm for multi-AUV can be summarized as follows. Firstly, the underwater environment and communication condition have been well presented in the countermeasure process using the interval information game; thus, the modeling of the dynamic maneuver counter-game of the multi-AUV is more accurate. Then, an improved fractional-order DE algorithm is proposed to improve the efficiency of achieving the optimal strategy of a real-time underwater countermeasure. The order of a fractional-order DE can affect the convergence rate and optimization error, which can also be tuned to satisfy different underwater requirements. But it should be noted that the standard of determining the optimal fractional order is not analytic or unified. The determining process needs more computation at the beginning of the proposed algorithm.

The remainder of this paper is organized as follows: Section 2 presents the preliminary of the fractional derivatives and DE algorithm; Section 3 presents the maneuver attributes, the advantage function, and the payoff matrix; Section 4 presents the payment ranking based on the four-parameter interval set, relative entropy, and the Nash equilibrium optimal solution; the fractional-order DE algorithm is presented in Section 5; Section 6 demonstrates the example of the multi-AUV counter-game; finally, the conclusions are stated in Section 7.

2. Preliminary

2.1. Fractional Derivatives

Fractional calculus is a generalization of the classical integer-order derivatives and integrals. The common definitions of fractional calculus are included in Riemann–Liouville (R-L) [22], Grunwald–Letnikov (G-L), and Caputo definitions [23,24], which have been applied in different fields, such as mathematics, engineering, computer science, etc. [25,26]. These definitions are introduced in the following:

Definition 1

([22]). The R-L fractional derivative of a continuous function $f:(0,+\infty )⟶\mathrm{R}$ with order $q>0$ is defined as:

$${}_0^R{D}_{t^{}}^{q}f\left(t\right)=\frac{1}{\mathsf{\Gamma }(k-q)}\frac{{\mathrm{d}}^k}{\mathrm{d}{\mathrm{t}}^k}{\int }_0^t\frac{f\left(s\right)}{{(t-s)}^{q-k+1}}\mathrm{ds},$$

(1)

where k is a positive integer and $k-1\le q\le k$, and $\mathsf{\Gamma }(·)$ is the gamma function, i.e.,

$$\mathsf{\Gamma }\left(x\right)={\int }_0^{+\infty }{t}^{x-1}{e}^{-t}\mathrm{d}t.$$

Definition 2

([27]). The Caputo fractional derivative of a continuous function $f:(0,+\infty )⟶\mathrm{R}$ with order $q>0$ is defined as:

$${}_0^C{D}_{t^{}}^{q}f\left(t\right)=\frac{1}{\mathsf{\Gamma }(k-q)}{\int }_0^t\frac{f^{\left(k\right)}\left(s\right)}{{(t-s)}^{q-k+1}}\mathrm{ds},$$

(2)

where k is an positive integer and $k-1\le q\le k$.

The G-L definition is a discretization definition, which is equivalent to the discretized R-L definition. When $0<q<1$ ($k=1$), the G-L definition or discretized R-L definition can be described by

$$\begin{array}{ccc}\hfill {}_0^{G,R}{D}_t^{q}f\left(t\right)& =& \frac{1}{h^q}\left(f\left(s_{n+1}\right)+\sum _{i=1}^{n+1}{(-1)}^i\left(\begin{array}{c}q\\ i\end{array}\right)f\left(s_{n+1-i}\right)\right),h\to 0^{+}.\hfill \end{array}$$

(3)

The Caputo definition owns the practical physical meaning in initial values and has more advantages than the R-L or G-L definition. The discretized Caputo definition is different from R-L and G-L ones, and satisfies the following relationship, i.e.,

$${}_0^C{D}_t^{q}f\left(t\right)={}_0^{G,R}{D}_t^{q}f\left(t\right)-\frac{t^{-q}}{\mathsf{\Gamma }(1-q)}{f}_0,0<q<1.$$

(4)

2.2. Differential Evolution Algorithm

The differential evolution (DE) algorithm is a kind of intelligent optimization method based on population difference inspiration. DE uses the difference between population individuals to produce a disturbance to individual evolution, and it also uses the greedy rule to search the whole optimization space to find the optimal solution. In other words, the population individuals are greedy for a better fitness, and will update to a solution with a better fitness, as shown in the following Equation (7). The update process of the population includes mutation, crossover, and selection, and finally finds the optimal solution. DE demonstrates the characteristics of a simple operation, high robustness, strong optimization ability, etc. [16]. The main population update algorithm in DE are introduced as follows:

(1) Mutation.

Mutation is the first step of the population update in DE. According to the DE/currentto-best strategy [28], the variation vector can be obtained as follows:

$$w_{i,g}=v_{i,g}+F_i(v_{best,g}-v_{i,g})+F_i(v_{r_1,g}-v_{r_2,g}),$$

(5)

where $w_{i,g}$ is the variation vector, $F_i$ is the scaling factor of the individual, $v_{i,g}$ represents the current individual vector, and $v_{best,g}$ represents the optimal individual of the population. $r_1$ and $r_2$ are two different integers, which are randomly selected from $1,2,\cdots ,NP$; $NP$ is the population. $v_{r_1,g}$ and $v_{r_2,g}$ represent the $r_1$th and $r_2$th individual vectors, respectively.

(2) Crossover.

Crossover is an exchange of each dimension vector between the mutated individual and the original individual. The crossover operation can be expressed as follows:

$$\begin{array}{c}\hfill u_{ij,g}=\left\{\begin{array}{c}\begin{array}{cc}{v}_{ij,g},& \begin{array}{ccc}if& rand[0, 1]<C{R}_i& \begin{array}{cc}or& j=j_{rand}\end{array}\end{array}\end{array}\hfill \\ \begin{array}{cc}{w}_{ij,g},& Otherwise\end{array}\hfill \end{array}\right.,\end{array}$$

(6)

where $u_{ij,g}$ is the jth components of the test vector $u_{i,g}$, $C{R}_i$ is the crossover rate, $j_{rand}$ is a random integer smaller than individual dimension D, and $rand[0, 1]$ is a random number between 0 and 1.

(3) Selection.

Selection operation is for choosing a better fitness between the newly generated test vector and the original target vector to become a member of the next population generation. It is a greedy selection operation. The selection operation can be described as follows:

$$\begin{array}{c}\hfill v_{i,g+1}=\left\{\begin{array}{c}\begin{array}{cc}{v}_{i,g},& f\left(u_{i,g}\right)<f\left(v_{i,g}\right)\end{array}\hfill \\ \begin{array}{cc}{u}_{i,g},& Otherwise\end{array}\hfill \end{array}\right.,\end{array}$$

(7)

where $v_{i,g+1}$ is the next generation individual.

3. Multi-AUV Maneuver Countermeasure Model

3.1. Maneuver Strategies

In order to establish the interval information payoff matrix, the evaluation of multi-AUV maneuver attributes is presented according to the situation information of different confronting sides. The counter-game trajectory of the multi-AUV is treated as a combination of each maneuver actions. The enemy multi-AUV and our multi-AUV systems are regarded as two players in the game.

The game model of multi-AUV systems based on uncertain information can be expressed as:

$$G=\left\{N,S,\stackrel{\sim }{U}\right\},$$

(8)

where $N=\left\{N_1, N_2\right\}$ are two players in the game, terminal 1 represents our multi-AUV system with a AUVs, and terminal 2 represents the enemy multi-AUV system with d AUVs; $S=\left\{s_{1i}^k,s_{2j}^k\right\}$ are the strategy spaces of game players, $s_{1i}^k$ represents that we choose the i-th maneuver strategy, and $s_{2j}^k$ represents the enemy choosing the j-th maneuver strategy at the k-th stage; $\stackrel{\sim }{U}=\left\{{\stackrel{\sim }{u}}_1\left(s^{k,i}\right),{\stackrel{\sim }{u}}_2\left(s^{k,j}\right)\right\}$ are the benefit intervals corresponding to each strategy that may be selected by the multi-AUV system participating in this game. The action of the multi-AUV system at the k-th stage game can be represented by an information set; thus, the maneuver strategy is actually the action rules of the multi-AUV system in each information set.

3.2. Advantage Function

The main difference between the multi-AUV counter-game and other autonomous robots’ counter-game is the information transmission. Due to the marine environment, information in the multi-AUV game process is mainly received through an underwater acoustic. The shallow water acoustic channel is a channel with time-space-frequency variation [19]. It has strong multi-path interference, high environmental noise, large transmission loss, and a serious Doppler shift effect [19]. Therefore, the information provided in the multi-AUV confrontation process has strong uncertainties. It is difficult to quantify the threat degree of each side accurately during the countermeasure process [29]. Hence, in this paper, each attribute is represented by an interval information set in the countermeasure process. The advantage evaluation function that can evaluate the payment of each AUV consists of two parts, including the situation advantage and energy efficiency advantage.

3.2.1. Situation Advantage Function

In order to attack the enemy multi-AUV system, it is necessary to occupy a favorable attack position and minimize the attack risk of our multi-AUV system. The situation advantage function consists of angle advantage, speed advantage, and distance advantage [30].

The angle advantage function $A_{ag}$ can be expressed as:

$$\begin{array}{c}\hfill A_{ag}=\left\{\frac{1-\left|AA\right|/{180}^{\circ }+\left|ATA\right|/{180}^{\circ }}{2}\right\},\end{array}$$

(9)

where $\left|AA\right|<{180}^{\circ }$ is the eye angle of both game players and $\left|ATA\right|<{180}^{\circ }$ is the target entry angle.

The speed advantage function $A_s$ is achieved as:

$$\begin{array}{c}\hfill A_s=\left\{\begin{array}{c}\begin{array}{cc}0.1,& v_{n_{1i}}\le 0.6v_{n_{2j}}\end{array}\hfill \\ \begin{array}{cc}0.5+\frac{v_{n_{1i}}}{v_{n_{2j}}},& 0.6v_{n_{2j}}<v_{n_{1i}}<1.5v_{n_{2j}}\end{array}\hfill \\ \begin{array}{cc}1,& v_{n_{1i}}\ge 1.5v_{n_{2j}}\end{array}\hfill \end{array}\right.,\end{array}$$

(10)

where $v_{n_{1i}}, v_{n_{2j}}$ are the speed vectors of $n_{1i}$ and $n_{2j}$ in the game. The above Equation (10) indicates that a higher attack speed will lead to a greater attack advantage. According to Ref. [30], when the speed ratio $\frac{v_{n_{1i}}}{v_{n_{2j}}}$ is larger than 1.5, $n_{1i}$ owns an absolute speed advantage. In addition, when the speed ratio $\frac{v_{n_{1i}}}{v_{n_{2j}}}$ is less than 0.6, the speed advantage of $n_{1i}$ is negligible.

Then, the distance advantage $A_{dis}$ can be obtained as:

$$\begin{array}{c}\hfill A_{dis}=e^{-{\left(\left(D_{ij}-R_0\right)/\left(R_{max}-R_{min}\right)\right)}^2},\end{array}$$

(11)

where $D_{ij}$ is the distance between different AUVs; $R_0=(R_{max}+R_{min})/2$, $R_{max}$ is the maximum attack distance and $R_{min}$ is the minimum attack distance. When $D_{ij}>R_{max}$, the distance advantage is considered to be zero; with the decrease in distance, the distance advantage increases gradually; when $D_{ij}=R_0$, the distance advantage reaches the maximum value; with the further reduction in distance, the distance advantage gradually decreases.

Hence, the overall situation advantage function $W_A$ can be achieved by:

$$\begin{array}{c}\hfill W_A=k_1{A}_{ag}+k_2{A}_s+k_3{A}_{dis},\end{array}$$

(12)

where $k_1,k_2,k_3$ are weighting coefficients, and $k_1+k_2+k_3=1$.

3.2.2. Energy Efficiency Advantage Function

The energy efficiency advantage C is mainly measured by seven factors, including maneuverability B, firepower P, target detection capability T, manipulating ability ${\epsilon }_1$, viability ${\epsilon }_2$, voyage ${\epsilon }_3$, and confrontation ability ${\epsilon }_4$ [30]:

$$\begin{array}{c}\hfill C=\left[lnB+ln\left(\sum P+1\right)+ln\left(\sum T+1\right)\right]{\epsilon }_1{\epsilon }_2{\epsilon }_3{\epsilon }_4.\end{array}$$

(13)

The energy efficiency advantage function in Equation (13) is quite different from the situation advantage function. In order to simplify the total advantage function, the energy efficiency advantage function is transformed into the following form according to the actual situation:

$$\begin{array}{c}\hfill W_C=\left\{\begin{array}{c}\begin{array}{cc}0,& C_{N_{1a}}/C_{N_{2d}}<0.3\end{array}\hfill \\ \begin{array}{cc}0.25,& 0.3\le C_{N_{1a}}/C_{N_{2d}}<1\end{array}\hfill \\ \begin{array}{cc}0.5,& C_{N_{1a}}/C_{N_{2d}}=1\end{array}\hfill \\ \begin{array}{cc}0.75,& 1\le C_{N_{1a}}/C_{N_{2d}}<1.5\end{array}\hfill \\ \begin{array}{cc}1,& C_{N_{1a}}/C_{N_{2d}}\ge 1.5\end{array}\hfill \end{array}\right.,\end{array}$$

(14)

where $C_{N_{1a}},C_{N_{2d}}$ are the energy efficiency of AUVs from different sides. According to Ref. [30], the energy efficiency advantage function owns five steps depended on by the energy efficiency ratio $C_{N_{1a}}/C_{N_{2d}}$ shown in Equation (14), which indicates that a higher energy efficiency could provide a greater advantage.

3.3. Payoff Matrix Based on Interval Information

Payment in the game refers to the final gain or loss of players in the strategic choice. In the multi-AUV counter-game, the income of our AUV must be the loss of the enemy’s AUV. Therefore, the game in this paper belongs to the category of the two-person-zero-sum game.

Due to the abovementioned various underwater interference factors, the multi-AUV system is often unable to obtain all kinds of information accurately in an actual underwater maneuver countermeasure. After a reasonable analysis of the counter-game situation, each interference factor often changes within a certain interval. Therefore, the payoff matrix of each multi-AUV system is established based on interval information.

According to the situation and energy efficiency advantage functions proposed in the last section, the overall advantage function of our multi-AUV system, including situation advantage and energy efficiency advantage, can be obtained as:

$$\begin{array}{c}\hfill {\stackrel{\sim }{W}}_1={\delta }_1{\stackrel{\sim }{W}}_A+{\delta }_2{\stackrel{\sim }{W}}_C,\end{array}$$

(15)

where ${\delta }_1,{\delta }_2$ are weighting coefficients, and ${\delta }_1+{\delta }_2=1$; $\stackrel{\sim }{W}=\left[W^L,W^R\right]$, ${\stackrel{\sim }{W}}_A=\left[{W_A}^L,{W_A}^R\right]$, ${\stackrel{\sim }{W}}_C=\left[{W_C}^L,{W_C}^R\right]$ represent the advantage functions with upper and lower boundaries.

In the same way, the overall advantage function ${\stackrel{\sim }{W}}_2$ of the enemy side in the game can be achieved by exchanging the situation information parameters of both sides.

The payment function of the multi-AUV game under uncertain information is established as:

$$\begin{array}{c}\hfill \stackrel{\sim }{f}=\sum _{i=1}^m{x}_{ij}\stackrel{\sim }{W_1}-\sum _{j=1}^n{y}_{ji}\stackrel{\sim }{W_2},\end{array}$$

(16)

where $x_{ij},y_{ji}$ are binary decision variables, $x_{ij}=1$ represents our ith AUV attacks the jth AUV of the enemy, and $x_{ij}=0$ means our ith AUV does not attack the jth AUV of the enemy; in the same way, $y_{ji}$ represents whether the jth AUV of the enemy attacks our ith AUV or not.

Then, the payment matrix under uncertain information is obtained as:

$$\begin{array}{ccc}\hfill \stackrel{\sim }{F}& =\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_m\end{array}\stackrel{\begin{array}{cccc}\begin{array}{c}{d}_1\end{array}& \begin{array}{c}{d}_2\end{array}& \begin{array}{c}\cdots \end{array}& \begin{array}{c}{d}_n\end{array}\end{array}}{\left[\begin{array}{cccc}\stackrel{\sim }{f_{11}}& \stackrel{\sim }{f_{12}}& \cdots & \stackrel{\sim }{f_{1n}}\\ \stackrel{\sim }{f_{21}}& \stackrel{\sim }{f_{22}}& \cdots & \stackrel{\sim }{f_{2n}}\\ ⋮& ⋮& \ddots & ⋮\\ \stackrel{\sim }{f_{m1}}& \stackrel{\sim }{f_{m2}}& \cdots & \stackrel{\sim }{f_{mn}}\end{array}\right]}\hfill & \hfill =\left[\begin{array}{cccc}\left[f_{11}^L,f_{11}^R\right]& \left[f_{12}^L,f_{12}^R\right]& \cdots & \left[f_{1n}^L,f_{1n}^R\right]\\ \left[f_{21}^L,f_{21}^R\right]& \left[f_{22}^L,f_{22}^R\right]& \cdots & \left[f_{2n}^L,f_{2n}^R\right]\\ ⋮& ⋮& \ddots & ⋮\\ \left[f_{m1}^L,f_{m1}^R\right]& \left[f_{m2}^L,f_{m2}^R\right]& \cdots & \left[f_{mn}^L,f_{mn}^R\right]\end{array}\right],\end{array}$$

(17)

where $a_1,a_2,\dots ,a_m$ are the maneuvering strategies of our multi-AUV system; $d_1,d_2,\dots ,d_n$ are the maneuver strategies of the enemy multi-AUV system; $\stackrel{\sim }{u_{mn}}$ represents the gain when our multi-AUV system uses the m-th strategy, and the enemy multi-UUV system uses the n-th strategy.

4. Optimal Solution of Multi-AUV Dynamic Maneuver Countermeasure

4.1. Payment Interval Ranking Based on Relative Entropy

For the comparison of interval information sets, one cannot compare the size from the perspective of quantity, such as real numbers. The sorting method based on the possibility degree has the possibility of failure and that based on geometric distance may cause serious information loss [31]. In order to avoid these shortcomings, a method combining the four parameter interval set and relative entropy is proposed in this paper.

According to the interval information given in the last subsection, the payment interval $\left[f_{mn}^L,f_{mn}^R\right]$ is obtained from the comprehensive information of both sides in the game. However, the payments do not consider the distribution of points in the interval. In fact, the interior point of payment interval set cannot be simply regarded as uniform distribution. It should change according to the change of the underwater game situation. For one strategy $x_i$, when the confrontation situation is favorable to the attacker, the payment of the attacker tends to $f^R$ inevitably; if not, it will tend to $f^L$. In order to fully explore the information of the advantage matrix, the payment interval is transformed into four-parameter interval sets in this subsection.

The four-parameter interval set is similar to the trapezoid fuzzy set [32]. Combined with the advantage function in the last section, the payment interval $\left[f_{mn}^L,f_{mn}^R\right]$ is transformed into a four-parameter interval set $\left[f_{mn}^L,f_{mn}^{ML},f_{mn}^{MR},f_{mn}^R\right]$, where $f_{mn}^{ML}=f_{mn}^L+W_{mn}^R(f_{mn}^R-f_{mn}^L)$, $f_{mn}^{MR}=f_{mn}^L+W_{mn}^L(f_{mn}^R-f_{mn}^L)$, and $W_{mn}$ is the normalized advantage function with $0<W_{mn}^L<W_{mn}^R<1$.

The basic idea of this ranking method is using information entropy to measure the difference between AUV’s own revenue and the maximum revenue (minimum revenue) under different strategies, and to choose the strategy with the smallest difference between AUV’s own revenue and the maximum revenue (or the strategy with the largest difference between AUV’s own revenue and minimum revenue). In fact, the highest return indicates that the AUV has completed the scheduled task without casualties; the lowest return indicates that the AUV has failed to complete the scheduled task with the largest casualties.

First, the Kullback–Leibler distance concept is introduced. For two systems P and Q, the relative information entropy for them in state $P_i$ and $Q_i$ can be expressed as [33]:

$$\begin{array}{c}\hfill M_i=P_i{log}_2^{P_i/Q_i}+(1-P_i){log}_2^{(1-P_i)/(1-Q_i)},\end{array}$$

(18)

where the unit of $M_i$ is the bit.

In order to overcome the meaninglessness of Equation (18) when $Q_i=0$ or $Q_i=1$, the relative information entropy is improved as [33]:

$$\begin{array}{c}\hfill H_i=P_i{log}_2^{P_i/[1/2(P_i+Q_i)]}+(1-P_i){log}_2^{(1-P_i)/[1-1/2(P_i+Q_i)]},\end{array}$$

(19)

in which the smaller $H_i$ means the smaller difference between $P_i$ and $Q_i$. Equation (18) measures the relative information entropy of the two systems under the specific attribute; when the information entropy is zero, it means that the two systems are identical under the evaluation criteria of specific attributes.

Definition 3.

For two four-parameter interval sets $\stackrel{\sim }{p}=\left[p^L,p^{ML},p^{MR},p^R\right]$ and $\stackrel{\sim }{q}=$$\left[q^L,q^{ML},q^{MR},q^R\right]$, the relative information entropy of $\stackrel{\sim }{p}$ and $\stackrel{\sim }{q}$ is defined as:

$$\begin{array}{c}\hfill E(\stackrel{\sim }{p},\stackrel{\sim }{q})=\sum _{i=L,ML,MR,R}^{}{\omega }_i(p_i{log}_2^{p_i/[1/2(p_i+q_i)]}+(1-p_i){log}_2^{(1-p_i)/[1-1/2(p_i+q_i)]}),\end{array}$$

(20)

where ${\omega }_i$ is the weighting coefficient. Equation (20) is asymmetrical to $\stackrel{\sim }{p}$ and $\stackrel{\sim }{q}$, which is not corresponding to the actual submarine counter-game. Therefore, the definition of the relative information entropy is improved in the following:

$$\begin{array}{c}\hfill IE(\stackrel{\sim }{p},\stackrel{\sim }{q})=E(\stackrel{\sim }{p},\stackrel{\sim }{q})+E(\stackrel{\sim }{q},\stackrel{\sim }{p}),\end{array}$$

(21)

where $IE(\stackrel{\sim }{p},\stackrel{\sim }{q})$ presents the improved relative information entropy between two four-parameter interval sets, $\stackrel{\sim }{p}$ and $\stackrel{\sim }{q}$.

Property 1.

$IE(\stackrel{\sim }{p},\stackrel{\sim }{q})\ge 0$, if and only if when $\stackrel{\sim }{p}=\stackrel{\sim }{q}$, the equal sign holds.

Property 2.

$IE(\stackrel{\sim }{p},\stackrel{\sim }{q})=IE(\stackrel{\sim }{q},\stackrel{\sim }{p})$.

Here, we give the proof of the tenable condition of Property 1.

Proof. 

For any $i\in \{L,ML,MR,R\}$, define the corresponding component of relative information entropy $I{E}_i$ as

$$\begin{array}{c}\hfill \begin{array}{c}I{E}_i=p_i{log}_2^{p_i/[1/2(p_i+q_i)]}+(1-p_i){log}_2^{(1-p_i)/[1-1/2(p_i+q_i)]}\hfill \\ =-p_i{log}_2^{[1/2(p_i+q_i)/p_i]}-(1-p_i){log}_2^{[1-1/2(p_i+q_i)]/(1-p_i)}\hfill \end{array}.\end{array}$$

Because $f\left(x\right)={log}_2^x$ is a convex function, thus it is yielded from Jensen inequality that:

$$\begin{array}{c}\hfill \begin{array}{c}I{E}_i\ge -{log}_2^{\left\{p_i\frac{1/2(p_i+q_i)}{p_i}+(1-p_i)\frac{1-1/2(p_i+q_i)}{1-p_i}\right\}}={log}_2^1=0\hfill \end{array}\end{array}$$

if and only if when $p_i=q_i$, the equality holds, thus $I{E}_i(\stackrel{\sim }{p},\stackrel{\sim }{q})=0$. In the same way, it can be proven that $I{E}_i(\stackrel{\sim }{q},\stackrel{\sim }{p})=0$ when $p_i=q_i$. This completes the proof of Property 1. □

If the relative information entropy between the payment of strategy i and the maximum payment is $d_{max}^i$, that between the payment of strategy i and the minimum payment is $d_{min}^i$, which can be achieved from Equation (21). Therefore, the relative closeness of the payment of strategy i can be expressed as:

$$\begin{array}{c}\hfill C_i=\frac{d_{min}^i}{d_{max}^i+d_{min}^i}.\end{array}$$

(22)

The multi-AUV system maneuver strategies can be sorted according to $C_i$ in the last equation. In the countermeasure process, the strategy with the largest C value gets the highest priority; when the C values of different strategies are the same, the strategy with a smaller $d_{max}$ gets the higher priority.

4.2. State Transition of Payment Functions

When one multi-AUV system completes the $k-1$-th game stage and reaches the k-th stage, the multi-AUV strategy is $i=F\left(h\right)$, which means the i-th strategy is used in this stage. At this moment, the game situation of the last stage is ${\stackrel{\sim }{S}}^{k-1}$, and the estimated confrontation situation after the current stage $s^k$ is ${\stackrel{\sim }{S}}^k$. Over time, when the multi-AUV system completes the maneuver strategy $s^k$, the situation state of the k-th stage transfers to:

$$\begin{array}{c}\hfill \begin{array}{c}{\stackrel{\sim }{W}}_A\left(s^k\right)={\stackrel{\sim }{W}}_A\left(s^{k-1}\right)+\Delta {\stackrel{\sim }{W}}_A\left(s_i^k\right)\hfill \\ {\stackrel{\sim }{W}}_C\left(s^k\right)={\stackrel{\sim }{W}}_C\left(s^{k-1}\right)+\Delta {\stackrel{\sim }{W}}_C\left(s_j^k\right)\hfill \end{array},\end{array}$$

(23)

where ${\stackrel{\sim }{W}}_A\left(s^k\right)$ is the situation advantage state interval of the current strategy. $\Delta {\stackrel{\sim }{W}}_A\left(s_i^k\right)$ represents the variation of the state intervals of ${\stackrel{\sim }{W}}_A\left(s^k\right)$ and ${\stackrel{\sim }{W}}_A\left(s^{k-1}\right)$. $\Delta {\stackrel{\sim }{W}}_A\left(s_i^k\right)$ could be determined easily by the $k-1$-th state ${\stackrel{\sim }{W}}_A\left(s^{k-1}\right)$ and the definition of i-th strategy. For example, if the i-th strategy is to keep the current speed, $\Delta {\stackrel{\sim }{W}}_A\left(s_i^k\right)$ is the product of the speed vector in ${\stackrel{\sim }{W}}_A\left(s^{k-1}\right)$ and the stage interval time. ${\stackrel{\sim }{W}}_C\left(s^k\right)$ is the energy situation advantage state interval of the current strategy within the j-th strategy.

4.3. Nash Equilibrium Optimal Solution

The payment of the complete information zero-sum-game is replaced by the interval information set in this paper; thus, the payment of the interval information zero-sum-game discussed here is similar to that of the complete information zero-sum-game. For the multi-AUV attack and defense game, there is no guaranteed saddle point in the pure strategy game; thus, the countermeasure algorithm of the attacker and the defender can only choose the strategy randomly with a certain probability in the strategy set. Therefore, the Nash equilibrium for the mixed strategy is discussed here.

Definition 4.

For game $G=\left\{N,S,\stackrel{\sim }{U}\right\}$, define $x_i,y_j$ as the probabilities when players $N_1,N_2$ choose strategies $s_{1i}^k,s_{2j}^k$ at the k-th stage of the game from strategy sets $S_1,S_2$. The mixed strategies of the players in this game can be expressed as follows.

$$\begin{array}{c}\hfill \begin{array}{c}x=\left(x_1,x_2,\cdots ,x_m\right),{\displaystyle \sum _{i=1}^m}{x}_i=1,x_i\ge 0.\hfill \\ y=\left(y_1,y_2,\cdots ,y_n\right),{\displaystyle \sum _{j=1}^n}{y}_j=1,y_j\ge 0.\hfill \end{array}\end{array}$$

(24)

Definition 5.

$\underset{x\in S_m}{max}\underset{y\in S_n}{min}IE(x,y)=\underset{y\in S_n}{min}\underset{x\in S_m}{max}IE(x,y)=IE(x^{*},y^{*})=v$, so $x^{*},y^{*}$ are the optimal mixed strategies of players $N_1,N_2$ at this stage of the game, $(x^{*},y^{*})$ is the optimal mixed situation, and v is the expected benefit to the players.

According to the interval information game proposed, $IE(x,y)$ in this paper is an interval information set. Assume $f_u=\left[f_u^L,f_u^R\right],u=1,2,\dots mn$ is the interval payment when our multi-AUV system chooses maneuver strategy $x_m$ and the enemy multi-AUV system chooses maneuver strategy $y_n$ at this stage of the game.

According to Definition 3 and Definition 4, the Nash equilibrium [34] of our multi-AUV system under the mixed strategy $x=\left(x_1,x_2,\cdots ,x_m\right)$ can be achieved as follows:

$$\begin{array}{c}\hfill \stackrel{\sim }{v}=\underset{x}{max}\underset{1\le j\le n}{min}\sum _{i=1}^m\stackrel{\sim }{F_{ij}}{x}_i.\end{array}$$

(25)

On considering the actual submarine environment constraints, Equation (25) can be transformed into an optimization problem with interval uncertain parameters, as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\stackrel{\sim }{v}=\underset{x}{max}{\stackrel{\sim }{v}}_R\left(x\right)\hfill \\ s.t.\left\{\begin{array}{c}{\displaystyle \sum _{i=1}^m}\stackrel{\sim }{F_{ij}}{x}_i\ge {\stackrel{\sim }{v}}_R\left(x\right)\hfill \\ {\displaystyle \sum _{i=1}^m}{x}_i=1\hfill \\ x_i\ge 0\hfill \end{array}\right.,\begin{array}{c}i=1,2,\dots ,m\\ j=1,2,\dots ,n\end{array}.\hfill \end{array}\end{array}$$

(26)

Therefore, the optimal mixed strategy of the multi-AUV maneuver counter-game can be obtained by solving this optimization problem.

5. Fractional-Order Differential Evolution Algorithm

DE algorithm owns the characteristics of simple operation, high robustness, strong optimization ability, etc. [16]. However, slow convergence and falling into a local extremum may occur when the DE algorithm is applied in the optimization process, which makes satisfying the real-time counter-game requirements difficult. Aiming to address this problem, a fractional-order differential evolution (FDE) algorithm is presented in this section.

(1) Fitness function.

Normally, the optimal strategy of game player $N_1$ is to maximize payoff benefit under the constraint conditions, while the other game player $N_2$ is the opposite. Therefore, the fitness function here could be the optimization objective function presented in Equation (26).

(2) Mutation.

According to Equations (3)–(5), the variation vector of FDE mutation is proposed within the Caputo definition:

$$\begin{array}{c}\hfill w_{i,g}=h^q\left[v_{i,g}+F_i(v_{best,g}-v_{i,g})+F_i(v_{r_1,g}-v_{r_2,g})\right]+\frac{{(g+1)}^{-q}{v}_{i,0}}{\mathsf{\Gamma }(1-q)}-\sum _{k=1}^{g+1}{s}_k{v}_{i,g+1-k},\end{array}$$

(27)

where

$$s_0=1,s_k=\left(1-\frac{q+1}{k}\right)s_{k-1},$$

h is the iteration step of $v_{i,g}$, and the other parameters are same with the definition in Equation (5).

In particular, the variation vector (27) with $h=1$ becomes

$$\begin{array}{c}\hfill w_{i,g}=v_{i,g}+F_i(v_{best,g}-v_{i,g})+F_i(v_{r_1,g}-v_{r_2,g})+\frac{{(g+1)}^{-q}{v}_{i,0}}{\mathsf{\Gamma }(1-q)}-\sum _{k=1}^{g+1}{s}_k{v}_{i,g+1-k}.\end{array}$$

(28)

Remark 1.

The variation vector (27) of the FDE mutation is derived by the combination of Equation (5) and Caputo fractional derivatives. In fact, Equation (5) is a special case with $h=1$ of the following discrete iteration:

$$\frac{w_{i,g}-v_{i,g}}{h}=F_i(v_{best,g}-v_{i,g})+F_i(v_{r_1,g}-v_{r_2,g}).$$

The first-order difference $\frac{w_{i,g}-v_{i,g}}{h}$ is replaced by the Caputo fractional one (4), i.e.,

$$\begin{array}{c}\frac{1}{h^q}\left(w_{i,g}+\sum _{i=1}^{g+1}{(-1)}^i\left(\begin{array}{c}q\\ i\end{array}\right)v_{i,g+1-k}\right)-\frac{{\left[(g+1)h\right]}^{-q}}{\mathsf{\Gamma }(1-q)}{v}_{i,0}\hfill \\ =F_i(v_{best,g}-v_{i,g})+F_i(v_{r_1,g}-v_{r_2,g}).\hfill \end{array}$$

In addition, denote

$$\begin{array}{c}\hfill s_k={(-1)}^k\left(\begin{array}{c}q\\ k\end{array}\right),k=1,2,\dots ,g+1,\end{array}$$

where $s_k$ can be calculated through a recursive scheme, as follows

$$s_0=1,s_k=\left(1-\frac{q+1}{k}\right)s_{k-1}.$$

Then, the variation vector (27) of the FDE mutation can be obviously deduced.

Note that in Equation (27), the scaling factor $F_i$ is for scaling each base vector and generating a new variation vector. A larger $F_i$ can search for a potentially optimal solution in a larger range. On the contrary, a smaller one can speed up convergence and improve accuracy. Meanwhile, when the fitness of each individual is relatively superior, it is preferable for $F_i$ to be small to reduce the disturbance of the better individuals; on the contrary, when the fitness of each individual is relatively poor, it is preferred to expand the search range of the solution; thus, a larger $F_i$ may be applied. Combined with the game algorithm proposed in this paper, the scaling faction $F_i$ here is determined according to the evolution time and the difference between the best and worst individuals, as follows:

$$F_i=(F_{max}-F_{min})\times \frac{\mid f_{best}-f_i\mid }{\mid f_{best}-f_{worst}\mid }\times \Delta g+F_{min},$$

(29)

where $\Delta g=(G-g)/G$, G is the maximum number of iterations, g is the current number of iterations; $f_{best}$ is the best fitness, $f_{worst}$ is the worst fitness, and $f_i$ is the current individual fitness. $F_{max}$ and $F_{min}$ are the maximum and minimum values of F, respectively. If the fitness difference between the current individual and the optimal individual is large, it means that the individual is far away from the optimal individual in space. A larger value of $F_i$ signifies that the disturbance to the individual is larger, which in turn implies that the search scope of the algorithm is expanded, and the global search ability is enhanced. If the fitness difference is small, $F_i$ may acquire a smaller value, and the disturbance to the individual is also small, which means that the search is only carried out in the small area near the individual, so as to enhance the ability of the algorithm development. Moreover, at a later stage in the evolution, the value of $\Delta g$ prefers to be relatively small, which can make the search in the neighborhood of the current individual and ensure the accuracy of the algorithm.

(3) Crossover.

As shown in Equation (6), the crossover operation is described by

$$u_{ij,g}=\left\{\begin{array}{c}\begin{array}{cc}{v}_{ij,g},& \begin{array}{ccc}if& rand[0, 1]<C{R}_i& \begin{array}{cc}or& j=j_{rand}\end{array}\end{array}\end{array}\hfill \\ \begin{array}{cc}{w}_{ij,g},& Otherwise\end{array}\hfill \end{array}\right..$$

Note that $C{R}_i$ is the crossover rate that determines the crossover probability of the mutated individual and the original individual on each dimension vector. The larger $C{R}_i$ of the individual with poor adaptability can accelerate the change of the individual structure. Therefore, it is preferred that a smaller $C{R}_i$ be used in the late evolution stage to reduce the disturbance of the target individual to the experimental individual and ensure the convergence speed of the algorithm. The designed crossover rate is as follows:

$$\begin{array}{c}\hfill C{R}_i=\left\{\begin{array}{cc}C{R}_{min},& \stackrel{-}{f}>f\left(v_{i,g}\right)\\ C{R}_{min}+(C{R}_{max}-C{R}_{min})\times \Delta g,& \stackrel{-}{f}\le f\left(v_{i,g}\right)\end{array}\right.,\end{array}$$

(30)

where $f^{-}$ is the average fitness of the current population, $C{R}_i$ is the current crossover rate, and $C{R}_{max}$ and $C{R}_{min}$ are the maximum and minimum values of $CR$, respectively. Equation (30) shows that when the fitness of the target individual $v_{i,g}$ is smaller than the average fitness, the target individual is relatively superior. A smaller $C{R}_i$ should be chosen, and more information of the test vector is obtained from the target vector $v_{i,g}$. Otherwise, more information of the test vector $u_{i,g}$ is obtained from the variation vector $w_{i,g}$, which improves the diversity of population. $\Delta g$ could ensure a large $C{R}_i$ at an early stage of evolution, increase population diversity, and speed up convergence. In addition, a small $C{R}_i$ in the late evolution stage is conducive to finding the optimal solution.

(4) Selection.

According to Equation (7), the selection operation is re-mentioned.

$$v_{i,g+1}=\left\{\begin{array}{c}\begin{array}{cc}{v}_{i,g},& f\left(u_{i,g}\right)<f\left(v_{i,g}\right)\end{array}\hfill \\ \begin{array}{cc}{u}_{i,g},& Otherwise\end{array}\hfill \end{array}\right.,$$

where $v_{i,g+1}$ is the next generation individual.

6. Example

In this section, a multi-AUV countermeasure example is given to verify the effectiveness of the proposed algorithm. In the initialization process of the proposed algorithm, several parameters need to be determined as the following guideline.

• The initial states of game players ${\stackrel{\sim }{W}}_A\left(s^1\right)$ and ${\stackrel{\sim }{W}}_C\left(s^1\right)$, such as the initial positions, velocity, deflection angle, and pitch angle, as well as the time interval of the game steps and the weighting coefficients $k_1,k_2,k_3$ in Equation (12).
• The strategy spaces of both sides $S=\left\{s_{1i}^k,s_{2j}^k\right\}$, such as keep the current speed, turn left, turn right, pitch up, pitch down, etc.
• The maximum number of iterations G in the fractional-order DE, as well as the population $NP$, iteration step h, and fractional order q.

Suppose ‘A’ and ‘D’ are engaged in a two-vs.-two AUVs underwater counter-game. The initial positions of ‘A1’ and ‘A2’ are (0 m, 100 m, and 200 m) and (0 m, −100 m, and 200 m), and ‘D1’ and ‘D2’ are (800 m, 100 m, and 200 m) and (800 m, −100 m, and 200 m), respectively. The velocity, deflection angle, and pitch angle of ‘A1’ and ‘A2’ are 23 m/s, $-{60}^{\circ }$, $5^{\circ }$, 23 m/s, and ${60}^{\circ }$, $-5^{\circ }$, respectively. The velocity, deflection angle, and pitch angle of ‘D1’ and ‘D2’ are 25 m/s, ${120}^{\circ }$, $3^{\circ }$, 25 m/s, and $-{120}^{\circ }$, $-3^{\circ }$, respectively. Both sides have the same control ability, and the time interval of the game steps is 5 s. The weighting coefficients are denoted as $k_1=0.2,k_2=0.4,k_3=0.4$ referring to [30]. The strategy spaces of ‘A’ and ‘D’ are both supposed as keep the current speed, speed up, speed down, turn left, turn right, pitch up, and pitch down for each AUV. In addition, the maximum number of iterations is set as $G=500$, as well as the population $NP=100$, iteration step $h=1$, and fractional order $0<q\le 1$. Within all the above initial conditions, the multi-AUV maneuver countermeasure model is constructed according to Section 3 and Section 4, and the fractional-order differential evolution in Section 5 is employed in strategy optimization. For a different fractional order q, the optimization errors of the first game step are calculated and shown in Figure 1. When the optimization errors are close to 0, the convergence rates are not clearly visible in Figure 1. Thus, the base-10 logarithms ($lo{g}_{10}$ ) of the optimization errors of different fractional orders q are exhibited in Figure 2.

According to Figure 1 and Figure 2, the optimization error of $q=0.2$ is better than the others when the iteration is 300. However, the optimization error of $q=0.8$ is the most superior one when the iteration is 500. Thus, smaller order q brings faster optimization rate when the iteration number is not enough. When the iterations are raising, the optimization effects of the larger order visibly increase. Thus, according to the iteration number, the fractional-order DE algorithm could adjust the fractional order q to achieve the optimal strategy calculation.

It is obvious that ’D’ possesses some advantage in the beginning. It should also be noted that the maximum maneuver steps should be determined according to the effectiveness of the AUVs used in the confrontation. There are 50 steps in the game process, whose expected benefits are shown in Figure 3. According to the last section, the obtained expected benefits demonstrate that the Nash equilibrium of the interval information game is satisfied.

In order to compare the game performance, ‘A’ uses the cooperative dynamic maneuver countermeasure algorithm proposed in this paper, and ‘D’ uses the max-min countermeasure algorithm during the multi-AUV confrontation process [34]. The three-dimensional counter-game process with five main stages are shown in Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8. The red-dotted line represents the path of ‘A1’, the red solid line represents ‘A2’, and the blue-dotted and solid lines represent ‘D1’ and ‘D2’, respectively. The ‘*’ shows the initial position and ‘▵’ shows the current position. The confrontation ends when one side’s expected benefit reaches the absolute advantage. For stage 1 in Figure 4, ‘A’ possesses the dominant position and decides to attack ‘D’ actively, in which ‘A1’ tries to attack ‘D2’, and ‘A2’ is moving towards ‘D1’. For stage 2 in Figure 5, ‘A1’ misses the attack on ‘D2’, and tries to attack ‘D1’ with ‘A2’. ‘D2’ escapes the attack of ‘A1’, and tries to outflank ‘A2’. The situation changes when ‘D’ possesses the dominating position in stage 3. This can also be validated in Figure 3, in which the expected benefits change from positive to negative. In Figure 6, ‘D1’ and ‘D2’ try to achieve a converging attack to ‘A2’; ‘A1’ tries to attack ‘D1’ and supports ‘A2’. Then, in stage 4, the situation changes again, in which ‘A’ possesses the dominating position and the expected benefits change from negative to positive. ‘A2’ turns continuously and drives ‘D1’ away successfully, then ‘A1’ and ‘A2’ tries to attack ‘D2’, but ‘D1’ and ‘D2’ escape in two different directions. In the end, both ‘A1’ and ‘A2’ possess the dominating positions, thus ‘A’ achieves the absolute advantage and ends the game, which is illustrated in Figure 8. This example validates the effectiveness of the proposed multi-AUV dynamic maneuver countermeasure algorithm.

7. Conclusions

In this paper, the interval information set is introduced into game theory to study the dynamic maneuver countermeasure algorithm for multi-AUV systems. Marine environment characteristics, including different kinds of uncertainties, are expressed by the interval information sets. The maneuver countermeasure model of the multi-AUV system is established based on the forehead interval information sets, and the payment interval ranking is achieved based on the four-parameter interval set and relative entropy. Combined with the background and model characteristics, the optimal maneuver strategy satisfying the Nash equilibrium condition is obtained using the fractional-order DE algorithm in each step of the dynamic counter-game process. A multi-AUV dynamic counter-game example with several maneuver countermeasure steps is provided to illustrate the superiority and effectiveness of the proposed algorithm. Our future work shall focus on considering different communication conditions in underwater environments, which may bring about more constraints and computation burden in the countermeasure process. Moreover, some intelligent algorithms including the learning mechanism may be explored to obtain the optimal strategy of the multi-AUV game.