Leader–Follower Formation Reconfiguration Control for Fixed-Wing UAVs Using Multiplayer Stackelberg–Nash Game

Abstract

For the formation reconfiguration of fixed-wing unmanned aerial vehicles (UAVs), a hierarchical control decision-making method considering both convergence and optimality is studied. To begin with, the dynamic model of the fixed-wing UAVs is established, and the formation reconfiguration control problem formally constructed. Subsequently, based on information such as the initial positions of the UAVs and the expected geometric configuration, an integer programming issue is formulated to determine the destinations of the UAVs. After completing the aforementioned preparations, by incorporating the concept of hierarchical games, the formation guidance and control problem is consequently reformulated as a multiplayer Stackelberg–Nash game (SNG). Through rigorous analysis, the optimality of using the Stackelberg–Nash equilibrium solution as the UAV control commands was demonstrated. Furthermore, a novel policy iteration (PI) algorithm for solving this equilibrium based on fixed-point iteration is proposed. To guarantee the accurate execution of the control commands, an auxiliary control system is designed, thereby forming a closed-loop real-time control decision-making mechanism. The numerical simulation results illustrate that the UAVs can rapidly switch to the desired formation configuration, thus validating the effectiveness of the proposed method.

1. Introduction

In recent years, UAVs have played an increasingly significant role in various civilian-related areas [1], including emergency rescue [2,3], coastal surveying [4], and fire monitoring [5,6]. Compared with unmanned rotorcraft, fixed-wing UAVs exhibit significant advantages in terms of flight speed and range, making them irreplaceable in certain applications [7]. It is worth noting that forming a formation with multiple fixed-wing UAVs can further capitalize on the advantage of a larger search range, thereby enhancing mission execution efficiency [8]. In order to put this concept into practice, a reliable guidance and control system is required, which should be capable of achieving UAV formation reconfiguration while ensuring safety.

As one of the major challenges for the guidance and control system, formation reconfiguration [9] can be considered an integration of a series of sub-tasks, including task allocation [10], trajectory planning [11], and trajectory tracking [12]. Among these sub-tasks, trajectory planning is probably the most complex, as it requires the rapid generation of a feasible and secure scheme based on the actual scenario. In addition, since fixed-wing UAVs are incapable of hovering, formation reconfiguration must be accomplished during the flight, which inevitably increases the operational complexity. Even so, significant advancements have been achieved in the field of formation reconfiguration control across the following three aspects. First, a variety of schemes have been proposed for the control decision-making mechanism of formations, including but not limited to the “leader–follower” [13,14], “virtual structure” [15,16], artificial potential field technique [17,18], and others. These methods each possess their own advantages in terms of rapidity, operational complexity, and damage resistance. Numerous investigations have carried out comparative analyses, and concluded that the selection of a specific method depends on the actual scenario [19]. Second, substantial studies have been performed in communication or information transmission between UAVs [20]. Typical issues in this area include dealing with communication delays and packet loss [21], reducing communication frequency and transmitted information while ensuring control performance [22]. In addition, the safe flight of a UAV formation has also attracted scholars’ attention. During flight, UAVs need to handle possible external disturbances such as gusts and turbulence [23], bypass no-fly zones [24] and ensure mutual collision avoidance [25]. If these difficulties are successfully addressed, it will effectively enhance the survival rate of the UAV swarm.

It is worth noting that the aforementioned studies seldom delve into the optimality of formation decision-making. In fact, it is essential to discuss the optimality of formation control under the premise of ensuring convergence. The optimal/suboptimal control of the formation requires making the trajectory of each UAV as direct as possible, thereby reducing operational complexity and decreasing the probability of risk. When each UAV is considered an intelligent agent with independent decision-making capabilities, all of them become engaged in a multiplayer game. If each individual aims to minimize its travel distance, upon reaching game equilibrium, the optimal strategies will be generated. This deduction enlightens us to apply game theory to solving the optimal control problem of UAV formations. Among various game models, one that is similar to the leader–follower formation control stands out, namely the Stackelberg–Nash game (SNG) [26].

The concept of a Stackelberg game was initially proposed to address the problems in static competition economics [27]. Fundamental theoretical research on the Stackelberg game, such as the existence and uniqueness of solutions [28,29], has already been subject to rigorous analysis, yielding comprehensive and conclusive results. Recently, by incorporating practical factors such as random diffusion [30], time delay [31], and noise observation [32], a relatively comprehensive theoretical framework has emerged. Moreover, with the advancement of artificial intelligence technologies [33,34,35], the challenges in solving Stackelberg–Nash equilibrium are being overcome, making the online application of the SNG a reality. It is precisely due to the breakthrough in the aforementioned theoretical research and computational methods that the SNG has gradually been extended to certain industrial domains. In fields such as electricity trading processes [36], collaborative control of unmanned surface vehicles [37], and coordinated pursuit of UAVs [38], the SNG has demonstrated considerable application potential.

However, the existing achievements cannot be directly utilized for formation reconfiguration of fixed-wing UAVs. Note that the existing Stackelberg games are mainly applied to the “one leader and one follower” scenario. Extending them to multiplayer scenarios to accommodate more general situations requires further investigation. Subsequently, the flight dynamics model of fixed-wing UAVs is complex, and few effective schemes have yet emerged regarding how to integrate them into the SNG framework. Furthermore, achieving hierarchical distributed decision-making of control strategies based on the real-time flight states poses significant challenges.

As mentioned earlier, the existing distributed control algorithms for UAVs seldom account for optimality. While the existing Stackelberg game theory offers insights into hierarchical optimal control, it cannot be directly applied to multi-UAV scenarios. To solve this contradiction, this paper is committed to proposing a distributed guidance and control methodology with the SNG serving as the core decision-making mechanism. This method is employed to adjust the relative trajectories of fixed-wing UAVs, ultimately realizing the optimal formation reconfiguration control. To this end, we have designed a state variable suitable for multiple players to describe the progress of formation reconfiguration. Based on this, decisions of the leader and the followers are integrated into a unified framework, thus laying the foundation for the introduction of a multiplayer SNG. To solve the Stackelberg–Nash equilibrium efficiently, inspired by the fixed-point iteration method [39], a novel two-level policy iteration algorithm is proposed. Compared with other artificial intelligence methods, for example, integral reinforcement learning [26,35], this approach avoids integral calculations and thus enhances the computational efficiency. In addition to the crucial decision-making algorithm, this work also incorporates task allocation and tracking control to ensure the completeness of the guidance and control system.

The paper is structured as follows. Section 2 introduces the preliminaries, including the communication topology, the dynamical model of fixed-wing UAVs, and then formulates the formation reconfiguration problem. Section 3 establishes the control framework and specifies the destinations of all the UAVs. Section 4 designs the SNG-based control decision-making mechanism as well as the auxiliary tracking control subsystem. Section 5 verifies the effectiveness of the designed method through numerical simulations. Finally, some concluding remarks are given in Section 6.

2. Preliminaries and Problem Formulation

2.1. Network Communication Topology

In this paper, we consider a cluster of $\left(N+1\right)$ UAVs consisting of one leader and N followers. The network communication topology among the UAVs can be described by a weighted graph $\mathcal{G}=\left(\mathcal{V},\mathcal{A}\right)$, where $\mathcal{V}=\left\{\mathcal{L},\mathcal{F}\right\},\mathcal{L}=\left\{0\right\},\mathcal{F}=\left\{1,\cdots ,N\right\}$ denotes the index set of the $\left(N+1\right)$ UAVs, and $\mathcal{A}=\left[a_{ij}\right]\in {\mathbb{R}}^{(N+1)\times (N+1)}$ is a weighted adjacency matrix. It is worth noting that $a_{ij}>0(i\ne j)$ indicates that the j-th UAV can receive information transmitted by the i-th UAV, whereas the reverse does not necessarily hold. In this article, the information exchange between each follower and the leader is bidirectional (i.e., $a_{0i}>0,a_{i0}>0,i\in \mathcal{F}$), while there is no information exchange among all followers (i.e., $a_{ij}=0,i\ne j,i,j\in \mathcal{F}$). Since a UAV does not communicate with itself, we have $a_{ii}=0,i\in \mathcal{V}$.

As evident from the preceding discussion, the leader, serving as the communication center, holds an exceptionally critical role in formation control. To avoid task failure resulting from the potential fault of the leader, this work refrains from permanently designating a leader and instead temporarily assigns one based on the actual scenario. Given that each UAV possesses the potential to be the leader, all UAVs are uniformly identical in both performance and structure.

2.2. Dynamical Model of Fixed-Wing UAVs

The UAV model presented in this work, as illustrated in Figure 1, is derived by scaling down the classic manned airplane, the Cessna 172, with a reduction ratio of 0.35. To describe the flight of UAVs, in addition to the body frame illustrated in Figure 1, the inertial frame, airflow frame, trajectory frame, and relative motion frame are also employed [40].

By assuming that the UAV performs lateral maneuvering through bank-to-turn, and neglecting the effects of Earth’s rotation and flattening, the simplified centroid dynamic equations of a single UAV can be derived as follows [41]:

$${\displaystyle \frac{dx}{dt}}=Vcos\gamma cos\psi$$

(1)

$${\displaystyle \frac{dy}{dt}}=Vcos\gamma sin\psi$$

(2)

$${\displaystyle \frac{dz}{dt}}=Vsin\gamma$$

(3)

$$m{\displaystyle \frac{dV}{dt}}=Tcos\alpha -D-mgsin\gamma$$

(4)

$$mV{\displaystyle \frac{d\gamma }{dt}}=\left(Tsin\alpha +L\right)cos\sigma -mgcos\gamma$$

(5)

$$mVcos\gamma {\displaystyle \frac{d\psi }{dt}}=(Tsin\alpha +L)sin\sigma$$

(6)

where x, y and z represent the position coordinates of the UAV in the inertial frame, m is the mass of the UAV, g is the local gravitational acceleration, V is the velocity, L and D are the lift and drag defined in the airflow frame, T represents the thrust, $\alpha$ is the angle of attack, $\sigma$ is the bank angle, and $\gamma$ and $\psi$ are the trajectory inclination and declination, respectively.

For such low-speed aircraft, the aerodynamic forces (i.e., the lift L and drag D) can be written as [41]

$$L=\overline{q}S{C}_L$$

(7)

$$D=\overline{q}S{C}_D$$

(8)

where $\overline{q}$ is the dynamic pressure, S represents the reference area. Within the flight envelope of the fixed-wing UAVs considered in this paper, the aerodynamic coefficients $C_L$ and $C_D$ can be modeled as [41]

$$C_L=C_L^0+C_L^{\alpha }\alpha$$

(9)

$$C_D=C_D^0+\kappa C_L^2$$

(10)

where $C_L^0$, $C_L^{\alpha }$, $C_D^0$ and $\kappa$ are the parameters that can be calculated from aerodynamic data.

To facilitate subsequent research on the relative motion of UAVs, the velocity vector of UAVs is projected onto the inertial frame, thereby enabling the definition of $V_x\triangleq Vcos\gamma cos\psi$, $V_y\triangleq Vcos\gamma sin\psi$, and $V_z\triangleq Vsin\gamma$. Then Equations (1)–(3) can be simplified as

$${\displaystyle \frac{d\mathit{X}}{dt}}=\mathit{V}$$

(11)

where $\mathit{X}\triangleq {[x,y,z]}^T$ and $\mathit{V}\triangleq {[V_x,V_y,V_z]}^T$.

Since fixed-wing UAVs are incapable of hovering in the air, the motion can be divided into baseline motion and additional motion during the formation reconfiguration. For safety considerations, it is assumed that the process occurs in the cruise stage. Consequently, the baseline motion of a UAV is characterized by horizontal uniform-speed flight, during which aerodynamic forces, thrust, and gravity are in equilibrium. We define $\overline{U}\left(U\in \left\{V_x,V_y,V_z,V,\gamma ,\psi ,T,\alpha ,\sigma \right\}\right)$ as the trim state, and $\mathrm{\Delta }U$ as the state of additional motion. Notably, the relative motion frame maintains the trim state, with its initial position defined by the user. Then the kinematic equations of the additional motion can be derived as

$${\displaystyle \frac{d}{dt}}\mathrm{\Delta }\mathit{X}=\mathrm{\Delta }\mathit{V}$$

(12)

Considering that this work centers on the formation control of UAVs, the attitude dynamics of UAVs is ignored for simplicity. Specifically, the inputs to the UAV system encompass the angle of attack $\alpha$, bank angle $\sigma$, and thrust T, while the outputs consist of the positions $\mathit{X}={\left[x,y,z\right]}^T$ within the inertial coordinate framework.

2.3. Control Objectives

The objective of this study is to realize the automatic reconfiguration of the fixed-wing UAVs into a predefined configuration. The crux of this task lies in converging the current UAV formation to the desired configuration, which can be formulated as the following mathematical problem.

Define ${\mathit{d}}_i\triangleq {\left[d_{xi},d_{yi},d_{zi}\right]}^T,i\in \mathcal{F}$ as the desired relative position of the i-th follower with respect to the leader. The deviations between the current formation configuration and the expected one can be quantified as the variable $\mathit{w}$, viz.

$$\mathit{w}=\left[\begin{array}{c}{w}_x\\ w_y\\ w_z\end{array}\right]=\left[\begin{array}{c}{\displaystyle \frac{1}{2}}{\sum }_{i=1}^N{\left(\mathrm{\Delta }{x}_0-\mathrm{\Delta }{x}_i-d_{xi}\right)}^2\\ {\displaystyle \frac{1}{2}}{\sum }_{i=1}^N{\left(\mathrm{\Delta }{y}_0-\mathrm{\Delta }{y}_i-d_{yi}\right)}^2\\ {\displaystyle \frac{1}{2}}{\sum }_{i=1}^N{\left(\mathrm{\Delta }{z}_0-\mathrm{\Delta }{z}_i-d_{zi}\right)}^2\end{array}\right],i\in \mathcal{F}$$

(13)

where $w_x\ge 0$, $w_y\ge 0$, $w_z\ge 0$, and “=” holds if and only if the UAV formation converges to the expected configuration.

Taking the derivative with respect to time yields the dynamic model of $\mathit{w}$

$${\displaystyle \frac{d\mathit{w}}{dt}}={\mathit{B}}_0\mathrm{\Delta }{\mathit{V}}_0+\sum _{i=1}^N{\mathit{B}}_i\mathrm{\Delta }{\mathit{V}}_i,i\in \mathcal{F}$$

(14)

where ${\mathit{B}}_0=\mathrm{diag}\left[{\sum }_{i=1}^N\left(\mathrm{\Delta }{x}_0-\mathrm{\Delta }{x}_i-d_{xi}\right),{\sum }_{i=1}^N\left(\mathrm{\Delta }{y}_0-\mathrm{\Delta }{y}_i-d_{yi}\right),{\sum }_{i=1}^N\left(\mathrm{\Delta }{z}_0-\mathrm{\Delta }{z}_i-d_{zi}\right)\right]$, ${\mathit{B}}_i=\mathrm{diag}\left[-\left(\mathrm{\Delta }{x}_0-\mathrm{\Delta }{x}_i-d_{xi}\right),-\left(\mathrm{\Delta }{y}_0-\mathrm{\Delta }{y}_i-d_{yi}\right),-\left(\mathrm{\Delta }{z}_0-\mathrm{\Delta }{z}_i-d_{zi}\right)\right],i=1,\cdots ,N$.

It can be found that if a control strategy is provided to ensure the convergence of the dynamic system from the given initial state to the origin, then the UAV formation reconstruction task can be accomplished. Therefore, the control problem for the formation reconfiguration of fixed-wing UAVs is formulated.

3. Preparations for Formation Reconfiguration Control

3.1. Control Framework Design

In the discussion in the previous section, the problem of formation reconfiguration control for fixed-wing UAVs has been formulated. It is worth emphasizing that this control process entails far more than merely designing distributed control laws. The process can be divided into the following steps: first, select the leader; second, determine the relative position of each UAV in the desired configuration; and finally, drive each UAV to its desired position. Therefore, the control framework for UAV formation reconfiguration can be designed as depicted in Figure 2. The contents, including control decision-making based on the SNG and auxiliary control systems, will be elaborated on in the next section.

3.2. Task Assignment

Prior to executing the formation reconfiguration task, it is essential to specify the destinations for each UAV. The first thing at this stage is to determine the position of the leader in the desired configuration. In general, the UAV positioned closest to the geometric center of the desired configuration is selected as the leader. This choice not only ensures a balanced spatial distribution but also facilitates more efficient communication with other UAVs.

Once the position of the leader within the desired configuration is determined, the relative positions of the followers can be therefore calculated (i.e., $\mathcal{D}=\left\{{\mathit{d}}_1,{\mathit{d}}_2,\cdots ,{\mathit{d}}_N\right\}$; the order is user-defined). The challenge is then to sequentially match the UAVs with those in the desired configuration, that is, to determine the destination of each UAV. To reduce the adjustment complexity of the cluster and thereby lower the collision probability, the overall moving distance of the UAVs should be minimized. To acquire a reasonable task allocation scheme, the following optimization problem can be established.

Assign sequential initial indices to the $\left(N+1\right)$ UAVs, specifically $\mathcal{S}=\left\{s_1,s_2,\cdots ,s_{N+1}\right\}$. Manage to find the appropriate leader $s_k$ and the sequence $\left\{n_i\right\}$ such that the algebraic sum of displacements is minimized.

$$\begin{array}{cc}\hfill & \underset{k,\left\{n_i\right\}}{min}R=\hfill \\ \hfill & \sum _{i=1}^N{\left[{\left(x_{s_k}\left(t_0\right)-x_{n_i}\left(t_0\right)-d_{xi}\right)}^2+{\left(y_{s_k}\left(t_0\right)-y_{n_i}\left(t_0\right)-d_{yi}\right)}^2+{\left(z_{s_k}\left(t_0\right)-z_{n_i}-d_{zi}\left(t_0\right)\right)}^2\right]}^{\frac{1}{2}}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& k\in \left\{1,2,\cdots ,N+1\right\}\hfill \\ \hfill & n_i\in \mathcal{S},n_i\ne s_k\hfill \end{array}$$

(15)

By solving the optimization problem (15), the leader’s index can be determined as $s_k$. Simultaneously, the destinations for all the followers can be set via the correspondence between $s_{n_i}$ and $d_i$. It is evident that the total number of leader–follower task allocation schemes amounts to $(N+1)!$. Therefore, when N is relatively small, the exhaustive method can be employed to solve the problem, whereas for larger values of N, it becomes necessary to consider incorporating swarm intelligence algorithms to address the increased complexity.

4. SNG-Based Leader–Follower Formation Reconfiguration Control

4.1. SNG-Based Optimal Control

After determining the relative positions of the UAVs in the desired configuration, the remaining task—also the most challenging part—is to ensure that the UAVs form the preset geometric shape. Following the goal of minimizing the moving distance proposed in Section 3, the value function can be defined as

$$J_i(\mathit{w},\mathrm{\Delta }{\mathit{V}}_i)={\int }_t^{\infty }{r}_i\left(\mathit{w}\left(\tau \right),\mathrm{\Delta }{\mathit{V}}_i\right)d\tau ,i\in \mathcal{V}$$

(16)

where $r_i\in \mathbb{R}$ is the utility function of the i-th UAV.

Since the control process of the formation involves all the UAVs, the value function (16) of any UAV is inevitably influenced by the actions of other participants. Thus one can define the utility functions for the leader and followers as follows:

$$r_0\left(\mathit{w},\mathrm{\Delta }{V}_i\right)={\parallel \mathit{w}\parallel }_{{\mathit{Q}}_0}^2+{\parallel \mathrm{\Delta }{V}_0+\sum _{j=1}^N{\mathit{C}}_j\mathrm{\Delta }{\mathit{V}}_j\parallel }_{{\mathit{R}}_0}^2,i\in \mathcal{V},j\in \mathcal{F}$$

(17)

$$r_j\left(\mathit{w},\mathrm{\Delta }{V}_i\right)={\parallel \mathit{w}\parallel }_{{\mathit{Q}}_i}^2+{\parallel \mathrm{\Delta }{V}_j+{\mathit{D}}_j\mathrm{\Delta }{\mathit{V}}_0\parallel }_{{\mathit{R}}_j}^2,i\in \mathcal{V},j\in \mathcal{F}$$

(18)

where ${\parallel \mathit{w}\parallel }_{{\mathit{Q}}_i}\triangleq (1/2){\mathit{w}}^T{\mathit{Q}}_i\mathit{w}$, ${\mathit{Q}}_i\in {\mathbb{R}}^{3\times 3}$, ${\mathit{Q}}_i>\mathbf{0}$, ${\mathit{C}}_j,{\mathit{D}}_j\in {\mathbb{R}}^{3\times 3}$ respectively represent the coupling coefficient matrices of the follower j with respect to the leader and the leader with respect to the follower j.

It should be noted that each UAV manages to adjust its control strategy to minimize its respective value function, which is interrelated with others. Consequently, the control decision-making process can be regarded as a multiplayer Stackelberg–Nash game. The following presents the relevant definitions.

Definition 1

(Best Response). A best response is the best decision a player can make to optimize the corresponding value function, given the strategies of the other players.

Definition 2

(Stackelberg–Nash Equilibrium). Define a mapping $T_{fj}:{\mathbb{V}}_0\to {\mathbb{V}}_j$, and $T_{-fj}=\left\{T_{fi}\left({\mathit{V}}_0\right)|i\in \mathcal{F}\mathit{and}i\ne j\right\}$. If there exists a $T_{fj}$, such that for all $\mathrm{\Delta }{\mathit{V}}_j\in {\mathbb{V}}_j$,

$$J_j\left({\mathit{w}}_0,\mathrm{\Delta }{\mathit{V}}_0,T_{fj}(\mathrm{\Delta }{\mathit{V}}_0),T_{-fj}(\mathrm{\Delta }{\mathit{V}}_0)\right)\le J_j\left({\mathit{w}}_0,\mathrm{\Delta }{\mathit{V}}_0,\mathrm{\Delta }{\mathit{V}}_j,T_{-fj}(\mathrm{\Delta }{\mathit{V}}_0)\right)$$

(19)

and simultaneously, if there exists a $\mathrm{\Delta }{\overline{\mathit{V}}}_0\in {\mathbb{V}}_0$, such that for all $\mathrm{\Delta }{\mathit{V}}_0\in {\mathbb{V}}_0$,

$$J_0\left({\mathit{w}}_0,\mathrm{\Delta }{\overline{\mathit{V}}}_0,T_{fj}(\mathrm{\Delta }{\overline{\mathit{V}}}_0),T_{-fj}(\mathrm{\Delta }{\overline{\mathit{V}}}_0)\right)\le J_0\left({\mathit{w}}_0,\mathrm{\Delta }{\mathit{V}}_0,T_{fj}(\mathrm{\Delta }{\mathit{V}}_0),T_{-fj}(\mathrm{\Delta }{\mathit{V}}_0)\right)$$

(20)

then $\left\{\mathrm{\Delta }{\overline{\mathit{V}}}_0,{\overline{\mathit{V}}}_j=T_{fj}\left({\overline{\mathit{V}}}_0\right),j\in \mathcal{F}\right\}$ is called the Stackelberg–Nash equilibrium.

Definition 3

(Multiplayer Stackelberg–Nash Game). If there exist appropriate strategies for all the players that stabilize the system (14) while constituting a Stackelberg–Nash equilibrium, then the leader and followers are said to be in a multiplayer Stackelberg–Nash game.

After transforming the issue into a multiplayer game problem, we then manage to solve the control decisions by employing the techniques of optimal control and policy iteration. At first we need to introduce the definitions of the optimal policy and its corresponding value function. For the leader,

$$J_0^{*}\left(\mathit{w}\left(t\right)\right)=\underset{\mathrm{\Delta }{\mathit{V}}_0}{min}{\int }_t^{\infty }{r}_0(\mathit{w},\mathrm{\Delta }{\mathit{V}}_0,\mathrm{\Delta }{\mathit{V}}_j^{*})d\tau$$

(21)

$$\mathrm{\Delta }{\mathit{V}}_0^{*}\left(\mathit{w}\left(t\right)\right)=\arg \underset{\mathrm{\Delta }{\mathit{V}}_0}{\min }{J}_0^{*}\left(\mathit{w}\left(t\right)\right)$$

(22)

For the followers,

$$J_j^{*}\left(\mathit{w}\left(t\right)\right)=\underset{\mathrm{\Delta }{\mathit{V}}_j}{min}{\int }_t^{\infty }{r}_i(\mathit{w},\mathrm{\Delta }{\mathit{V}}_0^{*},\mathrm{\Delta }{\mathit{V}}_j,{\mathit{V}}_{-j}^{*})d\tau$$

(23)

$${\mathit{V}}_j^{*}\left(\mathit{w}\left(t\right)\right)=\arg \underset{\mathrm{\Delta }{\mathit{V}}_j}{\min }{J}_j^{*}\left(\mathit{w}\left(t\right)\right)$$

(24)

where $\mathrm{\Delta }{\mathit{V}}_{-j}={\left\{\mathrm{\Delta }{\mathit{V}}_i\right\}}_{i\in \mathcal{F},i\ne j}$.

For all the players ($i\in \mathcal{V}$), define the Hamiltonian as

$${\mathcal{H}}_i\left(\mathit{w},\nabla J_i,\mathrm{\Delta }{V}_i\right)=r_i\left(\mathit{w},\mathrm{\Delta }{\mathit{V}}_i\right)+\nabla J_i^T\left(\sum _{i=0}^N{\mathit{B}}_i\left(\mathit{w}\right)\mathrm{\Delta }{\mathit{V}}_i\right)$$

(25)

where $\nabla J_i\triangleq \left(\partial J_i\left(\mathit{w}\right)/\partial \mathit{w}\right)$.

According to Definition 2, the optimal policy for the follower j is to minimize the value function (23) given the information of $\mathrm{\Delta }{\mathit{V}}_0$ and $T_{-fj}(\mathrm{\Delta }{\mathit{V}}_0)$, viz.,

$$\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0)=\arg \underset{\mathrm{\Delta }{\mathit{V}}_j}{\min }{\mathcal{H}}_j\left(\mathit{w},\nabla {\overline{J}}_j,\mathrm{\Delta }{\mathit{V}}_0,\mathrm{\Delta }{\mathit{V}}_j(\mathrm{\Delta }{\mathit{V}}_0),T_{-fj}^{*}(\mathrm{\Delta }{\mathit{V}}_0)\right)$$

(26)

where $\nabla {\overline{J}}_j=\mathrm{\Delta }{J}_j^{*}$ if $\mathrm{\Delta }{\mathit{V}}_0=\mathrm{\Delta }{\mathit{V}}_0^{*}$.

Similarly, the optimal strategy $\mathrm{\Delta }{\mathit{V}}_0$ for the leader can be derived by minimizing the value function (21) given the information of $\mathrm{\Delta }{\mathit{V}}_j(\mathrm{\Delta }{\mathit{V}}_0)$, viz.,

$$\mathrm{\Delta }{\mathit{V}}_0^{*}=\arg \underset{\mathrm{\Delta }{\mathit{V}}_0}{\min }{\mathcal{H}}_0\left(\mathit{w},\nabla J_j^{*},\mathrm{\Delta }{\mathit{V}}_0,\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0)\right)$$

(27)

By applying the stationary condition, the optimal policies for the leader and the followers can be solved as

$$\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0)=-{\mathit{D}}_j\mathrm{\Delta }{\mathit{V}}_0-{\displaystyle \frac{1}{2}}{\mathit{R}}_i^{-1}{\mathit{B}}_i^T\left(\mathit{w}\right)\nabla {\overline{J}}_j$$

(28)

$$\mathrm{\Delta }{\mathit{V}}_0^{*}={\displaystyle \frac{1}{2}}{\left({\mathit{K}}^T{\mathit{R}}_0\mathit{K}\right)}^{-1}{\left(\sum _{j=1}^N{\mathit{B}}_j{\mathit{D}}_j-{\mathit{B}}_0\right)}^T\nabla J_0^{*}+{\displaystyle \frac{1}{2}}{\mathit{K}}^{-1}\sum _{j=1}^N\left({\mathit{C}}_j{\mathit{R}}_j^{-1}{\mathit{B}}_j^T\nabla J_j^{*}\right)$$

(29)

where $\mathit{K}={\mathit{I}}_{3\times 3}-{\sum }_{j=1}^N{\mathit{C}}_j{\mathit{D}}_j\ne \mathbf{0}$ with ${\mathit{I}}_{3\times 3}$ being an identity matrix.

Combining Equations (28) and (29), one can derive the best response for each follower as

$$\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0^{*})=-{\mathit{D}}_j\mathrm{\Delta }{\mathit{V}}_0^{*}-{\displaystyle \frac{1}{2}}{\mathit{R}}_i^{-1}{\mathit{B}}_i^T\left(\mathit{w}\right)\nabla J_j^{*}$$

(30)

It is worth noting that the optimal value functions satisfy the following coupled Hamilton–Jacobi–Bellman (HJB) equations:

$$r_0(\mathit{w},\mathrm{\Delta }{\mathit{V}}_0^{*},\mathrm{\Delta }{\mathit{V}}_j^{*})+\nabla J_0^{T*}\left(\sum _{i=0}^N{\mathit{B}}_i\left(\mathit{w}\right)\mathrm{\Delta }{\mathit{V}}_i^{*}\right)=0$$

(31)

and

$$r_j(\mathit{w},\mathrm{\Delta }{\mathit{V}}_0^{*},\mathrm{\Delta }{\mathit{V}}_j^{*})+\nabla J_j^{T*}\left(\sum _{i=0}^N{\mathit{B}}_i\left(\mathit{w}\right)\mathrm{\Delta }{\mathit{V}}_i^{*}\right)=0$$

(32)

where $j\in \mathcal{F}$ and $i\in \mathcal{V}$.

The subsequent theorem guarantees that the strategies derived by solving the coupled HJB equations enable the UAV cluster to achieve Stackelberg–Nash equilibrium.

Theorem 1.

Consider the multiplayer SNG with the dynamic system (14). Let the functions $P_i\left(\mathit{w}\right),i\in \mathcal{V},P_i\left(\mathbf{0}\right)=0$ be positive semi-definite and satisfy the HJB Equations (31) and (32). Suppose that the control policies for the leader and followers are given by Equations (29) and (30). Then the control policy profile $\left\{\mathrm{\Delta }{\mathit{V}}_0^{*},\mathrm{\Delta }{\mathit{V}}_1^{*},\cdots ,\mathrm{\Delta }{\mathit{V}}_N^{*}\right\}$ is the Stackelberg–Nash equilibrium with the corresponding value function being $P_i\left(\mathit{w}\right)$, and the closed-loop system (14) is asymptotically stable.

Proof. 

Stability: Consider $P_0\left(\mathit{w}\right)$ as the Lyapunov candidate. Taking the derivative of $P_0\left(\mathit{w}\right)$ along the trajectory $\dot{\mathit{w}}={\mathit{B}}_0\mathrm{\Delta }{\mathit{V}}_0^{*}+{\sum }_{j=1}^N\mathrm{\Delta }{\mathit{B}}_j{\mathit{V}}_j^{*}$, we have

$${\dot{P}}_0\left(\mathit{w}\right)=\nabla P_0^T\left({\mathit{B}}_0\mathrm{\Delta }{\mathit{V}}_0^{*}+\sum _{j=1}^N{\mathit{B}}_j\mathrm{\Delta }{\mathit{V}}_j^{*}\right)=-{\parallel \mathit{w}\parallel }_{{\mathit{Q}}_0}^2-{\parallel \mathrm{\Delta }{V}_0^{*}+\sum _{j=1}^N{\mathit{C}}_j\mathrm{\Delta }{\mathit{V}}_j^{*}\parallel }_{{\mathit{R}}_0}^2<0$$

(33)

This leads to the asymptotic stability of the closed-loop system.

Optimality: According to the assumed conditions, it can be deduced that

$$\begin{array}{cc}\hfill J_0\left(\mathit{w}\left(0\right),\mathrm{\Delta }{\mathit{V}}^{*}\right)=& {\int }_0^{\infty }{r}_0(\mathit{w},\mathrm{\Delta }{\mathit{V}}^{*})dt+P_0\left(\mathit{w}\left(0\right)\right)+{\int }_0^{\infty }{\dot{L}}_0\left(\mathit{w}\right)dt\hfill \\ \hfill =& {\int }_0^{\infty }{\mathcal{H}}_0\left(\mathit{w},\nabla P_0,\mathrm{\Delta }{\mathit{V}}^{*}\right)dt+P_0\left(\mathit{w}\left(0\right)\right)\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill J_j\left(\mathit{w}\left(0\right),\mathrm{\Delta }{\mathit{V}}^{*}\right)=& {\int }_0^{\infty }{r}_j(\mathit{w},\mathrm{\Delta }{\mathit{V}}^{*})dt+P_j\left(\mathit{w}\left(0\right)\right)+{\int }_0^{\infty }{\dot{L}}_j\left(\mathit{w}\right)dt\hfill \\ \hfill =& {\int }_0^{\infty }{\mathcal{H}}_j\left(\mathit{w},\nabla P_j,\mathrm{\Delta }{\mathit{V}}^{*}\right)dt+P_j\left(\mathit{w}\left(0\right)\right),j\in \mathcal{F}\hfill \end{array}$$

(35)

By utilizing contradiction, one can derive

$$J_0(\mathit{w}\left(0\right),{\mathit{V}}^{*})=P_0\left(\mathit{w}\left(0\right)\right)$$

(36)

$$J_j(\mathit{w}\left(0\right),{\mathit{V}}^{*})=P_j\left(\mathit{w}\left(0\right)\right),j\in \mathcal{F}$$

(37)

The Hamiltonian can be rewritten as

$$\begin{array}{cc}\hfill & {\mathcal{H}}_0\left(\mathit{w},\nabla P_0,\mathrm{\Delta }\mathit{V}\right)\hfill \\ \hfill =& {\mathcal{H}}_0\left(\mathit{w},\nabla P_0,\mathrm{\Delta }{\mathit{V}}^{*}\right)+\nabla P_0^T\left[\sum _{i=0}^N{\mathit{B}}_i\left(\mathrm{\Delta }{\mathit{V}}_i-\mathrm{\Delta }{\mathit{V}}_i^{*}\right)\right]+r_0(\mathit{w},\mathrm{\Delta }\mathit{V})-r_0(\mathit{w},\mathrm{\Delta }{\mathit{V}}^{*})\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill & {\mathcal{H}}_j\left(\mathit{w},\nabla P_j,\mathrm{\Delta }\mathit{V}\right)\hfill \\ \hfill =& {\mathcal{H}}_j\left(\mathit{w},\nabla P_j,\mathrm{\Delta }{\mathit{V}}^{*}\right)+\nabla P_j^T\left[\sum _{i=0}^N{\mathit{B}}_i\left(\mathrm{\Delta }{\mathit{V}}_i-\mathrm{\Delta }{\mathit{V}}_i^{*}\right)\right]+r_j(\mathit{w},\mathrm{\Delta }\mathit{V})-r_j(\mathit{w},\mathrm{\Delta }{\mathit{V}}^{*})\hfill \end{array}$$

(39)

Substituting (38) and (39) into (34) and (35) yields

$$\begin{array}{cc}\hfill & J_0(\mathit{w}\left(0\right),\mathrm{\Delta }{\mathit{V}}_0,\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0))={\int }_0^{\infty }\left\{\nabla P_0^T\left[{\mathit{B}}_0\left({\mathit{V}}_0-{\mathit{V}}_0^{*}\right)+\sum _{j=1}^N{\mathit{B}}_j\left({\mathit{V}}_j^{*}\left({\mathit{V}}_0\right)-{\mathit{V}}_j^{*}\right)\right]\right.\hfill \\ \hfill & \left.+\parallel \mathrm{\Delta }{V}_0+\sum _{j=1}^N{\mathit{C}}_j\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0){\parallel }_{{\mathit{R}}_0}^2-{\parallel \mathrm{\Delta }{V}_0^{*}+\sum _{j=1}^N{\mathit{C}}_j\mathrm{\Delta }{\mathit{V}}_j^{*}\parallel }_{{\mathit{R}}_0}^2\right\}dt+P_0\left(\mathit{w}\left(0\right)\right)\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill & J_j(\mathit{w}\left(0\right),\mathrm{\Delta }{\mathit{V}}_0^{*},\mathrm{\Delta }{\mathit{V}}_j,\mathrm{\Delta }{\mathit{V}}_{-j}^{*})={\int }_0^{\infty }\left\{\nabla P_j^T\left[{\mathit{B}}_j\left({\mathit{V}}_j-{\mathit{V}}_j^{*}\right)\right]\right.\hfill \\ \hfill & \left.+\parallel \mathrm{\Delta }{V}_j+{\mathit{D}}_j\mathrm{\Delta }{\mathit{V}}_0^{*}{\parallel }_{{\mathit{R}}_j}^2-{\parallel \mathrm{\Delta }{V}_j^{*}+{\mathit{D}}_j\mathrm{\Delta }{\mathit{V}}_0^{*}\parallel }_{{\mathit{R}}_j}^2\right\}dt+P_j\left(\mathit{w}\left(0\right)\right)\hfill \end{array}$$

(41)

From the definition, we have

$${\mathcal{H}}_0\left(\mathit{w},\nabla P_0,\mathrm{\Delta }{\mathit{V}}^{*}\right)\le {\mathcal{H}}_0\left(\mathit{w},\nabla P_0,\mathrm{\Delta }{\mathit{V}}_0,\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0)\right)$$

(42)

$${\mathcal{H}}_j\left(\mathit{w},\nabla P_j,\mathrm{\Delta }{\mathit{V}}^{*}\right)\le {\mathcal{H}}_j\left(\mathit{w},\nabla P_j,\mathrm{\Delta }{\mathit{V}}_0^{*},\mathrm{\Delta }{\mathit{V}}_j,\mathrm{\Delta }{\mathit{V}}_{-j}^{*}\right)$$

(43)

Based on (42) and (43), we obtain

$$\begin{array}{cc}\hfill & \nabla P_0^T\left[\sum _{i=0}^N{\mathit{B}}_i\mathrm{\Delta }{\mathit{V}}_i^{*}\right]+{\parallel \mathrm{\Delta }{V}_0^{*}+\sum _{j=1}^N{\mathit{C}}_j\mathrm{\Delta }{\mathit{V}}_j^{*}\parallel }_{{\mathit{R}}_0}^2\hfill \\ \hfill \le & \nabla P_0^T\left[{\mathit{B}}_0{\mathit{V}}_0+\sum _{j=1}^N{\mathit{B}}_j\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0)\right]+{\parallel \mathrm{\Delta }{V}_0+\sum _{j=1}^N{\mathit{C}}_j\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0)\parallel }_{{\mathit{R}}_0}^2\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill & \nabla P_j^T\left[\sum _{i=0}^N{\mathit{B}}_i\mathrm{\Delta }{\mathit{V}}_i^{*}\right]+{\parallel \mathrm{\Delta }{V}_j^{*}+{\mathit{D}}_j\mathrm{\Delta }{\mathit{V}}_0^{*}\parallel }_{{\mathit{R}}_j}^2\hfill \\ \hfill \le & \nabla P_j^T\left[{\mathit{B}}_j{\mathit{V}}_j+\sum _{i=0,i\ne j}^N{\mathit{B}}_i\mathrm{\Delta }{\mathit{V}}_i^{*}\right]+{\parallel \mathrm{\Delta }{V}_0^{*}+{\mathit{D}}_j\mathrm{\Delta }{\mathit{V}}_j\parallel }_{{\mathit{R}}_j}^2\hfill \end{array}$$

(45)

Combining (40) and (41) and (44) and (45), we can finally derive

$$P_0\left(\mathit{w}\left(0\right)\right)\le J_0\left(\mathit{w}\left(0\right),\mathrm{\Delta }{\mathit{V}}_0,\mathrm{\Delta }{\mathit{V}}_j^{*}(\mathrm{\Delta }{\mathit{V}}_0)\right)$$

(46)

$$P_j\left(\mathit{w}\left(0\right)\right)\le J_j\left(\mathit{w}\left(0\right),{\mathit{V}}_0^{*},\mathrm{\Delta }{\mathit{V}}_j,\mathrm{\Delta }{\mathit{V}}_{-j}^{*}\right)$$

(47)

Therefore, the profile $\left\{{\mathit{V}}_0^{*},{\mathit{V}}_1^{*},\cdots {\mathit{V}}_N^{*}\right\}$ constitutes the Stackelberg–Nash equilibrium, and $P_i\left(\mathit{w}\left(0\right)\right)$ is the optimal value function. This completes the proof.    □

Although Theorem 1 shows that solving the coupled HJB Equations (31) and (32) can lead to the optimal control policies (29) and (30), this process involves considerable computational challenges. In the next subsection, an efficient algorithm is proposed to achieve the synchronous solution of the coupled HJB equations.

4.2. Leader–Follower Formation Control Algorithm

Let ${\mathit{\chi }}_i\triangleq \nabla J_i^{*}$, which serves as a critical intermediate variable in deriving the optimal policy. Motivated by fixed-point iteration [39], the HJB Equations (31) and (32) can be rewritten in the following form:

$${\left[\sum _{j=0}^N{\mathit{B}}_j\left(\mathit{w}\right)\mathrm{\Delta }{\mathit{V}}_j\right]}^T{\mathit{\chi }}_i=r_i\left(\mathit{w},\mathrm{\Delta }\mathit{V}\left({\mathit{\chi }}_i\right)\right)$$

(48)

where ${\sum }_{j=0}^N{\mathit{B}}_j\left(\mathit{w}\right)\mathrm{\Delta }{\mathit{V}}_j\ne \mathbf{0}$, and $\mathit{w}$ is a parameter.

Let ${\mathrm{\Gamma }}_i=r_i\left(\mathit{w},\mathrm{\Delta }\mathit{V}\left({\mathit{\chi }}_i\right)\right)$, and $\mathit{\beta }={\left[{\sum }_{j=0}^N{\mathit{B}}_j\left(\mathit{w}\right)\mathrm{\Delta }{\mathit{V}}_j\right]}^T$. Note that one can calculate ${\mathit{\chi }}_j$ via the least square method, viz.

$${\mathit{\chi }}_i=\mathcal{G}\left({\mathit{\chi }}_i\right)={\left({\mathit{\beta }}^{\mathit{T}}\mathit{\beta }\right)}^{-1}{\mathit{\beta }}^T{\mathrm{\Gamma }}_i$$

(49)

By adjusting the user-defined matrices ${\mathit{Q}}_i$ and ${\mathit{R}}_i$, one can ensure that $\underline{\mathit{\chi }}\le \mathcal{G}\left(\mathit{\chi }\right)\le \overline{\mathit{\chi }}$ for any $\mathit{\chi }\in \left[\underline{\mathit{\chi }},\overline{\mathit{\chi }}\right]$, and $\left|\frac{d\mathcal{G}}{d\mathit{\chi }}\right|<1$. Once the above two conditions are satisfied, we can construct an iterative sequence $\left\{{\mathit{\chi }}_i^l\right\}$ eventually converging to ${\mathit{\chi }}_j$. In other words, the coupled HJB Equations (31) and (32) can be solved via policy iteration, as presented in the subsequent algorithm.

Note that the approximate optimal policies obtained from Algorithm 1 represent the desired velocities for the UAVs’ additional motions. To enable UAVs to complete the formation reconfiguration task, it is essential to translate these commands into executable actions, specifically thrust, angle of attack, and tilt angle. This will be detailed in the subsequent section.

Algorithm 1 Policy iteration algorithm for multiplayer Stackelberg–Nash game

1: Initialization with $\mathrm{\Delta }{\mathit{X}}_i\left(t_0\right),i\in \mathcal{V}$. Let $l=0$, and ${\mathit{\chi }}_i^l={\left[1,1,1\right]}^T$. 2: Update the policy for the leader by $$\mathrm{\Delta }{\mathit{V}}_0^{l+1}={\displaystyle \frac{1}{2}}{\left({\mathit{K}}^T{\mathit{R}}_0\mathit{K}\right)}^{-1}{\left(\sum _{j=1}^N{\mathit{B}}_j{\mathit{D}}_j-{\mathit{B}}_0\right)}^T{\mathit{\chi }}_0^l+{\displaystyle \frac{1}{2}}{\mathit{K}}^{-1}\sum _{j=1}^N\left({\mathit{C}}_j{\mathit{R}}_j^{-1}{\mathit{B}}_j^T{\mathit{\chi }}_j^l\right)$$ (50) 3: Update the policies for the followers $j\in \mathcal{F}$ by $$\mathrm{\Delta }{\mathit{V}}_j^{l+1}=-{\mathit{D}}_j\mathrm{\Delta }{\mathit{V}}_0^l-{\displaystyle \frac{1}{2}}{\mathit{R}}_j^{-1}{\mathit{B}}_j^T\left(\mathit{w}\right){\mathit{\chi }}_j^l$$ (51) 4: Update ${\mathit{\chi }}_0$ for the leader by $$r_0(\mathit{w},\mathrm{\Delta }{\mathit{V}}_i^{l+1})+{\mathit{\chi }}_0^{l+1}\left(\sum _{i=0}^N{\mathit{B}}_i\left(\mathit{w}\right)\mathrm{\Delta }{\mathit{V}}_j^{l+1}\right)=0$$ (52) 5: Update ${\mathit{\chi }}_j$ for the followers by $$r_j(\mathit{w},\mathrm{\Delta }{\mathit{V}}_i^{l+1})+{\mathit{\chi }}_j^{l+1}\left(\sum _{i=0}^N{\mathit{B}}_i\left(\mathit{w}\right)\mathrm{\Delta }{\mathit{V}}_i^{l+1}\right)=0$$ (53) 6: Let $l=l+1$. Repeat procedures 2–5 until $\parallel {\mathit{\chi }}_i^l-{\mathit{\chi }}_i^{l-1}\parallel \le {\varsigma }_1$ (${\varsigma }_1$ is a small positive constant). Choose the final control policies $\mathrm{\Delta }{\mathit{V}}_0^l$ and $\mathrm{\Delta }{\mathit{V}}_j^l$ as the approximate optimal policies for the leader and followers.

4.3. Auxiliary Controller Design

In the subsequent discussion, to avoid ambiguity, the desired velocity for the additional motions calculated by Algorithm 1 is denoted as $\mathrm{\Delta }{\mathit{V}}_{i,d}$. Given the cruising velocity $\left[{\overline{V}}_x,{\overline{V}}_y,{\overline{V}}_z\right]$, the expected velocity for the i-th UAV can be determined by

$$\left\{\begin{array}{cc}\hfill & V_{xi,d}={\overline{V}}_x+\mathrm{\Delta }{V}_{xi,d}\hfill \\ \hfill & V_{yi,d}={\overline{V}}_y+\mathrm{\Delta }{V}_{yi,d}\hfill \\ \hfill & V_{zi,d}={\overline{V}}_z+\mathrm{\Delta }{V}_{zi,d}\hfill \end{array}\right.$$

(54)

By converting the expected velocity for the i-th UAV into the trajectory frame, one can obtain

$$\left\{\begin{array}{cc}\hfill & V_{i,d}=\sqrt{V_{xi,d}^2+V_{yi,d}^2+V_{zi,d}^2}\hfill \\ \hfill & {\gamma }_{i,d}={sin}^{-1}{\displaystyle \frac{V_{zi,d}}{V_{i,d}}}\hfill \\ \hfill & {\psi }_{i,d}={cot}^{-1}{\displaystyle \frac{V_{xi,d}}{V_{yi,d}}}\mathrm{or}{\psi }_{i,d}={tan}^{-1}{\displaystyle \frac{V_{yi,d}}{V_{xi,d}}}\hfill \end{array}\right.$$

(55)

It is worth noting that there are two approaches to solving ${\psi }_{i,d}$ in order to prevent singularity.

Define $\mathit{e}$ as the normalized tracking error

$${\mathit{e}}_i\triangleq {\left[{\displaystyle \frac{V_i-V_{i,d}}{{\overline{V}}_i}},{\gamma }_i-{\gamma }_{i,d},\psi -{\psi }_{i,d}\right]}^T$$

(56)

Then a PD controller can be designed to achieve the tracking control of the desired velocity for each UAV, viz.

$${\mathit{u}}_i=-{\mathcal{K}}_p{\mathit{e}}_i-{\mathcal{K}}_d{\dot{\mathit{e}}}_i$$

(57)

where ${\mathit{u}}_i={\left[T_i,{\alpha }_i,{\sigma }_i\right]}^T$, ${\mathcal{K}}_p$ and ${\mathcal{K}}_d$ are the PD controller gains.

Therefore, we can conclude the algorithm for the UAV formation reconfiguration control in Algorithm 2.

Algorithm 2 SNG-based control algorithm for the UAV formation reconfiguration

1: Initialization with the state being ${\mathit{X}}_i\left(0\right),i\in \mathcal{V}$. 2: Determine the destinations for all the UAVs. 3: Obtain the desired velocity commands ${\mathit{V}}_{i,d}$ by operating Algorithm 1. 4: Calculate the auxiliary control gains (57). 5: Propagate the state via (1)–(6) and (13). 6: Repeat procedures 3–5 until $\mathit{w}\le {\varsigma }_2$ (${\varsigma }_2$ is a predefined positive constant).

Remark 1.

Due to the similarity of the performance functions among the followers, adding an additional follower does not substantially increase the calculation time of Algorithm 1. Nevertheless, the computational complexity of the task allocation problem exhibits a “dimension curse” as the number of followers increases; thus, the number of UAVs cannot be expanded indefinitely.

5. Simulation Results

In this section, numerical simulations will be conducted to validate the effectiveness of the proposed scheme for UAV formation reconfiguration.

The relevant parameters of the model, flight, and aerodynamic coefficients are presented in

Table 1. Model parameters.

Items	Values
Mass (kg)	$85.68$
Reference Area (${\mathrm{m}}^2$)	$2.508$
Length (m)	$3.048$
Semi-Span (m)	$1.829$
Center of Gravity	$37\%$

,

Table 2. Fight parameters.

Items	Values
Velocity of the Baseline Motion (m/s)	${[30.45,0,0]}^T$
Origin of the Relative Motion Frame (in the Inertial Frame) (m)	${[500.0,0,0]}^T$
Trim Angle of Attack (deg)	$7.681$
Trim Bank Angle (deg)	0
Thrust for Cruising (N)	$240.4$

and

Table 3. Aerodynamic coefficients.

Items	Values
$C_L^0$	$0.1211$
$C_L^{\alpha }$	$5.397$
$C_D^0$	$0.0102$
$\kappa$	$0.1073$

, respectively. The external dimensions of the original Cessna 172 are provided in [42]. In this paper, the aircraft is scaled down and utilized as the fixed-wing UAVs. The center of mass is positioned at 37% of the length from the nose of the aircraft, which is ahead of the aerodynamic focus, thereby ensuring the static stability. The fight scenarios presented in Table 2 are user-defined. This article sets the flight altitude at approximately 500 m, primarily considering that cruising at this altitude is well-suited for tasks such as fire warning. The trim angle and thrust are calculated based on the cruising state and aerodynamic coefficients given in Table 3. Since all the UAVs are in a cruising state in the beginning, the initial bank angle is set to 0 deg. It should be particularly noted that the aerodynamic parameters are generated using the aerodynamic analysis software Tornado.

Assume that the total number of UAVs is 7. Set the initial positions of the UAVs numbered $s_1$ to $s_7$ in the inertial frame as

${\mathit{X}}_{s1}={[38.38,21.82,512.38]}^T$ m, ${\mathit{X}}_{s2}={[25.56,17.20,493.64]}^T$ m, ${\mathit{X}}_{s3}={[29.06,7.90,484.28]}^T$ m, ${\mathit{X}}_{s4}={[4.22,19.70,509.82]}^T$ m, ${\mathit{X}}_{s5}={[-16.00,4.92,486.92]}^T$ m, ${\mathit{X}}_{s6}={[-5.12,38.98,532.92]}^T$ m, ${\mathit{X}}_{s7}={[-4.04,-10.42,528.44]}^T$ m.

The desired formation configuration is an approximate regular hexagon in a horizontal plane and the leader is located at the geometry center. Set the expected relative positions with respect to the leader as

${\mathit{d}}_1={\left[-30,0,0\right]}^T\mathrm{m}$, ${\mathit{d}}_2={\left[-20,-20,0\right]}^T\mathrm{m}$, ${\mathit{d}}_3={\left[20,20,0\right]}^T\mathrm{m}$, ${\mathit{d}}_4={\left[30,0,0\right]}^T\mathrm{m}$, ${\mathit{d}}_5={\left[20,-20,0\right]}^T\mathrm{m}$, ${\mathit{d}}_6={\left[-20,20,0\right]}^T\mathrm{m}$.

Other relevant parameter settings are as follows. We choose ${\mathit{Q}}_i=\mathrm{diag}[1,1,1]$, ${\mathit{R}}_i=\mathrm{diag}[1,1,1]$, ${\mathit{C}}_j=\mathrm{diag}[0.2,0.2,0.2]$ and ${\mathit{D}}_j=\mathrm{diag}[0.4,0.4,0.4]$, where $i\in \mathcal{V},j\in \mathcal{F}$. The gains of the auxiliary control system is set to ${\mathcal{K}}_p=\left[{[100,1,10]}^T,{[100,1,10]}^T,{[100,1,10]}^T\right]$ and ${\mathcal{K}}_d=\left[{[30,0.3,3]}^T,{[30,0.3,3]}^T,{[30,0.3,3]}^T\right]$. The thresholds related to the termination of Algorithm 1 and Algorithm 2 are set to ${\varsigma }_1=1\times {10}^{-5}$ and ${\varsigma }_2=1{\mathrm{m}}^2$, respectively.

Based on the initial positions of the UAVs and the desired configuration, the exhaustive method, which involves a total of $7!=5040$ possible combinations, is employed for task allocation. By solving the optimization problem (15), the UAV originally numbered s4 was determined as the leader. The corresponding relationships between the other original numbers $\mathcal{S}$ and new numbers $\mathcal{D}$ are $s_1=2$, $s_2=1$, $s_3=6$, $s_4=0$, $s_5=4$, $s_6=5$ and $s_7=3$.

Next, the computer simulation of the formation reconfiguration is conducted using the MATLAB R2024a platform. Figure 3 illustrates the deviations between the current formation configuration and the desired configuration. As can be seen from the evolution of the key indicator $\mathit{w}$, at the initial moment, the projections on the $x,y,z$ directions of the inertial coordinate frame are $323.02{\mathrm{m}}^2$, $357.29{\mathrm{m}}^2$ and $1162.68{\mathrm{m}}^2$, respectively. As defined earlier, since $\mathit{w}$ represents the sum of the squares of the relative distances in the three directions, it is always non-negative, and the unit is ${\mathrm{m}}^2$. As time progresses, $\mathit{w}$ rapidly decreases in all three directions until it converges to within ${\varsigma }_2=1{\mathrm{m}}^2$ at 33.54, which corresponds to the termination condition of Algorithm 2. At 40 s, this indicator reaches $\mathit{w}(t=40)={[0.682,0.738,0.762]}^T{\mathrm{m}}^2$. Thus, we can conclude that the deviations between the current formation configuration and the desired configuration gradually converge to zero over time.

Figure 4 presents the flight trajectories of the fixed-wing UAVs in the inertial frame. For aesthetic purposes, the image is moderately stretched in the y and z directions. After the departure, within a $1200\mathrm{m}$ distance ahead, the UAV formation transitions to the predetermined configuration. The entire reconfiguration process is completed while in motion, and the flight trajectory of each UAV remains relatively smooth.

Figure 5 illustrates the 3D trajectories from the perspective of relative motion. The reference point of the relative motion frame is located at the initial position ${[500,0,0]}^T \mathrm{m}$, and the moving velocity remains constant at ${[30.45,0,0]}^T \mathrm{m}/\mathrm{s}$. The relative motion trajectories further corroborate that the UAVs precisely execute the flight plan, successfully achieving the predetermined configuration. It is noted that the flight trajectories do not show any crossing or close contact, which indicates that a safe distance is maintained between the UAVs throughout the entire flight process.

The coordinates of the leader and the followers in the relative motion frame are respectively given by Figure 6 and Figure 7. As can be observed from these several groups of “position–time” curves, following the trajectory adjustment, the UAVs eventually become relatively static, thereby forming a stable geometric configuration. It is worth noting that there is almost no overshoot in any of the UAV trajectories. This outcome arises from the independent decision-making process of each UAV aimed at achieving its shortest path, further indicating the optimality of the method proposed in this paper.

Moreover, Figure 8, Figure 9, Figure 10 and Figure 11 show the control inputs for the leader and followers during the formation reconfiguration stage, specifically including thrust T, angle of attack $\alpha$, and bank angle $\sigma$. As can be seen in these figures, the control inputs remain executable and eventually converge to their respective trim values.

6. Conclusions

This article studies the control strategies for formation reconfiguration of fixed-wing UAVs. In particular, a PI-based hierarchical control algorithm (i.e., Algorithm 1) is proposed to address the multiplayer SNG problem, and on this basis, a formation decision-making framework is established. To accurately implement the speed commands generated by the algorithm, an auxiliary control subsystem is developed, thereby ensuring the stable operation of the leader–follower flight control system. Furthermore, a task allocation mechanism has been constructed for the UAV swarm, thus laying a solid foundation for subsequent formation control. Specifically, based on the initial distribution of the UAVs and information of the desired configuration, with the shortest path serving as the indicator, the destinations of the UAVs are therefore determined.

This paper marks the beginning of our research on formation reconfiguration and optimal control, focusing on proposing a hierarchical decision-making framework for achieving the shortest path and conducting a theoretical analysis of the scheme’s feasibility. Owing to space constraints, we do not account for the information transmission delay, or external disturbances such as gusts and turbulence. These factors will be progressively addressed in our future research.