Optimization of Predefined-Time Agent-Scheduling Strategy Based on PPO

Abstract

In this paper, we introduce an agent rescue scheduling approach grounded in proximal policy optimization, coupled with a singularity-free predefined-time control strategy. The primary objective of this methodology is to bolster the efficiency and precision of rescue missions. Firstly, we have designed an evaluation function closely related to the average flying distance of agents, which provides a quantitative benchmark for assessing different scheduling schemes and assists in optimizing the allocation of rescue resources. Secondly, we have developed a scheduling strategy optimization method using the Proximal Policy Optimization (PPO) algorithm. This method can automatically learn and adjust scheduling strategies to adapt to complex rescue environments and varying task demands. The evaluation function provides crucial feedback signals for the PPO algorithm, ensuring that the algorithm can precisely adjust the scheduling strategies to achieve optimal results. Thirdly, aiming to attain stability and precision in agent navigation to designated positions, we formulate a singularity-free predefined-time fuzzy adaptive tracking control strategy. This approach dynamically modulates control parameters in reaction to external disturbances and uncertainties, thus ensuring the precise arrival of agents at their destinations within the predefined time. Finally, to substantiate the validity of our proposed approach, we crafted a simulation environment in Python 3.7, engaging in a comparative analysis between the PPO and the other optimization method, Deep Q-network (DQN), utilizing the variation in reward values as the benchmark for evaluation.

1. Introduction

With the rapid development of technology, the application of agents in the field of rescue has become increasingly widespread [1,2,3]. However, how to efficiently dispatch agents for rescue missions and ensure their accurate and rapid arrival at designated locations remains a significant issue facing current research. Traditional agent-dispatching methods often lack precise considerations for real-time performance and average arrival time [4,5], while control strategies may experience performance degradation in complex environments [6]. Therefore, the study of new methods for agent rescue dispatching and control is of great significance for improving rescue efficiency and reducing rescue costs.

In the context of rescue operations, the promptness of agent arrival holds paramount importance in determining the overall effectiveness of the rescue efforts [7,8,9]. Men et al. [10] proposed a multi-objective emergency response facility location (ERFsL) model for emergency management, taking into account the spatial characteristics of facilities and the potential risks associated with catastrophic interlocking chemical accidents. Regarding the application of swarm intelligence algorithms in positioning, Shalli Rani et al. [11] proposed a method that utilizes the krill herd algorithm for the non-ranging distributed processing of mobile target nodes in a maritime rescue network. Addressing the emergency dispatch issue for large-scale forest fires with multiple rescue centers (depots) and limited firefighting resources, Wang et al. [12] aim to minimize the total completion time of all firefighting tasks by determining the optimal rescue routes for firefighting teams from multiple rescue centers. Nonetheless, previous research has frequently neglected the need for the precise estimation and optimization of agent arrival times, thereby compromising the efficiency of scheduling strategies. By constructing an average distance function, we are able to more precisely evaluate the arrival time of agents under different scheduling strategies, thus optimizing the scheduling plan and ensuring that the agents arrive at the rescue site in the shortest possible time.

Amidst the complexity and volatility of rescue environments, a rational and swift dispatch strategy serves to optimize resource allocation and elevate utilization efficiency [13,14,15]. Traditional scheduling optimization methods often rely on mathematical programming techniques [16,17], such as linear programming, integer programming, and mixed-integer linear programming. These methods formulate scheduling problems as optimization problems with the objective of minimizing a specific cost function, such as makespan, tardiness, or total completion time. They utilize established algorithms, including simplex methods, branch-and-bound, and heuristic search algorithms, to find optimal or near-optimal solutions. Yue et al. [18] established an evaluation model for electric vehicle (EV) indicators, determined the weight of each indicator using the optimal combination weight method based on accelerated genetic algorithms, and proposed an optimized scheduling strategy to achieve optimal EV fleet scheduling. For the large-scale and time-consuming agile Earth observation satellite scheduling problem, Du et al. [19] propose a data-driven parallel scheduling method that incorporates a probability prediction model, a task allocation strategy, and a parallel scheduling approach. In the IEEE Time-Sensitive Networking (TSN) Task Group, Min et al. [20] proposed two metaheuristic optimization methods to enhance the scheduling success rate of the Time-Aware Shaper (TAS). However, these traditional methods can suffer from limitations, particularly when dealing with complex real-world scheduling problems characterized by uncertainty, dynamic changes, and non-linear constraints [21]. They may struggle to scale to large-scale instances, require significant computational resources, and often make assumptions that do not fully capture the complexities of real-world systems. Furthermore, traditional methods may not be flexible enough to accommodate emerging trends and requirements, such as real-time decision making and integration with machine learning techniques. To overcome the previously mentioned limitations, a scheduling strategy optimization method based on the PPO algorithm is designed. Combining the characteristics of the PPO algorithm, an optimization method suitable for agent rescue scheduling is designed, achieving the efficient optimization of the agent-scheduling strategy and improving rescue efficiency.

Predefined-time control can ensure that the intelligent agent arrives at the designated location within the pre-set time limit during the rescue process, thus ensuring the timeliness and effectiveness of the rescue operation [22,23,24]. To enable the direct preset of the upper bound of the settling time, ref. [25] first proposed a new concept called predefined-time stability. Predefined-time control can make the control system achieve stability before the user-preset time, which has aroused the research interest of scholars. Refs. [26,27] separately studied the predefined-time control design issues for first-order and second-order systems. In [28], a predefined-time leader–follower consensus protocol was established to achieve the predefined-time convergence of tracking errors. The authors of [29] investigated the predefined-time bipartite consensus control problem for multiple quadrotors under malicious attacks. However, the aforementioned predefined-time control design methods are proposed for control systems that satisfy the matching conditions and cannot be used to address the problem for non-strict feedback nonlinear systems. To overcome the aforementioned constraints, we formulate a precision-oriented fuzzy adaptive tracking control approach with non-singular predefined time, specifically tailored for nonlinear systems. This approach synergistically leverages the strengths of fuzzy control and adaptive control, empowering the agent with the capability to precisely calibrate control parameters during its dynamic motion, thereby effectively navigating through the complexities of diverse environmental scenarios.

The core innovations are highlighted as follows:

(1) In optimizing the dispatch of rescue agents, we have formulated an evaluation function centered around the average flying distance of the agents. This function serves as a rigorous assessment criterion for rescue mission scheduling, accurately quantifying the time taken by agents to reach their intended destinations.

(2) In the research on agent-task allocation, this paper designs a task allocation method based on the proximal policy optimization algorithm. This algorithm can adaptively adjust the task allocation strategy, thereby enhancing the collaborative efficiency of the agent team and improving the quality of task completion.

(3) Addressing the issue of controlling the arrival of agent at target locations, this paper proposes a singularity-free predefined-time fuzzy adaptive tracking control strategy tailored for nonlinear systems. The agents are capable of accurately reaching their destinations from the starting points with enhanced efficiency within a predefined time.

2. Preliminaries and Problem Description

2.1. Preliminaries

The following nonlinear system is to be taken into account:

$$\dot{x}\left(t\right)=f(x,t),x\left(0\right)=x_0,$$

(1)

where $x\in R^n$ signifies the state vector of the system and $f(x,t):R^n\to R^n$ is designated as a continuous function.

Definition 1.

When the state trajectory $x(t,x_0)$ of system (1) fulfills the condition $∥x(x_0,t)∥\le \epsilon$ after a predetermined time $T_d$ has elapsed, the equilibrium point is referred to as practical predefined-time stable (PPTS), where ε is a positive constant.

Lemma 1

(See [30]). Given a Lyapunov function $V\left(x\right)$ that pertains to system (1) and satisfies the inequality

$$\dot{V}\le -{\displaystyle \frac{\pi }{\theta T_d}}{V}^{1+{\displaystyle \frac{\theta }{2}}}-{\displaystyle \frac{\overline{\rho }\pi }{\theta T_d}}{V}^{1-{\displaystyle \frac{\theta }{2}}}+\mathsf{\Theta },$$

(2)

where $T_d$ is the predefined time and is positive, θ and $\overline{\rho }$ are design parameters with $0<\theta <1$ and $\overline{\rho }=\sqrt{2}+1$, and D is a positive constant. Under these circumstances, the equilibrium point of system (1) is deemed PPTS with the predefined time $T_d$. Additionally, the residual set of the state trajectory $x(t,x_0)$ can be defined as

$$x\in \{V\left(x\right):V\left(x\right)\le 2\theta T_d\mathsf{\Theta }/\pi \}.$$

(3)

2.2. Problem Description

In this paper, we are devoted to exploring the pivotal question of how to allocate resources and satisfy demands in rescue operations. Precisely, our operation involves the utilization of n rescue agents, all of which possess the capability to transport rescue packages with a standardized load capacity of m units. Furthermore, there are p disaster-affected rescue locations, each exhibiting varying degrees of need for rescue resources, which we quantify as $q_i$, where $i=1,2,\dots ,n$ signifies the individual rescue location. Concerning the issue of agent scheduling, we introduce a binary matrix $A\in {\mathbb{R}}^{n\times p}$ to depict the scheduling results. Specifically, $a_{ij}=1$ when agent i is dispatched to relief site j, and $a_{ij}=0$ otherwise. Concurrently, we establish a distance matrix $D\in {\mathbb{R}}^{n\times p}$, wherein $d_{ij}$ represents the flying distance between agent i and relief site j. The fundamental aim of this research is to achieve the minimization of the average flying distance for agents to reach relief sites while maintaining the satisfaction of all relief site requirements.

Based on the description provided, the evaluation function for the task-scheduling approach presented in this paper is formulated to accurately assess its performance, which is expressed as:

$$\begin{array}{cc}\hfill & min{\displaystyle \frac{{\sum }_{i=1}^n{\sum }_{j=1}^p{a}_{ij}{d}_{ij}}{n}}\hfill \\ \hfill & st.m\sum _{i=1}^n{a}_{ij}\ge q_{j}j=1,2,\dots ,p.\hfill \end{array}$$

(4)

The dynamic model chosen for the nonlinear multi-agent system is outlined below:

$$\begin{array}{c}\hfill \left\{\begin{array}{ccc}{\dot{x}}_i& =& f_i\left(x\right)+x_{i+1}+d_i\left(t\right),\\ {\dot{x}}_n& =& f_n\left(x\right)+u+d_n\left(t\right),\\ y& =& x_1,\end{array}\right.\end{array}$$

(5)

where u and y represent the input and output of the system, respectively, $x_i(i=1,2,\dots ,n)$ represents the system state variable, $d_i\left(t\right)$ represents the external disturbance bounded by an unknown positive constant $d_i^{\ast }$, $f_i\left(x\right)={\mathsf{\Phi }}_i^{\ast \mathrm{T}}\lambda \left(x\right)+{\epsilon }_i$ denotes the unknown system nonlinear dynamic, and ${\epsilon }_i$ is the minimum approximation error.

3. Scheduling Policy Optimization Method Based on PPO

In rescue operations involving agents, the scheduling strategy of agents is crucial for enhancing rescue effectiveness. Especially under resource constraints, the challenge lies in how to allocate agents fairly and efficiently. To address this issue, this paper introduces the PPO algorithm and explores its application in optimizing agent-scheduling strategies. The PPO algorithm, with its distinctive advantages such as swift learning speed and robust convergence performance, has offered formidable support for tackling this intricate problem. Its capability to efficiently traverse the vast search space and converge to optimal solutions, even in high-dimensional problems, underscores its applicability and significance in addressing complex decision-making challenges within the context of agent rescue operations.

To address the agent-scheduling strategy optimization problem, we adopt a Markov Decision Process (MDP) framework. In detail, the specifics are as follows:

State: The state is framed as an $n\times p$ matrix, encoding the enclosure status between agents and targets. Specifically, if the i-th agent is scheduling the j-th target at time t, ${\left(S_t\right)}_{ij}$ takes a value of 1; otherwise, it is 0.

Action: The action space comprises a vector $Ac$ of length p, where each element represents the allocation of agents to targets. Specifically, $Ac\left[i\right]$ indicates the identity of the agent assigned to surround the i-th target, thereby defining the scheduling strategy.

Reward: To optimize the multiple target assignment process using PPO, our primary goal is to maximize the efficiency of surrounding. This is achieved by defining a reward function as outlined in (4).

Return: The total return $R_t$ is calculated as the discounted sum of all future rewards, starting from $t+1$ and extending indefinitely. This is formally expressed as $R_t={\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}$, where $\gamma$ serves as the discount factor, set to 0.9 in this context, balancing the importance of immediate and future rewards.

The scheduling strategy optimization method based on PPO aims to iteratively seek the optimal scheduling strategy. The optimization of the agent-scheduling policy based on PPO is shown in Algorithm 1. It entails a systematic reinforcement learning process that initializes policy and value networks, iteratively collects data through agent–environment interactions, updates policy using a clipped surrogate loss function and importance sampling for stability and efficiency, concurrently refines the value function, and repeats this cycle to gradually improve scheduling strategies, harnessing PPO’s strengths in stability, efficiency, and flexibility. The specific process is as follows:

Algorithm 1 The optimization of the agent-scheduling policy based on PPO.

1: Initialize:   2: Number of agents: n   3: Policy network $\pi \left(a\right|s;\theta )$ with parameters $\theta$   4: Value network $V(s;{\theta }^{\prime })$ with parameters ${\theta }^{\prime }$   5: Replay buffer D   6: Learning rates ${\alpha }_{\mathrm{actor}}$, ${\alpha }_{\mathrm{critic}}$   7: Discount factor $\gamma$   8: Clipping factor $ϵ$   9: Number of iterations T  10: Batch size B  11: for $t=1$ to T do  12:     Clear micro-batch data $B_t$ for the current iteration  13:     for each episode do  14:         Initialize environment state s (e.g., locations and demands of all sites)  15:         for each agent $u=1$ to n do  16:            Select action $a_u$ (allocation of rescue packages) based on $\pi \left(a\right|s;\theta )$  17:            Execute $a_u$ (allocate packages to sites)  18:            Receive reward $r_u$ based on satisfaction of demands  19:            Observe next state $s^{\prime }$  20:            Store $(s,a_u,r_u,s^{\prime })$ in D  21:            Update $s\leftarrow s^{\prime }$  22:         end for  23:     end for  24:     for each training step do  25:         Sample a batch $B_t$ randomly from D  26:         for each $(s,a_u,r_u,s^{\prime })\in B_t$ do  27:            Compute advantage function $A(s,a_u)$  28:            Compute clipped surrogate objective $L_t\left(\theta \right)$  29:            Update policy network parameters $\theta$ using gradient ascent: $\theta \leftarrow \theta +{\alpha }_{\mathrm{actor}}{\nabla }_{\theta }{L}_t\left(\theta \right)$  30:            Compute value network target $V_{\mathrm{target}}=r_u+\gamma V(s^{\prime };{\theta }^{\prime })$  31:            Update value network parameters $\theta$ using mean squared error loss: $\theta \leftarrow \theta -{\alpha }_{\mathrm{critic}}{\nabla }_{\theta }{(V(s;\theta )-V_{\mathrm{target}})}^2$  32:         end for  33:     end for  34:     Evaluate the current policy $\pi \left(a\right|s;\theta )$ on a validation set  35:     if performance does not improve significantly then  36:         Adjust hyperparameters or improve the network structure  37:     end if  38: end for  39: Output: The trained policy network $\pi \left(a\right|s;\theta )$ for agent rescue package allocation

Step 1. Initialization of Policy Network: In the initial phase, we establish a randomly initialized policy network, typically a neural network with parameters $\theta$. This network takes the current state s as input and outputs two values: the probability distribution $\pi \left(a\right|s;\theta )$ for the action a (i.e., the scheduling decision) and the state value function $V(s;\theta )$. The objective of the policy network is to maximize the expected value of the cumulative rewards.

Step 2. Data Collection: In each iteration, we interact with the environment using the current policy network. Given the state $s_t$, the policy network selects an action $a_t$ based on the probability distribution $\pi \left(a\right|s_t;\theta )$ and executes it. The environment then responds with the next state $s_{t+1}$ and an immediate reward $r_t$. We collect these data points $(s_t,a_t,r_t,s_{t+1})$ to form trajectories, which are utilized for subsequent policy updates.

Step 3. Calculation of Advantage Function: The advantage function $A_{\pi }(s,a)$ represents the additional reward that can be obtained by taking action a in a given state s compared to the average reward obtained by following the current policy $\pi$. It can be as follows:

$$A_{\pi }(s,a)=Q_{\pi }(s,a)-V_{\pi }\left(s\right)$$

(6)

where $Q_{\pi }(s,a)$ is the action-value function, representing the expected cumulative reward obtained by following policy $\pi$ after taking action a in state s and $V_{\pi }\left(s\right)$ is the state-value function, representing the expected cumulative reward obtained by following policy $\pi$ in state s.

In this paper, we estimate the advantage function using the Generalized Advantage Estimation (GAE) approach. GAE estimates the advantage function by introducing the $\phi$-return method, which combines multi-step Temporal Difference (TD) errors. The TD error ${\delta }_t$ is defined as follows:

$${\delta }_t=r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right),$$

(7)

where $r_t$ is the immediate reward, $V\left(s_{t+1}\right)$ and $V\left(s_t\right)$ are the value function estimates for the next and current states, respectively, and $\gamma$ is the discount factor.

GAE calculates $A_t^{\mathrm{GAE}\left(\phi \right)}$ as a weighted sum of multi-step TD errors, specifically

$$A_t^{\mathrm{GAE}\left(\phi \right)}=\sum _{l=0}^{k-1}{\left(\gamma \phi \right)}^l{\delta }_{t+l},$$

(8)

where k is the number of steps considered and $\gamma$ is the GAE’s hyperparameter that balances bias and variance. When $\gamma$ is close to 1, GAE favors multi-step returns, thus reducing variance; when $\gamma$ is close to 0, GAE favors one-step returns, thus reducing bias.

Step 4. Update of Policy Network: Based on the PPO algorithm, we utilize the collected data and computed advantage function values to update the policy network. The PPO algorithm ensures stable updates by limiting the magnitude of changes between the old and new policies. Specifically, it employs a clipped surrogate objective function to update the parameters $\theta$ of the policy network, which can be expressed as follows:

$$L^{\mathrm{clip}}\left(\theta \right)={\mathrm{E}}_t\left[min\left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t\right)\right],$$

(9)

where $r_t\left(\theta \right)={\displaystyle \frac{\pi \left(a_t\right|s_t;\theta )}{\pi \left(a_t\right|s_t;{\theta }_{old})}}$ is the ratio of the probability of taking action $a_t$ under the new policy to that under the old policy, ${\widehat{A}}_t$ is an estimate of the advantage function $A_{\pi }(s_t,a_t)$, and $ϵ$ is a hyperparameter that controls the degree of change between the old and new policies.

We iteratively repeat the process from Step 2 to Step 4 until a predefined number of iterations is reached or a convergence criterion is satisfied. During this iterative procedure, the policy network gradually converges to an optimal scheduling strategy, which maximizes the expected cumulative reward.

4. Predefined-Time Control Design and Stability Analysis

We define the error of the system as follows:

$$\left\{\begin{array}{c}{\mu }_1=y-y_d,\hfill \\ \\ {\mu }_i=x_i-{\beta }_{i-1},\hfill \end{array}\right.$$

(10)

where ${\mu }_1$ represents the tracking error, $y_d$ stands for the desired command signal, ${\mu }_i$ refers to the virtual error variable, and ${\beta }_{i-1}$ signifies the intermediate control variable.

The following Lyapunov function is to be chosen:

$$V_1={\displaystyle \frac{1}{2}}{\mu }_1^2+{\displaystyle \frac{1}{2}}{\tilde{\mathsf{\Phi }}}_1^2,$$

(11)

where ${\tilde{\mathsf{\Phi }}}_1={\mathsf{\Phi }}_1^{\ast }-\widehat{\mathsf{\Phi }}$, $\widehat{\mathsf{\Phi }}$ is the estimated value of parameter ${\mathsf{\Phi }}_1^{\ast }$.

From (5), (10), and (11), one can obtain the following:

$${\dot{V}}_1={\mu }_1\left[{\mathsf{\Phi }}_1^{\ast \mathrm{T}}{\lambda }_1\left(x\right)+{\epsilon }_1+{\mu }_2+{\beta }_1+d_1\left(t\right)-{\dot{y}}_d\right]-{\tilde{\mathsf{\Phi }}}_1{\dot{\widehat{\mathsf{\Phi }}}}_1.$$

(12)

Additionally, in order to achieve the desired functionality, we can establish the intermediate control variable ${\beta }_1$ and a strategy for the adaptive update of ${\widehat{\mathsf{\Phi }}}_1$ as specified below:

$$\begin{array}{cc}\hfill {\beta }_1=& -{\displaystyle \frac{\phi \pi }{\theta T_d}}{\mu }_1^{1+\theta }tanh\left({\displaystyle \frac{\phi \pi }{\theta T_d}}{\displaystyle \frac{{\mu }_1^{2+\theta }}{{\nu }_1}}\right)+{\dot{y}}_d\hfill \\ \hfill & -{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\mu }_1^{1-\theta }tanh\left({\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\displaystyle \frac{{\mu }_1^{2-\theta }}{{\nu }_1}}\right)-{\mu }_1{\chi }_1{\widehat{\mathsf{\Phi }}}_1,\hfill \end{array}$$

(13)

$${\dot{\widehat{\mathsf{\Phi }}}}_1=-\gamma {\displaystyle \frac{\phi \pi }{\theta T_d}}{\widehat{\mathsf{\Phi }}}_1^{1+\theta }-\overline{\gamma }{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\widehat{\mathsf{\Phi }}}_1^{1-\theta }+{\mu }_1^2{\chi }_1,{\widehat{\mathsf{\Phi }}}_1\left(0\right)>0,$$

(14)

where $\theta$ is determined by dividing the positive even integer ${\theta }_1$ (greater than the positive odd integer ${\theta }_2$) by ${\theta }_2$. The time constant $T_d$ serves as a positive design parameter. The values of $c_0,\phi ,\overline{\phi },\gamma$, and $\overline{\gamma }$ are set to $\sqrt{2}+1,{\left(2n\right)}^{{\displaystyle \frac{\theta }{2}}}/2^{1+{\displaystyle \frac{\theta }{2}}},1/2^{1-{\displaystyle \frac{\theta }{2}}}$, $\gamma =2+\theta ,2-\theta$.

By incorporating Equations (13) and (14) into Equation (12) and subsequently applying the corresponding inequality, we obtain

$$\left|w\right|-wtanh(w/\nu )\le \tau \nu ,$$

(15)

where $\nu$ is a positive, intentionally chosen parameter and $\tau$ is equal to 0.2785, the desired outcome can be achieved.

$$\begin{array}{cc}\hfill {\dot{V}}_1\le & -{\displaystyle \frac{\phi \pi }{\theta T_d}}{\mu }_1^{2+\theta }tanh\left({\displaystyle \frac{\phi \pi }{\theta T_d}}{\displaystyle \frac{{\mu }_1^{2+\theta }}{{\nu }_1}}\right)+\gamma {\displaystyle \frac{\phi \pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_1{\widehat{\mathsf{\Phi }}}_1^{1+\theta }+{\mu }_1{\mu }_2\hfill \\ \hfill & -{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\mu }_1^{2-\theta }tanh\left({\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\displaystyle \frac{{\mu }_1^{2-\theta }}{{\nu }_1}}\right)+\overline{\gamma }{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_1{\widehat{\mathsf{\Phi }}}_1^{1-\theta }+{\displaystyle \frac{3}{4}}\hfill \\ \hfill \le & -{\displaystyle \frac{\phi \pi }{\theta T_d}}\left|{\mu }_1^{2+\theta }\right|+\gamma {\displaystyle \frac{\phi \pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_1{\widehat{\mathsf{\Phi }}}_1^{1+\theta }+{\mu }_1{\mu }_2\hfill \\ \hfill & -{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}\left|{\mu }_1^{2-\theta }\right|+\overline{\gamma }{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_1{\widehat{\mathsf{\Phi }}}_1^{1-\theta }+{\mathsf{\Theta }}_1,\hfill \end{array}$$

(16)

where ${\mathsf{\Theta }}_1={\displaystyle \frac{3}{4}}+2\tau {\nu }_1$.

In a similar manner to (11), we have selected the following Lyapunov function:

$$V_i=V_{i-1}+{\displaystyle \frac{1}{2}}{\mu }_i^2+{\displaystyle \frac{1}{2}}{\tilde{\mathsf{\Phi }}}_i^2,$$

(17)

Subsequently, we are able to derive

$$\begin{array}{c}{\dot{V}}_i={\dot{V}}_{i-1}+{\mu }_i\{{\mathsf{\Phi }}_i^{\ast \mathrm{T}}{\lambda }_i\left(x\right)+{\epsilon }_i+{\mu }_{i+1}+{\beta }_i+d_i\left(t\right)-H_{i-1}\\ -{\displaystyle \sum _{j=1}^{i-1}}{\displaystyle \frac{\partial {\beta }_{i-1}}{\partial x_j}}\left[{\mathsf{\Phi }}_j^{\ast \mathrm{T}}{\lambda }_j\left(x\right)+{\epsilon }_j+d_j\left(t\right)\right]\}-{\tilde{\mathsf{\Phi }}}_i{\dot{\widehat{\mathsf{\Phi }}}}_i.\end{array}$$

(18)

Similarly, we define the intermediate control variables ${\beta }_i$ and the adaptive update strategies $\widehat{{\mathsf{\Phi }}_i}$ as follow:

$$\begin{array}{c}{\beta }_i=-{\displaystyle \frac{\phi \pi }{\theta T_d}}{\mu }_i^{1+\theta }tanh\left({\displaystyle \frac{\phi \pi }{\theta T_d}}{\displaystyle \frac{{\mu }_i^{2+\theta }}{{\nu }_i}}\right)+H_{i-1}-{\mu }_i{\chi }_i{\widehat{\mathsf{\Phi }}}_i\\ -{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\mu }_i^{1-\theta }tanh\left({\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\displaystyle \frac{{\mu }_i^{2-\theta }}{{\nu }_i}}\right)-{\mu }_{i-1},\end{array}$$

(19)

$${\dot{\widehat{\mathsf{\Phi }}}}_i=-\gamma {\displaystyle \frac{\phi \pi }{\theta T_d}}{\widehat{\mathsf{\Phi }}}_i^{1+\theta }-\overline{\gamma }{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\widehat{\mathsf{\Phi }}}_i^{1-\theta }+{\mu }_i^2{\chi }_i,{\widehat{\mathsf{\Phi }}}_i\left(0\right)>0.$$

(20)

By incorporating Equations (19) and (20) into Equation (18) and utilizing the inequality stated in (15), one can obtain

$$\begin{array}{cc}\hfill {\dot{V}}_i\le & -\sum _{j=1}^{i-1}{\displaystyle \frac{\phi \pi }{\theta T_d}}\left|{\mu }_j^{2+\theta }\right|+\sum _{j=1}^i\gamma {\displaystyle \frac{\phi \pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1+\theta }\hfill \\ \hfill & -\sum _{j=1}^{i-1}{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}\left|{\mu }_j^{2-\theta }\right|+\sum _{j=1}^i\overline{\gamma }{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1-\theta }\hfill \\ \hfill & -{\displaystyle \frac{\phi \pi }{\theta T_d}}{\mu }_i^{2+\theta }tanh\left({\displaystyle \frac{\phi \pi }{\theta T_d}}{\displaystyle \frac{{\mu }_i^{2+\theta }}{{\nu }_i}}\right)\hfill \\ \hfill & -{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\mu }_i^{2-\theta }tanh\left({\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\displaystyle \frac{{\mu }_i^{2-\theta }}{{\nu }_i}}\right)+{\mu }_i{\mu }_{i+1}+{\mathsf{\Theta }}_{i-1}+{\displaystyle \frac{3i}{4}}\hfill \\ \hfill & \le -\sum _{j=1}^i{\displaystyle \frac{\phi \pi }{\theta T_d}}\left|z_j^{2+\theta }\right|+\sum _{j=1}^i\gamma {\displaystyle \frac{\phi \pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1+\theta }+{\mu }_i{\mu }_{i+1}\hfill \\ \hfill & -\sum _{j=1}^i{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}\left|z_j^{2-\theta }\right|+\sum _{j=1}^i\overline{\gamma }{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1-\theta }+{\mathsf{\Theta }}_i,\hfill \end{array}$$

(21)

where ${\mathsf{\Theta }}_i={\mathsf{\Theta }}_{i-1}+{\displaystyle \frac{3i}{4}}+2\tau {\nu }_i$.

Consistent with our approach, we adopt the Lyapunov function specified below:

$$V_n=V_{n-1}+{\displaystyle \frac{1}{2}}{\mu }_n^2+{\displaystyle \frac{1}{2}}{\tilde{\mathsf{\Phi }}}_n^2,$$

(22)

then we can obtain

$$\begin{array}{cc}\hfill {\dot{V}}_n=& {\dot{V}}_{n-1}+{\mu }_n\left\{{\mathsf{\Phi }}_n^{\ast \mathrm{T}}{\lambda }_n\left(x\right)+{\epsilon }_n+u+d_n\left(t\right)-H_{n-1}\right.\hfill \\ \hfill & \begin{array}{cc}& -\sum _{j=1}^{n-1}{\displaystyle \frac{\partial {\beta }_{n-1}}{\partial x_j}}\left[{\mathsf{\Phi }}_j^{\ast \mathrm{T}}{\lambda }_j\left(x\right)+{\epsilon }_j\left(x\right)+d_j\left(t\right)\right]\}-{\tilde{\mathsf{\Phi }}}_n{\dot{\widehat{\mathsf{\Phi }}}}_n.\hfill \end{array}\hfill \end{array}$$

(23)

Similarly, we define the predefined-time controller u and the adaptive update strategies $\widehat{{\mathsf{\Phi }}_n}$ as follows:

$$\begin{array}{c}u=-{\displaystyle \frac{\phi \pi }{\theta T_d}}{\mu }_n^{1+\theta }tanh\left({\displaystyle \frac{\phi \pi }{\theta T_d}}{\displaystyle \frac{{\mu }_n^{2+\theta }}{{\nu }_n}}\right)+H_{n-1}-{\mu }_n{\chi }_n{\widehat{\mathsf{\Phi }}}_n\\ -{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\mu }_n^{1-\theta }tanh\left({\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\displaystyle \frac{{\mu }_n^{2-\theta }}{{\nu }_n}}\right)-z_{n-1},\end{array}$$

(24)

$${\dot{\widehat{\mathsf{\Phi }}}}_n=-\gamma {\displaystyle \frac{\phi \pi }{\theta T_d}}{\widehat{\mathsf{\Phi }}}_n^{1+\theta }-\overline{\gamma }{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\widehat{\mathsf{\Phi }}}_n^{1-\theta }+{\mu }_n^2{\chi }_n,{\widehat{\mathsf{\Phi }}}_n\left(0\right)>0.$$

(25)

Substituting (24) and (25) into (23) and applying the inequality in (15), one can obtain

$$\begin{array}{cc}\hfill {\dot{V}}_n\le & -\sum _{j=1}^{n-1}{\displaystyle \frac{\phi \pi }{\theta T_d}}\left|{\mu }_j^{2+\theta }\right|+\sum _{j=1}^n\gamma {\displaystyle \frac{\phi \pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1+\theta }\hfill \\ \hfill & -\sum _{j=1}^{n-1}{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}\left|{\mu }_j^{2-\theta }\right|+\sum _{j=1}^n\overline{\gamma }{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1-\theta }\hfill \\ \hfill & -{\displaystyle \frac{\phi \pi }{\theta T_d}}{\mu }_n^{2+\theta }tanh\left({\displaystyle \frac{\phi \pi }{\theta T_d}}{\displaystyle \frac{{\mu }_n^{2+\theta }}{{\nu }_n}}\right)\hfill \\ \hfill & -{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\mu }_n^{2-\theta }tanh\left({\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\displaystyle \frac{{\mu }_n^{2-\theta }}{{\nu }_n}}\right)+{\mathsf{\Theta }}_{n-1}+{\displaystyle \frac{3n}{4}}\hfill \\ \hfill \le & -\sum _{j=1}^n{\displaystyle \frac{\phi \pi }{\theta T_d}}\left|{\mu }_j^{2+\theta }\right|+\sum _{j=1}^n\gamma {\displaystyle \frac{\phi \pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1+\theta }\hfill \\ \hfill & -\sum _{j=1}^n{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}\left|{\mu }_j^{2-\theta }\right|+\sum _{j=1}^n\overline{\gamma }{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1-\theta }+{\mathsf{\Theta }}_n,\hfill \end{array}$$

(26)

where ${\mathsf{\Theta }}_n={\mathsf{\Theta }}_{n-1}+{\displaystyle \frac{3n}{4}}+2\tau {\nu }_n$.

The following inequalities can be derived from [31,32]:

$$\gamma {\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1+\theta }\le {\mathsf{\Phi }}_j^{\ast (2+\theta )}-{\widehat{\mathsf{\Phi }}}_j^{2+\theta }\le 2{\mathsf{\Phi }}_j^{\ast (2+\theta )}-{\tilde{\mathsf{\Phi }}}_j^{2+\theta },$$

(27)

$$\overline{\gamma }{\tilde{\mathsf{\Phi }}}_j{\widehat{\mathsf{\Phi }}}_j^{1-\theta }\le {\mathsf{\Phi }}_j^{\ast (2-\theta )}-{\widehat{\mathsf{\Phi }}}_j^{2-\theta }\le 2{\mathsf{\Phi }}_j^{\ast (2-\theta )}-{\tilde{\mathsf{\Phi }}}_j^{2-\theta }.$$

(28)

Substituting (27) and (28) into (26), one has

$$\begin{array}{cc}\hfill {\dot{V}}_n\le & -\sum _{j=1}^n{\displaystyle \frac{\phi \pi }{\theta T_d}}\left|{\mu }_j^{2+\theta }\right|-\sum _{j=1}^n{\displaystyle \frac{\phi \pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j^{2+\theta }\hfill \\ \hfill & -\sum _{j=1}^n{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}\left|{\mu }_j^{2-\theta }\right|-\sum _{j=1}^n{\displaystyle \frac{\overline{\phi }{c}_0\pi }{\theta T_d}}{\tilde{\mathsf{\Phi }}}_j^{2-\theta }+\mathsf{\Theta },\hfill \end{array}$$

(29)

where $\mathsf{\Theta }={\mathsf{\Theta }}_n+{\sum }_{j=1}^n{\displaystyle \frac{\pi {\left(2n\right)}^{\frac{\theta }{2}}}{2^{\frac{\theta }{2}}\theta T_d}}{\mathsf{\Phi }}_j^{\ast (2+\theta )}+{\sum }_{j=1}^n{\displaystyle \frac{c_0\pi }{2^{-\frac{\theta }{2}}\theta T_d}}{\mathsf{\Phi }}_j^{\ast (2-\theta )}.$

According to [33,34], for any $x_i\in R$, there exist constants p and q satisfying $0<p\le 1$ and $1<q\le \infty$, for which the following inequalities are valid:

$${\left(\sum _{i=1}^n\left|x_i\right|\right)}^p\le \sum _{i=1}^n|x_i{|}^p,{\left(\sum _{i=1}^n\left|x_i\right|\right)}^q\le n^{q-1}\sum _{i=1}^n{\left|x_i\right|}^q.$$

(30)

Then, one can obtain

$$\begin{array}{cc}\hfill {\dot{V}}_n\le & -{\displaystyle \frac{\pi }{\theta T_d}}{\left[\sum _{j=1}^n\left({\displaystyle \frac{1}{2}}{\mu }_j^2+{\displaystyle \frac{1}{2}}{\tilde{\mathsf{\Phi }}}_j^2\right)\right]}^{1+{\displaystyle \frac{\theta }{2}}}\hfill \\ \hfill & -{\displaystyle \frac{c_0\pi }{\theta T_d}}{\left[\sum _{j=1}^n\left({\displaystyle \frac{1}{2}}{\mu }_j^2+{\displaystyle \frac{1}{2}}{\tilde{\mathsf{\Phi }}}_j^2\right)\right]}^{1-{\displaystyle \frac{\theta }{2}}}+\mathsf{\Theta }\hfill \\ \hfill =& -{\displaystyle \frac{\pi }{\theta T_d}}{V}_n^{1+{\displaystyle \frac{\theta }{2}}}-{\displaystyle \frac{c_0\pi }{\theta T_d}}{V}_n^{1-{\displaystyle \frac{\theta }{2}}}+\mathsf{\Theta }.\hfill \end{array}$$

(31)

5. Simulation Setup And Result Analysis

5.1. Simulation Setup

To validate the effectiveness of the theoretical derivation and the proposed agent-scheduling optimization strategy, this section designs and executes a series of simulation experiments using the Python programming language. To comprehensively consider the potential impact of the number of agents and the scale of rescue areas on algorithm performance, we specifically set up two experimental scenarios with different characteristics to demonstrate the versatility and adaptability of the algorithm under various conditions. In both scenarios, each agent is set to carry one rescue package as the task load. Specifically, scenario one simulates a multi-agent system consisting of ten agents and three rescue areas, with the initial positions of the agents and the locations of the rescue targets detailed in

Table 1. The initial position of the agent in Scenario 1.

Agent	Lat	Lon	Agent	Lat	Lon
1	0.0426	1.0557	6	4.5103	7.9949
2	2.8584	0.2695	7	1.4984	4.0147
3	4.7161	0.6012	8	0.5418	4.5941
4	7.7187	7.4370	9	1.7561	9.4918
5	5.9442	8.8786	10	8.4732	8.7486

and

Table 2. The information of rescue areas in Scenario 1.

Rescue Area	Lat	Lon	Demand
1	6.4831	2.1476	3
2	9.4930	0.1211	3
3	1.8088	1.8772	4

, respectively. Scenario two further expands the system complexity by introducing fourteen agents and four rescue areas, and, similarly, the initial deployment of agents and the locations of task targets are set according to

Table 3. The initial position of the agent in Scenario 2.

Agent	Lat	Lon	Agent	Lat	Lon
1	6.1470	3.8101	8	7.1160	2.0503
2	6.3711	4.7446	9	3.0778	9.8086
3	4.4253	0.9577	10	0.1026	4.6604
4	6.1416	0.5733	11	4.6038	8.5465
5	5.6571	5.3323	12	4.5247	6.3166
6	3.9005	9.0885	13	4.7600	2.2003
7	5.3337	7.0734	14	7.1359	6.1904

and

Table 4. The information of rescue areas in Scenario 2.

Rescue Area	Lat	Lon	Demand
1	2.1660	2.5708	3
2	0.4580	1.7551	3
3	6.1767	8.2907	4
4	5.2463	2.7081	4

.

This paper simulation platform is $PyTorch$, and the specific parameter configuration of PPO is given in

Table 5. Hyperparameter configuration.

Notation	Meaning
Time steps	20,000
Number of agents	10/14
Number of rescue areas	3/4
Actor learning rate	0.01
Critic learning rate	0.001
Discount	0.9

. And the structure of the PPO networks is given in

Table 6. Structure of the PPO networks.

Policy Network	Value Network
Input(c,state_dim)	Input(c,state_dim)
Linear(state_dim,hidden_dim)	Linear(state_dim,hidden_dim)
relu	softmax
Linear(hidden_dim,1)	Linear(hidden_dim,1)

To demonstrate the outstanding performance of the method proposed in this study, we conducted an in-depth comparison between the PPO algorithm and the classic DQN, a traditional optimization technique. This move aims to validate the significant advantages of the PPO algorithm in terms of optimization efficiency and effectiveness through direct performance evaluation.

For the singularity-free predefined-time fuzzy adaptive tracking control strategy based on nonlinear systems, in (13) and (14) and (24) and (25), we set $T_d=5$, $\theta ={\displaystyle \frac{{\theta }_1}{{\theta }_2}}={\displaystyle \frac{6}{75}}$ and ${\nu }_1={\nu }_2=0.001$.

$${\lambda }_c\left(x\right)={\displaystyle \frac{{\prod }_{i=1}^b{\varpi }_{B_i^c}\left(x_i\right)}{{\sum }_{c=1}^{\iota }\left[{\prod }_{i=1}^b{\varpi }_{B_i^c}\left(x_i\right)\right]}}.$$

(32)

The fuzzy membership functions are selected as

$${\varpi }_{B_1^c}={\mathrm{e}}^{-{\displaystyle \frac{{(x_1-3+c)}^2}{16}}},{\varpi }_{B_2^c}={\mathrm{e}}^{-{\displaystyle \frac{{(x_2-6+2c)}^2}{4}}},$$

(33)

where the number of fuzzy rules is $\iota =5$ and $b=1,2$.

5.2. Result Analysis

Figure 1 and Figure 2 comprehensively exhibit the comparative results of agent scheduling under two distinct scenarios, aiming to delve into and compare the performance of two scheduling strategies. Specifically, Figure 1a visually presents the detailed layout of task allocation among agents under the random scheduling strategy in a scenario with ten agents and three rescue areas, where the average distance between agents is 6.61. In contrast, Figure 1b reveals the optimized state of task allocation and a more compact layout among agents in the same scenario after adopting the PPO-optimized scheduling strategy, with the average distance reduced to 5.40. Furthermore, Figure 2a,b showcase the contrast between the effects of random scheduling and PPO-optimized scheduling in a complex scenario expanded to fourteen agents and four rescue areas, with the latter significantly decreasing the average distance between agents from 4.88 to 3.72. This series of comparative analyses not only highlights the superiority of the PPO algorithm in shortening the average distance from agents to rescue areas, but also validates the algorithm’s wide applicability and efficiency in solving problems through the setting of two different scenarios.

This figure, compared to the results under the random scheduling strategy, demonstrates a remarkably optimized effect. This finding not only validates the effectiveness of the PPO strategy in scheduling optimization, but also provides a strong basis for our future research and applications. By comparing and analyzing the performance of these two strategies, we can gain a clearer understanding of the differences in resource allocation and efficiency optimization between different strategies, providing valuable references for future research and applications.

This paper comparatively analyzes the performance of the PPO algorithm against the DQN algorithm, aiming to demonstrate the superiority of the proposed approach. Figure 3 vividly illustrates the distinct differences in average return values and respective trend characteristics between the two. Specifically, the figure encapsulates the learning progression of both algorithms under Scenario 2, as devised in this study, providing quantitative evidence for assessing their efficiency and stability. Observing the DQN algorithm, representative of traditional methods, its average return exhibits a gradual decline from a peak of 10.5 during the initial 15,000 iterations, subsequently stabilizing over the next 5000 iterations to converge at approximately 7.9. This phenomenon underscores DQN’s potential for overfitting in early learning phases and its relatively slower convergence rate, limiting performance enhancements in complex environments which require rapid adaptation and optimization. In contrast, the PPO algorithm demonstrates remarkable convergence efficiency and optimization prowess. As depicted in Figure 3, PPO’s average return swiftly peaks at 9.5 before steadily declining and converging to around 6.2 after approximately 12,500 iterations, advancing 2500 iterations ahead of DQN. This outcome underscores PPO’s efficiency in strategy optimization and underscores its unique advantage in maintaining training stability through constrained policy update steps. PPO’s ability to swiftly identify and stabilize on a relatively superior strategy underscores its capacity to offer more efficient results.

The simulation results of the non-singular predefined-time fuzzy adaptive tracking control strategy based on nonlinear systems are presented in detail in Figure 4, Figure 5, Figure 6 and Figure 7. Figure 4 presents a comparison of the trajectories of the system output signal y and the desired command signal $y_d$, demonstrating a high degree of overlap between the two signals, which visually showcases the dynamic response of the system. Figure 5 shows the trajectory of the tracking error, indicating that before the preset time $T_d=5$, the tracking error ${\mu }_1$ can stably converge to a very small range near zero. This result fully validates the effectiveness and accuracy of the control algorithm. As evident from Figure 4 and Figure 5, the proposed preset-time control algorithm in this study exhibits excellent tracking control performance. Figure 6 presents the trajectory of the control system’s input signal u, while Figure 7 depicts the evolution trajectory of the adaptive parameter $\mathsf{\Phi }$. This demonstrates that the proposed preset-time control method in this paper enables the agent to reach its destination within the desired time frame for rescue.

6. Conclusions

In this paper, we introduce innovative scheduling and control strategies for the rescue missions of intelligent agents. We construct a function related to the average distance of the intelligent agents and design an optimization method for the scheduling strategy based on the PPO algorithm, achieving efficient response to rescue missions. Furthermore, we propose a singularity-free predefined-time fuzzy adaptive tracking control strategy based on nonlinear systems, ensuring that the intelligent agents can accurately reach designated locations within the predetermined time. Finally, two independent experimental scenarios are designed to comprehensively evaluate the universality and efficiency of the proposed method. Specifically, to highlight the superiority of the PPO algorithm in optimizing scheduling strategies, the PPO algorithm is directly compared with the classical DQN algorithm. The experimental results show that, when compared with the scheduling strategy optimized by the DQN algorithm, the strategy generated by the PPO algorithm significantly shortens the average travel distance for the agent to complete the task by 1.7 units, while achieving a 27% increase in efficiency, effectively demonstrating the superior performance of the PPO algorithm in the field of scheduling optimization. These innovative approaches not only improve the rescue efficiency, but also enhance the reliability of the intelligent agents in complex environments.