Reinforcement Learning-Based Energy-Saving Path Planning for UAVs in Turbulent Wind

Abstract

The unmanned aerial vehicle (UAV) is prevalent in power inspection. However, due to a limited battery life, turbulent wind, and its motion, it brings some challenges. To address these problems, a reinforcement learning-based energy-saving path-planning algorithm (ESPP-RL) in a turbulent wind environment is proposed. The algorithm dynamically adjusts flight strategies for UAVs based on reinforcement learning to find the most energy-saving flight paths. Thus, the UAV can navigate and overcome real-world constraints in order to save energy. Firstly, an observation processing module is designed to combine battery energy consumption prediction with multi-target path planning. Then, the multi-target path-planning problem is decomposed into iterative, dynamically optimized single-target subproblems, which aim to derive the optimal discrete path solution for energy consumption prediction. Additionally, an adaptive path-planning reward function based on reinforcement learning is designed. Finally, a simulation scenario for a quadcopter UAV is set up in a 3-D turbulent wind environment. Several simulations show that the proposed algorithm can effectively resist the disturbance of turbulent wind and improve convergence.

1. Introduction

1.1. Background

In recent years, the unmanned aerial vehicle (UAV) technology has gradually matured, demonstrating diverse potentials and gaining widely attention in engineering research and market applications [1,2]. The endurance time and path planning of UAVs is crucial for information acquisition and task execution of path planning. In practical applications, such as power inspection, aerial photography, and logistics, UAVs need to have sufficient flight range and endurance time to complete the task of path planning [3]. However, the endurance issue has always been a major factor restricting the development of UAV technology research and market applications. Therefore, how to effectively improve the endurance time of UAVs has become an urgent problem to be solved in the industry.

UAVs can be classified into several common types based on their mechanical structures, including multi-rotor drones, fixed-wing drones, and unmanned helicopters. Multi-rotor drones and unmanned helicopters are characterized by a flexible attitude, the capability to ascend and descend vertically, the ability to hover freely, low requirements for the environment, and a site for take-off and landing [4]. However, they have some limitations in terms of load capacity, flight speed, and endurance. Therefore, it becomes crucial for UAV energy-saving path planning to understand how the power consumption of drones changes with different flight states.

The traditional battery energy consumption management system of UAVs cannot be directly applied to drones. Due to different practical application scenarios, for example, the UAV has different motion modes and thus different energy consumption losses during autonomous landing and horizontal flight. Furthermore, the influence of three-dimensional(3-D) turbulent wind environmental factors during the flight of the UAV makes the traditional battery energy consumption management system for the UAV. At present, researchers have established battery energy consumption management systems for fixed-wing UAVs and single-rotor UAVs or provided heuristic battery energy consumption management solutions for multi-rotor UAVs [5]. The power consumption model of the UAV has been extensively studied by domestic and foreign scholars [6], mainly divided into three categories: ① the data-driven heuristic battery energy consumption management model. In [7], a quadcopter battery energy consumption management model based on collected data-driven approaches is proposed. By collecting information such as flight speed data and heading angle, the algorithm infers the battery energy-loss model under flight conditions, and then designs an energy consumption management model. However, this algorithm fails to consider designing the battery energy consumption management model from the perspective of task optimization, such as path-planning tasks. Similarly, given the case of a UAV, a power consumption model involving its speed and working conditions is presented in [8] based on experimental results. However, the power consumption models presented in [8,9] are solely related to speed, overlooking the impact of other crucial factors on the power consumption of UAVs, such as the number of rotors and payload weight. Therefore, the lack of more accurate closed-form expressions limits their applications. In addition, by analyzing battery performance, some power consumption models for electric UAVs have been derived in [5,10,11], and the aforementioned shortcomings have been identified. Therefore, in order to theoretically investigate the power consumption of UAVs and improve the applicability of the obtained models, theoretical power consumption models based on kinematics and aircraft theory are considered. ② Battery energy consumption management model based on UAV kinematics theory. In [9], the authors propose a general fixed-wing energy-loss model based on speed and acceleration from the perspective of UAV kinematics theory, but this model is not suitable for rotor drones. In [12], a closed-form power consumption model is derived for a single-rotor UAV under constant-speed 1-D forward flight, which is used to study energy-efficient UAV communications. This model is then extended in [13,14] to derive analytical models for a single-rotor UAV performing 2-D forward flight. Furthermore, the model in [12] was validated in [15] by fitting measured data from UAVs collected from extensive experiments. Additionally, a power consumption model for a single-rotor UAV during vertical flight was derived in [14]. All these works provide valuable guidance for establishing UAV power consumption models. However, considering that multi-rotor UAVs are the most popular type of UAV today, the existing works [5,7,8,10] only provide heuristic power consumption models without rigorous mathematical derivations, which limits their application in many research areas. Therefore, it becomes necessary to derive a theoretical power consumption model for multi-rotor UAVs. Furthermore, previous studies [9,13,14] only considered the flight status of UAVs in 1-D or 2-D scenarios, overlooking the general 3-D scenario. This prompts the need to investigate corresponding power consumption characteristics in order to study the power consumption behavior of UAVs in 3-D scenarios. ③ Battery energy consumption management model for multi-rotor drones. The authors in [16] proposed a closed-form mathematical expression model for battery energy loss in 1-D or 2-D real-world flight scenarios, which can be extended to 3-D real-world scenarios. However, this algorithm neglects the research on the closed-loop learning framework between path planning and data collection (factors such as the UAV’s motion model and 3-D turbulent wind), making it difficult to effectively guide the UAV in adaptive path planning under limited energy consumption. Evidently, the battery energy consumption management system for UAVs has evolved from 1-D to 3-D real-world scenarios, extending from a data-collection and data-driven energy management model to a generalizable closed-form mathematical expression for calculating battery energy loss, thereby effectively optimizing energy consumption.

With the advancement of artificial intelligence algorithms, the optimization of existing UAV power consumption model strategies has also become more intelligent. For instance, Hung et al. [17] controlled the UAV’s propulsion system using a fuzzy-based equivalent energy consumption minimization strategy and a control strategy based on the optimal operating curve of the engine. Lee et al. [18] and Khayyam et al. [19] have designed rule-based power management systems and neural network control structures, respectively. Bongermino et al. [20,21] proposed a real-time iterative algorithm based on dynamic programming. Although Lei Tao et al. [22] analyzed various management strategies such as state machines, fuzzy logic, dynamic programming, and minimum equivalent fuel consumption, their methods are still limited to rule-based energy management strategies, and the shortcomings of such methods have not been fully addressed. In [23], the authors adopted the multiagent deep deterministic policy gradient (MADDPG) algorithm and prioritized experience replay (PER) to optimize the energy consumption and cooperation strategy of the UAV-assisted multiaccess edge computing (MEC) system.

In summary, researchers have devoted significant effort to studying the management strategies for UAV battery energy consumption and have achieved remarkable results. However, both the complex 3-D turbulent wind environmental factors and the changes in the UAV’s own flight status can affect the battery energy consumption management system. Traditional research has rarely combined path-planning targets with known acquisition parameters for autonomous learning.

Reinforcement learning (RL) is a recently emerged artificial intelligence (AI) technique that describes and solves the problem of an agent achieving specific goals through interactions with the environment by learning to maximize rewards. Therefore, this paper proposes a reinforcement learning-based energy-saving path-planning algorithm (ESPP-RL) for the UAV in turbulent wind. Based on RL, the ESPP-RL algorithm considers the 3-D turbulent wind environmental and the UAV’s flight movement patterns (including horizontal flight, vertical descent, and vertical ascent), introducing local observation conditions to enable the policy network to efficiently process observation features of dynamic dimensions, thus making path planning adaptive.

1.2. Motivation and Contribution

In recent years, RL has been successfully applied in the fields of robot planning, control, and navigation. RL has been shown to be a powerful tool for solving path-planning problems. Through interaction with the environment, the agent can learn how to make optimal decisions in uncertain environments. However, most existing RL methods still face challenges when dealing with high-dimensional state spaces and dynamic environments. Moreover, ensuring the robustness of the algorithm under various wind intensities while maintaining real-time performance is also a current research hotspot. A reinforcement learning-based energy-saving path-planning algorithm (ESPP-RL) in a turbulent wind environment is proposed. The main contributions of this paper are as follows:

• Considering the complex factors of 3-D turbulent wind and the UAV’s motion models (including horizontal flight, vertical descent, and vertical ascent), an optimal energy-saving path planning for the UAV on the basic of the framework of RL technology is proposed.
• Utilizing end-to-end training, combined with carefully crafted state features and reward functions, an efficient single-policy network is introduced, which not only achieves adaptive path planning but also optimizes dynamic battery management strategies to tackle challenges under partially observable conditions. The policy network adeptly handles dynamic and dimensional observational features, significantly enhancing the UAV’s adaptability in path planning under limited battery capacity and improving energy-saving efficiency.
• Conducting sets of experiments on the built simulation platform, compared to other algorithms, the ESPP-RL algorithm demonstrates superior performance and notable robustness advantages under a complex 3-D turbulent wind environment.

2. System Model and Problem Formulation

2.1. UAV Model

Firstly, a dynamic analysis of autonomous drones is crucial. In this paper, the dynamics model of the four-rotor UAV is taken as the analysis object, which has six degrees of freedom. As shown in Figure 1, the geodetic coordinate system and AUV coordinate system are set up to describe the motion and force of the UAV. $O_B{X}_B{Y}_B{Z}_B$ is the geodetic coordinate system and $O_E{X}_E{Y}_E{Z}_E$ is the body coordinate system used to describe the motion of the drone for the 3-D environment.

It is assumed that for small roll and pitch rotation, the angular dynamics are considered to be consistent across frames. For the 3-D environment, the linear position of the UAV in the inertial coordinate system is $d={\left[ x , y , z\right]}^{\mathrm{T}}$ and the linear velocity in the body coordinate system is $v={\left[ u , v , w\right]}^{\mathrm{T}}$, the attitude vector is $\Theta ={\left[ \phi , \theta , \psi \right]}^{\mathrm{T}}$, and the angular speed vector is $\Omega$. The kinetics can be simplified into [24,25,26]:

$$\dot{\mathbf{d}}=\mathbf{R}\mathbf{v}$$

(1)

$$\dot{\Theta }=\Omega$$

(2)

$$f=-T_h{e}_3+R^{-1}G{e}_3+R^{-1}\mathit{\delta }$$

(3)

$$G\dot{v}=fg-G\Omega \times v$$

(4)

$$\dot{\Omega }=J^{-1}(\tau -\Omega \times J\Omega )$$

(5)

where $G$ is the gravity of the vehicle and g is the acceleration of gravity. The transformation matrix R is responsible for converting the data from the volume coordinate system to the inertial coordinate system. The vector f contains all the linear forces acting on the quadrotor, director vector $e_3={\left[0,0,1\right]}^{\mathrm{T}}$, while $\mathit{\delta }={\left[\delta x,\delta y,\delta z\right]}^{\mathrm{T}}$ describes all the external disturbances acting on the vehicle. The total thrust T_h is generated by four motors and can be expressed as $T_h={\displaystyle {\sum }_{j=1}^4{T}_h^j}$.

In the hovering state of the four-rotor UAV, it is presumed that the weight of each rotor is uniform. Since the four-rotor UAV flies in a 3-D scene, its own gravity and the total thrust are balanced, so the thrust on each rotor is equal. In order to simplify the calculation, the rotor-related parameters (rotor solidity $h$, rotor disk area ${\mathrm{A}}_s$) on each rotor are the same, and the weight of each rotor of a quadrotor UAV is the same, where $C_T$ is the thrust coefficient and $\rho$ is the air density. The main notations used in this section are summarized in

Table 1. List of main notations.

Parameter	Meaning	Value
$\rho$	Air density in kg/m3	1.168
$h$	Rotor solidity	0.045
${\mathrm{A}}_s$	Rotor disk area in m2	0.214
$C_T$	Thrust coefficient	0.001195
$k$	Incremental correction faction to induced power	0.11
${\gamma }_v$	Fuselage drag coefficient during vertical flight in kg/m	0.220168
${\gamma }_h$	Fuselage drag coefficient in horizontal flight in kg/m	0.005256
$\lambda$	Profile drag coefficient	0.011
$G$	UAV weight in Newtons	20

.

The quadrotor UAV flies horizontally with a constant speed $V_u$ in the 3-D environment, which is decomposed into three speeds on the world coordinate axis, $V_x$, $V_y$, and $V_z$. When a quadrotor UAV is hovering, the energy consumption on each rotor is equal, so the total energy consumption during hovering can be expressed as the sum of the energy consumption of the four rotors. The energy consumption of the jth rotor $P_{hov}^j$ and the total energy consumption during hovering can be formulated as [16]:

$$P_{hov}^j={\left({\displaystyle \frac{G}{4C_T}}\right)}^{3/2}{\displaystyle \frac{\lambda h}{8\sqrt{\rho {\mathrm{A}}_s}}}+\left(1+k\right){\left({\displaystyle \frac{G}{4}}\right)}^{3/2}{\displaystyle \frac{1}{\sqrt{2\rho {\mathrm{A}}_s}}}$$

(6)

$${\displaystyle {\sum }_j^4{P}_{hov}^j}=4\times \left({\left({\displaystyle \frac{G}{4C_T}}\right)}^{3/2}{\displaystyle \frac{\lambda h}{8\sqrt{\rho {\mathrm{A}}_s}}}+\left(1+k\right){\left({\displaystyle \frac{G}{4}}\right)}^{3/2}{\displaystyle \frac{1}{\sqrt{2\rho {\mathrm{A}}_s}}}\right)$$

$$\begin{array}{c}=\underset{P_{bl}}{\underbrace{{\left({\displaystyle \frac{G}{C_T}}\right)}^{3/2}{\displaystyle \frac{\lambda h}{16\sqrt{\rho {\mathrm{A}}_s}}}}}+\underset{P_{in}}{\underbrace{\left(1+k\right)G^{3/2}{\displaystyle \frac{1}{2\sqrt{2\rho {\mathrm{A}}_s}}}}}\end{array}$$

(7)

where $k$ represents the incremental correction factor to induced power. The total energy consumption during hovering can be composed of two parts: one is the rotor profile power $P_{bl}$ and the other is the induced power $P_{in}$.

When a quadrotor UAV flies horizontally with a constant speed, its energy consumption can be expressed as [16]:

$$\mathbf{\Delta }{P}_h\left({\mathrm{V}}_h\right)={\displaystyle \frac{3}{4}}\delta \sqrt{{\displaystyle \frac{G\rho {\mathrm{A}}_s}{C_T}}}h{‖{\mathrm{V}}_h‖}^2+P_{in}\left[{\left(\sqrt{1+{\displaystyle \frac{{‖{\mathrm{V}}_h‖}^4}{4v_0^4}}}-{\displaystyle \frac{{‖{\mathrm{V}}_h‖}^2}{2v_0^2}}\right)}^{1/2}-1\right]+4\rho {\gamma }_h{‖{\mathrm{V}}_h‖}^3$$

(8)

When a quadcopter UAV is flying vertically, there are two situations: vertical ascent and vertical descent. In both cases, the problem of velocity direction and thrust direction is dealt with by the sgn function. Therefore, in vertical flight, due to the influence of fuselage resistance, the energy loss is [16]:

$$\mathbf{\Delta }{P}_v\left({\mathrm{V}}_v\right)={\displaystyle \frac{1}{2}}G‖{\mathrm{V}}_v‖+2sgn\left({\mathrm{V}}_v\right)\rho {\gamma }_v{‖{\mathrm{V}}_h‖}^3+\left({\displaystyle \frac{G}{2}}+2sgn\left({\mathrm{V}}_v\right)\rho {\gamma }_v{‖{\mathrm{V}}_v‖}^2\right)$$

$$\begin{array}{c}\times \sqrt{\left(1+{\displaystyle \frac{2{\gamma }_{v}sgn\left({\mathrm{V}}_v\right)}{\rho {\mathrm{A}}_s}}\right){‖{\mathrm{V}}_v‖}^2+{\displaystyle \frac{G}{2\rho {\mathrm{A}}_s}}}+sgn\left(‖{\mathrm{V}}_v‖-1\right){\displaystyle \frac{G}{2}}\sqrt{{\displaystyle \frac{G}{2\rho {\mathrm{A}}_s}}}\end{array}$$

(9)

2.2. Turbulent Wind Model

In changeable and challenging 3-D space, the flight trajectory and attitude of autonomous drones are not only limited by their internal dynamics model but also by various factors in the surrounding environment. In particular, the wind field plays a crucial role in the course control and stability of the UAV. The following content will explore the theoretical basis of constructing a turbulent wind model in depth. In a turbulent wind scenario, the motion of a quadrotor UAV is affected by wind resistance. The turbulent wind disturbance model is noted as follows [27]:

$${\delta }_{tur}=-{\displaystyle \frac{1}{2}}\rho {\mathit{B}}_{tur}{\mathrm{A}}_s{(\dot{\mathbf{d}}-{\mathit{v}}_{\omega -tur})}^{2}sign(\dot{\mathbf{d}}-{\mathit{v}}_{\omega -tur})$$

(10)

where ${\mathit{B}}_{tur}={[x_{tur},y_{tur},z_{tur}]}^{\mathrm{T}}$ represents the drag coefficient and ${\mathit{v}}_{\omega -tur}$ is the turbulent wind speed in the volume coordinate system.

2.3. Reinforcement Learning Model

Reinforcement learning algorithms acquire rewards in the environment through continuous trial and error, guiding the training and optimization direction of the algorithms based on the reward values, ultimately enabling the RL algorithms to prioritize actions or paths that maximize the reward values. RL is a type of learning approach that acquires corresponding reward values by observing the environment and taking actions and ensures the convergence of the algorithm through maximizing reward expectations.

As shown in Figure 2, RL is generally described by a Markov process which consists of components such as state $s$, action $a$, policy $\pi$, and reward $r$, and achieves state transitions in a probabilistic manner based on the state transition function. At each time step $t$, the agent acquires the current state information $s_t$ from the environment; based on the state $s_t$ and policy $\pi$, the agent selects an action $a_{t+1}$ to output within the action space. According to the state transition function $p\left(s_{t+1}|s_t,a\right)$, the environment then returns a reward value $r$ and the state for the next time step $s_{t+1}$. Ultimately, the policy $\pi$ aims to optimize itself by maximizing the expected reward.

A Markov decision process (MDP) is the foundation of RL algorithms, which expresses the state transition probabilities and processes after making a decision [28]. The MDP is given by the tuple $\left(S,A,p,r\right)$, including the state space $S$, action space $A$, state transition probability $p$, and reward value $r$. Among them, the state transition probability, denoted as $p:S\times S\times A$, represents the probability density of transitioning to the next state when an action is taken $a_t\in A$ in the current state $s_t\in S$. For every completed transfer of state, the environment generates an immediate reward value $r$ based on the state and action, and the range of reward values can be expressed as $r:S\times A\to \left[r_{\min },r_{\max }\right]$.

3. Algorithm

Based on a UAV motion model and 3-D turbulent wind model, this paper presents the development of a UAV’s motion simulation platform and introduces a UAV path-planning scheme ESPP-RL. The specific technical content is as follows.

3.1. Turbulent Wind Environment

To increase the realism of the simulation environment and enhance the practical deployability of the algorithm on a UAV, this paper constructs a turbulent wind environment based on PyBullet. As a physics simulation library, PyBullet integrates rich physical and mathematical constraints, enabling complex real-time physical calculations and rendering. It has been widely applied in the fields of robot motion simulation and deep learning. Therefore, PyBullet serves as the foundational tool for environment construction, catering to the dynamic environmental computation needs of UAVs during complex task execution.

A continuous 3-D environment space of dimensions 100 × 100 × 100 is constructed within PyBullet, and subsequently, a 3-D turbulent wind environment is simulated within it. The model data of turbulent wind are derived utilizing formula (10), and subsequent interpolation processing is applied to this data, ensuring that a corresponding turbulent wind strength $I(u_I,v_I,w_I)$ is assigned to any position within the spatial domain. And $u_I,v_I,w_I$ represent the linear velocities of turbulent wind in the x-, y-, and z-axes, respectively.

3.2. State Transition Function

For a quadcopter UAV, this paper sets the action space to four dimensions $a\in ({\Omega }_u,{\Omega }_v,{\Omega }_w,T_a)$, including the angular velocity outputs ${\omega }_u,{\omega }_v,{\omega }_w$ and thrust $T_a$ in the x-, y-, and z-axes in the body coordinate system. The thrust component on each axis is expressed as:

$${[T_u,T_v,T_w]}^{\mathrm{T}}={[{\Omega }_u,{\Omega }_v,{\Omega }_w]}^{\mathrm{T}}\ast T_a$$

(11)

$$m{[a_u,a_v,a_w]}^{\mathrm{T}}={[T_u,T_v,T_w]}^{\mathrm{T}}-{[G,G,G]}^{\mathrm{T}}$$

(12)

where $\left[a_u,a_v,a_w\right]$ represents the acceleration of the UAV affected by thrust $[T_u,T_v,T_w]$ in the horizontal, vertical, and vertical axis directions. In the context of environmental observation, the state space of the environment is assumed to be partially observable, and the state space acquired by the UAV at a given time step is defined as:

$$S=[\mathbf{d},\mathbf{v}, \Theta ,\Omega ,p_{tar},I]$$

(13)

which includes the linear position component $\mathbf{d}$, linear velocity component $\mathbf{v}$, attitude angle component $\Theta$, angular velocity component $\Omega$, and the target’s position information of the UAV $p_{tar}$. It also includes the current turbulence wind strength $I(u_I,v_I,w_I)$ at the location. The state transition is influenced by the UAV state transition equation, the algorithm’s action output, and the turbulence wind interference. Upon the execution of an action, the subsequent position $d^{\prime }={\left[ x^{\prime } , y^{\prime } , z^{\prime }\right]}^{\mathrm{T}}$ of the UAV is determined by a confluence of factors: the UAV’s current velocity ${\left[{\mathrm{v}}_u,{\mathrm{v}}_v,{\mathrm{v}}_w\right]}^{\mathrm{T}}$, the acceleration generated by the algorithmic output, and the turbulence wind strength.

$$\left[\begin{array}{l}{x}^{\prime }\\ y^{\prime }\\ z^{\prime }\end{array}\right]=\left[\begin{array}{l}x\\ y\\ z\end{array}\right]+\left[\begin{array}{l}{\mathrm{v}}_u+a_u\\ {\mathrm{v}}_v+a_v\\ {\mathrm{v}}_w+a_w\end{array}\right]+\left[\begin{array}{l}{u}_I\\ v_I\\ w_I\end{array}\right]$$

(14)

3.3. Reward Function

In reinforcement learning, the purpose of algorithm optimization is to maximize the reward expectation and then affect the gradient updating direction of the neural network. Therefore, the design of a reward function is very important to the RL algorithm, which directly determines the effect of environment-guided algorithm training. To this end, this paper carefully designed the reward function, including the following four items: distance reward $re{w}_{dis}$, collision reward $re{w}_{collision}$, energy consumption reward $re{w}_{energy}$, and goal reward $re{w}_{target}$.

In order to improve the efficiency of the algorithm to explore the environment, distance reward is designed to guide the algorithm to approach the target point region. The distance reward $re{w}_{dis}$ can be expressed by the formula:

$$re{w}_{dis}=\min DIS(P_{UAV},P_{ta{r}_i})/siz{e}_{env},i=1,2,\dots n$$

(15)

where $DIS(P_{UAV},P_{ta{r}_i})$ represents the Euclidean distance of the UAV from the target and $siz{e}_{env}$ corresponds to the environmental dimensions. The collision reward is a reward for the auxiliary algorithm to perceive the environmental boundary and obstacles, and the value of the obstacle reward changes according to the perception of the agent, where $r_{perception}$ is the obstacle perception radius of the UAV and $r_{fail}$ is the collision radius. The collision reward $re{w}_{collision}$ can be expressed by the formula:

$$re{w}_{collision}=\left\{\begin{array}{l}0(DIS(P_{UAV},P_{\mathrm{obstacle}})>r_{perception}\\ -1/DIS(P_{UAV},P_{\mathrm{obstacle}}),r_{fail}<DIS(P_{UAV},P_{\mathrm{obstacle}})<r_{perception}\\ -100,DIS(P_{UAV},P_{\mathrm{obstacle}})<r_{fail}\end{array}\right.$$

(16)

In order to make the UAV complete the task more efficiently, this paper designs the energy consumption reward $re{w}_{energy}$, which is used to evaluate the number of running steps and action quality of the algorithm. ${\mathbf{\Delta }}_h(P_{UAV},P_{tar})$ is the vertical update component of the UAV and the target position after the execution of each step and ${\mathbf{\Delta }}_v(P_{UAV},P_{tar})$ corresponds to the horizontal update component $\psi =1.2$. The energy consumption reward $re{w}_{collision}$ can be expressed by the formula:

$$re{w}_{energy}=-0.1+\psi {\mathbf{\Delta }}_h(P_{UAV},P_{tar})+{\mathbf{\Delta }}_v(P_{UAV},P_{ta{r}_i})$$

(17)

When the UAV reaches the target point, a larger positive reward will be given, for example, $re{w}_{target}=100$, and the episode will end. Therefore, the comprehensive reward function is expressed as:

$$re{w}_{total}=re{w}_{dis}+re{w}_{collision}+re{w}_{energy}+re{w}_{target}$$

(18)

3.4. Reinforcement Learning Algorithm

A significant feature of UAV systems is energy limitation, which requires the path planning of the UAV to have a higher flexibility of algorithms. How to self-adapt the battery management, select the movement path, and adapt to the change-of-wind field can effectively improve the intelligence and adaptability of the algorithm. The traditional path-planning algorithm has limited capability and difficulty in dealing with a UAV multi-target path planning problem in a dynamic environment. Therefore, a reinforcement learning-based energy-saving path-planning algorithm in a turbulent wind environment is designed, The framework of the proposed ESPP-RL algorithm is shown in Figure 3.

In general, the ESPP-RL algorithm includes an observation processing module, a neural network module, and an action optimizer. This section will explain each module of the ESPP-RL algorithm. Algorithm 1 summarizes the training process.

Firstly, the power consumption of the UAV in 3-D space is expressed as $P_{total}(V_{total})=P_{hov}+\mathbf{\Delta }{P}_h(V_h)+\mathbf{\Delta }{P}_v(V_v)$, where the total power is decomposed into three components: hover power $P_{hov}$, vertical descent and ascent power $P_h$, and horizontal flight power $P_v$. The energy consumed by the UAV moving to the target point is:

$$E=P_{hov}(\mathbf{\Delta }{t}_h+\mathbf{\Delta }{t}_v)+\mathbf{\Delta }{P}_h(V_h)\mathbf{\Delta }{t}_h+\mathbf{\Delta }{P}_v(V_v)\mathbf{\Delta }{t}_v$$

(19)

Under multi-target path planning, the position of the UAV changes dynamically and the corresponding expectation of total energy consumption can be expressed as:

$$E_{total}=\min {\displaystyle \sum _i{P}_{hov}^i(\mathbf{\Delta }{t}_h^i+\mathbf{\Delta }{t}_v^i)+\mathbf{\Delta }{P}_h^i(V_h^i)\mathbf{\Delta }{t}_h^i+\mathbf{\Delta }{P}_v^i(V_v^i)\mathbf{\Delta }{t}_v^i}$$

(20)

$$E_{total}=\min {\displaystyle \sum _i\left\{\begin{array}{c}{P}_{hov}^i(\mathbf{\Delta }{t}_h^i+\mathbf{\Delta }{t}_v^i)+\left\{\frac{3}{4}\delta \sqrt{\frac{G\rho {\mathrm{A}}_s}{C_T}}h{‖{\mathrm{V}}_h^i‖}^2+P_{in}\left[{\left(\sqrt{1+\frac{{‖{\mathrm{V}}_h^i‖}^4}{4v_0^4}}-\frac{{‖{\mathrm{V}}_h^i‖}^2}{2v_0^2}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}-1\right]+4{\gamma }_{r\perp }{‖{\mathrm{V}}_h^i‖}^3\right\}\mathbf{\Delta }{t}_h^i\\ +\left\{\begin{array}{c}\frac{1}{2}G‖{\mathrm{V}}_v^i‖+2{\gamma }_{\perp }sgn\left({\mathrm{V}}_v^i\right){‖{\mathrm{V}}_v^i‖}^3+\left(\frac{G}{2}+2{\gamma }_{\perp }sgn\left({\mathrm{V}}_v^i\right){‖{\mathrm{V}}_v^i‖}^2\right)\\ \times \sqrt{\left(1+\frac{2{\gamma }_{\perp }sgn\left({\mathrm{V}}_v^i\right)}{\rho {\mathrm{A}}_s}\right){‖{\mathrm{V}}_v^i‖}^2+\frac{G}{2\rho {\mathrm{A}}_s}}+sgn\left(‖{\mathrm{V}}_v^i‖-1\right)\frac{G}{2}\sqrt{\frac{G}{2\rho {\mathrm{A}}_s}}\end{array}\right\}\mathbf{\Delta }{t}_v^i\end{array}\right\}}$$

(21)

According to the energy consumption expectation $E_{total}$ and the position state transfer of the UAV, the observation processing module decomposed the multi-target path-planning problem into a dynamic single-objective path-planning subproblem, calculated and stored the solution of the subproblem, and iteratively optimized to obtain the discrete path optimal solution of the energy consumption expectation $E_{total}$. At the same time, the observation processing module splices the optimal solution of the discrete path with the environment observation and inputs it into the neural network module.

Algorithm 1: ESPP-RL Algorithm
1	Initialize the multi-target path-planning environment for the UAV
2	$\mathrm{Initialize} \mathrm{the} \mathrm{state} \mathrm{action} \mathrm{value} \mathrm{network} Q_{{\theta }_1}$$, Q_{{\theta }_2}$$, \mathrm{target} \mathrm{action} \mathrm{state} \mathrm{value} \mathrm{network} Q_{{{\theta }_1}^{\prime }}$$Q_{{{\theta }_2}^{\prime }}$$, \mathrm{policy} \mathrm{network} {\pi }_{\varphi }$$, \mathrm{target} \mathrm{policy} \mathrm{network} {\pi }_{{\varphi }^{\prime }}$$, \mathrm{and} \mathrm{random} \mathrm{parameters} {\theta }_1,{\theta }_2,\varphi$
3	Initialize the replay buffer $\mathbf{{\rm B}}$
4	$\mathrm{Initial} \mathrm{the} \mathrm{network} \mathrm{parameters} {{\theta }_1}^{\prime }\leftarrow {\theta }_1,{{\theta }_2}^{\prime }\leftarrow {\theta }_2,{\varphi }^{\prime }\leftarrow \varphi$
5	Obtain information about the observation processing module
6	for t = 1 to T do
7	$\mathrm{Select} \mathrm{action} \mathrm{with} \mathrm{exploration} \mathrm{noise} a\sim {\pi }_{{\varphi }^{\prime }}(s)+\epsilon$$, \epsilon \sim clip(N(0,\sigma ),-c,c) \mathrm{and}$$\mathrm{observe} \mathrm{reward} r \mathrm{and}\mathrm{new} \mathrm{state} s^{\prime }$
8	$\mathrm{Store} \mathrm{transition} \mathrm{tuple} \left(s,a,r,s^{\prime }\right)$ in ${\rm B}$
9	Sample mini-batch of $N$$\mathrm{transitions} \left(s,a,r,s^{\prime }\right)$ from ${\rm B}$
10	$\tilde{a}\leftarrow {\pi }_{{\varphi }^{\prime }}(s^{\prime })+\epsilon$$,\epsilon \sim clip(N(0,\tilde{\sigma }),-c,c)$
11	$y\leftarrow r+\gamma {\min }_{i=1,2}{Q}_{{\theta }_i\prime }(s^{\prime },\tilde{a})$
12	$\mathrm{Update} \mathrm{critics} {\theta }_i\leftarrow argmi{n}_{{\theta }_i}{N}^{-1}{{\displaystyle \sum \left(y-Q_{{\theta }_i}\left(s,a\right)\right)}}^2$
13	if t mod d == True then
14	Update $\varphi$ by the deterministic policy gradient:
15	${\mathbf{\nabla }}_{\varphi }J(\varphi )=N^{-1}{\displaystyle \sum {\mathbf{\nabla }}_a}{Q}_{{\theta }_1}\left(s,a\right){|}_{a={\pi }_{\varphi }\left(s\right)}{\mathbf{\nabla }}_{\varphi }{\pi }_{\varphi }\left(s\right)$
16	Update target networks
17	${\varphi }^{\prime }\leftarrow \tau \varphi +\left(1-\tau \right){\varphi }^{\prime }$
18	${{\theta }_i}^{\prime }\leftarrow \tau {\theta }_i+\left(1-\tau \right){{\theta }_i}^{\prime }$
19	end if
20	end for
21	Input the action to the PID controller

The information of the observation processing module is inputted into the neural network module, which includes the state action value network $Q_{{\theta }_i}(s_t,a_t),i=1,2$, the target action state value network $Q_{{{\theta }_i}^{\prime }}(s_t,a_t),i=1,2$, the policy network ${\pi }_{\varphi }$, and the target policy network ${\pi }_{{\varphi }^{\prime }}$, with the corresponding network parameters being ${\theta }_1,{\theta }_2,{{\theta }_1}^{\prime },{{\theta }_2}^{\prime },\varphi ,{\varphi }^{\prime }$, respectively. The updated observation space input target strategy network ${\pi }_{{\varphi }^{\prime }}$ with maximum reward expectation, the target strategy network, is expressed as:

$${\pi }_{{\varphi }^{\prime }}=\arg \underset{{\pi }_{{\varphi }^{\prime }}}{\max }{\displaystyle \sum _t{\mathrm{{\rm E}}}_{(s_t,a_t)\sim {\rho }_{{\pi }_{{\varphi }^{\prime }}}}}[r(s_t,a_t)]$$

(22)

The network ${\pi }_{{\varphi }^{\prime }}$ selects the action output based on the observation $a\sim {\pi }_{{\varphi }^{\prime }}(s)+\epsilon$, where the action noise is represented by $\epsilon$, in order to increase the environmental exploration rate $\epsilon \sim clip(N(0,\sigma ),-c,c)$. The algorithm interacts with the environment to feedback the corresponding reward value obtained by the action $r(s_t,a_t)$. The state action value network $Q_{{\theta }_i}(s_t,a_t),i=1,2$ in the neural network module evaluates the action output effect of the current policy network according to the reward value $r(s_t,a_t)$. It is defined as:

$$Q_{{\theta }_i}(s_t,a_t)\triangleq r(s_t,a_t)+\gamma {\mathrm{{\rm E}}}_{s_{t+1}\sim p}[V(s_{t+1})],i=1,2$$

(23)

$$V(s_t)\triangleq {\mathrm{{\rm E}}}_{a_t\sim {\pi }_{{\varphi }^{\prime }}}[Q_{{\theta }_i}(s_t,a_t)]$$

(24)

where $\gamma$ represents the discount factor, which is used to represent the impact of future n steps on the value function of the current action $a_{t+n}$, And $V(s_t)$ is the state value function, which evaluates the expected value of the policy network action output in the current state. Further, this method introduces the target state action value network $Q_{{{\theta }_i}^{\prime }}(s_t,a_t),i=1,2$, designs the objective function to stabilize the algorithm training, and reduces the overestimation problem caused by the influence of the state value action network on the policy network. The specific objective function $y_{tar}$ is expressed as:

$$y_{tar}=r(s_t,a_t)+\gamma \left(\underset{i=1,2}{\min }{Q}_{{\theta }_i^{\prime }}(s_{t+1},{\pi }_{\varphi }(s_{t+1}))\right)$$

(25)

The state action value network parameters ${\theta }_i,i=1,2$ are updated according to the objective function $y_{tar}$:

$${\theta }_i\leftarrow \underset{{\theta }_i}{\arg \min }{N}^{-1}{\displaystyle \sum {(y_{tar}-Q_{{\theta }_i}(s_t,a_t))}^2}$$

(26)

At the same time, the strategy parameters and the target action value network parameters in the algorithm are updated in a delayed way to smooth the training process. Policy parameters $\varphi$ are updated by gradient optimization:

$$\varphi \leftarrow {\mathbf{\nabla }}_{\varphi }J(\varphi )=N^{-1}{\displaystyle \sum {\mathbf{\nabla }}_a}{Q}_{{\theta }_1}(s,a){|}_{a={\pi }_{\varphi }(s)}{\mathbf{\nabla }}_{\varphi }{\pi }_{\varphi }(s)$$

(27)

Finally, the target action value network parameters ${\theta }_i,i=1,2$ and target policy parameter ${\varphi }^{\prime }$ are updated, and $\eta$ represents the soft update weight.

$${{\theta }_i}^{\prime }\leftarrow \eta {\theta }_i+(1-\tau ){{\theta }_i}^{\prime }$$

(28)

$${\varphi }^{\prime }\leftarrow \eta \varphi +(1-\tau ){\varphi }^{\prime }$$

(29)

In addition, the action output of the neural network module will pass through an action optimizer. The action optimizer is composed of a PID controller, and the action output of the algorithm will be converted into the actual output parameters after the PID controller, in order to smooth the action output and ensure the relative stability of the UAV attitude.

4. Experiment

4.1. Experimental Setup

In this section, we focus on optimizing the path-planning performance of a four-axis UAV in a 3-D turbulent wind environment and optimizing the energy consumption performance while ensuring that the path planning is completed. The turbulent wind model is imported into a 3-D continuous space of dimensions 100 × 100 × 100 utilizing the PyBullet physics simulation library. By integrating the motion space model, state transition model, and reward function specific to the four-axis UAV, a comprehensive RL simulation environment is constructed. The algorithms are implemented on Ubuntu 18.04, with the hardware configuration including an Intel i9-11900k processor, NVIDIA GTX3080 graphics card, and software based on Python 3.8, Ray 1.8.0, and TensorFlow 2.3.1. And soft actor-critic (SAC) and twin delayed deep deterministic (TD3) algorithms are implemented within the framework as comparison algorithms. The algorithm parameters are shown in

Table 2. Algorithm parameters.

Algorithm	SAC	TD3	ESPP-RL
Learning rate	3 × 10−4	1 × 10−3	1 × 10−3
Total training steps (T)	1 × 10−6	1 × 10−6	1 × 10−6
Batch_size	1 × 10−5	2 × 10−6	1 × 10−5
Update weight ($\eta$)	0.005	0.005	0.005
Policy noise	1	1	1
Discount factor ($\gamma$)	0.99	0.99	0.99
Delayed update ($d$)	None	2	2
$\mathrm{PID} \mathrm{parameters} k_p,k_i,k_d$	None	None	4 × 10−2, 5 × 10−7, 1 × 10−4

.

The performance of various algorithms in path planning is quantified through the application of four key metrics, which serve as the basis for evaluating the results

• Steps: The UAV selects actions at fixed intervals according to the policy and then moves to the next position according to the state transition function. The number of steps reflects the number of interactions and the time consumed.
• Path length: This paper records the cumulative position offset of the agent in the environment and as the path length. The path length reflects the algorithm’s path-planning ability and the ability to resist wind-field interference.
• Target reach number: The target reach number can accurately reflect the adaptability of the agent to the environment and the quality of the decision. A high goal reach number means that the agent can effectively identify and utilize information in the environment to reach the goal quickly and with minimal cost. Secondly, this index is also an important basis for evaluating the stability of the algorithm.
• Reward value: The reward value is a core indicator to measure the performance of the agent. The reward value reflects the agent’s performance in the environment, i.e., the total reward it receives after performing a series of actions.

4.2. Analysis of Experimental Results

Sufficient training and experiments have been conducted on the SAC, TD3, and ESPP-RL algorithm. Since the training is an oscillating process, in order to better visualize the training results, appropriate smoothing processing is carried out. The training process is recorded in Figure 4, where Figure 4a represents the average number of actions of the algorithm in the training process, Figure 4b corresponds to the reward value of the UAV in a round, Figure 4c reflects the average distance of the UAV in each round, and, finally, Figure 4d represents the average number of target points that the algorithm can reach in the environment.

In the initial stage of algorithm training, the agent randomly selects actions to explore the environment, and when it touches the environmental boundary or obstacles, it will result in the end of the turn and obtain a large negative reward, which makes the average number of steps and reward value of the algorithm at the beginning of the training stage low. With the influence of the distance reward and random exploration, the UAV accidentally explores the target point and obtains a large target reward, and these exploration processes of the environment are combined into experiences and stored in the experience playback pool. The algorithm regularly extracts experience from the experience playback pool for training and updates its own network parameters. It can be seen that the ESPP-RL algorithm benefits from the observation processing module, with a faster training speed and higher cumulative reward value.

4.2.1. Single-Target Path-Planning Environment

To facilitate observation of the algorithm’s action strategy output performance, the environment is refined into a single-target path-planning scenario. The algorithm does not need to assign tasks to multiple objective points and mainly considers the path-planning effect when the target is determined. Specifically, a static target point and multiple obstacles are configured within the environment, requiring the UAV to navigate from a designated starting point, bypass the obstacles, and ultimately reach the target location. The experimental results show that all algorithms can avoid obstacles to reach the target point in the single-target path-planning task.

Based on the algorithmic execution outcomes presented in Figure 5 and

Table 3. The reward and steps of algorithms under a single-target path-planning environment task.

	SAC	TD3	ESPP-RL
Reward	123.96	156.86	160.32
Steps	135	66	68

, within the context of single-target path-planning tasks, all algorithms successfully avoided obstacles and reached the target. Notably, the TD3 and ESPP-RL algorithms exhibited comparable results, whereas the SAC algorithm demonstrated lower reward values and required more steps, which is indicative of higher energy consumption. More specifically, despite requiring a comparable number of steps as TD3, the ESPP-RL algorithm achieved higher reward values, attributed to its PID-regulated action output, which facilitated optimal maneuvers, resulting in reduced vertical deviations and consequently higher rewards. Conversely, as depicted in Figure 5a, while the SAC algorithm initially demonstrated comparable performance to other algorithms during the initial stages of path planning, it struggled to reach the target’s decision boundary towards the end due to disturbances caused by turbulent wind, hindering its ability to precisely navigate within the target’s vicinity.

4.2.2. Multi-Target Path-Planning Environment

To evaluate the decision-making capability and path optimization efficiency of the UAV, the number of target points is varied within the multi-target path-planning environment. The training results are shown in Figure 6 and

Table 4. The reward and steps of algorithms under a multi-target path-planning environment task.

		SAC	TD3	ESPP-RL
Three-target	Reward	326.85	397.66	421.14
	Steps	313	248	222
Four-target	Reward	407.31	487.98	528.40
	Steps	458	339	326
Five-target	Reward	461.73	587.59	615.77
	Steps	494	370	353

. In single-target path planning, emphasis is placed on the algorithm’s resilience against disturbances in turbulent wind. Conversely, in multi-target scenarios, the algorithm dynamically selects path nodes based on the locations of the targets, necessitating intricate decision-making.

As depicted in Table 4, the ESPP-RL algorithm surpasses other methods in terms of both step count and reward value when tasked with 3, 4, and 5 target points. This superiority stems from the algorithm’s imposition of energy consumption constraints, which are decomposed via Equation (17) to minimize overall energy expenditure during path planning, thereby yielding superior traversal solutions. In contrast, the SAC algorithm’s action output is significantly perturbed by turbulent wind, hindering its ability to swiftly reach target points, resulting in a higher cumulative step count and lower reward value. While the TD3 algorithm exhibits comparable performance to ESPP-RL in single-target path planning, it notably underperforms in multi-target tasks due to its focus on optimizing individual target paths, neglecting intertarget relationships, leading to locally optimal solutions in the broader context.

The proposed algorithm can output actions by combining the environment information and PID controller to ensure more accurate actions. Therefore, the algorithm converges faster under turbulent wind interference and maintains a high path-planning success rate, which reflects the strong adaptability of the proposed algorithm. The proposed algorithm has a smoother path. When approaching a moving target, its strong robustness enables it to accurately reach the target location, reducing the number of steps required and the cumulative distance. The experimental results demonstrate that in a multi-target path-planning environment, the ESPP-RL algorithm exhibits outstanding performance. It efficiently switches between multiple targets, planning both short and safe paths, resulting in a reduced accumulated distance. In summary, the ESPP-RL algorithm, with its precise action control, fast convergence speed, high success rate in path planning, smooth path characteristics, and robustness, demonstrates remarkable superiority in multi-target scenarios.

5. Conclusions

This paper proposes a reinforcement learning-based energy-saving path-planning algorithm for a UAV in complex 3-D turbulent wind. Based on the RL framework, the algorithm takes into account various flight models of the UAV, including vertical descent, vertical ascent, and horizontal flight, while paying particular attention to battery energy limitations. Through end-to-end training methods, the algorithm designs state features and reward functions for the entire mission process, utilizing a single policy network to achieve adaptive path planning and dynamic battery management. An optimization strategy suitable for locally observable conditions is also introduced, enabling the policy network to efficiently process dynamic dimensional observation features, thus improving the adaptability of path planning for drones with limited energy consumption. Experimental simulation results demonstrate that the proposed algorithm outperforms other drone algorithms in terms of robustness, especially in complex 3-D turbulent wind, ensuring that drones can complete tasks in a time-efficient and collision-free manner, further bridging the gap between drone theoretical research and practical applications.

Future research will include exploring multi-UAVs’ collaboration in complex 3-D turbulent wind to keep the UAVs stable, achieving better energy consumption, realizing autonomous obstacle avoidance and path planning. Additionally, we will strive to elevate the versatility of our algorithms to accommodate a wider range of UAV types, ensuring their applicability across diverse platforms and missions, and verifying the proposed control strategies with real UAVs.