**1. Introduction**

With the growing market for unmanned aerial vehicles (UAVs), a wide range of industries and organizations, including military, government, industrial, and recreational users, deploy this technology across the globe [1–3]. Among different types of UAVs, small unmanned aerial vehicles (sUAVs) [4] are attractive for military, aerial photography, and environmental monitoring applications due to their small size and flexible operation [5]. Considering the (i) hardware and weight constraints, (ii) limited onboard energy storage, and (iii) performance requirements for sUAVs, improving their endurance (maximizing their flight time) is of great importance for extending the duration of their missions, which could involve surveillance, search and rescue, disaster relief, traffic control, and precision agriculture, thereby motivating the development of novel propulsion systems and the implementation of optimal control policies for power and energy management. Among different propulsion systems for such sUAVs, a hybrid propulsion system consisting of a polymer electrolyte membrane fuel cell (PEMFC) and a battery has been proposed for long duration missions, e.g., in [6–9]. Other propulsion systems may incorporate energy harvesters, such as in [10]. In this paper we focus on novel approaches to the energy

**Citation:** Zhang, J.; Kolmanovsky, I.; Amini, M.R. Stochastic Drift Counteraction Optimal Control of a Fuel Cell-Powered Small Unmanned Aerial Vehicle. *Energies* **2021**, *14*, 1304. https://doi.org/10.3390/en14051304

Academic Editor: Aldo Sorniotti

Received: 31 January 2021 Accepted: 22 February 2021 Published: 27 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

management of sUAVs through optimal coordination between the PEMFC and battery for the previously proposed fuel cell hybrid propulsion system.

Rule-based (e.g., thermostat-like on-off control [11]), dynamic programming-based [12], and model predictive control (MPC) [13] methods have been considered for the energy management of hybrid aircraft. As in automotive energy management applications [14], the use of simple rule-based strategies may not provide optimal performance, while the conventional formulations of MPC and dynamic programming do not directly address the flight time maximization objective. Furthermore, deterministic variants of MPC and dynamic programming may require an accurate preview of the propeller power and climb angle over a long horizon and are computationally demanding if optimization has to be performed online. Similarly, the Pontryagin maximum principle (PMP)-based guidance solutions [15] need accurate characterization of the flight environment.

In this paper, we consider a different approach to the problem of endurance maximization for a hybrid UAV with a polymer electrolyte membrane fuel cell (PEMFC) based on an application of stochastic drift counteraction optimal control (SDCOC) [16], which directly addresses the problem of maximizing the time to constraint violation in a stochastic environment. In our case, the objective is to maintain the vehicle flying for a maximum amount of time by coordinating the fuel cell and the battery to provide the requested propeller power subject to the limited amount of fuel and battery state of charge (*SOC*) onboard the vehicle. The transitions in aircraft climb angle and propeller power are modeled stochastically by a Markov chain with the transition probabilities determined from historical data representing typical missions of an sUAV. Then, a control policy that minimizes a cost functional reflective of expected time-to-violate constraints is determined offline through value iterations; this control policy is then deployed onboard for the online coordination of the fuel cell and the battery in the sUAV.

In a preliminary conference paper [17] by the second author of this paper, the application of SDCOC for power management of a hybrid sUAV with a direct methanol fuel cell (DMFC) was considered. While the DMFC is often considered as a suitable power source for ground vehicles [18] and has certain advantages, PEMFCs are more appealing for air mobility applications [6,7] due to their relatively lower operating temperature, allowing for a quick start-up [19], higher efficiency (up to 60% [18,20]) and power density, and higher safety due to the use of the solid electrolyte [18].

Different from [17], in this paper, we consider the application of SDCOC to the power management of a hybrid sUAV with a PEMFC rather than a DMFC. To accommodate a different fuel cell and an sUAV, the fuel cell model is changed and improvements are made to the models used to compute the propeller power and thrust, as well as the evolution of the *SOC*.

More importantly, the lack of the ability of the PEMFC to rapidly change its power output imposes a stringent operating constraint (rate limit on PEMFC power output), which was not treated in [17], but is treated in this paper. This rate limit increases the complexity of the problem as an extra state needs to be introduced in the model and handled in SDCOC, and it also changes the optimal policies and the optimal response of the system. For instance, as the fuel cell is not able to support fast changes in power output, the optimal policy is shown to charge the battery to a turnpike value if starting from a low initial state of charge value. If starting from a high *SOC*, the battery energy is used till a turnpike value of the state of charge is reached with further discharge delayed to a later phase of the flight. In either case, the high frequency chattering of fuel cell load demand power in [17], which cannot be supported by the PEMFC, is eliminated.

Additionally, in this paper, the value iterations are implemented in Cython rather than MATLAB, with an order of magnitude speedup as compared to the MATLAB implementation. As value iterations are frequently used to solve dynamic programming problems in different applications and Python is becoming increasingly popular, our results on the ten-fold speedup with Cython without a substantive increase of the code complexity are

of reference value to other researchers considering the computational implementation of dynamic programming.

Furthermore, a discount factor is introduced into the cost function of SDCOC, and its impact on the convergence speed of the value iterations is illustrated. It is shown that this discount factor results in the faster convergence of value iterations, but the performance of the control policy (in terms of exit time) is decreased.

While the SDCOC theory was developed in [16], that reference did not address the fuel cell or sUAV application studied in this paper. Our approach to representing motor power demand and climb angle by a Markov chain with a finite number of states follows [21], which is the first (to the authors' knowledge) paper proposing the use of stochastic dynamic programming for automotive powertrain control applications; that paper also did not address the fuel cell or sUAV application studied in the present paper, nor the drift counteraction problem formulation.

The remainder of this paper is organized as follows: Section 2 describes the sUAV sub-systems and their models. Section 3 presents an integrated model of the hybrid system and defines the problem in a form suitable for SDCOC. Section 4 summarizes SDCOC, and Section 5 reports the results. Finally, Section 6 presents concluding remarks.

#### **2. Physical Description of the Systems and Model**

An sUAV with a series hybrid propulsion system, shown in Figure 1, was chosen in which the power supplied by the battery and the power supplied by the PEMFC are combined to meet the propeller motor power demand. The PEMFC uses hydrogen as the fuel, which is stored in the tank, and air from the atmosphere. A fraction of the energy generated by the PEMFC can be used to charge the battery. The fuel cell pack and battery pack are sized large enough so that they are able to meet the sUAV's mean power demand individually, should either one not be operating properly.

**Figure 1.** A diagram of a fuel cell-powered series hybrid small UAV (sUAV).

The model used in this paper for generating the SDCOC policies captures the battery's *SOC* dynamics, the fuel cell's hydrogen rate dynamics, and the fuel cell load power dynamics. Thus, the states of this model are the *SOC*, the mass of hydrogen remaining in the gas tank, and the fuel cell load demand power. The motor power of the sUAV and climb angle are treated as operating variables, and the SDCOC controller determines changes in the fuel cell load demand power. This system level model has been implemented by combining component submodels and characterizations available from the literature; our methodology is generic and can accommodate changes in these component models.

#### *2.1. sUAV Dynamics*

A control-oriented dynamic model of the sUAV is used for SDCOC law development. The sUAV is constrained to a longitudinal flight path in a vertical plane [22]. Table 1 defines the notations for the variables used in the model. Table A1 in Appendix A lists the model parameter values, partly based on [23–25]. The development of lightweight electric components (batteries, fuel cells, motors) for sUAVs is an active area of research; see, e.g., [26,27]. In our model, we assumed that such lightweight components are available to be consistent with the assumed sUAV's weight.


**Table 1.** List of variables used in the sUAV model.

Using a flat Earth coordinate system, the longitudinal equations of motion of the sUAV are given by:

$$\dot{v} = \frac{T\cos(\alpha) - D}{m} - g\sin(\gamma),\tag{1}$$

$$\gamma = \frac{\text{T}\sin(\alpha) + L}{m\upsilon} - \frac{\text{g}\cos(\gamma)}{\upsilon},\tag{2}$$

where *v* is the velocity of the sUAV and *γ* is the climb angle. The lift *L* and drag force *D* are characterized as:

$$L = \frac{1}{2} \rho\_{\dot{a}\dot{r}} v^2 \mathcal{S}\_{rcf} \mathcal{C}\_{L\prime} \ D = \frac{1}{2} \rho\_{\dot{a}\dot{r}} v^2 \mathcal{S}\_{rcf} \mathcal{C}\_{D\prime} \tag{3}$$

where *CL* = *CL*<sup>0</sup> + *CL*,*αα*, *CD* = *CD*<sup>0</sup> + *KCL* 2. Neglecting vertical acceleration (i.e., with *L* = *mg*), solving Equations (1) and (2) yields the thrust required by the sUAV,

$$T = \left[ \left( m\wp + mg\sin(\gamma) + \frac{1}{2}\rho\_{\mathrm{dir}}\wp^2 S\_{\mathrm{ref}}C\_{\mathrm{D0}} + \frac{2Km^2g^2}{\rho\_{\mathrm{dir}}\wp^2 S\_{\mathrm{ref}}} \right)^2 + \left( m\wp\gamma + mg\cos(\gamma) - mg \right)^2 \right]^{\frac{1}{2}}.\tag{4}$$

Here, *ρair* is a function of altitude. The power required by the sUAV is then given by:

$$P\_{slIAV} = T\upsilon.\tag{5}$$

## *2.2. Propeller Model*

The propeller model is used to relate the torque and angular velocity generated by the electric motor to the power required by the sUAV and the velocity of the sUAV, respectively [22]. With the propulsive efficiency given by *ηP*, the power required to drive the propeller is:

$$Pp = \frac{P\_{sIIAV}}{\eta\_P}.\tag{6}$$

According to the disk actuator theory, the ideal propeller power is:

$$P\_{P,ideal} = \frac{1}{2} T \upsilon \left( 1 + \sqrt{1 + \frac{8T}{\pi \rho\_{air} \upsilon^2 d\_P^{-2}}} \right).$$

In general, the actual power required would be about 15% greater than this [28], which means *PP* = 1.15*PP*,*ideal*. Thus, *η<sup>P</sup>* can be calculated as:

$$\eta\_P = \frac{P\_{sIAV}}{1.15 P\_{P,ideal}} = \frac{2}{1.15 + 1.15\sqrt{1 + \frac{8T}{\pi \rho\_{air} v^2 d\_P}}} \cdot \tag{7}$$

Combing Equation (7) with (5) and (6) yields:

$$P\_P = \frac{1.15 P\_{sIIAV}}{2} + \frac{1.15 P\_{sIAV}}{2} \sqrt{1 + \frac{8 P\_{sIIAV}}{\pi \rho\_{air} \upsilon^3 d\_P^2}}.$$
