Energy Management of Hybrid Electric Urban Bus by Off-Line Dynamic Programming Optimization and One-Step Look-Ahead Rollout

Abstract

Due to the growing air quality concern in urban areas and rising fuel prices, urban bus fleets are progressively turning to hybrid electric vehicles (HEVs) which show higher efficiency and lower emissions in comparison with conventional vehicles. HEVs can reduce fuel consumption and emissions by combining different energy sources (i.e., fuel and batteries). In this sense, the performance of HEVs is strongly dependent on the energy management strategy (EMS) which coordinates the energy sources available to exploit their potential. While most EMSs are calibrated for general driving conditions, this paper proposes to adapt the EMS to the specific driving conditions on a particular bus route. The proposed algorithm relies on the fact that partial information on the driving cycle can be assumed since, in the case of a urban bus, the considered route is periodically covered. According to this hypothesis, the strategy presented in this paper is based on estimating the driving cycle from a previous trip of the bus in the considered route. This initial driving cycle is used to compute the theoretical optimal solution by dynamic programming. The obtained control policy (particularly the cost-to-go matrix) is stored and used in the subsequent driving cycles by applying one-step look-ahead roll out, then, adapting the EMS to the actual driving conditions but exploiting the similarities with previous cycles in the same route. To justify the proposed strategy, the paper discusses the common patterns in different driving cycles of the same bus route, pointing out several metrics that show how a single cycle captures most of the key parameters for EMS optimization. Then, the proposed algorithm (off-line dynamic programming optimization and one-step look-ahead rollout) is described. Results obtained by simulation show that the proposed method is able to keep the battery charge within the required range and achieve near-optimal performance, with only a 1.9% increase in fuel consumption with regards to the theoretical optimum. As a reference for comparison, the equivalent consumption minimization strategy (ECMS), which is the most widespread algorithm for HEV energy management, produces an increase in fuel consumption with respect to the optimal solution of 11%.

1. Introduction

With the growing concern about greenhouse gases and air pollution on the environment, hybrid electric vehicles (HEV) are being considered as one of the alternatives towards clean mobility [1]. One of the advantages of vehicle hybridization is that the combination of internal combustion engines (ICE) and electric motors (EM) provides an additional degree of freedom to the powertrain that allows meeting the driver power demands more efficiently. In addition, the use of batteries and EMs allows energy recovery during braking [2]. Among mobility alternatives, urban buses present a high potential of electrification (including hybridization) due to their high mileage covered, known routes and strict requirements for air pollution control in cities. The previous reasons are leading European cities and manufacturers who invest efforts in electrified powertrains for buses [3] such as pure electric, fuel cell or HEV and in other sustainable alternatives such as hydrogen engines.

HEV performance depends on powertrain configuration, and accordingly, there is an extensive literature discussing HEV powertrain architectures (parallel, series, power split, etc.) and sizing [4,5,6]. However, the additional degree of freedom of HEVs, i.e., the power split between battery and fuel tank, addressed by energy management strategies (EMS), also plays a key role in vehicle performance. For this reason, several approaches have been investigated in the literature. An extensive review of HEV energy management can be found in [7]. As expected, the literature points out that the performance and optimal EMS are closely related with driving factors such as driver behavior, road slope, traffic and weather conditions that can influence the energy distribution in HEV [8]. Another important aspect that can be extracted from the available literature is that while the EMS can be addressed by the application of heuristic techniques, there is room for optimal control theory, leading the EMSs to specifically address the minimization of the vehicle energy consumption [7]. In this group, some studies are focused on offline optimization, where the EMS of a given HEV in a pre-defined driving cycle is optimized. Among the techniques applied for offline optimization, dynamic programming (DP) [9] and methods based on Pontryagin’s minimum principle (PMP) [10,11,12] are the most widespread. DP is widely applied in offline optimization problems; this method consists of subdividing the nonlinear dynamic problem into subproblems in a discretized time, and then proceeding backwards generating a cost-to-go function depending on the state value and control action chosen. Then, the optimal cost-to-go is calculated for every sample time and stored in a matrix. When the entire problem has been examined, the optimal solution is obtained by tracking the path with the lowest cost-to-go. The major issue of the previous techniques is the requirement of a priori knowledge of the driving cycle, something that only happens in homologation cycles or very particular vehicle applications. For this reason, DP is widely used as an optimal benchmark solution to compare with the results of other approaches, as well as to extract control parameters for other control strategies [13].

Several approaches have been investigated in the literature in order to overcome the driving cycle dependency to develop methods that can be applied for control purposes, leading to near-optimal results. Most of them are based on the equivalent consumption minimization strategy (ECMS), that is based on applying, in any time-step, the control action that minimizes a weighted function of the energy consumption from the fuel tank and the batteries [14]. Among them, adaptive-ECMS (A-ECMS) [15,16] and adaptive-PMP (A-PMP) [17] provide online implementable solutions that consist in updates to the co-state, or the equivalence factor in the case of ECMS as driving conditions change, making possible to reach near-optimal solutions [7]. The task of updating the co-state is implemented in several ways and with different information sources. Ref. [18] shows the adaption based on driving cycle prediction implementing an on-the-fly algorithm for the estimate of the equivalent factor according to the driving conditions, combining past and predicted data to periodically refresh the equivalent factor. Ref. [19] uses an adaptive-ECMS in an HEV with continuously variable transmission based on driving pattern recognition. The proposed method considers the type of driving pattern and the battery state of charge (SOC) to adapt the equivalent factor. Ref. [17] presents an online co-state adaptation rule based on the SOC. In a similar way, authors in [20] employ a Markov chain model for vehicle-speed forecasting, and adapt the equivalent factor to the expected driving cycle. The main advantages of the previous approaches are the physical interpretability and their low computational cost; on the other hand, they need substantial calibration effort (specially for the equivalent factor) and sacrifice optimality in order to be applicable in cases where the conditions of the driving cycle are not well defined [21].

Another method developed to tackle the limitations of the driving-cycle information dependency of DP is the model predictive control (MPC). The main advantages of MPC are its ability to deal with system constraints and the capability of leveraging driving cycle estimations in a prediction horizon (e.g., traffic data, preceding vehicles and traffic lights) for optimization [22,23,24].

Although previous techniques have been developed, in general, for the passenger-car application, most of them have been adapted to the particular case of urban buses. In this sense, ref. [25] compare different architectures for HEV hybrid buses and employ DP to perform this comparison with optimal energy management for every architecture. In the same line [26] propose the use of DP to calibrate a heuristic energy management strategy for a hybrid bus. Regarding methods that can be applied for control, several variations of the ECMS have been applied to the field of hybrid buses. For example, in [27], authors approach the optimization of emissions by applying ECMS with an engine map that combines fuel consumption and emissions. In [28], authors proposed a ECMS strategy where the equivalence factor is adapted to driving condition of the bus by means of neuro-fuzzy inference techniques. Similarly, in [29], the equivalence factor of the ECMS in a HEV bus is adapted by means of genetic algorithms. A model predictive control based on Pontryagin’s minimum principle is applied in [30] to a hybrid electric bus, with a fuel consumption that exceed the theoretical optimum of 6% according to simulation results.

In line with previous papers, this article presents an EMS for a parallel hybrid electric urban bus. The objective of the proposed strategy is to be applicable for control purposes and have a performance close to the theoretical optimum. As in previous literature [7], the criteria to judge the performance of the strategy is the fuel consumption and the deviation between the initial and final energy stored in the battery, which must be minimized to ensure the charge sustainability. To approximate the theoretical optimum, the proposed strategy is based on model-based optimization. On the other hand, to be applicable for control purposes, the proposed strategy must meet two main conditions:

• The computation time should be low enough to be applied in real time. To this aim, the developed EMS is based on a simplified powertrain model as in the previously cited papers.
• The strategy should not rely on future information. The proposed method exploits the periodicity in the route covered to reduce the dependence of the optimal control technique on the information about the future driving conditions. While previous literature estimates the future driving conditions from a receding horizon and then searches of an optimal control policy, the proposed methodology does not rely on the estimation of future driving conditions. Instead, the proposed methodology is based on the offline optimization of a previously covered driving cycle by dynamic programming, and then uses the obtained cost-to-go as an approximation for the cost-to-go in the actual route, which is a novelty with regards to previous works.

Once the cost-to-go is estimated, the optimal power-split at every time-step of the real route can be computed by applying the one-step look-ahead roll out algorithm [31]. The simulation results obtained show near-optimal performance for the proposed strategy, sustaining the energy in the battery and, in terms of fuel consumption, showing a 1.9% increase with regards to the theoretical optimum, while the standard ECMS, with the same information as the rollout, shows an increase of 11%.

According to the previous paragraph, the main novelties and contributions of the proposed strategy are:

• The paper shows that the resemblance of the different loops covered by an urban bus in its route can be used to obtain a control strategy that minimizes its fuel consumption while fulfilling the restrictions on the energy in the battery.
• The proposed strategy produces near-optimum results, improving the fuel consumption obtained with other algorithms in the literature and without relying on information about future driving conditions.
• Unlike most of the available literature on the EMS of HEVs, the proposed strategy does not require the estimation of the future driving conditions and does not follow the ECMS approach for optimization.

The paper is organized as follows: after this introduction in Section 1, the case study and the powertrain model used in the paper are presented in Section 2. Then, Section 3 introduces two standard algorithms (DP and ECMS) to be used as reference strategies for offline and online optimization in order to assess the performance of the proposed strategy. Section 4 analyzes the measured vehicle-speed profiles and the optimal controls from DP in order to find the main patterns. Section 5 describes the proposed algorithm based on the dynamic programming optimization of a reference loop previously registered and one-step look-ahead rollout. Finally, Section 6 and Section 7 discuss the obtained results and highlight the main paper conclusions, respectively.

2. Case Study and HEV Model

The case study is a hybrid electric bus covering the route 2 (“Universitats-Hosp. Dr.Peset”) from the public transport service in the city of Valencia (EMT-Valencia). The route consists of a loop of 15.1 km which is repeated by the bus between 9 and 10 times a day with an average duration of 5100 s and a standard deviation of 320 s. The vehicle position and speed of the bus in the considered route was registered during two consecutive working days, covering 287 km in total. A GPS in the vehicle outputs the vehicle position, while the vehicle speed is obtained from the vehicle measurements through the OBD port. To divide the total distance covered daily by the bus between the different loops, the starting and ending point of every loop is determined from the GPS signal. In particular, the starting point of the route is associated to the measurement sample with minimum Euclidean distance between its GPS coordinates (in UTM) and those of the theoretical starting location (7.275 × ${10}^5$ UTM Easting, 4,373,741 UTM Northing). The ending point of a given loop is associated to the starting point of the next loop. When the initial and final points of a given loop are determined, the distance covered is computed by integrating the vehicle-speed signal obtained from the ECU. Figure 1 shows the vehicle-speed traces in time (left plot) and space (right plot) bases along the 19 repetitions of the loop.

While it is difficult to observe any clear correlation between the speed profiles shown in Figure 1, not in time (left plot) nor in distance domain (right plot), it is intuitive that most of the vehicle speed disturbances, e.g., traffic lights or bus stops, and even the trip duration, are defined in a space basis. It is intuitive that the loop duration is defined in terms of distance instead of time, for this reason, important deviations in the time duration of the different loops can be observed in Figure 1 (left), while they show similar distances covered around the theoretical value of 15.1 km (right). Despite the route being the same in all the loops, some discrepancies with the exact distance covered appear due to differences in the driving trajectory, e.g., due to sporadic line changes to avoid obstacles and due to measuring uncertainties. In any case, Figure 1 shows a better correlation between the driving cycles in the distance domain and even some strips of high and low velocity can be observed in the space basis (right plot). This correlation is exploited by the proposed control strategy of the HEV bus, whose architecture follows the P2 scheme shown in Figure 2.

According to the previous paragraph, the objective of the energy management strategy can be formally defined as finding the control law $u\left(t\right)$ that minimizes the following cost:

$$J={\int }_{t_0}^{t_f}{P}_f\left(u\left(t\right),t\right)\mathrm{d}\mathrm{t}+{(E_b\left(t_f\right)-E_b\left(t_0\right))}^2$$

(1)

where $P_f$ is the power of the fuel consumed by the ICE, $t_0$ and $t_f$ are the starting and ending time of the trip, u is the control action, i.e., the power split between ICE and motor and $E_b$ is the energy stored in the battery. Note that the last term in Equation (1) is needed to ensure the battery charges sustainability. The key variables to determine the value of J are the energy stored in the battery at the end of the trip and the fuel consumption, so a powertrain model able to estimate the system evolution and the impact of control decisions on J is needed. According to the powertrain architecture in Figure 2, the power provided by the ICE and motor must be equal to the mechanical power required by the vehicle to follow the driving cycle; moreover, due to the mechanical link, the motor and ICE share the same speed and, therefore, the following relation between the torque provided by both machines should be fulfilled:

$$T_g=T_m+T_{ICE}$$

(2)

where $T_g$, $T_m$ and $T_{ICE}$ are the torque in the powertrain transmission, motor and ICE, respectively. Including the inertia of powertrain elements in an equivalent vehicle mass (m), the wheel torque required to drive the vehicle is given by:

$$T_w=(m\dot{v}-mg\mu cos\theta -mgsin\theta -\frac{1}{2}\rho A{c}_d{v}^2)R_w$$

(3)

where $\mu$ is the rolling coefficient, $\theta$ is the angle due to the road slope (that can be neglected in the considered route), $\rho$ is the air density, $A{c}_d$ the product of frontal area and aerodynamic coefficient of the vehicle, $R_w$ the wheel radius and, finally, v and $\dot{v}$ the vehicle speed and acceleration, respectively. Note that while motor and ICE speeds are proportional to the wheel speed through the gear ratio of the selected gear, their joint torque in Equation (2) ($T_g$) is proportional to the wheel torque through the inverse of the gear ratio. In this sense, if the demanded vehicle speed is known, the demanded acceleration can be computed and the requested torque for the powertrain ($T_g$) can be estimated from Equation (3) and the gear ratio. Then, defining the control action as $u=T_m$, Equation (2) can be reformulated as:

$$T_{ICE}=T_g-T_m$$

(4)

and, therefore, provided a vehicle-speed demand and a decision on u, the ICE torque is defined. Regarding the ICE, the model employed is based on the quasi-static (QS) approach introduced in [32], using experimental data to map the fuel consumption depending on the engine speed (${\omega }_g$) and torque ($T_{ICE}$), so:

$$m_f^{QS}=g({\omega }_g,T_{ICE})$$

(5)

The quasi-static approach assumes that the engine can move instantaneously from one operating condition to another without an increase in the fuel consumption and, particularly, there is not any fuel associated to start the ICE. In this sense, the simplicity of the quasi-static approach comes at the price of having frequent switching between ICE on and off that can be unrealistic. To avoid this issue, an additional cost is used for the engine start as:

$$m_f=\left\{\begin{array}{c}(1+\beta )m_f^{QS}ifIC{E}_{on}=0\\ m_f^{QS}ifIC{E}_{on}=1\end{array}\right.$$

(6)

where $\beta$ is a positive value to be calibrated and the integer variable $IC{E}_{on}$ represents if the engine is on (1) or off (0) according to the following dynamics:

$$IC{E}_{on}\left(t\right)=\left\{\begin{array}{c}1if{m}_f(t-dt)>0\\ 0else\end{array}\right.$$

(7)

Regarding the energy stored in the battery ($E_b$), its dynamic equation is:

$${\dot{E}}_b=-P_b$$

(8)

with $P_b$ being the battery power (considered positive when the battery is drained and negative when the battery is being charged according to the sign criteria defined in Figure 2). The battery power $P_b$ is obtained from the motor power demand and the motor efficiency by means of a model. The quasi-static approach is employed for the electric motor, while in this case the map used considers positive and negative torques to model battery recharging. Finally, a Thevenin electric equivalent circuit is proposed to model the battery dynamics as a resistance in series with a voltage source:

$$V=V_{oc}-I_b{R}_b$$

(9)

where $I_b$ is the current drawn from the battery, i.e., the variation on the battery charge (${\dot{Q}}_b=-I_b$) and R represents its internal resistance that depends on the battery state of energy ($SoE$), i.e., a measure of the energy level in terms of remaining energy relative to the total energy content of the fully charged battery ($E_{b,0}$):

$$E_{b,0}=V_{oc,0}{Q}_{b,0}$$

(10)

with $V_{oc,0}$ and $Q_{b,0}$ being the open circuit voltage and charge of the fully charged battery. The actual energy stored in the battery is represented by:

$$E_b=V_{oc}{Q}_b$$

(11)

In order to normalize the battery energy, the state of energy of the battery can be defined as:

$$SoE=\frac{E_b}{E_{b,0}}=\frac{V_{oc}{Q}_b}{V_{oc,0}{Q}_{b,0}}=SoC\frac{V_{oc}}{V_{oc,0}}$$

(12)

where $SoC$ is the state of charge of the battery, used in many works instead of the $SoE$.

3. Benchmarkt Solutions: ECMS and DP with a Priori Knowledge of the Driving Cycle

The equivalent consumption minimization strategy (ECMS) is the most frequently used method to address the optimization problem introduced in Section 2. The ECMS is based on using an equivalence factor for the battery power to take into account the potential of currently discharging the batteries and recharging them in the future or vice versa. Assuming that this equivalent factor between battery and fuel energy is constant, the integral problem represented by Equation (1) can be replaced by the minimization of a set of instantaneous objective functions:

$$c=P_f+s{P}_b$$

(13)

where the parameter s represents an equivalent factor between fuel and battery energy sources and accordingly balances the solution between the fuel and battery usage. The link between the ECMS and optimal control, and more precisely Pontryagin’s minimum principle, is carefully discussed in [33]. From Equation (13), it can be observed that high values of s impose a high weight to the battery energy in the objective function, then promoting the fuel use and battery charging, while low values of s entail a low impact of the battery use in the objective function, then favoring fuel savings at the expense of battery depletion. For a given driving cycle, there is an optimal value of s that leads to the fuel consumption minimization while sustaining the battery energy. Of course, the optimal value of the parameter s strongly depends on the driving conditions, and the same equivalence factor s may lead to battery depletion in a given cycle and battery overload in a different one. In cases where the driving cycle is a priori known, the optimal value of s may be found by iteratively searching the value that satisfies the charge sustainability ($SoE\left(t_f\right)=SoE\left(t_0\right)$). In the general case, where the driving cycle is unknown, different methods to calibrate and adapt the value of s depending on the operating conditions may be found in [18,21], but essentially, they rely on calibrating the value of s in a set of representative cycles, and then apply corrections depending on $SoE$ deviations from the reference value.

On the other hand, the optimization problem introduced in Section 2 can be formally cast in the optimal control framework aimed to the minimization of a generic cost function:

$$J=\psi \left(x\left(t_f\right)\right)+{\int }_{t_0}^{t_f}L(x\left(t\right),u\left(t\right),d\left(t\right))dt$$

(14)

where x is a vector containing the system states, u is the vector of control actions and d is the vector of disturbances that affect the system evolution. Note that the function J includes an integral term (L) and a terminal cost $\psi$ that penalizes deviations from a desired final state. From Equation (1), one can identify L in Equation (14) as the power of the fuel consumed ($P_f$) and $\psi \left(x\left(t_f\right)\right)$ as the deviation from the initial energy stored in the battery. The optimal control problem consists on finding a trajectory $u\left(t\right)$ minimizing the function J, i.e., $mi{n}_{u}J$. The system state evolves from its initial value $x\left(t_0\right)$ driven by disturbances and control actuations according to a set of first-order differential equations represented by a generic function f:

$$\dot{x}\left(t\right)=f(x\left(t\right),d\left(t\right),u\left(t\right))$$

(15)

In the case at hand, the states of the system are the battery state of energy ($SoE$) and the state of the ICE (on or off), while the control action is the motor torque (see Equation (4)), the disturbance is the vehicle speed and the function f is defined by Equations (2)–(12). The Bellman’s principle of optimality states that: “An optimal policy has the property that whatever the initial state and initial decisions are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decisions” [34], which informally states that any partial trajectory inside the optimal one is also optimal between its initial and final states and leads to the Hamilton-Jacobi–Bellman (HJB) equation:

$${\mathcal{J}}^{*}(x\left(t\right),t)=mi{n}_u\left\{{\int }_t^{t+\delta t}L(x\left(\tau \right),u\left(\tau \right),\tau )d\tau +{\mathcal{J}}^{*}(x(t+\delta t),t+\delta t)\right\}$$

(16)

where the optimal cost-to-go ${\mathcal{J}}^{*}$ from any arbitrary state $x\left(t\right)$ at time t ($t_0\le t\le t_f$) can be calculated as the cost of a differential problem with length $\delta t$ plus the optimal cost-to-go from the resulting state at $t+\delta t$ to the end. Dynamic programming is an algorithm exploiting the Bellman principle of optimality and the HJB equation to numerically solve an optimal control problem. The method is based on the discretization of the time span of the problem in n time-steps, so for any state value in a given time-step (k), the Bellman’s principle of optimality implies that:

$${\mathcal{J}}^{*}(x,k)=mi{n}_u\left\{L(x,u)+{\mathcal{J}}^{*}(x,k+1)\right\}$$

(17)

Among other methods, the x and u spaces can be discretized and the problem numerically solved, starting from the last time-step $k=n-1$ (so ${\mathcal{J}}^{*}(x,n)=\psi \left(x\left(n\right)\right)$) and proceeding backwards accumulating the cost-to-go for the whole x space until obtaining a resulting space of cost-to-go values for the optimal solution at the initial time-step as a function of the initial state ${\mathcal{J}}^{*}(x,1)$. Note that storing ${\mathcal{J}}^{*}(x,k)$ allows to reproduce not only the optimal solution from the initial state, but from any point in the $(x,k)$ space, which makes DP a powerful tool for the optimization of dynamic systems. Of course, this potential does not come for free and DP suffers from the so-called curse of dimensionality: the number of combinations to test during the problem solution and the number of elements that must be stored in memory rapidly increase with the problem length, number of states and actuators and their discretization. According to Section 2, there are only two states in the case at hand, i.e., the energy stored in the battery (see Equation (11)) and the state of the ICE (on or off, see Equation (7)), and one control (the motor power), so the problem is still affordable by DP. On the other side, solving the optimal control problem requires perfect knowledge of the disturbances, in the case at hand the vehicle speed profile. Since future driving conditions cannot be perfectly known, DP cannot be used for online control purposes, but its result can be used as a benchmark for the performance of other strategies. In this sense, assuming that the complete driving cycle consisting of all the loops covered by the bus in the two days of testing is known, the optimal solution can be found by DP and then be used as a reference for comparison of the control strategy proposed in this paper. Figure 3 shows the results obtained after solving by DP for the complete driving cycle. The upper plot and medium plots show the fuel consumption in the considered loops and in the complete cycle. An average value of 36.5 kg/100 km in the fuel consumption during the loops and a standard deviation of 1.5 kg/100 km points out that there are non-negligible differences between the driving cycles due to the driving conditions but also due to the SoE at the end of every loop. The lower plot shows the evolution of the SoE despite the different evolutions during the cycles; a clear pattern can be observed in most of them, consisting on an initial battery charging during the first part of the loop, then some discharging below the initial SoE and finally a recovery phase to reach similar values as the initial SoE. Finally, the contour lines in the bottom plot represent the cost-to-go (${\mathcal{J}}^{*}(x,k)$), in kg of fuel, depending on the vehicle position. Note that despite there being two different cost-to-go maps (one for the case when the engine state is on and other for the case when it is off), for the sake of simplicity, the figure only shows the case when the engine is on. It can be noticed that the iso-cost-to-go lines are almost parallel, pointing out again that there is some periodicity in the optimal solution with regards to the vehicle position.

4. Discussing Correlation between Bus Driving Cycles

While DP results in the previous section point out that the optimal control of the vehicle differs from one loop to other, they also show some hidden patterns related to the vehicle position in the loop. To analyze that, Figure 4 shows the Fourier transform in distance domain of the measured vehicle speed and optimal SoE evolution obtained by DP. It can be clearly noticed that in both cases the maximum in the signal amplitude appears at the loop distance, in the case at hand 15.1 km, and then shows that there is some repetition in the loops completed by the bus that can be exploited to generalize the controls optimized for the vehicle in a single loop to the rest of the bus journey.

Beyond the frequency analysis previously presented, one of the most widespread methods for driving-cycle characterization relies on the vehicle speed–acceleration probability distribution (SAPD) [35]. The intuition behind using this metric as a way to characterize driving cycles is rooted in the vehicle energy balance itself, stating that, in absence of road gradients and other perturbations such as wind, the vehicle power demands are essentially governed by the acceleration and the velocity in a nonlinear but straightforward way according to Equation (3). This fact has lead many authors to propose different metrics based on the vehicle speed and acceleration to:

• Characterize driving cycles and driving style. A simplified version of those metrics are the $v{a}_{pos,95}$ defined as the 95th percentile of the product between vehicle speed and positive acceleration used in the European regulations [36] to admit an RDE cycle as valid.
• Classify driving cycles according to their aggressiveness or provide estimations on the vehicle emissions [37] or powertrain controls to optimize performance [38].
• Predict future driving conditions in a short horizon for control optimization purposes. In this line, the probability distribution of the vehicle speed and acceleration (or its equivalent distribution between current and next time-step speed) is used to model driving conditions as a Markov process, where the probability of a given velocity in a time-step is only dependent on the velocity in the previous one. The literature on this topic is extensive, as can be found in [39].

Figure 5 shows the probability distribution of the vehicle speed and acceleration obtained for the 19 loops registered. Defining a state as the combination of vehicle speed and acceleration, the distribution matrix for every cycle was estimated from the frequency of the possible states during the cycle. For practical reasons, the vehicle speed and acceleration is discretized in intervals of 0.28 m/s and m/s${}^2$ corresponding to the measuring sensibility of 1 km/h and the sampling rate of 1 s. Let $N_{i,j}$ be the number of times the vehicle speed is $v_i$ and acceleration is $a_j$ during a cycle with duration N, the probability of being in state $(v_i,a_j)$ in the previous time can be estimated as:

$$P_{v_i,a_j}=\frac{N_{i,j}}{N}$$

(18)

where, since the sum of the probabilities of all outcomes must equal 1, then:

$$N=\sum _j\sum _i{N}_{i,j}$$

(19)

Note that the distribution of all the cycles share common characteristics as an attractor at $v=0$, $a=0$ due to the frequent stops of the vehicle, and a reduction in acceleration as speed increases due to vehicle power constraints or maximum velocities around 14 m/s due to urban speed limits (50 km/h). However, despite there being a clear relation between probability distributions shown in Figure 5, the quantification of how one of the cycles may be representative of the rest requires a deeper analysis beyond the visual inspection. The singular value decomposition (SVD) of a matrix X is a factorization into the product of three matrices ($X=US{V}^T$), such that U and V are orthogonal matrices and S is a diagonal matrix with decreasing, non-negative singular values ($\sigma$). The explanation of the method is beyond the scope of the present paper and can be found in general linear algebra or machine learning literature [40]. However it is interesting to recall that SVD provides the optimal (in the sense of minimum square error) low-rank approximation of a matrix, then providing the most dominant patterns in the data. Then, rearranging the velocity–acceleration distributions for every cycle in Figure 5 as columns of a matrix, and applying the SVD algorithm, it is possible to obtain the 19 linearly independent modes shown in Figure 6. Any of the vehicle SAPDs in Figure 5 are a linear combination of the modes in Figure 6 and provide a limited model order (consider only $r<19$ modes); the first r modes provide the best estimation of the distribution. One can observe how the first mode certainly resembles any of the distributions in Figure 6 (and actually is their average) and, as the mode number increases, this resemblance progressively disappears.

The singular values ($\sigma$), which are the elements in the diagonal of the S matrix, represent the weight of the modes on the description of the data in matrix X, and as they are ranked in decreasing order, the first singular values (and corresponding modes) contain the leading information. The last plot in Figure 6 analyzes the percentage of signal contained as the number of considered modes increases. It can be observed that the first singular value (corresponding to the average) accounts for 88% of the data variance and that with three modes this figure increases above 90%.

Both frequency and vehicle speed–acceleration analyses show that there is a close relation between the operating conditions reached by the bus in the different loops of the selected route. Consequently, an optimization of the vehicle energy management for one of the loops might offer useful information to apply in the rest.

5. Proposed Algorithm: Dynamic Programming Optimization of a Reference Loop and One-Step Look-Ahead Rollout

While the perfect knowledge of the driving conditions is not possible, the preceding section shows that there is a close relation between the different loops covered by the bus within the route. In this sense, this paper proposes to use an arbitrary loop to make a DP optimization and employ the result as a base policy for vehicle energy management. Of course, despite similarities, all the loops are not exactly the same and deviations may lead to undesired vehicle behavior, e.g., fully depleting the battery or exceeding its maximum charge. To avoid this issue, the base policy provided by the DP should be adapted to actual working conditions. In the present work, the rollout algorithm [31] is used to this aim.

Consider the optimal control problem proposed in Equation (1), where the trip duration and disturbance (vehicle speed sequence) is that of one loop of the studied bus line registered beforehand. Call the registered cycle $ref$. The DP solution for this particular problem can be obtained offline, as explained in Section 3, and provides the optimal cost-to-go from any state $x_j$ at time-step k denoted by ${\mathcal{J}}^{*}{(x_j,k)}_{ref}$ for this specific cycle. Since the magnitude conserved between loops is the distance, a space-based optimal cost-to-go can be built by transforming time to space basis by integration of the vehicle speed registered during cycle $ref$. In this sense, a matrix (${\mathcal{J}}^{*}{(x_j,s_i)}_{ref}$) mapping the optimal cost-to-go in the reference cycle at a given vehicle position in the loop ($s_i$), depending on its state ($SoE$, $IC{E}_{on}$), can be built from the DP solution.

Taking into account that the complete driving cycle consists of a sequence of similar loops, ${\mathcal{J}}^{*}{(x_j,s_i)}_{ref}$ is an approximation of the optimal cost-to-go from any state $x_j$ in a position $s_i$ in any of the loops that can be used as a base policy. Recalling the HJB Equation (16), the one-step look-ahead rollout algorithm proposes to replace the optimal cost-to-go from the next state by the base policy, then:

$$u_k=argmi{n}_u\left\{L(x_k,d_k,u)+{\mathcal{J}}^{*}{(f_k(x_k,u),s_{k+1})}_{ref}\right\}$$

(20)

where $f_k$ is the discrete version of the state function f in Equation (15) and ${\mathcal{J}}^{*}{(f_k(x_k,u),s_{k+1})}_{ref}$ is an approximation of the optimal cost-to-go from the state in the next time step, resulting after applying action u, that is obtained from the DP solution of the reference cycle. Even though the proposed algorithm provides a suboptimal solution due to the fact that ${\mathcal{J}}^{*}{(x_j,s_i)}_{ref}$ is only an approximation of ${\mathcal{J}}^{*}(x_j,k)$, their difference tends to vanish as the estimated driving cycle ($ref$) approaches the real one. The inspection of Equation (20) shows that the proposed rollout algorithm has two main properties:

• The obtained policy always improves the base policy since it uses information from the actual cycle in the next time-step ($d_k$) to choose $u_k$.
• The algorithm is able to consider the feedback on the current system state to adapt the control policy, for instance avoiding battery SoE excursions outside the desired range by the evaluation of L.

Figure 7 shows the diagram of the proposed algorithm particularized for the case study. The left side shows the offline part of the algorithm, consisting of solving by DP a previous driving cycle of the bus in the same route. In the case at hand, a single loop previously recorded is considered as the reference cycle. The cost-to-go of the reference cycle is obtained by DP. Then, the cost-to-go is stored in a map (${\mathcal{J}}^{*}{(\overline{SoE},{\overline{ICE}}_{on},\overline{s})}_{ref}$) that provides the minimum fuel that the bus consumes in the reference cycle depending on the SoE, state of the ICE (on or off) and vehicle distance. In this sense, $\overline{SoE}$, ${\overline{ICE}}_{on}$ and $\overline{s}$ are vectors containing the possible values of SoE, state of ICE and vehicle distance. On the right side, the algorithm that runs while the vehicle is covering its actual route is described. The model presented in Section 2 (Equations (2)–(12)) is used to estimate, at every time step (k), the fuel consumption ($m_{f,k}$) depending on a set of possible motor torques (${\overline{T}}_m$), as well as the state of the vehicle in the next time-step ($k+1$), consisting of the SoE ($So{E}_{k+1}\left({\overline{T}}_m\right)$), state of the engine ($IC{E}_{on,k+1}\left({\overline{T}}_m\right)$) and distance covered by the vehicle ($s_{k+1}\left({\overline{T}}_m\right)$). Note that the estimated fuel consumption and state of the vehicle in the next time-step depend on ${\overline{T}}_m$, so they are vectors with the same size. The estimated state of the vehicle is then used to interpolate in the cost-to-go map the minimum fuel consumption that the vehicle has from the next time-step to the end of the loop depending on the control action chosen (${\mathcal{J}}^{*}{\left({\overline{T}}_m\right)}_{ref}$). Finally, in the optimization block, the action to be applied $T_{m,k}$, is the one that minimizes, not only the current fuel consumption ($m_f\left({\overline{T}}_m\right)$), but the total fuel consumption until the end of the loop, i.e., the sum of $m_f\left({\overline{T}}_m\right)$ and the minimum fuel from the next time-step to the end. This control action is applied to the HEV (in the case at hand a model), which updates the vehicle state, and the process in the current driving cycle continues until the end of the trip.

6. Results and Discussion

The potential of the proposed strategy is assessed by comparing its performance with two extreme cases. On the one hand, the DP solution of the complete driving cycle is considered as best-case scenario, where the complete driving cycle is a priori known and the optimal power split can be computed. On the other hand, the widespread ECMS is employed as a reference technique with online capability. The s factor is calibrated with the reference loop ($ref$), and this value is applied during the complete driving cycle. In the case at hand, the first loop registered by the bus was considered as reference ($ref$) for both ECMS and the rollout algorithms, and was removed from the evaluated driving cycle. Since differences between the reference loop and the rest may lead the ECMS to fail in its aim of sustaining the charge, a feedback from $SoE$ is considered, so the reference s is used in the $SoE$ range between 0.3 and 0.7, while its value increases if $SoE$ falls below 0.3 and decreases if $SoE$ rises above 0.7.

Figure 8 shows the evolution of the fuel consumption and $SoE$ during the complete driving cycle for the three strategies compared. It can be observed how the three methods are able to sustain the charge. Of course, DP succeeds in keeping the $SoE$ within the allowed range of 0.3 to 0.7 and leads to a final $SoE$ of 0.48, near the initial value of 0.5, to minimize the cost in Equation (14), since it exploits the full knowledge of the complete driving cycle. The ECMS with the s parameter calibrated from a single loop frequently hits the $SoE$ limits, and only is able to keep its value within the required range by applying corrections to the s parameter. In any case, the ECMS shows how, since the beginning of the route, the strategy struggles to keep the $SoE$ above 0.3 and leads to a progressive deviation from the optimal fuel consumption obtained from DP. The result clearly shows that some calibration effort is needed to overcome this issue. Regarding the proposed strategy (rollout), results show the $SoE$ evolves well within the limits of 0.3 and 0.7, showing an evolution more similar to the DP and leading to a final $SoE$ of 0.47. Figure 9 shows the probability distribution of the difference between the optimal $SoE$ evolution obtained by DP and those obtained with the two evaluated methods (rollout and ECMS). It can be clearly noticed how despite both methods being online applicable and requiring the same information, i.e., the optimization of a single representative loop, the rollout algorithm provides better tracking of the optimal $SoE$ evolution. Due to the differences between optimal $SoE$ evolution and those obtained with the online-assessed capable strategies, at the end of the cycle, the ECMS shows an increase in fuel consumption of 11% with respect to the DP, while this figure is reduced to 1.9% by the proposed strategy based on rollout, as pointed out by the lower plot of Figure 8.

The fuel saving provided by the rollout strategy with respect to the standard ECMS is based on the operation of the ICE and batteries in more efficient conditions. Contour lines in Figure 10 show the efficiency levels in the ICE depending on the engine speed and demanded torque, while the color scale represents the observed frequency in the engine operating conditions obtained with ECMS (left), rollout (center) and DP (right). For instance, in the standard ECMS case, the ICE operates frequently outside the high-efficiency area, particularly in the area of middle load (with an efficiency around 40%) and low load (with an efficiency around 30%) due to the $SoE$ excursions towards 0.3 that force the engine to start in order to avoid battery depletion, while for the rollout case most operating conditions are placed in the high-efficiency region, and there are only a few operating points in the area of medium and low efficiency. In the same way that the rollout strategy leads to a similar evolution of $SoE$ as DP, the distribution of ICE operating points also approaches the DP solution.

On the basis of the precedent discussion, the one-step look-ahead rollout outperforms the standard ECMS. This is justified by the fact that although both strategies have the same calibration requirements, i.e., both need the optimization of a reference cycle, ECMS lumps all the information from the reference cycle in a single parameter s, while the rollout exploits all the cost-to-go from the reference cycle. In this sense, the control policy provided by the ECMS approach is based on a single constant without dependence (excepting the $SoE$ correction) of the system state. On the contrary, the rollout is richer and considers the impact of the system states ($SoE$ and $IC{E}_{on}$) and vehicle position in the control policy. Of course, this is performed at the expense of higher storage requirements, despite the dependence on the size of the problem, in the case at hand ${\mathcal{J}}^{*}{(x_j,s_i)}_{ref}$ has size 51 × 2 × 152, representing a discretization in the $SoE$ of 51 points between 0.3 and 0.7, considering two possible states for $IC{E}_{on}$ (on and off) and a 100 m discretization of the 15.1 km loop distance.

Regarding the applicability for control purposes, the standard ECMS and look-ahead rollout run faster than real time, and 875 and 409 times faster than real time, respectively, for the driving cycle of Figure 8 in a standard laptop (Intel(R) Core(TM) i7-8550U CPU @ 1.80 GHz 1.99 GHz with 8.00 GB RAM). The rollout algorithm requires more computation resources but is still far from the limit. Moreover, neither the rollout nor the ECMS require information about future conditions, so they have potential to be applied in control purposes. On the contrary, DP, despite running 390 times faster than real time, depends on a priori knowledge of the driving cycle, so cannot be used for control applications.

Although the proposed rollout algorithm performs better than the standard ECMS, there is a noticeable gap between its performance and DP. Expressions (16) and (20) used for DP and rollout, respectively, highlight the two main simplifications that are the reason of such performance differences:

• The integral term Equation (16) has a time interval ($\delta t$) of a single time-step (one-step look ahead).
• The optimal cost to go from future state ($x(t+\delta t)$) to the end (${\mathcal{J}}^{*}(x(t+\delta t),t+\delta t)$) is approximated by the cost-to-go computed in the reference loop.

The first simplification is important to allow online application since power demands beyond the next time-step usually cannot be assumed. However, with the advent of vehicle to vehicle (V2V) and vehicle to infrastructure (V2I) technologies, one can expect an accurate extension of the look-ahead horizon. To analyze the impact of having information about future driving conditions on the proposed control algorithm, Figure 11 shows the fuel consumption and the deviation from the optimal $SoE$ evolution of the rollout algorithm assuming perfect driving cycle knowledge in different horizons. One can observe that increasing the look-ahead horizon allows approaching the optimal $SoE$ evolution and fuel consumption, but there is still a gap that cannot be closed. For this reason, look-ahead horizons beyond 250 s do not show a substantial improvement in the fuel consumption. The reason for that is the second simplification hypothesis. On the one hand, the rollout uses a reference cycle as estimator of the future driving conditions, on the other hand, the cost-to-go of the reference cycle is computed by applying a terminal cost to the end of the loop, while in DP, this terminal cost is only applied at the end of the complete route. For this reason, even when increasing the look-ahead horizon and obtaining an accurate estimation of the future driving conditions, the rollout strategy does not allow to reach the DP performance in terms of fuel consumption. In this sense, the 1.9% increase in fuel consumption obtained with the one-step look-ahead rollout strategy can be progressively reduced if accurate information about driving conditions in a future horizon is available, but even with long horizons such as 500 s, an increase of 0.5% remains due to the cost-to-go approximation and the impact of imposing the terminal cost at the end of the loop.

7. Conclusions

This paper proposes a new energy management strategy for a hybrid electric urban bus. The proposed strategy takes profit from the route pattern of urban buses by using information from the vehicle story for energy management optimization. While this is an approach widely discussed in the literature, this paper proposes a different alternative: the strategies available in the literature are based on the estimation of future driving conditions from the vehicle story, however, the solution proposed in this paper is based on using the optimal solution of a reference driving cycle, previously recorded, to estimate a cost-to-go that can be embedded in an online optimization. The proposed strategy is evaluated with data from a bus route in Valencia consisting of a loop of 15.1 km in urban conditions that is covered between 9 and 10 times per day. The recorded data and the optimal evolution of the $SoE$ was analyzed using the Fourier transform in space domain, showing the existence of patterns that are repeated at the characteristic length of the loop. In the same way, the singular value decomposition of the vehicle speed–acceleration probability distributions in several loops shows that there is a close relation between the operating conditions reached by the bus in the different loops of the route. The analysis suggests that a single loop can be representative of the driving cycle assuming periodicity. Following this idea, the optimal solution of a single loop is computed offline by DP; the cost-to-go generated by this solution is stored and used as an approximation for the cost-to-go from the next time-step in the online application of the HJB equation in the actual driving cycle. The proposed strategy is compared with the theoretical optimal obtained by DP and the online applicable ECMS. Simulations results show that the proposed method is able to keep the $SoE$ within the range delimited by the EMS and reduces the fuel consumption increase in the ECMS from 11% to 1.9%. The main contributions of the paper can be summarized as follows:

• It was shown that there exist patterns in the different loops covered by an urban bus that can be used for EMS optimization.
• A novel EMS strategy able to exploit those patterns was proposed and simulations offer near optimum results, improving the fuel consumption obtained with other algorithms in the literature calibrated with the same information.
• The proposed strategy does not require the estimation of future driving conditions.

Regarding future works, authors can observe the following possible lines:

• Upgrade the objective function to include other important aspects to be addressed by the EMS, such as emissions, drivability or battery aging.
• Consider other systems and actuators in the vehicle such as the thermal management to address temperature limitations in the powertrain elements and the impact of temperature on their efficiency.
• Extend the energy management to other elements in the bus such as the HVAC, which can be an important energy consumer. A holistic approach to all the energy flows in the vehicle can potentially improve its efficiency.
• Include other relevant aspects in the description of the driving cycle, for example the mass of the vehicle that can change substantially in the route depending on the number of passengers. One can expect a similar periodicity in the number of passengers on the bus, and the optimization can explore this information.

All the previous aspects involve to some extent increasing the model complexity, but the algorithm presented in Section 5 can be generalized to consider them at the expense of computation complexity.