Hypervolume-Based Multi-Objective Optimization Method Applying Deep Reinforcement Learning to the Optimization of Turbine Blade Shape

Abstract

A multi-objective turbine shape optimization method based on deep reinforcement learning (DRL) is proposed. DRL-based optimization methods are useful for repeating optimization tasks that arise in applications such as the design of turbines and automotive parts. In conventional research, DRL is applied only to single-optimization tasks. In this study, a multi-objective optimization method using improvements in hypervolume is proposed. The proposed method is applied to a benchmark problem and a turbine optimization problem. It succeeded in efficiently solving the problems, and Pareto optimal solutions are obtained.

1. Introduction

Deep reinforcement learning (DRL) utilizes deep neural networks (DNNs) in reinforcement learning algorithms. DRL is utilized in various industrial fields, mainly for control problems such as robotics [1,2,3], path planning [4,5], and automobile control [6,7]. DRL is also utilized in mechanical design applications [8], including the designs of airfoil shapes [9,10,11], turbine blades [12], and flow controls [13,14], as well as involving topology optimization [15].

In mechanical optimization, designers must find solutions to multi-objective optimization problems. In multi-objective optimization, it is necessary to obtain Pareto solutions [16]. Genetic algorithms, including the multi-objective GA (MOGA) [17], non-dominated sorting GA (NSGA) [18], and NSGA-II [19], are among the major methods used to perform multi-objective optimization [20]. Moreover, reinforcement learning has recently been utilized in mechanical design applications because of its high generalization capability. A properly trained RL agent has good generalization capability [9,12,21]. Hence, once trained, an RL agent is applicable to slightly different optimization problems. However, multi-objective DRL (MODRL) methods for mechanical designs have not been extensively studied.

Therefore, this study focuses on MODRL for use in multi-objective optimization problems. Several MODRL methods have been proposed for use in the field of computer science [5,22,23]. The weighted-sum method (WSM) is often used [24,25,26]. The WSM takes the weighted sum of the objective functions and treats the problem as a single-objective optimization problem. However, in using the WSM to obtain the Pareto front, we have to solve multiple weighted-sum problems by changing the weights. The computational cost of this procedure is high, and a non-convex Pareto front cannot be obtained using this procedure. Another commonly used method is the $\epsilon$-constrained method [27,28,29], which uses only one primary objective function and uses the others as constraints.

The hypervolume, which is the volume of the hypercubes generated by the Pareto solutions, is often considered to obtain good Pareto solutions. Moreover, it is often used as both an indicator to measure the goodness of the obtained Pareto solutions and an objective function to be maximized [30,31,32,33]. Good Pareto solutions can be obtained by maximizing the hypervolumes.

In DRL, an extensive computational cost is incurred in the model training. Once trained, the agent can adapt to various problems, but if the objective function changes, then the agent must be retrained. Therefore, if DRL is coupled with the WSM or $\epsilon$-constrained method, then the DRL agent must be trained multiple times to change the weights or constraints. In such cases, the computational costs are extremely high. In contrast, if the hypervolume-based method is coupled with DRL, then only one training phase is required, as the objective function and constraints do not change. Therefore, in this study, we propose a novel MODRL method that couples the hypervolume method with DRL.

The remainder of this paper is organized as follows. In Section 2, we formulate the proposed method by using the hypervolume. In Section 3, we describe the application of the proposed method to a benchmark problem. In Section 4, we formulate and solve the turbine optimization problem by using the proposed method. Finally, in Section 5, we conclude the paper.

2. Hypervolume-Based Multi-Objective Deep Reinforcement Learning

2.1. Deep Reinforcement Learning for Turbine Blades

RL obtains the optimal strategy for a Markov decision process (MDP). An MDP is a stochastic process that consists of states $s_t$, actions $a_t$, and rewards $r_t$, where t is the time step. In MDP, the state $s_t$ and action $a_t$ are related only to those values of the previous time step $t-1$. They are not related to the values of the other steps.

The proximal policy optimization (PPO) [34] algorithm was used in this study. PPO uses an actor network and a critic network. The actor network performs actions, whereas the critic network estimates the values of the actions and evaluates the policy of the actor network.

Ref. [12] proposed a DRL-based turbine optimization method that uses computational fluid dynamics (CFD) as an environment. In this method, an agent takes actions to modify turbine shapes. In this study, we coupled this DRL-based optimization method [12] with the hypervolume method.

2.2. Hypervolumes of Pareto Solutions

In multi-objective optimization, the solutions are divided into non-dominated solutions (i.e., Pareto solutions) and dominated solutions. We assume that the number of objective functions is d and that all functions are minimized. Let ${\mathit{z}}^i\in {\Re }^d$$(i=1,2,\dots ,n)$ be a solution, where i is the suffix and n is the number of solutions. A solution ${\mathit{z}}^{*}\in {\Re }^d$ is the dominant solution if

$$\begin{array}{c}\hfill z_k^{*}>{\mathit{z}}_k^i,\forall k(1\le k\le d)\end{array}$$

holds. Pareto solutions are defined as non-dominated solutions.

Let ${\mathcal{P}}_t$ be a set of Pareto solutions at time step t and ${\mathcal{P}}_t=\{{\mathit{p}}_1,{\mathit{p}}_2,\dots ,{\mathit{p}}_n\}$, where n is the number of Pareto solutions. In the two-dimensional case, ${\mathit{p}}_i=\left(x_i,y_i\right)$. A hypercube $C\left({\mathit{p}}_i\right)$ of solution ${\mathit{p}}_i$ is defined as a hypercube with ${\mathit{p}}_i$ and reference point ${\mathit{p}}_0$ as its diagonal points. In the two-dimensional case, $C\left({\mathit{p}}_i\right)$ is defined as

$$\begin{array}{c}\hfill C\left({\mathit{p}}_i\right)=\left\{\left(x,y\right)\mid x_i\le x\le x_0,y_i\le y\le y_0\right\}.\end{array}$$

The hypervolume V of the Pareto solutions $\mathcal{P}$ is defined as the hypervolume of the union of sets $C\left({\mathit{p}}_i\right)$:

$$\begin{array}{c}\hfill V\left(\mathcal{P}\right)=∥{\cup }_{i}C\left({\mathit{p}}_i\right)∥,\end{array}$$

where $\parallel ·\parallel$ denotes the hypervolume.

2.3. Reward Function

The reward function is defined according to three patterns: having the CFD computation fail, obtaining dominated solutions, and successfully finding a Pareto solution. The agent receives penalties in the first two patterns and rewards in the final pattern. An overview of rewards and penalties is provided in Figure 1, where $C_1$, $C_2$, $C_3$, and $C_4$ are the constants and we let $C_1=100$, $C_2=10$, $C_3=20$, and $C_4=50$.

• Pattern 1: Failed CFD

If the CFD computation does not converge, a fixed penalty is assigned as a reward.

$$\begin{array}{c}\hfill r_t=-C_1.\end{array}$$

(1)

• Pattern 2: Dominated solutions

If the CFD computation converges but is a dominated solution, then the agent obtains a penalty, according to the distance between the solution and the Pareto front.

$$\begin{array}{c}{r}_t=\left\{\begin{array}{cc}log\left(C_2-d\right)-C_3,\hfill & \mathrm{if}d<C_2\hfill \\ C_4\hfill & \mathrm{if}d\ge C_2,\hfill \end{array}\right.\hfill \end{array}$$

(2)

where d is the distance between the solution and the Pareto front.

• Pattern 3: Pareto solutions

If the obtained solution is a Pareto solution, the agent obtains a reward, according to the improvement in the hypervolume. The improvement is defined as follows:

$$\begin{array}{c}\hfill r_t={C_1}^2{\left(V\left({\mathcal{P}}_t\right)-V\left({\mathcal{P}}_{t-1}\right)\right)}^2.\end{array}$$

(3)

2.4. Optimization Algorithm

The optimization algorithm consists of two phases: training and optimization. The agent is trained in the training phase and used in the optimization phase. An overview of the training phase is provided in Figure 2. Each case corresponds to a single-flow condition and consists of 10 episodes, each of which corresponds to one initial solution. A CFD analysis was conducted on the initial shape. Subsequently, the state is input to the agent, and the agent modifies its shape. However, CFD computations sometimes fail because of unsuitable shape modifications. For example, a shape sometimes contains a self-intersection. The termination condition of each episode is the number of shape modifications reaching 50 or the CFD computation failing to converge. The trained agent was used in the optimization phase.

3. Benchmark Problem

3.1. Problem Definition

To demonstrate the proposed method, we solve the benchmark optimization problem defined in [35]. The benchmark problem is expressed as

$$\begin{array}{cccc}& \min .\hfill & \hfill & f_1=x_1,\hfill \end{array}$$

(4)

$$\begin{array}{cccc}& \min .\hfill & \hfill & f_2=gh,\hfill \end{array}$$

(5)

$$\begin{array}{cccc}& \mathrm{subject} \mathrm{to}\hfill & & g=1+10{\displaystyle \frac{{\sum }_{i=2}^N{x}_i}{N-1}},\hfill \end{array}$$

(6)

$$\begin{array}{cccc}& & & h=\left\{\begin{array}{cc}1-{\left({\displaystyle \frac{f_1}{g}}\right)}^{\alpha },\hfill & \mathrm{if}{f}_1\le g,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(7)

$$\begin{array}{cccc}& & & x_i\in \left[-1,1\right],i=1,\dots ,N.\hfill \end{array}$$

(8)

The Pareto front of the problem is analytically solved and written as

$$\begin{array}{cc}\hfill & f_1=x_1,\hfill \\ \hfill & f_2=1-{\left(f_1\right)}^{\alpha }.\hfill \end{array}$$

The Pareto front is upward concave if $\alpha >1$, and we use $\alpha =2$. If we use the reference point ${\mathit{p}}_0=(1,1)$, then the hypervolume is $V\left(\mathcal{P}\right)=1/3$.

3.2. Model Architecture

The state ${\mathit{s}}_t$ of the model is defined as follows:

$$\begin{array}{c}\hfill {\mathit{s}}_t=\left(f_1\left({\mathit{x}}_t\right),f_2\left({\mathit{x}}_t\right),v({\mathit{x}}_t,{\mathcal{P}}_t)\right),\end{array}$$

where $\mathit{v}({\mathit{x}}_t,{\mathcal{P}}_t)\in {\Re }^2$ denotes the vector from the current solution ${\mathit{x}}_t$ to the point nearest to the Pareto front. Therefore, the state vector is a four-dimensional vector. The action ${\mathit{a}}_t\in {\Re }^2$ is a vector consisting of a continuous variable within the range of $\left[-0.1,0.1\right]$ for each design variable $x_1$ and $x_2$.

The architectures of the actor and critic networks are presented in Figure 3. The dropout rate was $0.2$. The weights were initialized using HeNormal [36] for the hidden layers and GlorotNormal [37] for the last layer.

Table 1. Hyperparameters of the benchmark problem.

Hyperparameter	Value	Hyperparameter	Value
Episodes	1000	Batch size	512
Learning rate	${10}^{-3}$ to ${10}^{-5}$	Epochs	10
Discount rate $\gamma$	0.99	GAE discount rate $\lambda$	0.95
Dropout ratio	$0.2$	Optimization algorithm	Adam

lists the hyperparameters. Default settings were used for hyperparameters except that the learning rate was selected to achieve convergence.

The reward function is the same as that defined in Section 2.3, except for that of “Pattern 1”. Since CFD computation is not included in the benchmark problem, “Pattern 1: Failed CFD” (Equation (1)) is not considered.

3.3. Results

We trained the agent for 1000 episodes, as shown in Table 1. In each episode, the total hypervolume obtained by the agent was calculated, as shown in Figure 4. The moving average of the 10 episodes is also shown in red in Figure 4. The hypervolume after 1000 episodes of training was $0.324$, which corresponded to $97.2\%$ of the analytical solution ($1/3$). Moreover, Figure 4 shows that the score and hypervolume converged within 100 episodes.

The obtained solutions are presented in Figure 5. Without training (episode 0), the agent moved randomly, and Pareto solutions could not be obtained. In episode 1000, the agent succeeded in obtaining Pareto solutions. Although the analytical solution of the Pareto front was upwardly concave, the agent succeeded in obtaining a Pareto front.

4. Turbine Optimization Problem

4.1. Problem Definition

We optimized the turbine shape for cascaded turbine blades. We assumed repeated optimization tasks for the problem [12], which is a series of optimization problems whose flow conditions were slightly different from each other. Such optimization tasks are often present in industrial fields. In this study, we assume the flow conditions listed in

Table 2. Flow conditions of repeated optimization tasks of turbine blades.

Hyperparameter	Value
Inlet flow angle	0.3–0.5 [rad]
Inlet Mach number	0.2–0.35 [Mach]
Pitch	1.2–1.5 [rad]

. We randomly choose one flow condition from Table 2 and use it in one case. The objective function is to minimize the pressure loss and maximize the torque. The pressure loss is defined as

$$\begin{array}{c}\hfill P_{\mathrm{loss}}={\displaystyle \frac{P_{{\mathrm{T}}_{\mathrm{in}}}-P_{{\mathrm{T}}_{\mathrm{ex}}}}{P_{{\mathrm{T}}_{\mathrm{ex}}}-P_{{\mathrm{S}}_{\mathrm{ex}}}}},\end{array}$$

where $P_{{\mathrm{T}}_{\mathrm{in}}}$ and $P_{{\mathrm{T}}_{\mathrm{ex}}}$ are the total pressures at the inlet and outlet boundaries, respectively, and $P_{{\mathrm{S}}_{\mathrm{ex}}}$ is the static pressure at the outlet boundary. The torque generated by the turbine blade is denoted as F. The optimization problem is formulated as follows:

$$\begin{array}{cccc}& \min .\hfill & \hfill & P_{\mathrm{loss}},\hfill \end{array}$$

(9)

$$\begin{array}{cccc}& \min .\hfill & \hfill & -F,\hfill \end{array}$$

(10)

$$\begin{array}{cccc}& \mathrm{subject} \mathrm{to}\hfill & & |{\varphi }_{\mathrm{out}}-{\varphi }_{\mathrm{target}}|\le 0.5\left[\mathrm{deg}\right].\hfill \end{array}$$

(11)

The turbine blade shape is defined by adding thickness to the camber line. That is, the upper and lower sides of the blade are defined as $y=y_c+y_t$ and $y=y_c-y_t$, respectively. The camber line is defined as follows

$$\begin{array}{cc}\hfill & y_c=\left\{\begin{array}{cc}{\displaystyle \frac{m}{p^2}}\left(2px-x^2\right),\hfill & [0\le x\le p],\hfill \\ {\displaystyle \frac{m}{{\left(1-p\right)}^2}}\left[\left(1-2p\right)+2px-x^2\right],\hfill & [p<x\le 1],\hfill \end{array}\right.\hfill \end{array}$$

where m and p denote the maximum camber and its position, respectively. Subsequently, the shape was rotated according to the origin and scaled such that the chord length became 1 to set a positive stagger angle. The thickness $y_t$ is defined as

$$\begin{array}{c}\hfill y_t={\displaystyle \frac{t}{20}}(0.29690x^{{\displaystyle \frac{1}{2}}}-0.12600x-0.{35160}^2+0.28430x^3-0.10150x^4),\end{array}$$

where t is the maximum thickness ratio.

The camber line and thickness distribution are then approximated using B-spline curves. The B-spline curves and control points are presented in Figure 6. Eventually, the blade shape is represented by $y={y_c}^{\mathrm{B}}+{y_t}^{\mathrm{B}}$ and $y={y_c}^{\mathrm{B}}-{y_t}^{\mathrm{B}}$, where ${y_c}^{\mathrm{B}}$ and ${y_t}^{\mathrm{B}}$ are the B-spline curves of the camber line and thickness distribution, respectively. The control points of the B-spline curves are used as the design variables.

4.2. CFD Computation

The CFD computation is carried out using the MISES software suite, which is a collection of programs for cascade analysis and design [38]. An example computational grid is presented in Figure 7. The pressure distribution is obtained by using MISES. The pressure values are sampled from the pressure distributions on the pressure and suction sides, which are represented by ${\mathit{p}}_{\mathrm{p}}$ and ${\mathit{p}}_{\mathrm{s}}$, respectively. In total, 120 samples are collected from each side. The coordinates of the sampled points are represented by $\mathit{s}$, which are the coordinates of the streamline. The pressure values ${\mathit{p}}_{\mathrm{p}}$ and ${\mathit{p}}_{\mathrm{s}}$ are normalized using the peak pressure at the leading edge $p_0$. Hence, ${\mathit{p}}_{\mathrm{p}}/p_0$ and ${\mathit{p}}_{\mathrm{s}}/p_0$ are used as state variables, as explained in Section 4.3. Figure 8 shows the pressure distributions on the pressure and suction side surfaces.

4.3. Model Architecture

The state variable ${\mathit{s}}_t$ is constructed using the results of the CFD analysis. The elements of ${\mathit{s}}_t$ are as follows. In total, the number of dimensions of ${\mathit{s}}_t$ was 1093.

1.

Information of each case:

• Inlet Mach number,
• Inlet flow angle,
• Pitch,
• Target outlet flow angle.

2.

Geometric information:

• Coordinates of the camber line $({\mathit{x}}_c,{\mathit{y}}_c)$,
• Coordinates of the thickness distribution $({\mathit{x}}_t,{\mathit{y}}_t)$,
• Metal angle of the leading edge.

3.

Information of the flow field:

• Pressure distribution on the pressure side ${\mathit{p}}_p/p_0$,
• Pressure distribution on the suction side ${\mathit{p}}_s/p_0$,
• Outlet flow angle.

4.

Information of the Pareto front:

• Vector from the current point to the nearest Pareto front.

If the CFD computation failed to converge, information on the flow field could not be obtained. In this case, the following values were used: $\mathit{s}=\mathbf{0}$, ${\mathit{p}}_{\mathrm{p}}/p_0={\mathit{s}}_{\mathrm{p}}/p_0=\mathbf{0}$, and an outlet flow angle of 20 [deg].

The action ${\mathit{a}}_t$ is a vector in ${\Re }^{17}$ that corresponds to the

• Difference between the x and y coordinates of the four control points of a camber line,
• Difference between the x and y coordinates of the four control points of a thickness distribution,
• Difference of the stagger angle.

The range of ${\mathit{a}}_t$ was $[-1,1]$, and ${\mathit{a}}_t$ was scaled for each design variable. The scale factors for the camber line were $0.003$, $0.001$ for the thickness distribution, and $0.2$ for the stagger angle. As explained in Section 4.1, the turbine shape was generated using the control points and then rotated according to the stagger angle. The shape was then scaled such that the chord line became 1.

Figure 9 illustrates the DNN architectures of the actor and critic networks.

4.4. Results

The agent was optimized after training for 1000 episodes. One case was randomly chosen, and 10 initial solutions were used. The results of the objective values are shown in Figure 10, which indicates that the agent successfully finds Pareto solutions. The hypervolume history is also shown in Figure 11. When a new initial solution is set, the agent succeeds in finding new Pareto solutions. Hence, the hypervolume improves drastically.

The Pareto solutions obtained depend on the initial solution. Therefore, it is necessary to use multiple initial solutions.

5. Conclusions

A novel multi-objective optimization method using the hypervolume of the Pareto front is proposed for a turbine optimization problem. This method uses the improvement in hypervolumes as a reward. The proposed method was validated using a benchmark problem and then applied to a turbine optimization problem.

In the hypervolume-based method, the objective function and constraint conditions are not changed. Hence, only one DRL agent is required to obtain Pareto solutions. If we use the WSM or $\epsilon$-constrained method, then different DRL agents must be trained when the weights or constraints are changed. This generalization capability reduces the computational cost. Moreover, because DRL has a rich generalization capability, it can be used for multiple tasks.