Integrating Risk-Averse and Constrained Reinforcement Learning for Robust Decision-Making in High-Stakes Scenarios

Abstract

This paper considers a risk-averse Markov decision process (MDP) with non-risk constraints as a dynamic optimization framework to ensure robustness against unfavorable outcomes in high-stakes sequential decision-making situations such as disaster response. In this regard, strong duality is proved while making no assumptions on the problem’s convexity. This is necessary for some real-world issues, e.g., in the case of deprivation costs in the context of disaster relief, where convexity cannot be ensured. Our theoretical results imply that the problem can be exactly solved in a dual domain where it becomes convex. Based on our duality results, an augmented Lagrangian-based constraint handling mechanism is also developed for risk-averse reinforcement learning algorithms. The mechanism is proved to be theoretically convergent. Finally, we have also empirically established the convergence of the mechanism using a multi-stage disaster response relief allocation problem while using a fixed negative reward scheme as a benchmark.

1. Introduction

In certain sequential decision-making situations, e.g., disaster response or highly disruption-prone supply chains [1,2,3,4], decision outcomes are not only highly uncertain because of the gradual revelation of uncertainty, but their worst effects cannot be tolerated. To be robust against these worst-case outcomes, we need to use constraints to eliminate those decisions from the feasible solution space that may lead to them, while for additional robustness, we must use a dynamic risk-averse objective as a representation of decision makers’ risk-averseness (a behavioral consideration). In the literature, constraints are specifically used for robust decision-making, especially in the safe reinforcement learning domain [5,6,7], while the use of a risk-averse objective function is also prevalent for decision robustness [8,9,10,11,12]. Recently, the simultaneous use of constraints and risk aversion is also regarded as an effective approach for robust decision-making [1,13,14]. This leads us towards multi-stage risk-averse constrained optimization models. Studies have used stochastic dual dynamic programming (SDDP) for the solution of these kinds of models [9,15]. However, SDDP assumes the problem’s convexity and a finite number of random scenarios for its optimal convergence, which is restrictive as many real-world problems are non-convex, e.g., humanitarian relief allocation with deprivation costs [16], requiring multi-stage solution approaches which can handle non-convexities. Although [17] did not presume the problem’s convexity while using disciplined convex–concave programming for multi-stage risk-averse constrained optimization, optimal convergence was not established either theoretically or empirically. In this regard, as a model-free machine learning-based framework [18,19], reinforcement learning (RL) can be a better solution approach to non-convex problems. It has shown state-of-the-art performance in various real-world domains, e.g., cloud computing [20], finance [21,22], transportation [23,24], energy [25,26], inventory management [27,28], manufacturing or production scheduling [29], and routing [30]. However, in all of the above applications, robust and high-stack decision-making is not required, and thus expectation-based unconstrained RL is used. As we are considering high-stakes decision-making under uncertainty, risk-averse RL algorithms with constraint-handling abilities are required.

In this context, when it comes to constrained RL, besides other approaches such as mapping any action chosen by the policy to a constraint satisfying one which may need domain knowledge for its implementation [31], the most common and, so far, the best approach for solving constrained MDP is via Lagrangian dual (or augmented Lagrangian dual) optimization [32,33]. In this method, a constrained multi-stage optimization problem is converted into a min–max unconstrained problem, which is then solved by traditional RL algorithms. Besides its advantages, such as allowing non-convex constraints and not requiring any prior knowledge, it has theoretical guarantees for convergence as well. In this regard, it was proven that constrained MDPs have zero duality gap with no assumption on the problem’s convexity [34]. This result explains the performance of these primal-dual methods, such as primal-dual policy optimization (PPO) [35] and Lyapunove-based policy learning [36]. However, in all these methods, cumulative constraints are considered, the satisfaction of which does not mean the satisfaction of instantaneous constraints [37]. This is why an augmented Lagrangian-based mechanism for constrained RL has recently been developed for instantaneously constrained MDPs by [37], which extended the theoretical results from [34]. However, all of the above-constrained RL developments are in the realm of expectation-based RL [12,38], while no research has been performed on handling constraints in risk-averse RL. The reason behind this paucity of research is that the risk-averse RL is in its embryonic stage itself [12], adding constraints to it is the next stage of investigation, which no one has entered yet.

Research Contributions

Therefore, we have initiated the new research domain of constrained risk-averse RL to ensure optimal convergence of multi-stage risk-averse constrained mathematical programs, which can be easily reformulated into their version of a stochastic decision-making process, namely risk-averse Markov decision processes with constraints [39,40]. In doing so, we will not make any assumption on the problem’s convexity. However, some not-so-restrictive assumptions are made on the constraints and risk-averse objective. All the constraints used in this research are non-risk constraints. For the risk-averse MDPs with cumulative expectation-based constraints, we will assume those conditional spectral risk measures that satisfy conditions related to the derivative and integral of their risk-aversion function, given on page 7–9. Furthermore, for the incorporation of deterministic instantaneous constraints, the translation invariance property of coherent risk measures is made as one additional assumption on conditional spectral risk measures. This development will help the solution of real-world problems in many domains which are not yet considered solvable. For example, in humanitarian logistics, deprivation cost is an important metric for measuring human suffering due to deprivation from necessities. However, many researchers do not consider this cost to make the models tractable [41]. Despite the need for accurate and robust decision-making, no model in the disaster management literature has yet combined constrained optimization, risk aversion, and deprivation costs. This is due to the computational intractability issue which is solved by this research [42].

The main contributions of our research are summarized as follows:

• To prove the strong duality of spectral risk-averse MDPs with expectation-based cumulative and/or deterministic instantaneous constraints without making any assumption on the problem’s convexity.
• To propose a constraint handling mechanism for risk-averse reinforcement learning based on the strong duality results.
• To theoretically and empirically establish the convergence of the proposed constraint handling mechanism.

For ease of understanding, the graphical representation of this research is shown in Figure 1. Subsequently, we will first specify the preliminaries and mathematical notations in Section 2, which will be crucial for understanding our theoretical results in Section 3 and Section 4 for risk-averse MDPs with cumulative and instantaneous constraints, respectively. Based on the results, we will develop a constraint-handling mechanism in Section 5 where we will also prove the convergence of this newly developed approach. After this, in Section 6, we will empirically validate the convergence of the proposed mechanism using a multi-stage risk-averse constrained optimization problem. Findings will be discussed in Section 7, and finally, we will conclude in Section 8.

2. Preliminaries

Let $T$ be the index set of time periods(with indices starting from one) in a finite or infinite horizon MDP, then the key elements of the MDP are as follows:

2.1. Action Space

$a_t\in \mathcal{A}\subseteq {\mathbb{R}}^n \forall t\in T$ is the feasible action RL agent can take in the period $t\in T$ which belongs to the action space $\mathcal{A}\subseteq {\mathbb{R}}^n$ with $n$ being the action dimension or the number of decision or control variables (continuous or discrete). It is assumed to be compact.

2.2. State Space

$s_t\in \mathcal{S}\subseteq {\mathbb{R}}^m \forall t\in T\cup \left\{\left|T\right|+1\right\}$ is the state belonging to the state space $\mathcal{S}\subseteq {\mathbb{R}}^m$ with $m$ being the state dimension or the number of state variables (continuous or discrete). Similar to action space, it is also assumed to be compact. The state variables specify the state of the MDP in a particular period.

2.3. Policy Space

Here, $\pi \in \mathcal{P}$ is the feasible policy in the arbitrary policy space $\mathcal{P}$. $\mathcal{P}$ forms a measure space $\left(\mathcal{A}\subseteq {\mathbb{R}}^n,\mathcal{B}\left(\mathcal{A}\subseteq {\mathbb{R}}^n\right),\mathcal{P}\right)$ with $\mathcal{B}\left(\mathcal{A}\subseteq {\mathbb{R}}^n\right)$ being a collection of Boral sets of the action space $A\in {\mathbb{R}}^n$.

2.4. Reward and Constraint Functions

There are cost functions $g_{it}=g_i\left(s_t\in \mathcal{S},a_t\in \mathcal{A}\right), \forall i\in P\cup \left\{0\right\},t\in T$ producing cost signals which are received by the RL agent in each period. These are the mappings from the current time’s state $s_t\in \mathcal{S}\subseteq {\mathbb{R}}^m$ and the current time’s action $a_t\in A\subseteq {\mathbb{R}}^n$ to the real number line $\mathbb{R}$. These costs are considered to be random variables either due to stochastic policy $\pi \in \mathcal{P}$ and/or uncertainty in the MDP environment dynamics. In this regard, $P$ is the index set (with indices starting from one) of these cumulative or instantaneous constraint cost functions in a particular period $t\in T$, and the subscript “$0$” in (2) is regarded as the index of the reward function.

2.5. Discount Factor

$\gamma \in \left(0,1\right)$ is the discount factor which is assumed to be in the open interval $\left(0,1\right)$. This specifies the importance of the future rewards in the value of the current period’s state. The greater its value, the more important are the future rewards for the current state’s value.

2.6. Reference Equations

Before diving into the theory, we will first mention three Equations (1)–(3), which will be referred to further in this paper.

$$V_i\left(\pi \in \mathcal{P}\right)=\mathbb{E}\left[{\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{it}\right)}\right], \forall i\in P$$

(1)

$$M_0\left(\pi \in \mathcal{P}\right)=\mathbb{M}\left[g_{01}+\gamma \mathbb{M}\left[g_{02}+\dots +\gamma \mathbb{M}\left[g_{0\left|T\right|}\right]\right]\right]=\mathbb{M}\left[g_{0,1:\left|T\right|}\right]$$

(2)

where $\mathbb{E}\left[.\right]$ and $\mathbb{M}\left[.\right]$ represents expectation and conditional (conditioned on the realized trajectory up to the current state in an MDP) spectral risk measure operators, respectively, and the corresponding two equations, Equations (1) and (2), are the cumulative expectation and dynamic risk measure functions, respectively. In both Functions (1) and (2), periodic costs $g_{it}=g_i\left(s_t\in \mathcal{S},a_t\in \mathcal{A}\right), \forall i\in P,t\in T$ are used, which are discussed above in Section 2.4.

Another compact notation for the dual risk-averse objective we will use in Theorem 2 is as follows:

$$\mathbb{M}\left[g_{0:\left|P\right|,1:\left|T\right|}\right]=\mathbb{M}\left[g_{01}+{\displaystyle \sum _{i\in P}\left({\lambda }_{i1}{g}_{i1}\right)}+\gamma \mathbb{M}\left[g_{02}+{\displaystyle \sum _{i\in P}\left({\lambda }_{i2}{g}_{i2}\right)}+\dots +\gamma \mathbb{M}\left[g_{0\left|T\right|}+{\displaystyle \sum _{i\in P}\left({\lambda }_{i\left|T\right|}{g}_{i\left|T\right|}\right)}\right]\right]\right]$$

(3)

where, corresponding to every instantaneous constraint cost function $g_{it}, \forall i\in P,t\in T$ and ${\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T$ are the components of the Lagrangian vector ${\lambda }^{\prime }$ for the dual of “Problem 2”, referred to as “Dual 2” in Section “Strong Duality for “Problem 2””, on page 13.

Lastly, we defined $\mathcal{M}\left(\mathcal{S},\mathcal{A}\right)$ as the set of all measures defined onset of ordered pairs of states and actions $\left(\mathcal{S}\times \mathcal{A}\right)$, and $\mathcal{T}$ as occupation measures’ set originated from feasible policy space $\mathcal{P}$ as

$$\mathcal{T}=\left\{\rho \left(s\in \mathcal{S},a\in \mathcal{A}\right)\in \mathcal{M}\left(\mathcal{S},\mathcal{A}\right)|\rho \left(s\in \mathcal{S},a\in \mathcal{A}\right)=\left(1-\gamma \right){\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{p}_{\pi }\left(s_t=s\in \mathcal{S},a_t=a\in \mathcal{A}\right)\right)}\right\}$$

(4)

where $p_{\pi }\left(s_t=s\in \mathcal{S},a_t=a\in \mathcal{A}\right)$ is given as (A4) in Appendix A.

To initiate the field of constrained risk-averse RL, we will prove the following:

“Risk-averse MDPs with dynamic spectral risk objective and expectation-based cumulative constraints have zero duality gap in generic policy space $\mathcal{P}$ (as Theorem 1, in Section “Strong Duality for “Problem 1””, on page 6). Based on this Theorem 1, risk-averse MDPs with deterministic instantaneous constraints also have zero duality gap (as Theorem 2, in Section “Strong Duality for “Problem 2””, on page 12) without any assumption on the problem’s convexity.”

These results imply that these two classes of problems can be solved perfectly in the domain of dual variables where the space becomes convex (in terms of dual variables).

3. Risk-Averse Markov Decision Processes with Cumulative Constraints

In the literature [17], constrained risk-averse MDP refers to the MDP where both the objective function and constraints are risk-averse. However, we will use a varied version of it in which risk constraints [17] are substituted with cumulative expectation constraint(s) (9), which is the same as the cumulative constraints in constrained MDP [34]. The mathematical program for this varied version is as follows, which we will refer to as “Problem 1”:

$$\underset{\pi \in \mathcal{P}}{max}{M}_0\left(\pi \in \mathcal{P}\right)$$

(5)

Subject to the following:

$$s_{t+1}~p_t\left(\mathcal{S}\subseteq {\mathbb{R}}^m|s_t\in \mathcal{S}\subseteq {\mathbb{R}}^m,a_t\in \mathcal{A}\subseteq {\mathbb{R}}^m\right), \forall t\in T$$

(6)

$$a_t~\pi \in \mathcal{P}, \forall t\in T$$

(7)

$$s_1~p_0\left(\mathcal{S}\subseteq {\mathbb{R}}^m\right)$$

(8)

$$V_i\left(\pi \in \mathcal{P}\right)\le 0, \forall i\in P$$

(9)

where in “Problem 1” above, (5)–(8) are nested risk-averse objective, transition function set, policy inference, and initial state being sampled from its probability distribution, respectively.

Strong Duality for “Problem 1”

Let $\lambda$ be the Lagrangian vector, with its components ${\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P$ as Lagrangian multipliers corresponding to all the cumulative constraints (9) in “Problem 1” above. Then, the following (10) is the Lagrangian function:

$$L\left(\pi \in \mathcal{P},\left\{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P\right\}\right)=M_0\left(\pi \in P\right)+{\displaystyle \sum _{i\in P}\left({\lambda }_i\left(V_i\left(\pi \in \mathcal{P}\right)\right)\right)}$$

(10)

With fixed Lagrangian multipliers ${\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P$, if the Lagrangian function (10) is maximized over the policy space $\mathcal{P}$, then it will become the dual function, which forms a constrained optimization problem with (6)–(8). The solution of it provides an upper bound (for a maximization, or lower bound for a minimization problem) on the optimal value of “Problem 1” (primal problem).

In addition to optimizing (maximizing) policy space $\mathcal{P}$, if we also optimize (minimize) it over its Lagrangian multipliers ${\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P$ (dual variables) as in (11) below, the resulting min–max optimization problem will be the dual problem of “Problem 1”, which we will refer to later as “Dual 1”. Upon its solution, we will have the tightest (best) upper bound on the optimal solution of “Problem 1” [17]:

$$D^{*}=\underset{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P}{min}\underset{\pi \in \mathcal{P}}{max}L\left(\pi \in \mathcal{P},\left\{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P\right\}\right)$$

(11)

Subject to the following: (6)–(8).

In general, solving “Dual 1” does not result in the solution of “Problem 1” as there may not be any relation between the optimal values of both problems. However, if strong duality for “Problem 1” holds, i.e., the duality gap $\Delta$ between “Problem 1” (primal) and “Dual 1” (dual) is zero, then the optimal value of “Problem 1” $P^{*}$ will be equal to that of the “Dual 1” $D^{*}$, i.e., $P^{*}=D^{*}$. In Theorem 1 below, we will prove just that (with certain assumptions on the risk-aversion function in (5) given in Theorem 1’s proof)for both finite and infinite planning horizons with a discount factor $\gamma \in \left(0,1\right)$ using perturbation theory.

We define the perturbation function associated with “Problem 1” for any value of perturbation components ${\xi }_i, \forall i\in P$ of the perturbation vector $\xi$ as follows, which we will refer to as “The Perturbation function”:

$$P\left(\xi \right)\triangleq \underset{\pi \in \mathcal{P}}{max}{M}_0\left(\pi \in \mathcal{P}\right)$$

(12)

Subject to the following: (6)–(8)

$$V_i\left(\pi \in \mathcal{P}\right)\le {\xi }_i, \forall i\in P$$

(13)

Proposition 1. 

If we assume the Slater’s condition for “Problem 1”, and concavity of “The Perturbation function”, then strong duality will hold for “Problem 1”.

Theorem 1. 

If the costs $g_{it}, \forall i\in P\cup \left\{0\right\},t\in T$are bounded with the satisfaction of Slater’s condition for “Problem 1”, then strong duality holds for “Problem 1”.

The above theorem will be proved by connecting the strong duality of “Problem 1” with the convexity of “The Perturbation function” as follows in “Proposition 1” and further.

Proof. 

“Proposition 1” has been proven by [43] in Corollary 30.2.2, and as Slater’s condition holds by the assumption of Theorem 1, the only thing to prove here is the concavity of “The Perturbation function”, i.e., for any two perturbation vectors ${\xi }^1,{\xi }^2\in {\mathbb{R}}^{\left|P\right|}$ and $\mu \in \left(0,1\right)$, the following holds

$$P\left(\mu {\xi }^1+\left(1-\mu \right){\xi }^2\right)\ge \mu P\left({\xi }^1\right)+\left(1-\mu \right)P\left({\xi }^2\right), \forall \mu \in \left(0,1\right)$$

(14)

In the case when either perturbation ${\xi }^1,{\xi }^2\in {\mathbb{R}}^{\left|P\right|}$ makes “The Perturbation function” infeasible, then either $P\left({\xi }^1\right)=-\infty$ or $P\left({\xi }^2\right)=-\infty$ and, consequently, “The Perturbation function” is trivially concave. However, if the problem remains feasible for these perturbations ${\xi }^1,{\xi }^2\in {\mathbb{R}}^{\left|P\right|}$, then these perturbations have their respective optimal policies, ${\pi }_1\in \mathcal{P}$ and ${\pi }_2\in \mathcal{P}$, respectively, which can achieve values of respective perturbation functions such that $P\left({\xi }^1\right)=\underset{\pi \in \mathcal{P}}{max}{M}_0\left(\pi \in \mathcal{P}\right)=M_0\left({\pi }_1\in \mathcal{P}\right)$ is subject to $V_i\left({\pi }_1\in \mathcal{P}\right)\le {\xi }_i^1, \forall i=P$ and (6)–(8) and $P\left({\xi }^2\right)=\underset{\pi \in \mathcal{P}}{max}{M}_0\left(\pi \in \mathcal{P}\right)=M_0\left({\pi }_2\in \mathcal{P}\right)$ is subject to $V_i\left({\pi }_2\in \mathcal{P}\right)\le {\xi }_i^2, \forall i=P$ and (6)–(8).

Now, for proving the concavity of “The Perturbation function”, it is sufficient to show that a policy ${\pi }_{\mu }\in \mathcal{P}, \forall \mu \in \left(0,1\right)$ will exist which will satisfy $V_i\left({\pi }_{\mu }\in \mathcal{P}\right)\le \mu {\xi }_i^1+\left(1-\mu \right){\xi }_i^2, \forall i\in P,\mu \in \left(0,1\right)$, and $M_0\left({\pi }_{\mu }\in \mathcal{P}\right)=\mu M_0\left({\pi }_1\in \mathcal{P}\right)+\left(1-\mu \right)M_0\left({\pi }_2\in \mathcal{P}\right), \forall \mu \in \left(0,1\right)$ holds, so the policy ${\pi }_{\mu }\in \mathcal{P}, \forall \mu \in \left(0,1\right)$ is not only feasible but (15) naturally follows (which will result in the implication of (14)):

$$P\left(\mu {\xi }^1+\left(1-\mu \right){\xi }^2\right)\ge M_0\left({\pi }_{\mu }\in \mathcal{P}\right)=\mu M_0\left({\pi }_1\in \mathcal{P}\right)+\left(1-\mu \right)M_0\left({\pi }_2\in \mathcal{P}\right)=\mu P\left({\xi }^1\right)+\left(1-\mu \right)P\left({\xi }^2\right), \forall \mu \in \left(0,1\right)$$

(15)

So, we will prove the existence of such a policy ${\pi }_{\mu }\in \mathcal{P}, \forall \mu \in \left(0,1\right)$ for “Problem 1” to prove its strong duality.

• Expectation-Based Constraints

The manipulations of expectation-based constraints (9) we followed in this theorem are from [34]. However, due to notational differences, we have included all of these manipulations in terms of our notation in Appendix A for the readers’ ease of understanding.

• Risk-Averse Objective

Now, for the spectral risk-averse objective, we can rewrite $M_0\left(\pi \in P\right)$ in (5) as follows:

$$M_0\left(\pi \in \mathcal{P}\right)={\displaystyle \underset{{\left(\mathcal{S}\times \mathcal{A}\right)}^{\left|T\right|}}{\int }\left(\phi \left(F\left(R^{ds}\left(d^{trj}\in D^{trj}\right)\right)\right){\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{0t}\right)}{p}_{\pi }\left(d^{trj}\in D^{trj}\right)\right)}d{s}_{1}d{a}_{1}d{s}_{2}d{a}_2\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}$$

(16)

In (16) above, $R^{ds}\left(d^{trj}\in D^{trj}\right)={\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{0t}\right)}$, which is the mapping from a trajectory $d^{trj}\in D^{trj}$ to its discounted sum of rewards. $F\left(.\right)$ is the cumulative probability distribution function that maps the discounted sum of rewards ${\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{0t}\right)}$ (associated with a certain trajectory $d^{trj}\in D^{trj}$) to the closed interval $\left[0,1\right]$. $\phi \left(.\right)$ is the risk aversion function (in a spectral risk measure, which depends on the decision maker’s risk-aversion) mapping the output of $F\left(.\right)$ to the interval $\left[0,1\right]$. $\phi \left(.\right)$ and, its single variable and cross derivatives of all orders (with respect to any state and decision variables) is assumed to exist and be piecewise $C^{\infty }$ function in terms of all state and decision variables. In the end, $p_{\pi }\left(d^{trj}\in D^{trj}\right)$ is the probability of a particular trajectory $d^{trj}\in D^{trj}$, given a policy $\pi \in \mathcal{P}$ and $\left|T\right|$ being the cardinality of the set $T$ (number of periods in the planning horizon).

Since the cost functions $g_{0t}$ are bounded (by the assumption of Theorem 1), the dominance convergence theorem (DCT) will hold, allowing us to reverse the order of integration and summation in (16). With the use of conditional probabilities and Markov’s property of the MDPs, we can rewrite (16) above as follows:

$$M_0\left(\pi \in \mathcal{P}\right)={\displaystyle \sum _{t\in T}\left(\begin{array}{c}{\gamma }^{t-1}{\displaystyle \underset{{\left(\mathcal{S}\times \mathcal{A}\right)}^{\left|T\right|}}{\int }\left(\phi \left(F\left(R^{ds}\left(d^{trj}\in D^{trj}\right)\right)\right)g_{0t}{\displaystyle \prod _{u\in T\backslash \left\{1\right\}}\left(p\left(s_u|s_{u-1},a_{u-1}\right)p_{\pi }\left(a_u|s_u\right)\right)p_0\left(s_1\right)p_{\pi }\left(a_1|s_1\right)}\right)}\\ d{s}_{1}d{a}_{1}d{s}_{2}d{a}_2\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\end{array}\right)}$$

(17)

In (17), $p\left(s_u|s_{u-1},a_{u-1}\right)$ represents the probability of state $s_u$ given the previous state $s_{u-1}$ and action $a_{u-1}$. $p_{\pi }\left(a_u|s_u\right)$ is the probability of the action $a_u$ given $s_u$ and $\pi \in \mathcal{P}$ are current state and policy, respectively, and $p_0\left(s_1\right)$ is the probability of the initial state $s_1$.

Now let

$$f_1=\phi \left(F\left(R^{ds}\left(d^{trj}\in D^{trj}\right)\right)\right)$$

(18)

$$f_2=g_{0t}{\displaystyle \prod _{u\in T\backslash \left\{1\right\}}\left(p\left(s_u|s_{u-1},a_{u-1}\right)p_{\pi }\left(a_u|s_u\right)\right)p_0\left(s_1\right)p_{\pi }\left(a_1|s_1\right)}$$

(19)

Then, (17) above can be written as follows:

$$M_0\left(\pi \in \mathcal{P}\right)={\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{\displaystyle \underset{{\left(\mathcal{S}\times \mathcal{A}\right)}^{\left|T\right|}}{\int }\left(f_1{f}_2\right)}d{s}_{1}d{a}_{1}d{s}_{2}d{a}_2\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\right)}, \forall i\in P$$

(20)

Now, using integration by part formula, we can write (20) as follows:

$$M_0\left(\pi \in \mathcal{P}\right)={\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{\displaystyle \underset{{\mathcal{S}}^{\left|T\right|-1}}{\int }{\displaystyle \underset{{\mathcal{A}}^{\left|T\right|}}{\int }\left(f_1{\displaystyle \underset{\mathcal{S}}{\int }{f}_{2}d{s}_1}-{\displaystyle \underset{\mathcal{S}}{\int }\left({f_1}^{\prime }{\displaystyle \underset{\mathcal{S}}{\int }{f}_2}d{s}_1\right)d{s}_1}\right)}d{a}_{1}d{s}_{2}d{a}_2\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}}\right)}$$

(21)

By applying integration by part formula recursively on the second term each time it is being generated upon the application of integration by part formula, we can write the above expression as follows:

$$M_0\left(\pi \in \mathcal{P}\right)={\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{\displaystyle \underset{{\mathcal{S}}^{\left|T\right|-1}}{\int }{\displaystyle \underset{{\mathcal{A}}^{\left|T\right|}}{\int }\left(\begin{array}{l}{\left(-1\right)}^{1-1}{f_1}^{\left(1-1\right)}{\displaystyle \underset{\mathcal{S}}{\int }{f}_{2}d{s}_1}+{\left(-1\right)}^{2-1}{f_1}^{\left(2-1\right)}{\displaystyle \underset{{\mathcal{S}}^2}{\int }{f}_{2}d{s}_{1}d{s}_1}\\ +{\left(-1\right)}^{3-1}{f_1}^{\left(3-1\right)}{\displaystyle \underset{{\mathcal{S}}^3}{\int }{f}_{2}d{s}_1}d{s}_{1}d{s}_1+{\left(-1\right)}^{4-1}{f_1}^{\left(4-1\right)}{\displaystyle \underset{{\mathcal{S}}^4}{\int }{f}_{2}d{s}_1}d{s}_{1}d{s}_{1}d{s}_1\\ +{\left(-1\right)}^{5-1}{f_1}^{\left(5-1\right)}{\displaystyle \underset{{\mathcal{S}}^5}{\int }{f}_{2}d{s}_1}d{s}_{1}d{s}_{1}d{s}_{1}d{s}_1+\dots \\ +{\left(-1\right)}^{\left(n-1\right)}{f_1}^{\left(n-1\right)}{\displaystyle \underset{{\mathcal{S}}^n}{\int }{f}_2{\left(d{s}_1\right)}^n}+{\left(-1\right)}^n{\displaystyle \underset{\mathcal{S}}{\int }\left({f_1}^n{\displaystyle \underset{{\left(s_1\in \mathcal{S}\right)}^n}{\int }{f}_2{\left(d{s}_1\right)}^n}\right)d{s}_1}\end{array}\right)}d{a}_{1}d{s}_{2}d{a}_2\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}}\right)}$$

(22)

If we continue the application of integration by part formula recursively infinite times, we will have the following expression if we assume that an infinite derivative exists for the risk aversion function $f_1$ (concerning $s_1$) and ${\displaystyle \underset{\mathcal{S}}{\int }\left({f_1}^n{\displaystyle \underset{{\mathcal{S}}^n}{\int }{f}_2{\left(d{s}_1\right)}^n}\right)d{s}_1}\to 0$ as $n\to \infty$ (which holds in the case of CVaR and mean-CVaR given their piecewise constant risk aversion functions $f_1$ (which will have zero infinite derivatives for each piecewise component after the splitting of the integration interval), while for the rest of spectral risk measures, it may or may not hold depending on the risk aversion function itself along with the cumulative probability distribution function [44]):

$$M_0\left(\pi \in \mathcal{P}\right)={\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{\displaystyle \underset{{\mathcal{S}}^{\left|T\right|-1}}{\int }{\displaystyle \underset{{\mathcal{A}}^{\left|T\right|}}{\int }\left(\begin{array}{l}{\left(-1\right)}^{1-1}{f_1}^{\left(1-1\right)}{\displaystyle \underset{\mathcal{S}}{\int }{f}_{2}d{s}_1}+{\left(-1\right)}^{2-1}{f_1}^{\left(2-1\right)}{\displaystyle \underset{{\mathcal{S}}^2}{\int }{f}_{2}d{s}_{1}d{s}_1}\\ +{\left(-1\right)}^{3-1}{f_1}^{\left(3-1\right)}{\displaystyle \underset{{\mathcal{S}}^3}{\int }{f}_{2}d{s}_1}d{s}_{1}d{s}_1+{\left(-1\right)}^{4-1}{f_1}^{\left(4-1\right)}{\displaystyle \underset{{\mathcal{S}}^4}{\int }{f}_{2}d{s}_1}d{s}_{1}d{s}_{1}d{s}_1\\ +{\left(-1\right)}^{5-1}{f_1}^{\left(5-1\right)}{\displaystyle \underset{{\mathcal{S}}^5}{\int }{f}_{2}d{s}_1}d{s}_{1}d{s}_{1}d{s}_{1}d{s}_1+\dots +{\left(-1\right)}^{\left(n-1\right)}{f_1}^{\left(n-1\right)}{\displaystyle \underset{{\mathcal{S}}^n}{\int }{f}_2{\left(d{s}_1\right)}^n}\\ +{\left(-1\right)}^n{f_1}^n{\displaystyle \underset{{\mathcal{S}}^{n+1}}{\int }{f}_2{\left(d{s}_1\right)}^{n+1}}+\dots +{\left(-1\right)}^{\infty }{f_1}^{\infty }{\displaystyle \underset{{\mathcal{S}}^{\infty }}{\int }{f}_2{\left(d{s}_1\right)}^{\infty }}\end{array}\right)}d{a}_{1}d{s}_{2}d{a}_2\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}}\right)}$$

(23)

As a side note, it is important to search and develop risk-aversion functions for which we can write (23) for (22), which will be the direction of our future research.

Let $c=-1$ and $h_1={\displaystyle \underset{\mathcal{S}}{\int }{f}_{2}d{s}_1}$, $h_2={\displaystyle \underset{{\mathcal{S}}^2}{\int }{f}_{2}d{s}_{1}d{s}_1}$, $h_3={\displaystyle \underset{{\mathcal{S}}^3}{\int }{f}_{2}d{s}_1}d{s}_{1}d{s}_1$, ……, $h_n={\displaystyle \underset{{\mathcal{S}}^n}{\int }{f}_2{\left(d{s}_1\right)}^n}$, ……, $h_{\infty }={\displaystyle \underset{{\mathcal{S}}^{\infty }}{\int }{f}_2{\left(d{s}_1\right)}^{\infty }}$.

Now, to integrate all the individual terms with respect to the first action $a_1$, the expression (23) can be written as follows:

$$M_0\left(\pi \in \mathcal{P}\right)={\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{\displaystyle \underset{{\left(\mathcal{S}\times \mathcal{A}\right)}^{\left|T\right|-1}}{\int }\left(\begin{array}{l}{c}^{1-1}{\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(1-1\right)}{h}_1\right)d{a}_1}+c^{2-1}{\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(2-1\right)}{h}_2\right)d{a}_1}d{a}_1+c^{3-1}{\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(3-1\right)}{h}_3\right)d{a}_1}\\ +c^{4-1}{\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(4-1\right)}{h}_4\right)d{a}_1}+c^{5-1}{\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(5-1\right)}{h}_5\right)d{a}_1}+\dots \\ +c^{\left(n-1\right)}{\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(n-1\right)}{h}_n\right)d{a}_1}+c^n{\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(n\right)}{h}_{n+1}\right)d{a}_1}+\dots \\ +c^{\infty }{\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(\infty \right)}{h}_{\infty }\right)d{a}_1}\end{array}\right)d{s}_{2}d{a}_2\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}}\right)}$$

(24)

In the above expression, the terms ${\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(1-1\right)}{h}_1\right)d{a}_1}$, ${\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(2-1\right)}{h}_2\right)d{a}_1}$, ${\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(3-1\right)}{h}_3\right)d{a}_1}$, …, ${\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(\infty \right)}{h}_{\infty }\right)d{a}_1}$ can be expanded individually (similar to the expansion of (20)) by recursive application of integration by part formula (infinite times) given the derivatives of all orders of function $f_1$ are well-defined (with respect to $a_1$) and ${\displaystyle \underset{\mathcal{A}}{\int }\left({f_1}^{\left(p-1\right)\left(m\right)}{\displaystyle \underset{{\mathcal{A}}^m}{\int }{h}_p{\left(d{a}_1\right)}^m}\right)d{a}_1}\to 0, \forall p\in \left\{1,2,\dots ,\infty \right\}$ as $m\to \infty$, where, in ${f_1}^{\left(p-1\right)\left(m\right)}$, $\left(p-1\right)$ is the order of $f_1$ derivative with respect to $s_1$, and $m$ is the order of $f_1$ derivative with respect to $a_1$.

After the expansion of these terms, we will have similar terms (as the individual terms present in (23)). The integrals (with respect to other states and actions, e.g., $s_2,a_2,s_3,a_3\dots$) of these resulting terms can be expanded similarly by using the mechanism of recursive by-part integration as used above with $s_1$ and $a_1$.

Let

$$f_3={\displaystyle \underset{{\left(\mathcal{S}\times \mathcal{A}\right)}^{\left|T\right|}}{\int }\left(g_{0t}{\displaystyle \prod _{u\in T\backslash \left\{1\right\}}\left(p\left(s_u|s_{u-1},a_{u-1}\right)p_{\pi }\left(a_u|s_u\right)p_0\left(s_1\right)p_{\pi }\left(a_1|s_1\right)\right)}\right)d{s}_{1}d{a}_{1}d{s}_{2}d{a}_2\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}}$$

(25)

After recursive integration (with the application of by-part integration formula recursively infinite times) with respect to states and actions in all the time periods $s_1,a_1,s_2,a_2,s_3,a_3,\dots ,s_{\left|T\right|},a_{\left|T\right|}$ and given the fact that all the resulting multiple integral terms will always have an integral with respect to each state and action $s_1,a_1,s_2,a_2,s_3,a_3,\dots ,s_{\left|T\right|},a_{\left|T\right|}$, we will have the expression for $M_0\left(\pi \in \mathcal{P}\right)$ in the following form:

$$\begin{array}{l}{M}_0\left(\pi \in \mathcal{P}\right)=\\ {\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}\left(\begin{array}{l}{c}_1\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_3}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\\ +c_2\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_3}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)+\dots \\ +c_n\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_3}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\end{array}\right)\right)}\end{array}$$

(26)

Note that, in (26), $\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}$ (in each term after $c_1,c_2,\dots ,c_n$) are different with respect to each other as they may have derivatives of different orders with respect to various states and actions. Here, the multiple integral terms in (26) have one integral (with respect to all state and action variables $s_1,a_1,s_2,a_2,s_3,a_3,\dots ,s_{\left|T\right|},a_{\left|T\right|}$) less than that of the expression resulting after the application of the by-part integration formula recursively infinite times on $\left(f_1{f}_2\right)$ in (20) since one integral (with respect to each state and action variable) will be consumed by the expression of $f_3$(25). As we have performed while moving from (A2) to (A3) in Appendix A, the expression (25) can be reduced to the following:

$$f_3={\displaystyle \underset{{\left(\mathcal{S}\times \mathcal{A}\right)}^t}{\int }\left(g_{0t}{\displaystyle \prod _{u\in \left\{x\in T|x>1,x\le t\right\}}\left(p\left(s_u|s_{u-1},a_{u-1}\right)p_{\pi }\left(a_u|s_u\right)p_0\left(s_1\right)p_{\pi }\left(a_1|s_1\right)\right)}\right)}d{s}_1\dots d{s}_{t}d{a}_1\dots d{a}_t$$

(27)

Let

$$\begin{array}{l}{f}_4={\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{0t}{\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{p}_{\pi }\left(s_t=s\in \mathcal{S},a_t=a\in \mathcal{A}\right)\right)}\right)}dsda=\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\frac{1}{\left(1-\gamma \right)}{\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{0t}\left(1-\gamma \right){\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{p}_{\pi }\left(s_t=s\in \mathcal{S},a_t=a\in \mathcal{A}\right)\right)}\right)}dsda\end{array}$$

(28)

From DCT and (A4) in Appendix A, we can write the following form of (26) above (similar to (A5) in Appendix A) by interchanging positions of integrals and summation.

$$\begin{array}{ll}{M}_0\left(\pi \in \mathcal{P}\right)& =c_1\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_4}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\\ & +c_2\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_4}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)+\dots \\ & +c_n\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_4}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\end{array}$$

(29)

Let $f_5$ and the Equation of occupancy measure (A6) in Appendix A above, we can write the following:

$$f_5={\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{0t}\left(1-\gamma \right){\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{p}_{\pi }\left(s_t=s\in \mathcal{S},a_t=a\in \mathcal{A}\right)\right)}\right)}dsda={\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{0t}\rho \left(s\in \mathcal{S},a\in \mathcal{A}\right)\right)}dsda,\forall i\in P$$

(30)

Then, we can write (29) naturally as follows, which is the objective of the “The Perturbation function”:

$$\begin{array}{l}{M}_0\left(\pi \in \mathcal{P}\right)=c_1\frac{1}{1-\gamma }\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_5}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\\ \hspace{1em}\hspace{1em}+c_2\frac{1}{1-\gamma }\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_5}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)+\dots \\ \hspace{1em}\hspace{1em}+c_n\frac{1}{1-\gamma }\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_5}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\end{array}$$

(31)

Concavity of the Perturbation Function

From (A7) in Appendix A and (31), “The Perturbation function” itself can be written in its mathematical program below (given that (31) is to be optimized over the set of feasible occupation measures $\mathcal{T}$):

$$P\left(\xi \right)\triangleq \underset{\pi \in \mathcal{P}}{max}{M}_0\left(\pi \in \mathcal{P}\right)=\underset{\rho \left(.\right)\in \mathcal{T}}{max}{M}_0\left(\pi \in \mathcal{P}\right)$$

(32)

Subject to: (6)–(8)

$$\frac{1}{1-\gamma }{\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{it}\rho \left(s\in \mathcal{S},a\in \mathcal{A}\right)\right)}dsda\le {\xi }_i \forall i\in P$$

(33)

With the only consideration being the risk-averse objective, the occupation measure set $\mathcal{T}$ (in (4)) being convex and compact (compact means closed and bounded) is directly followed by Theorem 3.1 in [45], which makes us able to write the following:

$${\rho }_{\mu }\left(s\in \mathcal{S},a\in \mathcal{A}\right)=\mu {\rho }_1\left(s\in \mathcal{S},a\in \mathcal{A}\right)+\left(1-\mu \right){\rho }_2\left(s\in S,a\in \mathcal{A}\right), \forall \mu \in \left(0,1\right)$$

(34)

Now, for the feasible two policies, ${\pi }_1\in \mathcal{P}$ and ${\pi }_2\in \mathcal{P}$, which are optimal for the perturbation vectors ${\xi }^1$ and ${\xi }^2$, respectively, considered above, let their corresponding two occupation measures be ${\rho }_{‘1}\left(s\in \mathcal{S},a\in \mathcal{A}\right)$ and ${\rho }_{‘2}\left(s\in \mathcal{S},a\in \mathcal{A}\right)$, respectively. Then, from the convexity of the set $\mathcal{T}$, there must be a feasible policy ${\pi }_{\mu }\in \mathcal{P}$ whose occupation measure is given by ${\rho }_{\mu }\left(s,a\right)$. From (A7) in Appendix A, linearity of constraint integrals and constraint satisfaction by ${\rho }_1\left(s\in \mathcal{S},a\in \mathcal{A}\right)$ and ${\rho }_2\left(s\in \mathcal{S},a\in \mathcal{A}\right)$ with slacks ${\xi }_i^1, \forall i\in P$ and ${\xi }_i^2, \forall i\in P$, respectively, ${\rho }_{\mu }\left(s\in \mathcal{S},a\in \mathcal{A}\right)$ will satisfy the constraints with slack $\mu {\xi }_i^1+\left(1-\mu \right){\xi }_i^2, \forall i\in P,\mu \in \left(0,1\right)$.

From (A7) in Appendix A and (34), we can write the following:

$$\begin{array}{l}{\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{it}{\rho }_{\mu }\left(s\in \mathcal{S},a\in \mathcal{A}\right)\right)}dsda={\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{it}\left(\mu {\rho }_1\left(s\in \mathcal{S},a\in \mathcal{A}\right)+\left(1-\mu \right){\rho }_2\left(s\in \mathcal{S},a\in \mathcal{A}\right)\right)\right)}dsda\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\mu {\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{it}{\rho }_1\left(s\in \mathcal{S},a\in \mathcal{A}\right)\right)}dsda+\left(1-\mu \right){\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{it}{\rho }_2\left(s\in \mathcal{S},a\in \mathcal{A}\right)\right)}dsda,\forall i\in P\end{array}$$

(35)

For the risk-averse objective, let $f_{{\xi }^1}={\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{0t}{\rho }_1\left(s\in \mathcal{S},a\in \mathcal{A}\right)\right)}dsda$ and $f_{{\xi }^2}={\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{0t}{\rho }_2\left(s\in \mathcal{S},a\in \mathcal{A}\right)\right)}dsda$. Also, consider $f_{{\xi }^{\mu }}={\displaystyle \underset{\mathcal{S}\times \mathcal{A}}{\int }\left(g_{0t}{\rho }_{\mu }\left(s\in \mathcal{S},a\in \mathcal{A}\right)\right)}dsda$. From (34) above, $f_{{\xi }^{\mu }}=\mu f_{{\xi }^1}+\left(1-\mu \right)f_{{\xi }^2}$, and we can write, for “The Perturbation Function” objective, $P\left(\mu {\xi }^1+\left(1-\mu \right){\xi }^2\right)$, corresponding to the perturbation vector $\mu {\xi }^1+\left(1-\mu \right){\xi }^2$ as the following:

$$\begin{array}{l}P\left(\mu {\xi }^1+\left(1-\mu \right){\xi }^2\right)\\ \ge \left(\begin{array}{l}{c}_1\frac{1}{1-\gamma }\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_{{\xi }^{\mu }}}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\\ +c_2\frac{1}{1-\gamma }\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_{{\xi }^{\mu }}}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)+\dots \\ +c_n\frac{1}{1-\gamma }\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }{f}_{{\xi }^{\mu }}}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_{1}d{s}_2\dots d{s}_{2}d{a}_2\dots d{a}_2\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\end{array}\right)\\ =\left(\begin{array}{l}{c}_1\frac{1}{1-\gamma }\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }\left(\mu f_{{\xi }^1}+\left(1-\mu \right)f_{{\xi }^2}\right)}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_1\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\\ +c_2\frac{1}{1-\gamma }\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }\left(\mu f_{{\xi }^1}+\left(1-\mu \right)f_{{\xi }^2}\right)}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_1\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)+\dots \\ +c_n\frac{1}{1-\gamma }\frac{dd\dots ddd\dots d{f}_1}{d{s}_{1}d{s}_2\dots d{s}_{\left|T\right|}d{a}_{1}d{a}_2\dots d{a}_{\left|T\right|}}\left({\displaystyle \underset{\mathcal{A}}{\int }\cdots {\displaystyle \underset{\mathcal{A}}{\int }{\displaystyle \underset{\mathcal{S}}{\int }\cdots {\displaystyle \underset{\mathcal{S}}{\int }\left(\mu f_{{\xi }^1}+\left(1-\mu \right)f_{{\xi }^2}\right)}}}}d{s}_1\dots d{s}_{1}d{a}_1\dots d{a}_1\dots d{s}_{\left|T\right|}\dots d{s}_{\left|T\right|}d{a}_{\left|T\right|}\dots d{a}_{\left|T\right|}\right)\end{array}\right)\end{array}$$

(36)

From (31) and the linearity of the integrals, (36) above can be written as follows:

$$P\left(\mu {\xi }^1+\left(1-\mu \right){\xi }^2\right)\ge \mu M_0\left({\pi }_1\in \mathcal{P}\right)+\left(1-\mu \right)M_0\left({\pi }_2\in \mathcal{P}\right)$$

(37)

Since $M_0\left({\pi }_1\in \mathcal{P}\right)=P\left({\xi }^1\right)$ and $M_0\left({\pi }_2\in \mathcal{P}\right)=P\left({\xi }^2\right)$, the following holds

$$P\left({\mu }^1{\xi }^1+\left(1-\mu \right){\xi }^2\right)\ge \mu P\left({\xi }^1\right)+\left(1-\mu \right)P\left({\xi }^2\right),\forall \mu \in \left(0,1\right)$$

(38)

which means that “The Perturbation function” is concave, which then implies, from “Proposition 1”, that strong duality holds for “Problem 1”.$\square$

4. Risk-Averse Markov Decision Processes with Instantaneous Constraints

Derived from “Problem 1”, the problem class that we will consider in this section consists of deterministic instantaneous constraints (40) instead of cumulative expectation constraints (9). This problem will be referred to as “Problem 2”, and its mathematical program is given as follows:

$$\underset{\pi \in \mathcal{P}}{max}{M}_0\left(\pi \in \mathcal{P}\right)$$

(39)

Subject to the following: (6)–(8)

$${\gamma }^{t-1}{g}_{it}\le 0, \forall i\in P,t\in T$$

(40)

The assumption of deterministic constraints is justified in many real-world contexts. Not only do risk constraints [5] make the problem complex, but they are unnecessary in many real-world problems due to the very reason for the constraint utilization itself (in the presence of risk-averseness), which is to completely eliminate extreme realizations of certain stochastic variables regardless of risk-averseness or decision makers’ behavioral aspects. In addition, if the robustification of feasible solution space is required using box uncertainty sets, it will make the constraints deterministic in a robust counterpart program.

Strong Duality for “Problem 2”

Now based on Theorem 1 in Section 3, we will prove two theorems (Theorems 2 and 3) for proving strong duality for the “Problem 2” above. Based on these theorems, we will develop a constraint handling mechanism in Section 5 for solving “Problem 2”, where we will also perform a theoretical convergence analysis of this algorithm in Section 5.5.

In this regard, we will first assume the following, which we will refer to as “Assumption 1”:

Assumption 1. 

The feasible policy space  $\mathcal{P}$ of “Problem 2” has a non-empty relative interior (Slater’s condition holds), and if ${\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{it}\right)}\le 0, \forall i\in P$ holds for any feasible policy $\pi \in \mathcal{P}$ , then ${\gamma }^{t-1}{g}_{it}\le 0, \forall i\in P,t=T$ also holds for that policy.

Now, on the basis of our above assumption, we will prove the following:

Theorem 2. 

Under Assumption 1, strong duality holds for “Problem 2”.

Proof. 

Naturally, for any policy $\pi \in \mathcal{P}$, the condition ${\gamma }^{t-1}{g}_{it}\le 0, \forall i\in P,t=T$ implies condition ${\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_i\left(s_t,a_t\right)\right)}\le 0, \forall i\in P$, and from Assumption 1, ${\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{it}\right)}\le 0, \forall i\in P$ implies ${\gamma }^{t-1}{g}_{it}\le 0, \forall i\in P,t=T$, which means the following proposition holds

“${\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{it}\right)}\le 0, \forall i\in P$ if and only if $g_{it}\le 0, \forall i\in P,t=T$”.

Let ${\lambda }^{\prime }$ be the Lagrangian vector, with its components ${\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T$, then the dual of “Problem 2”, which we will refer to as “Dual 2”, is as follows:

$$\underset{{\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T }{min}\underset{\pi \in \mathcal{P}}{max}\left(M_0\left(\pi \in \mathcal{P}\right)+{\displaystyle \sum _{t\in T}{\displaystyle \sum _{i\in P}\left({\gamma }^{t-1}{\lambda }_{it}{g}_{it}\right)}}\right)$$

(41)

Subject to the following: (6)–(8)

It is important to note here that $\lambda$ (with its components ${\lambda }_i, \forall i\in P$) is the Lagrangian vector for “Problem 1” while ${\lambda }^{\prime }$ is the Lagrangian vector for the “Problem 2” with its components ${\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T$.

As Theorem 1 implies

$$\begin{array}{l}{P}^{*}=D^{*}=\underset{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P}{min}\underset{\pi \in \mathcal{P}}{max}L\left(\pi ,\lambda \right)=\\ \underset{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P}{min}\underset{\pi \in \mathcal{P}}{max}\left(M_0\left(\pi \in \mathcal{P}\right)+{\displaystyle \sum _{i\in \mathcal{P}}\left({\lambda }_i\left(V_i\left(\pi \in P\right)\right)\right)}\right), \forall \xi \end{array}$$

(42)

If we consider all the constraint cost functions $g_{it}, \forall i\in P,t\in T$ in (9) as deterministic and the risk measure $\mathbb{M}$ in (5) satisfies translation invariance (a property of coherent risk measures [46]), we can write (42) above as follows (referring to (3)), which can also be regarded as the min–max objective in “Dual 1” in (11):

$$P^{*}=\underset{{\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T}{min}\underset{\pi \in \mathcal{P}\left(\mathcal{S}\right)}{max}\left(\mathbb{M}\left[g_{0:\left|P\right|,1:\left|T\right|}\right]\right)$$

(43)

As a side note, the reason for making the above form of the dual objective of “Problem 1” is because, during RL training (for optimization of “Problem 2”), our model-free decision-making agent (heuristic) will obtain $g_{0t}+{\displaystyle \sum _{i\in P}\left({\lambda }_{it}{g}_{it}\right)}, \forall t\in T$ as values of reward signals from the MDP environment (in each time period) and from that value, the RL agent has to stay within the feasible solution region (constraint satisfaction) in addition to maximizing the objective function.

Let ${\pi }^{*}\in \mathcal{P}$ be the optimal policy for “Problem 2”, then given Lagrangian multipliers ${\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T$ of its dual, we will write the following (by construction given the translation invariance of the dynamic risk measure $\mathbb{M}\left[.\right]$):

$$\begin{array}{l}\underset{\pi \in \mathcal{P}}{max}\mathbb{M}\left[g_{0:\left|P\right|,1:\left|T\right|}\right]\ge \left(\mathbb{M}\left[g_{0:\left|P\right|,1:\left|T\right|}|{\pi }^{*}\right]\right)\ge \\ \left(\mathbb{M}\left[{\gamma }^{t-1}{\displaystyle \sum _{t\in T}{g}_{it}}|s_0,{\pi }^{*}\right]\right)+{\displaystyle \sum _{t\in T}{\displaystyle \sum _{i\in P}\left({\gamma }^{t-1}{\lambda }_{it}\left(g_{it}|{\pi }^{*}\right)\right)}}\ge P^{*}\end{array}$$

(44)

From the above, we can make the following implication:

$$\underset{{\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T}{min}\underset{\pi \in \mathcal{P}}{max}\left(\mathbb{M}\left[g_{0:\left|P\right|,1:\left|T\right|}\right]\right)\ge P^{*}$$

(45)

Suppose ${\lambda }_{it}^{*}, \forall i\in P,t\in T$ are the optimal Lagrangian multipliers in“Dual 2”, then we can write as follows given Theorem 1:

$$\underset{\pi \in \mathcal{P}}{max}\left(\mathbb{M}\left[g_{0:\left|P\right|,1:\left|T\right|}|{\lambda }_{it}^{*}, \forall i\in P,t\in T\right]\right)=P^{*}$$

(46)

As a result, the following directly follows from the above:

$$\underset{{\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T}{min}\underset{\pi \in \mathcal{P}}{max}\left(\mathbb{M}\left[g_{0:\left|P\right|,1:\left|T\right|}\right]\right)\le P^{*}$$

(47)

Hence, from (45) and (47), the strong duality condition is satisfied for “Problem 2”.□

Due to time-varying Lagrangian multipliers ${\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T$, with the increase in the number of time periods, the number of Lagrangian multipliers will grow, and, because this, memory complexity may become the point of concern. Also, for infinite time horizon problems, time-varying Lagrangian multipliers become infeasible. To counter this effect, we will prove Theorem 3 below.

Note: It is worth noting that the time invariance of Lagrangian multipliers (in the context of Problem 2) directly follows from the fact that we have used ${\lambda }_{it}={\lambda }_i, \forall t\in T$ while proving Theorem 2 from Theorem 1 (on page 15) when we moved from (42) to (43). However, if we start from the statement of Theorem 2, we need to prove the time invariance of Lagrangian multipliers ${\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T$ in “Dual 2”, which we will perform in Theorem 3 below (which is a redundant theoretical result).

Theorem 3. 

Suppose $\left(\left\{{\lambda }_{it}^{*}, \forall i\in P,t\in T\right\},{\pi }^{*}\in \mathcal{P}\right)$ to be the optimal solution for “Dual 2” and suppose $\left(\left\{{\lambda }_i^{*}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P\right\},{\overline{\pi }}^{*}\in \mathcal{P}\right)$ to be the optimal solution for the following Dual mathematical program having instantaneous constraint cost functions $g_{it}, \forall i\in P,t\in T$ and time-invariant Lagrangian multipliers ${\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P$:

$$\underset{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P}{min}\underset{\pi \in \mathcal{P}}{max}\left(M_0\left(\pi \in P\right)+{\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{\displaystyle \sum _{i\in P}\left({\lambda }_i{\gamma }^{t-1}{g}_{it}\right)}\right)}\right)$$

(48)

Subject to the following: (6)–(8)

Now, suppose that in “Dual 2”, ${\lambda }_{it}={\lambda }_i^{*}, \forall i\in P,t\in T$, under Assumption 1, $\left(\left\{{\lambda }_{it}={\lambda }_i^{*}, \forall i\in P,t\in T\right\},{\pi }^{*}\in \mathcal{P}\right)$ is also optimal for “Problem 2” or “Dual 2”.

Proof. 

By definition (under Assumption 1), for policy $\pi \in \mathcal{P}$, satisfying instantaneous constraints ${\gamma }^{t-1}{g}_{it}\le 0, \forall i\in P,t=T$ implies that $\pi \in \mathcal{P}$ will also satisfy cumulative constraints ${\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{it}\right)}\le 0, \forall i\in P$.

Then, by assuming under closed loop dynamics, $s_{t+1}~p_z\left(.|s_t,a_t\right), \forall t\in T$ and $a_t~\pi \left(s_t\right), \forall t\in T$, that ${\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{it}\right)}\le 0, \forall i\in P$ implies ${\gamma }^{t-1}{g}_{it}\le 0, \forall i\in P,t=T$ (under Assumption 1), and“Problem 1” shares the same feasible domain with “Dual 2” given the dynamic risk measure’s translation invariance property on the objective (39) and the fact that constraint cost functions $g_{it}, \forall i\in P,t\in T$ are deterministic. From Theorem 1, strong duality will hold for the problem given above (in the statement of Theorem 3), which then implies with Theorem 2 that the optimal solution for the above-motioned problem, which has time-invariant Lagrangian multipliers, is also the optimal solution for “Dual 2”. □

Note: It is important to mention that there can be optimal solutions $\left(\left\{{\lambda }_{it}\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P,t\in T\right\},{\pi }^{*}\in \mathcal{P}\right)$ for “Dual 2”, which has time variant Lagrangian multipliers in addition to the optimal solution $\left({\lambda }_i^{*}\in \mathbb{R}\backslash {\mathbb{R}}^{+},{\overline{\pi }}^{*}\in \mathcal{P}\right)$, which has time-invariant Lagrangian multipliers as given by Theorem 3 above.

5. Augmented Lagrangian-Based Constraint Handling Mechanism

In this section, based on Theorems 1 and 2, we will develop a risk-averse RL constraint-handling algorithm (Algorithm 1) for optimizing “Problem 2” (risk-averse MDP with instantaneous deterministic constraints). In the end, we will also theoretically prove its convergence. However, to take advantage of Algorithm 1, “Assumption 1” must be satisfied during its execution because Theorem 2 is based on the assumption. While the assumption can be automatically satisfied in the case of some problems, we ensure its satisfaction during the execution of Algorithm 1 by utilizing a so-called “Clipping method” proposed by [47]. This method is described as follows.

5.1. Clipping Method

In this method, values of constraints for the states where constraints are not violated are reduced to zero, i.e., if constraints are satisfied, then we will consider the constraint violation penalty as zero. For an illustration of the technique, consider a set of instantaneous constraints ${\gamma }^{t-1}{g}_{it}\le 0, \forall i\in P,t\in T$ as (40) in “Problem 2”. Then, to form a reward signal, ${\gamma }^{t-1}{g}_{it}, \forall i\in P,t=T$ will only be added as a constraint violation penalty to the reward value ${\gamma }^{t-1}{g}_{0t}, \forall t\in T$ if it is greater than zero; otherwise, it will not be added or considered zero. This can be performed by using functions or mappings which map ${\mathbb{R}}^{+}$ to ${\mathbb{R}}^{+}$ while mapping ${\mathbb{R}}^{-}$ to zero, e.g., $ReLU\left({\gamma }^{t-1}{g}_{0t}\right), \forall i\in P,t=T$ (in the case of which $ReLU\left(x\right)=x, \forall x\in {\mathbb{R}}^{+}\cup \left\{0\right\}$; otherwise, $ReLU\left(x\right)=0, \forall x\in {\mathbb{R}}^{-}$). Other non-negative activation functions like SoftPlus and Segmoid can also be used [47].

5.2. Surrogate Objective

As RL is mostly used for non-convex multi-stage optimization, in order to handle constraints in this setting, it is better to use augmented Lagrangian instead of the Lagrangian formulation [48], which we will use in our constraint-handling algorithm by first designing a surrogate objective and then a surrogate reward function, from the augmented Lagrangian formulation of “Problem 2”.

Let ${\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P$ be the Lagrangian multipliers, which are invariant to time, and ${\zeta }_i\in {\mathbb{R}}^{+}, \forall i\in P$ be the time-invariant quadratic penalty coefficients corresponding to each of the instantaneous constraints (40) in “Problem 2”. Based on the strong duality result for “Problem 2” (Theorem 2) and the Lagrangian-based objective (41) of “Dual 2” (with time-invariant Lagrangian multipliers), the augmented Lagrangian-based surrogate objective for “Problem 2” with clipping method (which will be directly optimized by an RL algorithm) is as follows:

$$S\left(\pi \in \mathcal{P},\left\{{\lambda }_i, \forall i\in P\right\},\left\{{\zeta }_i, \forall i\in P\right\}\right)=\mathbb{M}\left[{\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}{g}_{0t}+{\gamma }^{t-1}{\displaystyle \sum _{i\in P}\left({\lambda }_{i}ReLU\left(g_{it}\right)\right)}-{\left({\gamma }^{t-1}\right)}^2{\displaystyle \sum _{i\in P}\left(\frac{{\zeta }_i}{2}ReLU{\left(g_{it}\right)}^2\right)}\right)}\right]$$

(49)

5.3. Reward Function

Correspondingly, the instantaneous reward function (stage-wise objective component) is given as follows, where the first term is the discounted reward function while the second and third terms are the discounted linear and quadratic penalty terms for constraint violations, respectively, which will discourage the agent from violating hard constraints.

$$s_t\left(s_t\in \mathcal{S},a_t\in \mathcal{A},s_{t+1}\in \mathcal{S}\right)={\gamma }^{t-1}{g}_{0t}+{\gamma }^{t-1}{\displaystyle \sum _{i\in P}\left({\lambda }_{i}ReLU\left(g_{it}\right)\right)}-{\left({\gamma }^{t-1}\right)}^2{\displaystyle \sum _{i\in P}\left(\frac{{\zeta }_i}{2}ReLU{\left(g_{it}\right)}^2\right)}$$

(50)

In the above Equations (49) and (50), the ReLU function is used for the implementation of the clipping method described above to realize Assumption 1 during the execution of the risk-averse RL algorithm.

5.4. Developed Mechanism

Based on the surrogate reward function (50) above and [47], we have designed ”Algorithm 1” for the solution of “Problem 2” in the presence of only deterministic instantaneous constraints as follows.

Algorithm 1: Augmented Lagrangian-based constrained risk-averse RL
1	Take $u_{\zeta }\in [1,\infty )$ and dual assent step size $\Lambda \in {\mathbb{R}}^{+}$, and initialize ${\lambda }_i^0\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P$ and ${\zeta }_i^0\in {\mathbb{R}}^{+}, \forall i\in P$
2	Initialize RL policy ${\pi }^0\in \mathcal{P}$
3	For $e=0,1,\dots ,E$ (where, $E$ defines the number of iterations in which Lagrangian multipliers and quadratic penalty coefficients need to be updated)
4		${\pi }^e=\underset{{\pi }^e\in \mathcal{P}}{arg max}S\left(\pi \in \mathcal{P},\left\{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P\right\},\left\{{\zeta }_i\in {\mathbb{R}}^{+}, \forall i\in P\right\}\right)$
5		${\lambda }_i^{e+1}\leftarrow f_{step}\left({\lambda }_i^e-\Lambda \left({\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}Relu\left(g_{it}\right)\right)}|{\pi }_e\right)\right)$ ${\zeta }^{e+1}\leftarrow u_{\zeta }{\zeta }^e$
6	End

In the above algorithm, ${\lambda }_i^{e+1}\leftarrow f_{step}\left({\lambda }_i^e-\Lambda \left({\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}Relu\left(g_{it}\right)\right)}|{\pi }_e\right)\right)$ is the mapping $f_{step}\left(.\right):\mathbb{R}\to \mathbb{R}$ such that $f_{step}\left(x\right)=0$ if $x\ge 0$ and $f_{step}\left(x\right)=x$ if $x<0$. ${\pi }^e=\underset{{\pi }^e\in \mathcal{P}}{arg max}S\left(\pi \in \mathcal{P},\left\{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P\right\},\left\{{\zeta }_i\in {\mathbb{R}}^{+}, \forall i\in P\right\}\right)$ is the optimization of unconstrained surrogate objective function for “Problem 2”, which will be solved by the proposed risk-averse RL. Then, we will update Lagrangian multipliers via dual assent in dual-domain ${\lambda }_i^{e+1}\leftarrow f_{step}\left({\lambda }_i^e-\Lambda \left({\displaystyle \sum _{t\in T}\left({\gamma }^{t-1}Relu\left(g_{it}\right)\right)}|{\pi }_e\right)\right)$, while in tandem, we will also update the quadratic penalty coefficient ${\zeta }^{e+1}\leftarrow u_{\zeta }{\zeta }^e$ such that the penalty from both Lagrangian and quadratic penalty terms will increase with every update of ${\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P$ and, ${\zeta }_i\in {\mathbb{R}}^{+}, \forall i\in P$. For ease of understanding, the block diagram of the workings of the risk-averse actor–critic algorithm [12] with the proposed augmented lagrangian-based constraint handling mechanism is shown in Figure 2. Here, the upper shaded religion contains all the learning components inside the actor–critic algorithml while the shaded region at the bottom contains the proposed constraint handling mechanism, which updates the Lagrangian multipliers and quadratic penalty factors (inside the environment) after every fixed number of risk-averse actor–critic algorithm’s training iterations. For elucidation on conditionally elicitable dynamic risk measure and scoring function mentioned in the following diagram, readers are referred to [12].

5.5. Theoretical Results for Convergence

Now, we will perform the above algorithm’s convergence analysis by proving Theorem 4, but first, we will prove a proposition as follows, which will be used in proving Theorem 4:

Proposition 2. 

Consider following two optimization problems such that both of these have time-invariant Lagrangian multipliers ${\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P$ , and ${\zeta }_i\in {\mathbb{R}}^{+}$ , as follows under “Assumption 1”:

Problem A:

$$d_s^1\left({\lambda }_i, \forall i\in P\right)=\underset{\pi \in \mathcal{P}}{max}\mathbb{M}\left[g_{0:\left|P\right|,1:\left|T\right|}\right]$$

(51)

Subject to the following: (6)–(8)

$${\lambda }_i={\lambda }_{it}, \forall t\in T$$

(52)

Problem B:

$$d_s^2\left(\left\{{\lambda }_i, \forall i\in P\right\},\left\{{\zeta }_i, \forall i\in P\right\}\right)=\underset{\pi \in \mathcal{P}}{max}S\left(\pi \in P,\left\{{\lambda }_i, \forall i\in P\right\},\left\{{\zeta }_i, \forall i\in P\right\}\right)$$

(53)

Subject to the following: (6)–(8)

Let us fix Lagrangian multipliers for all the constraints, except the constraint $i\in P$, to any value ${\lambda }_{i^{\prime }}^f\in \mathbb{R}\backslash {\mathbb{R}}^{+}. \forall i^{\prime }\in P\backslash i$ (where these fixed values can be different for “Problem A” and “Problem B”) such that the Lagrangian multiplier ${\lambda }_i$ is the only dual variable in both “Problem A” and “Problem B”.

Suppose ${\lambda }_i^{1*}=\underset{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}}{\arg \min }{d}_s^1\left(\left\{{\lambda }_i, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right)$ and ${\lambda }_i^{2*}=\underset{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}}{\mathrm{argmin}}{d}_s^2\left(\left\{{\lambda }_i, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)$. Also, let ${\pi }^{*}\in \mathcal{P}$ be an optimal policy for “Problem 2” under which ${\gamma }^{t-1}{g}_{it}\le 0, \forall i\in P,t\in T$ holds; then, the following relation will also hold

$$\begin{array}{c}{d}_s^1\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right)\le d_s^2\left(\left\{{\lambda }_i^{2*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\le \\ d_s^2\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\le d_s^1\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right)\end{array}$$

(54)

Proof. 

As we know ${\zeta }_i\ge 0$, the following holds

$$\begin{array}{l}{d}_s^2\left(\left\{{\lambda }_i,{\lambda }_{i^{\prime }}^f\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i\in {\mathbb{R}}^{+}, i\in P\right\}\right)\le \\ d_s^2\left(\left\{{\lambda }_i,{\lambda }_{i^{\prime }}^f\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i=0, i\in P\right\}\right)=d_s^1\left({\lambda }_i,{\lambda }_{i^{\prime }}^f\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i^{\prime }\in P\backslash i\right)\end{array}$$

(55)

We can imply the following from the above, where instead of ${\lambda }_i$, we used ${\lambda }_i^{1*}$:

$$d_s^2\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\le d_s^1\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right)$$

(56)

Also, by definition

$$d_s^2\left(\left\{{\lambda }_i^{2*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\le d_s^2\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i=0, i\in P\right\}\right)$$

(57)

By assuming ${\pi }^{*}\in P$ isthe policy which is optimal for “Problem 2” and, thus, feasible, we can write the following for “Problem A” and “Problem B” (given strong duality holds for “Problem 2”):

$$\begin{array}{l}{d}_s^2\left(\left\{{\lambda }_i^{2*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\ge \\ S\left({\pi }^{*}\in P,\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)=\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{M}_0\left({\pi }^{*}\in P\right)=d_s^1\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right)\end{array}$$

(58)

From (56)–(58), we can write the following:

$$\begin{array}{l}{d}_s^1\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right)\le d_s^2\left(\left\{{\lambda }_i^{2*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\le \\ \hspace{1em}{d}_s^2\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\le d_s^1\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right)\end{array}$$

(59)

Since we have taken no assumption on the constraint $i\in P$ (whose Lagrangian multiplier (dual variable) is not fixed), the relation (59) above is true for all constraints which are indexed on the set $P$, i.e.,

$$\begin{array}{l}{d}_s^1\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right)\le d_s^2\left(\left\{{\lambda }_i^{2*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\le \\ \hspace{1em}\hspace{1em}{d}_s^2\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\le d_s^1\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right), \forall i\in P\end{array}$$

(60)

By direct implication from (60) above, we proved (54). □

Theorem 4. 

Suppose ${\lambda }_i^{2*}=\underset{{\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}}{arg min}{d}_s^2\left(\left\{{\lambda }_i, \forall i\in P\right\},\left\{{\zeta }_i, \forall i\in P\right\}\right)$ subject to: (6)–(8). ${\pi }^{*}\in \mathcal{P}$ is the optimal solution for “Problem 2” under which ${\gamma }^{t-1}{g}_i\le 0, \forall i\in P,t=T$, and the optimal policy of “Problem B” is ${\pi }_s^{2*}\left({\lambda }_i^{2*}, \forall i\in P\right)$, then as ${\zeta }_i\to \infty , \forall i\in P$, the following condition holds

$${‖M_0\left({\pi }^{*}\right)-M_0\left({\pi }_s^{2*}\left({\lambda }_i^{2*}, \forall i\in P\right)\right)‖}_2\to 0$$

(61)

Proof. 

Again, let us fix Lagrangian multipliers for all the constraints to any value ${\lambda }_{i^{\prime }}^f\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i^{\prime }\in P\backslash i$ except $i\in P$ such that the Lagrangian multiplier ${\lambda }_i\in \mathbb{R}\backslash {\mathbb{R}}^{+}$ for the constraint $i\in P$ is the only dual variables, while ${\zeta }_i$ can evolve.

Suppose there is a policy ${\pi }_s^{2*}\left({\lambda }_i^{2*}, i\in P\right)\in P$ optimal for “Problem B” with some ${\zeta }_i>0, i\in P$. Also, presume that the policy is infeasible for “Problem 1”:

$$V_i\left({\pi }_s^{2*}\left({\lambda }_i^{2*}, i\in P\right)\right)>0, \forall i\in P$$

(62)

Here, we will consider two cases to prove our proposition.

Consider the objective function $\mathbb{M}\left[g_{0,1:\left|T\right|}\right]$ of “Problem 1”. There are only two cases possible for the constraint $i\in P$:${\lambda }_i^{2*}<{\lambda }_i^{1*}$ or ${\lambda }_i^{2*}\ge {\lambda }_i^{1*}$.

Suppose ${\lambda }_i^{2*}<{\lambda }_i^{1*}$ holds. As ${\pi }_s^{2*}\left({\lambda }_i^{2*}, i\in P\right)$ is the policy optimal for “Problem B”, then the following expression can be written

$$\begin{array}{l}{d}_s^1\left(\left\{{\lambda }_i^{1*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\}\right)\ge \mathbb{M}\left[g_{0:\left|P\right|,1:\left|T\right|}|{\pi }_s^{2*}\left({\lambda }_i^{2*}, i\in P\right)\right]=\\ \mathbb{M}\left[g_{0,1:\left|T\right|}|{\pi }_s^{2*}\left({\lambda }_i^{2*}, i\in P\right)\right]+{\lambda }_i^{1*}{V}_i\left({\pi }_s^{2*}\left({\lambda }_i^{2*}, i\in P\right)\right)>\\ \mathbb{M}\left[g_{0,1:\left|T\right|}|{\pi }_s^{2*}\left({\lambda }_i^{2*}, i\in P\right)\right]+{\lambda }_i^{2*}{V}_i\left({\pi }_s^{2*}\left({\lambda }_i^{2*}, i\in P\right)\right)>\\ d_s^2\left(\left\{{\lambda }_i^{2*}, i\in P\right\},\left\{{\lambda }_{i^{\prime }}^f, \forall i^{\prime }\in P\backslash i\right\},\left\{{\zeta }_i, i\in P\right\}\right)\end{array}$$

(63)

The result from case 1 contradicts Proposition 2 above.

Hence, it is proved that ${\lambda }_i^{2*}\ge {\lambda }_i^{1*}$ for constraint $i\in P$.

As we do not assume anything for constraint $i\in P$, for any combination of the values of $\left\{{\lambda }_{i^{\prime }}^f\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i^{\prime }\in P\backslash i\right\}$, ${\lambda }_i^{2*}\ge {\lambda }_i^{1*}$ holds $\forall i\in P$.

Given the above, let ${\zeta }_i^{\prime }>0, \forall i\in P$ such that the following holds

$$\mathbb{M}\left[g_{0,1:\left|T\right|}|{\pi }_s^{2*}\left({\lambda }_i^{2*}, \forall i\in P\right)\right]-{\displaystyle \sum _{i\in P}\left(\frac{{\zeta }_i^{\prime }}{2}ReLU{\left(g_i\right)}^2|{\pi }_s^{2*}\left({\lambda }_i^{2*}, \forall i\in \mathcal{P}\right)\right)}<\mathbb{M}\left[g_{0,1:\left|T\right|}|{\pi }^{*}\in P\right]$$

(64)

For Proposition 2 and the strong duality of “Problem 2”, the following is implied

$$d_s^2\left(\left\{{\lambda }_i^{2*}, \forall i\in P\right\},\left\{{\zeta }_i^{\prime }, \forall i\in P\right\}\right)=\mathbb{M}\left[g_{0,1:\left|T\right|}|{\pi }^{*}\in P\right]$$

(65)

and

$$\begin{array}{l}\mathbb{M}\left[g_{0,1:\left|T\right|}|{\pi }^{*}\in P\right]>\\ \mathbb{M}\left[g_{0,1:\left|T\right|}|{\pi }_s^{2*}\left({\lambda }_i^{2*}, \forall i\in P\right)\right]-{\displaystyle \sum _{i\in P}\left(\frac{{\zeta }_i^{\prime }}{2}ReLU{\left(g_i\right)}^2|{\pi }_s^{2*}\left({\lambda }_i^{2*}, \forall i\in P\right)\right)}\ge \\ \mathbb{M}\left[g_{0,1:\left|T\right|}|{\pi }_s^{2*}\left({\lambda }_i^{2*}, \forall i\in P\right)\right]+{\displaystyle \sum _{i\in P}\left({\lambda }_i^{2*}{\displaystyle \sum _{t\in T}ReLU\left(g_{it}\right)}|{\pi }_s^{2*}\left({\lambda }_i^{2*}, \forall i\in P\right)\right)}-\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\displaystyle \sum _{i\in P}\left(\frac{{\zeta }_i^{\prime }}{2}ReLU{\left(g_{it}\right)}^2|{\pi }_s^{2*}\left({\lambda }_i^{2*}, \forall i\in P\right)\right)}\end{array}$$

(66)

From this, we can say that ${\pi }_s^{2*}\left({\lambda }_i^{2*},\forall i\in P\right)$ becomes suboptimal for “Problem B” as ${\zeta }_i\to {\zeta }_i^{\prime }, \forall i\in P$.

The above implies that as ${\zeta }_i\to \infty , \forall i\in P$, any infeasible policy for “Problem 2”becomes suboptimal for “Problem B”.

Hence, as ${\zeta }_i\to \infty \forall i\in P$, ${‖M_0\left({\pi }^{*}\right)-M_0\left({\pi }_s^{2*}\right)‖}_2\to 0$. □

6. Numerical Example

In this section, we showcase the implications of our theoretical results. For this, all the computational experiments were conducted with a 78.16 GB hard disk, 12.68 GB RAM, Intel(R) Xeon(R) CPU @ 2.00GHz dual-core processor, Ubuntu operating system, and Python 3.10. For deep learning implementation, feed-forward neural networks were used for both actor and critic with two and six layers, respectively. They were implemented in Pytorch and trained on the Tesla T4 (CUDA Version: 12.0) graphical processing unit (GPU). In this regard, a non-convex, non-differentiable, multi-stage disaster response relief allocation problem [49] with three affected areas was considered as a numerical example, the detailed formulation of which is given in Appendix B. It was assumed that all the affected areas have uniform demand stochasticity, and relief is supplied to them from a capacitated local response center. Dynamic conditional value at risk (DCVaR) is used as a risk-averse stochastic ordering for the stochastic objective. The problem has an upper bound (constraint) on the deprivation level of the third affected area. This is because this affected area is assumed to be relatively more vulnerable to disruption than the rest [49,50]. These disruptions can be in the form of secondary disasters inside the affected areas or logistical blockages hindering relief supply, which will have devastating effects (increase in deprivation from life-sustaining resources) on the affected population. So, as a preparation tactic against these disruptions, deprivation levels are kept low a priori by using the constraint (A18) in Appendix B, the form of which is assumed to be $\left[\mathrm{Re}LU\left(g_{it}^{\prime }\right)|\pi \right]\le 0, \forall i\in P$, where $g_{it}^{\prime }=s_{i\omega \left|T\right|+1}^{\prime }-T_i, \forall i\in P$ (given at the end of Appendix B). Therefore, if we assume the quadratic penalty coefficient to be zero, the classical Lagrangian dual method will be obtained [32,34,35,36]. In this regard, using our formulation, we train the agent to be risk-averse in the face of uncertainty while also requiring it to satisfy the hard constraint (A18) in the Appendix via Algorithm 1. All the Lagrangian multipliers ${\lambda }_i^0\in \mathbb{R}\backslash {\mathbb{R}}^{+}, \forall i\in P$ and quadratic penalty factors ${\zeta }_i^0\in {\mathbb{R}}^{+}, \forall i\in P$ are initialized as “0”. Before every dual ascent (with the dual ascent step size $\Lambda =0.005$), we find the locally optimal solution using a conditionally elicitable risk-averse actor–critic algorithm from [12]. By setting $u_{\zeta }=1.1$, we rapidly increase the quadratic penalty coefficient ${\zeta }_1\in {\mathbb{R}}^{+}$ as the dual ascent iteration number grows. We update ${\lambda }_1\in \mathbb{R}\backslash {\mathbb{R}}^{+}$ and ${\zeta }_1\in {\mathbb{R}}^{+}$ every 25 algorithmic epochs (each epoch has 666,675 MDP episodes) of the risk-averse actor–critic algorithm. As we increase ${\lambda }_1\in \mathbb{R}\backslash {\mathbb{R}}^{+}$ and ${\zeta }_1\in {\mathbb{R}}^{+}$, we see that the CVaR of episodic rewards converges, as shown in Figure 3a–c, and the constraint violations decrease until they vanish, as shown in Figure 4a–c), in the case of all values of risk-aversion confidence levels (0.6, 0.8 and 0.99) considered in our simulations.

We also show constraint violations and convergence curves (with a risk-averse confidence level of 0.99) when we used negative reward scheming during training. Here, the fixed constraint violation penalty factor is multiplied by constraint violation and then added to the reward function [51,52,53]. As we can see in Figure 5, for negative reward scheming, in case of a larger fixed penalty factor of “10⁰”, the constraint violations vanish to zero but, as shown in Figure 6, the CVaR episodic rewards converge to a higher cost (red curve) relative to that of the augmented Lagrangian-based approach (blue curve). This is because we are not optimizing over the fixed penalty factors, which makes these penalty factors not the optimal Lagrangian coefficients of the dual problem. Consequently, the optimal solution of the regularized unconstrained problem is different from that of the primal problem. Also, larger constraint violation factors earlier in training may hinder the RL agent’s exploration leading it to converge to the local optima of primal and/or regularized unconstrained problem. On the other hand, in the case of a lower fixed penalty factor of 10⁻¹, although the reward value converges to lower costs, as shown in Figure 7, constraint violations do not reach zero, as shown in Figure 8. This issue is resolved by our proposed constraint-handling mechanism where at the beginning of the training, the action space is less constrained and the agent can freely learn, but as the training progresses, the constraint violations gradually accentuate with the increase in the Lagrangian multipliers and quadratic penalty factors in (49) and (50), respectively.

7. Discussion

Throughout this work, we have developed a duality theory for risk-averse reinforcement learning problems with non-risk constraints. Specifically, in the case of an arbitrary policy space, the duality gap is zero, and thus, the primal problem is perfectly solvable in its dual domain, where it becomes convex in terms of dual variables. This is significant as it establishes that the constrained problem and the regularized unconstrained problem chase the same Prato optimal front and, therefore, are equivalent. However, despite these theoretical results, these problems may be computationally intractable. This is due to the requirement of the consideration of arbitrary policy in cases in which the evaluation of the dual function becomes practically infeasible. In place of arbitrary policy spaces, linear approximations or neural networks are used, which may cause weak duality or even widen the duality gap. Despite this approximation issue, the theoretical results, the resulting algorithm, and the empirical demonstrations provide a way to solve the constrained risk-averse policy optimization problem without the need to conduct an exhaustive search over the penalty weights (or factors) for constraint violation as was performed in [54,55,56].

Methodical Contribution of This Research

This research develops a model-free feed-back policy optimization method (Algorithm 1), which makes it now possible to optimize non-convex versions of constrained risk-averse multi-stage mathematical programs. This is useful for robust decision-making in high-stakes environments [13]. Previously, only linear or convex versions of these programs weresolvable [8,9,17,42], but now convergence to the optimal solution is possible for non-convex and non-smooth versions as well using our developed model-free method, which we have empirically validated as well in Section 6. In addition, our method can also handle constraints with state variables in them which cannot be handled by existing methods like risk-averse stochastic dual dynamic programming [57].

8. Conclusions

In this work, we have combined two embryonic fields of constrained and risk-averse RL. In this regard, a duality theory is developed for MDPs with non-risk cumulative and instantaneous constraints, and spectral risk objectives. We have established strong duality with certain assumptions on the risk measures given on page 7–9 (related to the derivative and integral of the risk aversion function) and on page 13 (translation invariance of risk measures). The assumptions are not so restrictive in the context of problems considered in dynamic programming literature [5,6,7,23,58]. Based on these theoretical results, we have proposed an augmented Lagrangian-based constraint handling mechanism for risk-averse RL along with its theoretical guarantees and empirical demonstration for its viability. This mechanism can be used with any risk-averse RL algorithm for the solution of risk-averse MDPs with non-risk constraints.

However, our theoretical results do not imply that the proposed approach in Section 5 will always be able to solve all the considered classes of problems. This can be due to the intractability of the dual function, making it impossible to evaluate and/or the fact that our theoretical results are for a general policy space, which is impossible to consider realistically. In place of general policy, exploration usually takes place in linear or neural networks-based parameterized (restricted) (sub-) policy spaces, which makes the problem different from the original one and, consequently, the duality gap might not be zero. Therefore, our future direction of investigation will be to theoretically and/or empirically investigate how to further improve the algorithm’s convergence behavior in case of different parameterizations of the policy space. Lastly, the need for constraint satisfaction might come directly from within the RL algorithm, e.g., in the case of trust region policy optimization [59], where the divergence of the policy is constrained by a threshold. Therefore, strong duality must also be investigated under these constraints.