EPO Dosage Optimization for Anemia Management: Stochastic Control under Uncertainty Using Conditional Value at Risk

Abstract

Due to insufficient endogenous production of erythropoietin, chronic kidney disease patients with anemia are often treated by the administration of recombinant human erythropoietin (EPO). The target of the treatment is to keep the patient’s hemoglobin level within a normal range. While conventional methods for guiding EPO dosing used by clinicians normally rely on a set of rules based on past experiences or retrospective studies, model predictive control (MPC) based dosage optimization is receiving attention recently. The objective of this paper is to incorporate the hemoglobin response model uncertainty into the dosage optimization decision making. Two methods utilizing Conditional Value at Risk (CVaR) are proposed for hemoglobin control in chronic kidney disease under model uncertainty. The first method includes a set-point tracking controller with the addition of CVaR constraints. The second method involves the use of CVaR directly in the cost function of the optimal control problem. The methods are compared to set-point tracking MPC and Zone-tracking MPC through computer simulations. Simulation results demonstrate the benefits of utilizing CVaR in stochastic predictive control for EPO dosage optimization.

1. Introduction

One of the major side effects of chronic kidney disease (CKD) is the reduced ability to produce endogenous erythropoietin (EPO), which is a hormone that regulates the production of red blood cells in the body. When the natural production of EPO drops significantly, these patients suffer from a condition called anemia, which is characterized by a reduced mass of red blood cells and hemoglobin level. Recombinant human erythropoietin has become the standard for treating anemia in chronic kidney disease [1]. Recently, optimization of the EPO dosage has been considered to achieve reduced deviation of patient hemoglobin levels from a normal range (zone), while also reducing drug use and expense [2].

Zone model predictive controllers have become a popular area of research in the biomedical field [3]. They have been used successfully in clinical trials in the control of blood glucose in Type 1 Diabetes Mellitus [4,5]. They have gained significant popularity over set-point tracking model predictive control (MPC) for biomedical applications due to their ability to reject sensor noise, and maintain a higher degree of stability in the presence of large plant–model mismatch [6]. They also seem more appropriate for these applications as many of them do not have a target state, but rather a target zone.

The EPO dosing algorithms that have been developed and tested previously are open-loop devices. The patient will typically have their blood labs completed once every 2–4 weeks, and the EPO dose will be optimized for the time period in between sampling times. The EPO medications are approved manually by a clinician and administered during regular dialysis. It has been recognized by the clinical community that, while low Hb level leads to anemia, too high Hb levels can increase the risk of mortality for the patient [7]. Hence, effective methods are needed to determine the appropriate dose of EPO to maintain the target Hb level.

To develop an automatic decision support system and achieve effective anemia management, some of the first published papers on the use of advanced process control in CKD utilize set-point tracking MPC combined with neural network hemoglobin response modeling and report promising results in clinical trials [8,9,10]. The MPC algorithm in this case was an L1-Norm of the state deviation from the target. In another case [11], a pharmacokinetic and pharmacodynamic model was used to describe the patient’s hemoglobin response and then designed controllers based on Quantitative Feedback Theory. In the small pilot study, these researchers also showed promising results. Modeling of hemoglobin response to EPO administration was also done under uncertainty via a recent method called semi-blind Robust Identification [12]. The work presented in this paper differs significantly from these researchers. One of the more challenging aspects of hemoglobin control is the time-varying nature of the patient’s system dynamics. Patient’s can become EPO resistant [13], as well as have their health improve drastically, if endogenous EPO production increases [14]. To address the time-varying nature of the hemoglobin response, constrained recursive system identification is used. There exists a great deal of uncertainty in the model parameters, and the system is often subjected to acute disturbances, such as infection [15] and blood loss [16].

The aim of this work is to develop an advanced control system to calculate optimal EPO dosages for anemia patients, while considering process uncertainty in the control method. The overall block diagram of the control system is presented in Figure 1. The method uses an autoregressive with exogenous inputs (ARX) model with a model predictive controller. To address the process uncertainty, it is represented in the form of different scenarios and further utilized within a model predictive control formulation, with the use of Conditional Value at Risk (CVaR) technique. CVaR was introduced by Rockafellar and Uryasev and is widely used in the finance industry [17]. CVaR is a popular tool used in many risk management applications. For example, it has been shown to work well in heating, ventilation and air conditioning control systems [18]; portfolio/asset optimization [19]; and quantifying flood damage [20]. However, it has been studied in very few applications related to model predictive control. The existing methods for model predictive control under uncertainty mostly include robust MPC [21] and stochastic MPC based methods [22]. The two methods address the worst-case performance and the expected performance, respectively. CVaR is a method that addresses both of these criteria. In addition, CVaR provides an effective convex approximation of a probabilistic constraint used in stochastic MPC. Thus, in this paper, the applications of CVaR for handling both risk averse performance and the probabilistic constraint are studied.

The rest of the paper is organized as follows. Section 2 presents the hemoglobin response modeling through constrained optimization. Deterministic set-point and zone model predictive control algorithms are introduced in Section 3, for comparison purposes to the CVaR methods introduced later. An overview of Conditional Value at Risk in general is introduced in Section 4, which is a technique for addressing uncertainty. A set-point tracking model predictive controller using CVaR constraints on the upper and lower zone boundaries is then introduced in Section 5 and later compared to the set-point tracking MPC. A second controller, a zone-tracking model predictive controller using CVaR directly in the cost function, is introduced in Section 6 and later compared to the zone-tracking model predictive controller. In Section 7, the controllers are tested in computer simulations. The experimental designs are outlined for a simple ARX model based simulator as well as for a physiological model based simulator, using both Gaussian and uniform uncertainty distributions. Section 8 concludes the paper.

2. Hemoglobin Response Modeling

The system model used in the following derivations has an ARX model structure. This kind of model is one type that can be estimated through classic System Identification methods. The goal of System Identification is to map the response of a system’s output, to that of its input. The internal dynamics of the system are not known or estimated. This type of modeling is also referred to as black box modeling. In this work, the model parameters are obtained through constrained optimization. The model uses a weekly sampling time ($\Delta t=1$ week), the last Hb measurement $y_k$, and several past EPO doses $u_k$ (eight-week history of dosages in this work), to predict the future Hb measurement. The order of the $b\left(z\right)$ polynomial is 8 and was determined empirically [23]. The order of the $a\left(z\right)$ polynomial in the model is set to 1. Using a higher order for the $a\left(z\right)$ polynomial may lead to inaccuracies in the model estimation and predictions because of the large amount of measurement error, mixed with the infrequent measurements typically used in the medical field. It is possible that, with more accurate measurements, a larger number of $a_k$ parameters could yield better modeling results, but this work does not address this aspect.

Iron is necessary for the creation of red blood cells, but the effect of iron levels in the patient are ignored in this model. It is assumed that, if the treating clinician were to maintain stable and adequate iron levels within the patient, the response of the hemoglobin should be mostly related to the administration of EPO, and not related to fluctuating or low values of iron within the patient. If a patient were to have low iron levels, they will likely become EPO resistant which could effect the quality of the current model, should the change (from adequate to low levels) occur quickly. Future work should try to include the iron dynamics within the model, as there certainly are some dynamic interactions between these two systems.

The ARX model structure that maps the EPO administration events to the hemoglobin concentration is

$$y_{k+1}=a_1{y}_k+b_1{u}_k+\cdots +b_8{u}_{k-7}$$

(1)

where the input u represents the EPO doses divided by 5000 international unit (IU), and y represents the hemoglobin concentration in g/dL. Clinical data for an example patient are shown in Figure 2. The hemoglobin measurements (top figure) are shown as the actual measurements recorded in approximately two-week intervals. The EPO doses (bottom figure) are presented as one-week dose totals. Typically, a dialysis patient with late stage renal disease will receive EPO medications 1–3 times per week, depending on dose size. With traditional dosing protocols, the oscillatory behavior observed in the hemoglobin is quite common [24].

To estimate a patient’s hemoglobin response model, the cost function in Equation (2) is used. It is the weighted least squares of the error between model predictions and actual hemoglobin values. The parameter $\lambda$ is used as a weighting value ($0<\lambda <1$), to put a higher priority on newer measurements. The output y represents the hemoglobin measurements.

$$\underset{a_1,b_1,\cdots ,b_8}{min}\sum _{i=1}^{t_f}{\lambda }^{(t_f-i)}{({\widehat{y}}_{k+i}-y_{k+i})}^2$$

(2)

where ${\widehat{y}}_{k+i}$ and $y_{k+i}$ represent the predicted and actual hemoglobin value, respectively. Equation (3) represents the predicted hemoglobin using the historical data and the estimated model parameters ($b_1,\cdots ,b_8,a_1$) for a time horizon of 1 to $t_f$.

$$\left[\begin{array}{c}{\widehat{y}}_{k+1}\\ {\widehat{y}}_{k+2}\\ :\\ {\widehat{y}}_{k+tf}\end{array}\right]=\left[\begin{array}{cccc}{u}_k& \cdots & u_{k-7}& y_k\\ u_{k+1}& \cdots & u_{k-6}& y_{k+1}\\ :& :\\ u_{k+t_f-1}& \cdots & u_{k+t_f-8}& y_{k+t_f-1}\end{array}\right]\left[\begin{array}{c}{b}_1\\ b_2\\ :\\ b_8\\ a_1\end{array}\right]$$

(3)

which can be compactly written as

$$\widehat{Y}=X\theta$$

(4)

where $\theta$ is a vector of the optimized model parameters, Y is the measured hemoglobin, $\widehat{Y}$ is the predicted hemoglobin, and X is a matrix of the EPO dosage and measured hemoglobin concentration. Then, the objective function in Equation (2) can be written as

$$min{\theta }^{T}H\theta +2{\theta }^{T}f+c$$

where $H=X^{T}RX$, $f=-X^{T}RY$, and $c=Y^{T}RY$. The measurement weight in Equation (2) is represented by the weighting matrix R, which is a diagonal matrix of the weighting values $\lambda$ for each measurement.

Recursive modeling has been show to work well in cases where the model parameters are time varying, such as in Type 1 Diabetes Mellitus [25]. Recursive modeling is the process of estimating a new patient model after each updated measurement. In this work, the patient’s model parameters are relearnt after each measurement, based on a moving window of estimation data. In the absence of validation data, constraints are used in the modeling formulation to ensure model stability, as well as to enforce that the model parameters follow patterns that are well described by many patients [26]. The complete constrained ARX (C-ARX) model parameter estimation problem can be represented by Equation (5). Equation (5) assumes that the delay of the system is approximately two weeks. The peak time $b_k$ parameter is shown here with a delay of two weeks (Equation (5f)), and represents the largest contribution to the system of all the $b_k$ parameters. The peak time $b_k$ parameter can be shifted to the third or fourth week by reformulating constraints (Equation (5f,g)). Individual patient hemoglobin response times can vary. Some patients may begin to observe a substantial increase of hemoglobin in as little as two weeks, where some take much longer before a significant effect on the hemoglobin can be observed. The most appropriate way of dealing with the system’s unknown delay is to solve the same problem where the peak time $b_k$ parameter is at the second, third and fourth delay, and then choose the solution with the lowest cost function. Equation (5b) is used as a minimum and maximum range for the model estimates, as almost all the measurements should be within this range. Measurements that exist outside of this range may not be realistic, or may coincide with some abnormal event. Equation (5c) is used to enforce model stability. Equation (5d) is used to enforce a minimum response delay of two weeks. Equation (5e) enforces that the model parameters are always positive, because the response of the system to the inputs is always positive. Equations (5f)–(5i) are defined through experience, and are used to enforce a particular parameter shape learned in [26]. Finally, Equation (5i) is used to ensure the model relies somewhat on the inputs to the system, and was determined through trial and error.

$$\begin{array}{cc}\hfill min& {\theta }^{T}H\theta +2{\theta }^{T}f+c\hfill \end{array}$$

(5a)

$$\begin{array}{cc}\hfill s.t.& 7.0\le {\widehat{y}}_{k+i}\le 15.0i=1,\cdots ,t_f\hfill \end{array}$$

(5b)

$$\begin{array}{c}0.7\le a_1\le 0.99\hfill \end{array}$$

(5c)

$$\begin{array}{c}{b}_1=0\hfill \end{array}$$

(5d)

$$\begin{array}{c}{b}_k\ge 0k=2,\cdots ,8\hfill \end{array}$$

(5e)

$$\begin{array}{c}{b}_2\ge 0.05\hfill \end{array}$$

(5f)

$$\begin{array}{c}{b}_k<0.8b_{k-1}k=3,\cdots ,8\hfill \end{array}$$

(5g)

$$\begin{array}{c}{b}_8<0.005\hfill \end{array}$$

(5h)

$$\begin{array}{c}\sum _{k=1}^8{b}_k\ge 0.1\hfill \end{array}$$

(5i)

The above model is a constrained quadratic optimization problem and it is solved using the quadprog solver in MATLAB.

3. Deterministic Control Formulation: Setpoint MPC and Zone-MPC

The stochastic control model described in later sections is compared to the traditional method of implementing model predictive control (MPC), including a set-point tracking MPC and a zone-tracking MPC. First, the set-point tracking model is presented in Equation (6). The target hemoglobin level is represented by $y_{sp}$. Q is the state tuning parameter. N is the prediction horizon.

$$\begin{array}{cc}\hfill \underset{u}{min}& \frac{1}{2}\sum _{i=1}^{N}Q∥{\widehat{y}}_{k+i}-y_{sp}∥+\frac{1}{2}\sum _{i=0}^{N-2}∥\Delta u_{k+i}∥\hfill \end{array}$$

(6a)

$$\begin{array}{cc}\hfill s.t.& u_L\le u_{k+i}\le u_{H}i=0,\cdots ,N-2\hfill \end{array}$$

(6b)

$$\begin{array}{c}\Delta u_L\le \Delta u_{k+i}\le \Delta u_{H}i=0,\cdots ,N-2\hfill \end{array}$$

(6c)

Figure 3a is an illustration of set-point tracking MPC, where the output is driven to the set-point. Control input chatter is often observed in set-point tracking MPC.

The zone-tracking MPC formulation is presented in Equation (7). Zone control is realized through the constraints set on ${\delta }_{k+i}$ in Equation (7b). Set-point tracking MPC is a special case of zone control, where the zone limits $y_H$ and $y_L$ are equal.

$$\begin{array}{cc}\hfill \underset{u,\delta }{min}& \frac{1}{2}\sum _{i=1}^{N}Q∥{\widehat{y}}_{k+i}-{\delta }_{k+i}∥+\frac{1}{2}\sum _{i=0}^{N-2}∥\Delta u_{k+i}∥\hfill \end{array}$$

(7a)

$$\begin{array}{cc}\hfill s.t.& y_L\le {\delta }_{k+i}\le y_{H}i=1,\cdots ,N\hfill \end{array}$$

(7b)

$$\begin{array}{c}{u}_L\le u_{k+i}\le u_{H}i=0,\cdots ,N-2\hfill \end{array}$$

(7c)

$$\begin{array}{c}\Delta u_L\le \Delta u_{k+i}\le \Delta u_{H}i=0,\cdots ,N-2\hfill \end{array}$$

(7d)

In zone-MPC, it is common to see the output oscillate slightly between the upper and lower limits while the system stabilizes to a steady state within the control zone limits. Figure 3b is an illustration of zone-MPC.

It is worth pointing out that the cost function can be modeled using either absolute value or a quadratic function for both controllers presented above.

4. Conditional Value at Risk

The uncertainty considered in this work is in the form of process uncertainty on the output and can be represented by the variable $w_k$ in Equation (8).

$${\tilde{y}}_{k+1}=a_1{y}_k+b_1{u}_k+\cdots +b_8{u}_{k-7}+w_k$$

(8)

To address the uncertainty in the hemoglobin response to EPO dosage, this work studies two Conditional Value at Risk (CVaR) techniques for control under uncertainty. In general, Value at Risk (VaR) at confidence level $\beta$ ($Va{R}_{\beta }$) is defined as the maximum loss value that is assigned to some desired probability, $\beta$. That is to mean that a loss less than or equal to $Va{R}_{\beta }$ will occur $\beta \times 100$% of the time. What $Va{R}_{\beta }$ fail to account for, is the losses that occur $(1-\beta )\times 100\%$ of the time with a loss larger than $Va{R}_{\beta }$. CVaR at confidence level $\beta$ ($CVa{R}_{\beta }$) aims to minimize this loss and it is defined as the average of the loss exceeding $Va{R}_{\beta }$. For instance, a slightly higher $Va{R}_{\beta }$ may be tolerated, if the average value of the $\beta$-tail distribution is lower. This would ensure that when a loss greater than $Va{R}_{\beta }$ does occur, it will likely be smaller. A diagram is included in Figure 4 to facilitate a better understanding of the value in which CVaR seeks to minimize.

Another major advantage of using CVaR constraints over other types of chance constraints, is that the uncertainty can follow any distribution. With a known distribution, different scenarios are generated for each sampling instant by sampling the random distribution. In this fashion, the controller optimization problem can take into account multiple scenarios, which help to approximate the different possibilities of measurement and process uncertainties, to ultimately provide a more robust solution.

Next, the CVaR technique is incorporated into the MPC formulation in two different ways. The first method handles chance constraints through CVaR approximation, whereas the second method directly optimizes CVaR in the cost function. Both controllers were tested in computer simulations against traditional zone-MPC and set-point tracking MPC to show the benefits that the two methods may provide.

5. Stochastic Control Using CVaR Constraints

The uncertainty is located within the output, ${\tilde{y}}_k$, in the form of a random variable $w_k$. $w_k$ is assumed to follow a certain distribution. The controller is designed to take into account a process uncertainty from a known distribution. The optimization problem uses two chance constraints (Equations (9b) and (9c)) on the output of the system, one on the lower bound and the other one on the upper bound. Due to the conservative approximation of the chance constraint that the CVaR constraint results in, they will only be used for $k=3$, as shown below. If more constraints are used, the problem will become infeasible. $k=2$ is not used because $b_1=0$, which means that, if the hemoglobin were to fall outside of the zone, the controller would only have a single input to try and move the hemoglobin into the zone to avoid infeasibility. The result would be a very aggressive control move in these scenarios that should be avoided. It should also be noted that the cost function is computed with Equation (1), where ${\widehat{y}}_k$ represents the predicted hemoglobin without uncertainty. Chance constraints are used in solving optimization problems under uncertainty. There exists uncertainty in the output prediction ${\tilde{y}}_k$, which is computed through Equation (8). The meaning of the chance constraint is that utilizing some knowledge of the uncertainty in ${\tilde{y}}_k$, the constraint will be infeasible no more than $ϵ$ amount of the time. $ϵ$ is the probability of violation. The chance constrained set-point MPC formulation is proposed as following:

$$\begin{array}{cc}\hfill min& Q\sum _{k=1}^N|{\widehat{y}}_k-y_{sp}|+\sum _{k=0}^{N-2}|\Delta u_k|\hfill \end{array}$$

(9a)

$$\begin{array}{cc}\hfill s.t.& Pr\left\{{\tilde{y}}_k>y_H\right\}\le ϵk=3\hfill \end{array}$$

(9b)

$$\begin{array}{c}Pr\left\{{\tilde{y}}_k<y_L\right\}\le ϵk=3\hfill \end{array}$$

(9c)

$$\begin{array}{c}{u}_L\le u_{k+i}\le u_{H}i=0,\cdots ,N-2\hfill \end{array}$$

(9d)

$$\begin{array}{c}\Delta u_L\le \Delta u_{k+i}\le \Delta u_{H}i=0,\cdots ,N-2\hfill \end{array}$$

(9e)

Due to the conservatism of the CVaR constraints, only a single point along the prediction horizon will utilize the constraints. Note that the $\Delta u_k$ term is summed from 0 to $N-2$ because $b_1$ is always estimated as zero in the ARX model. The following derivations use a prediction horizon, N, of eight weeks, and the ARX model introduced previously. Equation (10) is used to demonstrate the derivation, which is the upper bound constraint [17].

$$Pr\left\{{\tilde{y}}_k>y_H\right\}\le ϵ$$

(10)

In the derivation below, the indicator function $1_{(0,\infty )}\left(u\right)$ is used. This function holds a value of 1 if $u>0$ and a value of 0 if $u\le 0$. For any positive parameter $\alpha$, we have $1_{(0,\infty )}\left(u\right)=1_{(0,\infty )}\left(\frac{1}{\alpha }u\right)$. Defining an upper bounding function $\varphi \left(u\right)$ for the indicator function, the following inequality can be written.

$$1_{(0,\infty )}\left(\frac{1}{\alpha }u\right)\le \varphi \left(\frac{1}{\alpha }u\right)$$

(11)

Replacing u with ${\tilde{y}}_k-y_H$, the following inequality can be written.

$$Pr\left\{{\tilde{y}}_k-y_H>0\right\}=\mathrm{E}\left[1_{(0,\infty )}({\tilde{y}}_k-y_H)\right]\le \mathrm{E}\left[\varphi \left(\frac{1}{\alpha }({\tilde{y}}_k-y_H)\right)\right]\le ϵ$$

(12)

If ${\tilde{y}}_k-y_H>0$, the constraint is violated and the indicator function holds a value of 1, and vice versa. $\varphi \left(u\right)$ is the upperbounding function for the indicator function and it is a conservative estimate of the probability of violation, $ϵ$. From this, any solution that satisfies the inequality will also satisfy the constraint. The focus remains on the right side of this inequality. (u)${}^{+}$ is defined as the maximum operator, which holds a maximum value between 0 and the input u. Applying the upper bounding function $\varphi \left(u\right)={(u+1)}^{+}$, a conservative approximation of the chance constraint is defined as

$$\mathrm{E}\left[\begin{array}{c}{(\frac{1}{\alpha }({\tilde{y}}_k-y_H)+1)}^{+}\end{array}\right]\le ϵ$$

(13)

multiply by $\frac{\alpha }{ϵ}$

$$\frac{1}{ϵ}\mathrm{E}\left[\begin{array}{c}{(({\tilde{y}}_k-y_H)+\alpha )}^{+}\end{array}\right]\le \alpha$$

(14)

moving $\alpha$ to the left hand side

$$-\alpha +\frac{1}{ϵ}\mathrm{E}\left[\begin{array}{c}{(({\tilde{y}}_k-y_H)+\alpha )}^{+}\end{array}\right]\le 0$$

(15)

To reduce conservatism, it is necessary to find the minimum $\alpha$ that satisfies the inequality. That is, it is desired to find the smallest upperbound. This is a CVaR constraint, but it is difficult to calculate its value in its present form.

$$\underset{\alpha }{min}-\alpha +\frac{1}{ϵ}\mathrm{E}\left[\begin{array}{c}{(({\tilde{y}}_k-y_H)+\alpha )}^{+}\end{array}\right]\le 0$$

(16)

The expectation operator is removed with the introduction of sampling to approximate the CVaR constraint. M is the total number of scenarios. For instance, if there were a single random variable per scenario, and the prediction horizon was 1, it would be necessary to sample the random variables’ distribution M times. ${\pi }_j$ is the probability that a single scenario will occur, and is equal to $\frac{1}{M}$. Note that $\tilde{y}$ changes to $y_{k,j}$ because the output is now an expected value with the uncertainty added in.

$$-\alpha +\frac{1}{ϵ}\sum _{j=1}^M{\pi }_j\left[\begin{array}{c}{((y_{k,j}-y_H)+\alpha )}^{+}\end{array}\right]\le 0$$

(17)

The expression inside of the max operator can be replaced by two separate constraints. Equation (18) replaces the max operator term with the variable $v_{j,k}$, and constraints are set on $v_{j,k}$ to satisfy the original max operator’s functionality. The combination of all three constraints in Equation (18) can be used to provide a conservative estimate of the original chance constraint in Equation (10) through the use of sampling [18]. These constraints are solved using many different scenarios of the random variable distribution, leading to it being part of a class of controllers called Scenario-Based MPC.

$$\begin{array}{cc}-{\alpha }_k+\frac{1}{ϵ}\sum _{j=1}^M{\pi }_j{v}_{j,k}\le 0\hfill & \hfill k=3\end{array}$$

(18a)

$$\begin{array}{cc}{v}_{j,k}\ge (y_{k,j}-y_H)+{\alpha }_k\hfill & \hfill j=1,...,M,k=3\end{array}$$

(18b)

$$\begin{array}{cc}{v}_{j,k}\ge 0\hfill & \hfill j=1,...,M,k=3\end{array}$$

(18c)

Similarly, for the lower bound constraints, we have the following approximation

$$\begin{array}{cc}-{\gamma }_k+\frac{1}{ϵ}\sum _{j=1}^M{\pi }_j{\omega }_{j,k}\le 0\hfill & \hfill k=3\end{array}$$

(19a)

$$\begin{array}{cc}{\omega }_{j,k}\ge (y_L-y_{k,j})+{\gamma }_k\hfill & \hfill j=1,...,M,k=3\end{array}$$

(19b)

$$\begin{array}{cc}{\omega }_{j,k}\ge 0\hfill & \hfill j=1,...,M,k=3\end{array}$$

(19c)

Finally, notice that the cost function in Equation (9) contains absolute terms. This equation can be manipulated to remove the absolute sign and the resulting linear program with all the constraints is outlined in Equation (20).

$$\begin{array}{cc}\hfill min& Q\sum _{k=1}^N{h}_k+\sum _{k=0}^{N-2}{g}_k\hfill \end{array}$$

(20a)

$$\begin{array}{ccc}\hfill s.t.& h_k\ge {\widehat{y}}_k-y_{sp}\hfill & \hfill k=1,\cdots ,N\end{array}$$

(20b)

$$\begin{array}{cc}{h}_k\ge -({\widehat{y}}_k-y_{sp})\hfill & \hfill k=1,\cdots ,N\end{array}$$

(20c)

$$\begin{array}{cc}{g}_k\ge \Delta u_k\hfill & \hfill k=0,\cdots ,N-2\end{array}$$

(20d)

$$\begin{array}{cc}{g}_k\ge -\Delta u_k\hfill & \hfill k=0,\cdots ,N-2\end{array}$$

(20e)

where ${\widehat{y}}_k$ is the Hb prediction from the ARX model without process uncertainty term.

Summarizing the above derivations, the complete deterministic approximation to the chance constrained set-point MPC formulation is given as follows:

$$\begin{array}{cc}\hfill min& Q\sum _{k=1}^8{h}_k+\sum _{k=0}^6{g}_k\hfill \end{array}$$

(21a)

$$\begin{array}{ccc}\hfill s.t.& -{\alpha }_k+\frac{1}{ϵ}\sum _{j=1}^M{\pi }_j{v}_{j,k}\le 0\hfill & \hfill k=3\end{array}$$

(21b)

$$\begin{array}{cc}{v}_{j,k}\ge (y_{k,j}-y_H)+{\alpha }_k\hfill & \hfill j=1,...,M,k=3\end{array}$$

(21c)

$$\begin{array}{cc}{v}_{j,k}\ge 0\hfill & \hfill j=1,...,M,k=3\end{array}$$

(21d)

$$\begin{array}{cc}-{\gamma }_k+\frac{1}{ϵ}\sum _{j=1}^M{\pi }_j{\omega }_{j,k}\le 0\hfill & \hfill k=3\end{array}$$

(21e)

$$\begin{array}{cc}{\omega }_{j,k}\ge (y_L-y_{k,j})+{\gamma }_k\hfill & \hfill j=1,...,M,k=3\end{array}$$

(21f)

$$\begin{array}{cc}{\omega }_{j,k}\ge 0\hfill & \hfill j=1,...,M,k=3\end{array}$$

(21g)

$$\begin{array}{cc}{h}_k\ge {\widehat{y}}_k-y_{sp}\hfill & \hfill k=1,\cdots ,N\end{array}$$

(21h)

$$\begin{array}{cc}{h}_k\ge -({\widehat{y}}_k-y_{sp})\hfill & \hfill k=1,\cdots ,N\end{array}$$

(21i)

$$\begin{array}{cc}{g}_k\ge \Delta u_k\hfill & \hfill k=0,\cdots ,N-2\end{array}$$

(21j)

$$\begin{array}{cc}{g}_k\ge -\Delta u_K\hfill & \hfill k=0,\cdots ,N-2\end{array}$$

(21k)

$$\begin{array}{cc}{u}_L\le u_k\le u_H\hfill & \hfill k=0,\cdots ,N-2\end{array}$$

(21l)

$$\begin{array}{cc}\Delta u_L\le \Delta u_k\le \Delta u_H\hfill & \hfill k=0,\cdots ,N-2\end{array}$$

(21m)

There are $4M+76$ constraints and the optimization vector consists of $2M+34$ variables. The controller optimization problem involves the use of M scenarios, where M typically holds a value larger than 100 and is selected by the user. A single scenario is simply the trajectory of the hemoglobin response while including uncertainty values drawn randomly from the known uncertainty distribution. It should quickly become apparent that scenario-based constraints such as these lead to a very large number of constraints, with $4M+2$ constraints related to the CVaR constraints alone. With this in mind, the cost function used in this design is a linear program and MATLAB’s linprog function was used along with its dual-simplex optimization algorithm. This controller is referred to as CVaR${}_1$ below.

6. Stochastic Control Using a CVaR Cost Function

The second method discussed in this work optimizes the control inputs using conditional value at risk directly within the cost function. In contrast to the first method, the solution to this optimization problem is always feasible. The controller derivation follows the work of [17,19] closely, but with the exception that the algorithm is modified to use a zone-MPC formulation. The optimal solution can be thought of as the solution that minimizes the average loss that occurs in the $\beta$-tail distribution of the probability density function (pdf) of the loss function. The controller formulation in Equation (22) uses the distribution of the random variable, $w_k$, to generate several different plausible scenarios of process uncertainty.

$$\begin{array}{cc}\hfill \underset{u,\delta }{min}& CVa{R}_{\beta }f(u,\delta ,w)\hfill \end{array}$$

(22a)

$$\begin{array}{ccc}\hfill s.t.& y_L\le {\delta }_{k+i}\le y_H\hfill & \hfill i=1,\cdots ,N\end{array}$$

(22b)

$$\begin{array}{cc}{u}_L\le u_{k+i}\le u_H\hfill & \hfill i=0,\cdots ,N-2\end{array}$$

(22c)

$$\begin{array}{cc}\Delta u_L\le \Delta u_{k+i}\le \Delta u_H\hfill & \hfill i=0,\cdots ,N-2\end{array}$$

(22d)

where the cost function is given as

$$f(u,\delta ,w)=Q\sum _{i=1}^N|{\tilde{y}}_{k+i}-{\delta }_{k+i}|+\sum _{i=0}^{N-2}|\Delta u_{k+i}|$$

(23)

The loss function is associated with decision variables $u_k$ and auxiliary variables ${\delta }_k$ and random variables $w_k$. Note that Q is the tuning parameter in this equation and zone control is facilitated through the use of the variable ${\delta }_k$. The conditional expectation of loss in the bad tail leads to the idea of Conditional Value at Risk. $CVa{R}_{\beta }$ is defined as the conditional expectation of the loss above ${\ell }_{\beta }$

$$CVa{R}_{\beta }f={\varphi }_{\beta }=\frac{1}{1-\beta }{\int }_{f\ge {\ell }_{\beta }}f(u,\delta ,w)p\left(w\right)dw$$

(24)

Rockafellar and Uryasev [17] showed that the $CVa{R}_{\beta }$ can be determined as follows

$${\varphi }_{\beta }=\underset{\ell }{min}\ell +\frac{1}{1-\beta }E{\left[f(u,\delta ,w)-\ell \right]}^{+}=\underset{\ell }{min}\ell +\frac{1}{1-\beta }{\int }_w{[f(u,\delta ,w)-\ell ]}^{+}p\left(w\right)dw$$

(25)

${\left[f\right]}^{+}$ denotes the max of $[f,0]$. ℓ is an auxiliary variable (can be viewed as $VA{R}_{\beta }$) to be optimized. The integral can be approximated by sampling of the random variable: $w_1,\cdots ,w_M$ with probability ${\pi }_1,\cdots ,{\pi }_M$

$${\varphi }_{\beta }=\underset{\ell }{min}\ell +\frac{1}{1-\beta }\sum _{j=1}^M{\pi }_j{[f(u,\delta ,w_j)-\ell ]}^{+}$$

(26)

The absolute terms from Equation (23) can be linearized by introducing variables $h_k$ and $g_k$. Note that $y_{k+1+i,j}$ represents the expected output for the scenario.

$$\begin{array}{cc}\hfill \underset{u_k,{\delta }_k,\ell ,v_j}{min}& Q\sum _{i=1}^N{h}_{k+i,j}+\sum _{i=0}^{N-2}{g}_{k+i}\hfill \end{array}$$

(27a)

$$\begin{array}{ccc}\hfill s.t.& h_{k+i,j}\ge y_{k+1+i,j}-{\delta }_{k+i,j}\hfill & \hfill i=1,\cdots ,N,j=1,\cdots ,M\end{array}$$

(27b)

$$\begin{array}{cc}{h}_{k+i,j}\ge -(y_{k+1+i,j}-{\delta }_{k+i,j})\hfill & \hfill i=1,\cdots ,N,j=1,\cdots ,M\end{array}$$

(27c)

$$\begin{array}{cc}{g}_{k+i}\ge \Delta u_{k+i}\hfill & \hfill i=0,\cdots ,N-2\end{array}$$

(27d)

$$\begin{array}{cc}{g}_{k+i}\ge -\Delta u_{k+i}\hfill & \hfill i=0,\cdots ,N-2\end{array}$$

(27e)

Then, by combining Equations (26) and (27), the CVaR cost can be written as

$$\underset{u_k,{\delta }_k,\ell }{min}\ell +{(1-\beta )}^{-1}\sum _{j=1}^M{\pi }_j{\left[Q\sum _{i=1}^N{h}_{k+i,j}+\sum _{i=0}^{N-2}{g}_{k+i}-\ell \right]}^{+}$$

(28)

This is equivalent to

$$\begin{array}{cc}\hfill \underset{u_k,{\delta }_k,\ell ,v_j}{min}& \ell +{(1-\beta )}^{-1}\sum _{j=1}^M{\pi }_j{v}_j\hfill \end{array}$$

(29a)

$$\begin{array}{ccc}\hfill s.t.& v_j\ge Q\sum _{i=0}^{N-1}{h}_{k+i,j}+\sum _{i=0}^{N-2}{g}_{k+i}-\ell \hfill & \hfill j=1,\cdots ,M\end{array}$$

(29b)

$$\begin{array}{cc}{v}_j\ge 0\hfill & \hfill j=1,\cdots ,M\end{array}$$

(29c)

The variable $v_j$ is introduced to allow the calculation of the max operator. The final form of the stochastic MPC problem is a linear program outlined in Equation (30) that minimizes the $CVa{R}_{\beta }$, where $k,j,i$ represent the sampling instant, scenario number and sampling instant along the prediction horizon, respectively.

$$\begin{array}{cc}\hfill \underset{u_k,{\delta }_k,\ell ,v_j}{min}& \ell +{(1-p)}^{-1}\sum _{j=1}^M{\pi }_j{v}_j\hfill \end{array}$$

(30a)

$$\begin{array}{ccc}\hfill s.t.& v_j\ge Q\sum _{i=0}^{N-1}{h}_{k+i,j}+\sum _{i=0}^{N-2}{g}_{k+i}-\ell \hfill & \hfill j=1,\cdots ,M\end{array}$$

(30b)

$$\begin{array}{cc}{v}_j\ge 0\hfill & \hfill j=1,\cdots ,M\end{array}$$

(30c)

$$\begin{array}{cc}{h}_{k+i,j}\ge y_{k+1+i,j}-{\delta }_{k+i,j}\hfill & \hfill i=1,\cdots ,N,j=1,\cdots ,M\end{array}$$

(30d)

$$\begin{array}{cc}{h}_{k+i,j}\ge -(y_{k+1+i,j}-{\delta }_{k+i,j})\hfill & \hfill i=1,\cdots ,N,j=1,\cdots ,M\end{array}$$

(30e)

$$\begin{array}{cc}{g}_{k+i}\ge \Delta u_{k+i}\hfill & \hfill i=0,\cdots ,N-2\end{array}$$

(30f)

$$\begin{array}{cc}{g}_{k+i}\ge -\Delta u_{k+i}\hfill & \hfill i=0,\cdots ,N-2\end{array}$$

(30g)

$$\begin{array}{cc}{y}_L\le {\delta }_{k+i,j}\le y_H\hfill & \hfill i=1,\cdots ,N,j=1,\cdots ,M\end{array}$$

(30h)

$$\begin{array}{cc}{u}_L\le u_{k+i}\le u_H\hfill & \hfill i=0,\cdots ,N-2\end{array}$$

(30i)

$$\begin{array}{cc}\Delta u_L\le \Delta u_{k+i}\le \Delta u_H\hfill & \hfill i=0,\cdots ,N-2\end{array}$$

(30j)

The above Equation results in $4MN+2M+6N$ constraints and an optimization vector length of $2MN+2N+M+1$. This controller is referred to as CVaR${}_2$ below.

7. Computer Simulation Results

The following section outlines the experimental design and the simulation results comparing the CVaR methods to set-point tracking MPC and zone-MPC for both Gaussian and uniform distributions, for both an ARX model based patient simulator and a pharmacokinetics and pharmacodynamics (PK/PD) model based simulator.

Both simulator cases use the same Gaussian and uniform distributions for the random variable $w_k$. The histograms are outlined in Figure 5. Figure 5a follows a Gaussian distribution of $N(0,0.2^2)$, while Figure 5b is a uniform distribution of $-0.4\le w_k\le 0.4$. In all cases, the controllers that use the CVaR technique were given the exact distribution of $w_k$ to calculate the scenarios for each controller iteration. For these simulations, each scenario would draw random variables for each sampling time in the future that is used in the controller design. For instance, in the first CVaR controller, the uncertainty is only needed up to $k+3$ for each sampling instant, meaning there are only three values drawn from the uncertainty distribution for each scenario. In the second CVaR controller, it draws the full eight weeks of values for each scenario.

The performance of the controllers are measured based on four metrics. The state performance metrics are the integrated output error (IOE) outside of the control zone, and the percent of points in the zone (PIZ). The input metrics are average weekly EPO dose (EPO/week) and the average weekly change in dose (avg $\Delta EPO$). The four performance metrics can be calculated using Equation (31).

$$\begin{array}{c}IOE=\sum _{k=1}^{tf}{f}_k,f_k=\left\{\begin{array}{c}Hg{b}_k-y_H,ifHg{b}_k>y_H\hfill \\ y_L-Hg{b}_k,ifHg{b}_k<y_L\hfill \\ 0,if{y}_L\le Hg{b}_k\le y_H\hfill \end{array}\right.\hfill \end{array}$$

(31a)

$$\begin{array}{c}PIZ=100\sum _{k=1}^{tf}\frac{c_k}{tf},c_k=\left\{\begin{array}{c}1,if{y}_L\le Hg{b}_k\le y_H\hfill \\ 0,if{y}_L>Hg{b}_{k}orHg{b}_k>y_H\hfill \end{array}\right.\hfill \end{array}$$

(31b)

$$\begin{array}{c}EPO/week=\sum _{k=1}^{tf}\frac{EP{O}_k}{tf}\hfill \end{array}$$

(31c)

$$\begin{array}{c}Avg\Delta EPO=\sum _{k=2}^{tf}\frac{|EP{O}_k-EP{O}_{k-1}|}{tf}\hfill \end{array}$$

(31d)

7.1. Test under An ARX Model Based Simulator

The ARX model based simulations were performed without time-varying parameters and recursive modeling. These simulations represent a nominal case, where the patient model does not change. A single constrained-ARX model was used. A simulation time of 2000 weeks was used, to obtain better statistical power of the results. A sampling time of one week was used. The controller tuning parameters are shown in

Table 1. Controller settings.

Controller Identifier	MPC	Zone-MPC	CVaR${}_1$	CVaR${}_2$
Q	0.8	10	0.8	0.3
M	-	-	500	500
$\Delta u_H$ (IU)	20,000	20,000	20,000	20,000
$\beta$	-	-	-	0.95
$ϵ$	-	-	0.1	-
$y_L$	10.5	10	10.5	10
$y_H$	10.5	11	10.5	11
Target	10.5	10–11	10.5	10–11
Constraint Limits	-	-	9.5–11.5	-

. The tuning parameters were defined empirically, to limit the Average Change in EPO parameters to be low. In practice, EPO is often administered in 1000 IU increments, so the controllers were tuned to have the average weekly change less than this value.

The results for the Gaussian distribution simulation are outlined in

Table 2. Simulation results under Gaussian distribution.

Performance Statistic	MPC	Zone-MPC	CVaR${}_1$	CVaR${}_2$
IOE ($\frac{g}{dLweek}$)	112.1	174.3	113.2	108.0
PIZ (%)	74.2	64.8	74.0	74.0
EPO/week (IU)	5449	5450	5450	5448
Avg $\Delta EPO$	603	489	938	541
Time per Iteration (sec)	0.005	0.005	0.091	0.588

. The CVaR formulations offer significant improvements over traditional zone-MPC, but their performance is only marginally better than set-point tracking MPC. The time to solve is included in the table. Due to the large number of added variables introduced for sampling in the CVaR controllers, the time per iteration also increases.

The results for the uniform distribution are outlined in

Table 3. Setpoint MPC and CVaR${}_1$ model performance under uniform distribution.

Performance Statistic	MPC	CVaR${}_1$
IOE ($\frac{g}{dLweek}$)	146.7	115.5
PIZ (%)	69.9	73.4
EPO/week (IU)	5435	5438
Avg $\Delta EPO$	701	1413

and

Table 4. Zone MPC and CVaR${}_2$ model performance under uniform distribution.

Performance Statistic	Zone-MPC	CVaR${}_2$
IOE ($\frac{g}{dLweek}$)	189.2	140.8
PIZ (%)	65.3	71.0
EPO/week (IU)	5433	5435
Avg $\Delta EPO$	592	698

. CVaR${}_1$ has a large increase in the state performance over set-point tracking MPC, but it comes at the cost of an aggressive control action. The overall statistics show the controller to be more aggressive, but in fact it is actually quite stable with regards to its control input action.

Figure 6 depicts a small window of the full 2000 week simulation results comparing MPC to CVaR${}_1$. CVaR${}_1$ behaves similar to zone-MPC, but the controller will be extra aggressive outside of the zone boundary as the hemoglobin approaches the constraint boundaries. The overall aggressiveness of the controller is much larger due to these larger moves, even though the doses are typically more stable. Figure 7 compares CVaR${}_2$ to zone-MPC. As compared to zone-MPC, the response is much better, as the CVaR is able to pre-emptively change the dose in anticipation of the hemoglobin leaving the zone, whereas zone-MPC only reacts to the hemoglobin leaving the zone, causing it to traverse beyond the zone boundaries more often.

7.2. Test under PK/PD Model Based Simulator

For this test, the patient simulator shown in Figure 8 was used which was designed in detail in [23]. The patient simulator represents a more realistic scenario to test the controllers with. The patient simulator uses a system of nonlinear delayed differential equations (DDE) based on pharmacokinetics and pharmacodynamics to model hemoglobin response.

The DDE model was proposed by Chait et al. [11]. Pharmacokinetics is the study of the movement of drugs within the body while pharmacodynamics is related to the mechanisms by which the drugs affect the body. The PK/PD model is described below.

$$\begin{array}{c}{E}_{en}=\frac{C{H}_{en}}{\mu K_{H}S-H_{en}}\hfill \end{array}$$

(32a)

$$\begin{array}{c}\frac{dE\left(t\right)}{dt}=\frac{-VE\left(t\right)}{K_m+E\left(t\right)}-\alpha E\left(t\right)+dose\left(t\right)\hfill \end{array}$$

(32b)

$$\begin{array}{c}\frac{dR\left(t\right)}{dt}=\frac{S(E_{en}+E(t-D))}{(C+E_{en}+E(t-D))}-4\frac{x_1\left(t\right)}{{\mu }^2}\hfill \end{array}$$

(32c)

$$\begin{array}{c}\frac{d{x}_1\left(t\right)}{dt}=x_2\left(t\right)\hfill \end{array}$$

(32d)

$$\begin{array}{c}\frac{d{x}_2\left(t\right)}{dt}=\frac{S(E_{en}+E(t-D))}{(C+E_{en}+E(t-D))}-4\frac{x_1\left(t\right)}{{\mu }^2}-4\frac{x_2\left(t\right)}{\mu }\hfill \end{array}$$

(32e)

In Equation (32), states $E\left(t\right)$ and $R\left(t\right)$ represent the pool of exogenous erythropoietin and the population of red blood cells (RBC) within the body, respectively. States $x_1\left(t\right)$ and $x_2\left(t\right)$ are internal states that aid in calculating $R\left(t\right)$. $E_{en}$ is the endogenous erythropoietin naturally produced by the body. $K_H$ is the average amount of hemoglobin per RBC (also known as the mean corpuscular hemoglobin, MCH). The value used here is fixed at 29.5 pg/cell, which is within the reference range of 27–33 pg/cell [11]. The hemoglobin value is directly proportionate to the RBC population; the hemoglobin value can be attained by multiplying the RBC population estimate by the MCH value. The function $dose\left(t\right)$ is a train of impulses, representing the EPO injections. The model has an initial condition as represented in Equation (33) and requires two measurements of hemoglobin ($H{b}_{1}andH{b}_2$) and the time in between those measurements ($t_{1}and{t}_2$) to estimate. The prior history of the exogenous EPO is unknown, and assumed to be zero for all $t\le 0$.

$$\begin{array}{c}\dot{R_0}=\frac{(H{b}_2-H{b}_1)}{K_H(t_2-t_1)}\hfill \end{array}$$

(33a)

$$\begin{array}{c}E\left(0\right)=0\hfill \end{array}$$

(33b)

$$\begin{array}{c}R\left(0\right)=\frac{H{b}_1}{K_H}\hfill \end{array}$$

(33c)

$$\begin{array}{c}{x}_1\left(0\right)=\frac{\mu (H_{en}-\mu K_H\dot{R_0})}{4K_H}\hfill \end{array}$$

(33d)

$$\begin{array}{c}{x}_2\left(0\right)=R\left(0\right)-\frac{4x_1\left(0\right)}{\mu }\hfill \end{array}$$

(33e)

Table 5. Parameters for the PK/PD Model.

Parameter	Description
$H_{en}$	Hemoglobin Level due to Endogenous Erythropoietin
$\mu$	Mean RBC life span
V	Maximal Exogenous Erythropoietin clearance rate
$K_m$	Exogenous Erythropoietin level that produces half maximal clearance rate
$\alpha$	Linear clearance constant
S	Maximal RBC production rate stimulated by $E_P$
C	Amount of EPO that produces half maximal RBC production rate
D	Time required for EPO-stimulated RBCs to start forming

contains parameters that are estimated for an individual patient using clinical data and nonlinear least-squares regression. The dde23 and lsqnonlin functions in MATLAB were used for DDE solution and parameter estimation, respectively.

The DDE model is solved in continuous time and sampled every two weeks. The output of the DDE model is the red blood cell population ($R_k$), which is multiplied by the mean corpuscular hemoglobin ($K_H$) to get the hemoglobin measurement. The hemoglobin measurement is then added to an integrating process uncertainty ($w_k$). It is important to note the difference between the ARX model simulator and PK/PD model simulator in regards to the way $w_k$ is used. In the ARX simulator case, $w_k$ is filtered with the patient model. In the PK/PD simulator case, the disturbance is integrating. The magnitudes are drawn from the same distributions in both cases.

The simulator also includes the possibility of acute step and ramp disturbances to simulate infections and blood losses frequently observed in actual patient data. Acute disturbances are facilitated through the use of the variable $A_D$, which holds a value of one unless a disturbance has occurred. If an acute disturbance occurs, it will hold a value less than 1, and the existing pool of red blood cells in the patient simulator model is multiplied by the fractional value to reduce it. A step disturbance is then modeled by an impulse in $A_D$, while an infection is modeled by a step function in $A_D$. Plant–model mismatch exists as the controller model is linear. The patient simulator was shown to mimic the real life hemoglobin system dynamics of patients well [23]. The total simulation time was 500 weeks. Recursive modeling is necessary to capture the time-varying nature of the system model. Although the patient model does not change significantly from sampling interval to sampling interval, the patient model is still re-estimated to provide continuity between the models at each sampling interval. A hemoglobin sampling time of two weeks is used, and two weekly dose total inputs solved by the controller are used during each sampling interval. The recursive modeling method applies a data re-sampling algorithm to get one week measurements (through linear interpolation), and weekly dose totals. The one-week re-sampled hemoglobin history is also smoothed, using a five-point moving average filter, before modeling occurs. The weighting parameter $\lambda$ is set to 0.965. The tuning parameters for each of the controllers are outlined in

Table 6. Controller settings.

Controller Identifier	MPC	Zone-MPC	CVaR${}_1$	CVaR${}_2$
Q	3	10	3	1.5
M	-	-	500	500
$\Delta u_H$ (IU)	20,000	20,000	20,000	20,000
$u_H$ (IU)	30,000	30,000	30,000	30,000
$\beta$	-	-	-	0.95
$ϵ$	-	-	0.3	-
$y_L$	11	10	10	10
$y_H$	11	12	12	12
Target	11	10–12	10–12	10–12
Constraint Limits	-	-	10–12	-

. The tuning parameters were again defined empirically. For these simulations, the Avg $\Delta EPO$ was tuned to be approximately equal in most cases to better compare the controller results to one other.

For reference, more information about the actual disturbances is shown in Figure 9 and Figure 10. The top figures are the summation of the disturbance over the course of the simulation. The center figures are the magnitude of the disturbance at each point. The third figures are the acute disturbances that occur during the simulation, which are the same in both cases. In the simulation, the acute disturbances include seven blood loss events, and three infection events in both cases. The PK/PD parameters used in the simulator are as follows: $\alpha =0.2718$, $C=22.45$, $D=6.33$, $H_{en}=7.81$, $K_m=76.05$, $\mu =91.54$, $S=0.00554$, and $V=1655$.

The results for the Gaussian uncertainty case are presented in

Table 7. Simulation results under Gaussian distribution.

Performance Statistic	MPC	Zone-MPC	CVaR${}_1$	CVaR${}_2$
IOE ($\frac{g}{dLweek}$)	25.99	30.47	25.04	39.05
PIZ (%)	80.0	76.5	81.3	80.0
EPO/week (IU)	20,044	20,059	20,399	19,795
Avg $\Delta EPO$	1333	805	1267	1343

. The four controllers’ performance statistics are shown in Figure 11 for a portion of the full 500-week simulation. The tuning parameters were chosen to try and keep the controller average $\Delta EPO$ approximately equal, so it is easier to compare the state performance between the controllers. However, the zone-MPC controller is unable to reach the same level of aggressiveness as the other three without becoming relatively unstable. Using the CVaR constraints improves the performance drastically over zone-MPC, and offers modest improvements over traditional MPC. Using CVaR in the cost function, offers similar state performance to that of set-point tracking MPC, with the same average control action. Using CVaR in the cost function allows the medication dosing to remain stable when the hemoglobin is within the zone. As it nears the borders, the controller will typically make large moves in the medication dose. Overall, the medication dose remains very stable, even though the average $\Delta EPO$ does not portray this feature in the statistics.

For the uniform distribution case, the overall results are shown in

Table 8. Simulation results under uniform distribution.

Performance Statistic	MPC	Zone-MPC	CVaR${}_1$	CVaR${}_2$
IOE ($\frac{g}{dLweek}$)	44.58	55.12	40.7	45.10
PIZ (%)	70.4	65.2	77.8	71.7
EPO/week (IU)	20,429	20,548	20,793	20,779
Avg $\Delta EPO$	1223	838	1236	1289

. A short window of the overall simulation can be seen in Figure 12. The CVaR constraints allow the controller to significantly improve the PIZ metric over the other controllers, while remaining with a similar average controller aggressiveness. Using CVaR in the cost function has an average change in dose similar to MPC and $CVa{R}_1$, but, looking at Figure 12, it is again easy to see that the controller rarely changes the medication dose, but, when it does so, it is typically a significant move.

8. Conclusions

This work explores the use of Conditional Value at Risk (CVaR) in addressing process uncertainty in hemoglobin level control in chronic kidney disease patients. CVaR is able to utilize any type of uncertainty distribution and does not rely on Gaussian assumptions. Two methods of utilizing CVaR were introduced, one where CVaR constraints are used, and the other where the cost function is the CVaR. Simulations were performed under both Gaussian and uniform type of uncertainty distributions. A simple ARX based and a more complex PK/PD based patient simulator were used in the control performance test. Simulation results show that adding CVaR constraints to the setpoint tracking MPC problem increased the state performance over all of the other controllers. Using CVaR directly in the cost function resulted in improvements over traditional zone-MPC. Moreover, it also offers the benefit of having the medication doses remain more stable over the course of the treatment, which may be a highly sought after feature of a control method by clinicians. Based on these results, using Conditional Value at Risk is a promising tool in improving control performance in hemoglobin concentration control and EPO dosage optimization. Finally, this work has only focused on the process uncertainty and the model parameter uncertainty is skipped, which will be investigated in future work.