Computing Stackelberg Equilibrium for Cancer Treatment

Abstract

Recent work by Kleshnina et al. has presented a Stackelberg evolutionary game model in which the Stackelberg equilibrium strategy for the leading player corresponds to the optimal cancer treatment. We present an approach that is able to quickly and accurately solve the model presented in that work.

1. Introduction and Problem Formulation

In this section we review the Stackelberg evolutionary game dynamic model of cancer evolution previously studied [1]. There are two players: a follower and a leader. The leader is a physician who selects amounts of two different drugs to use for therapy, $m_1$ and $m_2.$ The follower is a cancer population consisting of three cell types: 0-type denotes a cell that does not develop resistance to either drug, 1-type cells are resistant to just drug 1, and 2-type cells are resistant to just drug 2. The follower selects a population size for each type, denoted $x_0,x_1,x_2$, as well as a trait for each type, denoted $u_0,u_1,u_2.$ It is assumed that each of these variables are nonnegative, with the $u_i\in [0,1].$ It is also assumed that they are all implicitly functions of time $t.$ Note that $u_0$ does not appear anywhere in the analysis so can be ignored.

For each cell type i, there is a fitness function $G_i(u_i,\mathbf{m},\mathbf{x})$ that the follower is trying to maximize. We assume that the dynamics of the population $\mathbf{x}$ are governed by

$$\dot{\mathbf{x}}=\mathbf{G}\left(t\right)\mathbf{x}.$$

In order to ensure that we are in equilibrium of the ecological dynamics we must have that $\dot{x_i}=0$ for $i=0,1,2.$ Thus, the follower is selecting $u_1$ that maximizes $G_1$, $u_2$ that maximizes $G_2$, and $x_0,x_1,x_2$ that ensure equilibrium of the ecological dynamics. The leader, knowing that the follower will subsequently select their actions in this way, selects $m_1,m_2$ to maximize a quality of life function $Q(\mathbf{m},\mathbf{u},\mathbf{x}).$

Thus, we can formulate the problem of determining the optimal strategies for both players as follows:

$$\begin{array}{ccc}\hfill & \hfill \underset{{\mathbf{m}}^{*},{\mathbf{u}}^{*},{\mathbf{x}}^{*}}{max}& Q({\mathbf{m}}^{*},{\mathbf{u}}^{*},{\mathbf{x}}^{*})\hfill \\ \hfill & \hfill \mathrm{s}.\mathrm{t}.& {\dot{x}}_i^{*}=0 \mathrm{for} i=0,1,2\hfill \\ \hfill & \hfill & u_i^{*}\in {arg\; max}_{u_i}{G}_i(u_i,{\mathbf{m}}^{*},{\mathbf{x}}^{*}) \mathrm{for} i=1,2\hfill \\ \hfill & \hfill & {\mathbf{m}}^{*}\ge 0,{\mathbf{x}}^{*}\ge 0,0\le {\mathbf{u}}^{*}\le 1\hfill \end{array}$$

(1)

This general model is instantiated by the following functional forms for the fitness functions $G_0,G_1,G_2$ and quality of life function Q. Note that the model presentation is slightly different between the paper [1] and the implementation in the code repository [2]. We will be using the model presented in the code.

$$G_0=r_{\max }\left(1-{\displaystyle \frac{{\alpha }_{00}{x}_0+{\alpha }_{01}{x}_1+{\alpha }_{02}{x}_2}{K}}\right)-d-{\displaystyle \frac{m_1}{k_1}}-{\displaystyle \frac{m_2}{k_2}}$$

$$G_1=r_{\max }{e}^{-g_1{u}_1}\left(1-{\displaystyle \frac{{\alpha }_{10}{x}_0+{\alpha }_{11}{x}_1+{\alpha }_{12}{x}_2}{K}}\right)-d-{\displaystyle \frac{m_1}{b_1{u}_1+k_1}}-{\displaystyle \frac{m_2}{k_2}}$$

$$G_2=r_{\max }{e}^{-g_2{u}_2}\left(1-{\displaystyle \frac{{\alpha }_{20}{x}_0+{\alpha }_{21}{x}_1+{\alpha }_{22}{x}_2}{K}}\right)-d-{\displaystyle \frac{m_1}{k_1}}-{\displaystyle \frac{m_2}{b_2{u}_2+k_2}}$$

$$Q=Q^{\max }-c{\left({\displaystyle \frac{x_0+x_1+x_2}{K}}\right)}^2-w_1{m}_1^2-w_2{m}_2^2-r_1{u}_1^2-r_2{u}_2^2$$

The model has several parameters, whose interpretations are summarized in

Table 1. Interpretations of model parameters.

Parameter	Interpretation
$r_{\max }$	Max cell growth rate
$g_i$	Cost of resistance strategy (cell type) i
${\alpha }_{ij}$	Interaction coefficient between cell types i and j
K	Carrying capacity
d	Natural death rate
$k_i$	Innate resistance that may be present before drug exposure
$b_i$	Benefit of the evolved resistance trait in reducing therapy efficacy
$Q^{\max }$	Quality of life of a healthy patient
$w_i$	Toxicity of drug i
$r_i$	Effect of resistance rate of cell type i
c	Weight for impact of tumor burden vs. drug toxicities and resistance rates

. Note that in the code additional parameters $a_0,a_1,a_2,a_3$ are defined, with

$$\left[\begin{array}{ccc}{\alpha }_{00}& {\alpha }_{01}& {\alpha }_{02}\\ {\alpha }_{10}& {\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{20}& {\alpha }_{21}& {\alpha }_{22}\end{array}\right]=\left[\begin{array}{ccc}{a}_0& a_1& a_1\\ a_2& a_0& a_3\\ a_2& a_3& a_0\end{array}\right]$$

2. Prior Approach

In this section we present the approach described in Github repository created by the authors [2]. They first solve for expressions for $x_i$ in terms of $u_i,m_i$ such that the condition for equilibrium of the ecological dynamics is satisfied. This involves solving a system of three equations $G_i{x}_i=0$ with unknowns $x_1,x_2,x_3.$ They calculate the following analytical solution:

$$x_0^{*}={\displaystyle \frac{K(X_{01}+X_{02}+X_{03})}{r_{\max }(a_0-a_3)(a_0^2-2a_1{a}_2+a_0{a}_3)}},$$

where $X_{01},X_{02},X_{03}$ denote the following quantities:

$$X_{01}=\left(a_0^2-a_3^2\right)\left(-d-{\displaystyle \frac{m_1}{k_1}}-{\displaystyle \frac{m_2}{k_2+r_{\max }}}\right)$$

$$X_{02}=a_1(a_3-a_0)e^{g_1{u}_1}\left(-d-{\displaystyle \frac{m_2}{k_2}}+e^{-g_1{u}_1}{r}_{\max }-{\displaystyle \frac{m_1}{k_1+b_1{u}_1}}\right)$$

$$X_{03}=a_1(a_3-a_0)e^{g_2{u}_2}\left(-d-{\displaystyle \frac{m_1}{k_1}}+e^{-g_2{u}_2}{r}_{\max }-{\displaystyle \frac{m_2}{k_2+b_2{u}_2}}\right)$$

$$x_1^{*}={\displaystyle \frac{K(X_{11}+X_{12}+X_{13})}{r_{\max }(a_0-a_3)(a_0^2-2a_1{a}_2+a_0{a}_3)}}$$

$$X_{11}=a_2(a_3-a_0)\left(-d-{\displaystyle \frac{m_1}{k_1}}-{\displaystyle \frac{m_2}{k_2}}+r_{\max }\right)$$

$$X_{12}=(a_0^2-a_1{a}_2)e^{g_1{u}_1}\left(-d-{\displaystyle \frac{m_2}{k_2}}+e^{-g_1{u}_1}{r}_{\max }-{\displaystyle \frac{m_1}{k_1+b_1{u}_1}}\right)$$

$$X_{13}=(a_1{a}_2-a_0{a}_3)e^{g_2{u}_2}\left(-d-{\displaystyle \frac{m_1}{k_1}}+e^{-g_2{u}_2}{r}_{\max }-{\displaystyle \frac{m_2}{k_2+b_2{u}_2}}\right)$$

$$x_2^{*}={\displaystyle \frac{K(X_{21}+X_{22}+X_{23})}{r_{\max }(a_0-a_3)(a_0^2-2a_1{a}_2+a_0{a}_3)}}$$

$$X_{21}=a_2(a_3-a_0)\left(-d-{\displaystyle \frac{m_1}{k_1}}-{\displaystyle \frac{m_2}{k_2}}+r_{\max }\right)$$

$$X_{22}=(a_1{a}_2-a_0{a}_3)e^{g_1{u}_1}\left(-d-{\displaystyle \frac{m_2}{k_2}}+e^{-g_1{u}_1}{r}_{\max }-{\displaystyle \frac{m_1}{k_1+b_1{u}_1}}\right)$$

$$X_{23}=(a_0^2-a_1{a}_2)e^{g_2{u}_2}\left(-d-{\displaystyle \frac{m_1}{k_1}}+e^{-g_2{u}_2}{r}_{\max }-{\displaystyle \frac{m_2}{k_2+b_2{u}_2}}\right)$$

While it is not given in the model formulation, the code assumes that $m_1$ and $m_2$ fall in the interval $[\underline{m},\overline{m}],$ with $\underline{m}=0,$ $\overline{m}=1.$ They discretize the space so that $m_1$ and $m_2$ are multiples of $h=0.1,$ and iterate over all combinations, of which there are ${\displaystyle \frac{1}{h^2}}.$ For each combination of values they calculate the optimal values of $u_1,u_2$ as follows. They assume that the optimal value of $u_1$ is where ${\displaystyle \frac{\partial G_1}{\partial u_1}}=0$, and the optimal value of $u_2$ is where ${\displaystyle \frac{\partial G_2}{\partial u_2}}=0$. They calculate the following expressions for the partial derivatives:

$${\displaystyle \frac{\partial G_1}{\partial u_1}}=-g_1{r}_{\max }{e}^{-g_1{u}_1}\left(1-{\displaystyle \frac{{\alpha }_{10}{x}_0+{\alpha }_{11}{x}_1+{\alpha }_{12}{x}_2}{K}}\right)+{\displaystyle \frac{m_1{b}_1}{{(k_1+b_1{u}_1)}^2}}$$

$${\displaystyle \frac{\partial G_2}{\partial u_2}}=-g_2{r}_{\max }{e}^{-g_2{u}_2}\left(1-{\displaystyle \frac{{\alpha }_{10}{x}_0+{\alpha }_{11}{x}_1+{\alpha }_{12}{x}_2}{K}}\right)+{\displaystyle \frac{m_2{b}_2}{{(k_2+b_2{u}_2)}^2}}$$

To find values of $u_1,u_2$ for which these derivatives equal zero, they perform an optimization to minimize the quantity $|{\displaystyle \frac{\partial G_1}{\partial u_1}}| + |{\displaystyle \frac{\partial G_2}{\partial u_2}}|.$ They perform this optimization in Matlab using the fminsearchbnd function, using bounds $0\le u_1\le 1,$ $0\le u_2\le 1.$ Note that the above expressions for $x_i^{*}$ which are in terms of $m_i$ and $u_i$ are substituted into the expressions for ${\displaystyle \frac{\partial G_i}{\partial u_i}},$ so that $u_i$ is the only variable in the objective. This procedure results in optimal values $u_1^{*}(m_1,m_2),u_2^{*}(m_1,m_2)$ for each combination of values for $(m_1,m_2).$ The values for $x_0^{*},x_1^{*},x_2^{*}$ are determined by these values.

Finally, they iterate over all ${\displaystyle \frac{1}{h^2}}$ of these combinations of values to determine which one maximizes the value of $Q.$ This determines the optimal values of $m_1^{*},m_2^{*}$ for the leader, which in turn determine the optimal strategies for the follower. The overall procedure is summarized in Algorithm 1.

Algorithm 1 Prior approach

for $m_1=0$ to 1 by increments of h do        for $m_2=0$ to 1 by increments of h do              Calculate values of $u_1,u_2$ that minimize $|{\displaystyle \frac{\partial G_1}{\partial u_1}} |+| {\displaystyle \frac{\partial G_2}{\partial u_2}}|,$ where $0\le u_i\le 1$ and the $x_i$ satisfy the equations for the equilibrium of the ecological dynamics.        end for  end for  Calculate the values for $(m_1,m_2)$ out of the ${\displaystyle \frac{1}{h^2}}$ possibilities for which the corresponding optimal variables maximize the value of $Q.$

3. Limitations of Prior Approach

The prior approach has several significant limitations. The first is that it does not check that the calculated values for $x_i^{*}$ for given values of $u_i$ and $m_i$ are biologically sensible. Since the $x_i$ correspond to populations, they must be nonnegative.

Another limitation is that the coarse discretization of values for $m_i$ means that only a small number of possibilities are considered. This also means that the running time will potentially be large, since we must perform ${\displaystyle \frac{1}{h^2}}$ separate optimizations.

Another significant limitation is that it is assumed that $G_i$ is maximized when ${\displaystyle \frac{\partial G_i}{\partial u_i}}=0,$ and the boundary cases when it is maximized at $u_i=0$ or 1 are ignored.

A final limitation is that the procedure invokes the fminsearch algorithm in Matlab, which is not even guaranteed to find a local minimum, let alone a global minimum.

4. New Approach

We now describe our new approach that addresses the limitations of the prior approach. We will formulate a single quadratic program that corresponds to the full optimization problem and solve it using Gurobi’s nonconvex MIQCP solver which has a guarantee of global optimality (subject to numerical precision).

First we have the main decision variables $x_i,u_i,m_i$ with $x_i\ge 0,$ $m_i\ge 0$, $0\le u_i\le 1.$ The objective function Q is a quadratic function of these variables. Next we encode the conditions for equilibrium of the ecological dynamics. We must define several auxiliary variables to do this.

First define ${\eta }_i=g_i{u}_i,$ and ${\tau }_i=e^{{\eta }_i}$ for $i=1,2.$ For the latter, we use Gurobi’s addGenConstrExp function that uses a piecewise linear approximation for the exponential function. We set these variables to be nonnegative. To provide tighter upper bounds we can set ${\eta }_i\le g_i$, ${\tau }_i\le e^{g_i},$ since $u_i\le 1.$ We next define the auxiliary variable ${\gamma }_i={\displaystyle \frac{m_i}{k_i+b{u}_i}}.$ We can do this by including the quadratic constraint $k_i{\gamma }_i+b{u}_i{\gamma }_i-m_i=0$ for $i=1,2.$ Using these variables we can now encode the conditions for equilibrium of ecological dynamics using constraints that are quadratic in the variables.

Next we must encode the conditions that $u_i$ is a maximizer of $G_i.$ To do this we define several additional auxiliary variables. We define ${\sigma }_i=e^{-g_i{u}_i}$ by adding in the constraint ${\sigma }_i{\tau }_i=1.$ Next we define ${\beta }_i={\displaystyle \frac{1}{{(k_i+b_i{u}_i)}^2}}.$ We can do this by including the quadratic constraint ${\beta }_i-b_i^2{u}_i^2-2k_i{b}_i{u}_i-k_i^2=0.$ Finally we define ${\omega }_i={\displaystyle \frac{m_i}{{(k_i+b_i{u}_i)}^2}}$ by including the quadratic constraint ${\omega }_i{\beta }_i-m_i=0.$ Using these variables, we can now encode the expressions for ${\displaystyle \frac{\partial G_i}{\partial u_i}}$ that are quadratic in the variables.

Recall that we are trying to select $u_i\in [0,1]$ to maximize $G_i,$ for $i=1,2.$ We can do this by introducing two Lagrange multipliers ${\lambda }_{i1}\ge 0,$ ${\lambda }_{i2}\ge 0.$ Then the KKT optimality condition is equivalent to the following three constraints:

$${\displaystyle \frac{\partial G_i}{\partial u_i}}\left(u_i\right)+{\lambda }_{i1}-{\lambda }_{i2}=0$$

$${\lambda }_{i1}(u_i-0)=0$$

$${\lambda }_{i2}(u_i-1)=0$$

These constraints are all quadratic in the variables and ensure that we find $u_i\in [0,1]$ that maximizes $G_i$ regardless of whether it is at the boundary or at an interior solution with the derivative equal to zero.

Our full formulation is given below. Here the $X_{ij}$ correspond to the same quantities as before and are just defined to simplify presentation, not as new variables.

$$\begin{array}{ccc}& \hfill \underset{\mathbf{m},\mathbf{u},\mathbf{x}}{max}& Q^{\max }-{\displaystyle \frac{c(x_0^2+x_1^2+x_2^2+2x_0{x}_1+2x_0{x}_2+2x_1{x}_2)}{K^2}}-w_1{m}_1^2-w_2{m}_2^2-r_1{u}_1^2-r_2{u}_2^2\hfill \\ & \hfill \mathrm{s}.\mathrm{t}.& x_0={\displaystyle \frac{K(X_{01}+X_{02}+X_{03})}{r_{\max }(a_0-a_3)(a_0^2-2a_1{a}_2+a_0{a}_3)}}\hfill \\ & & X_{01}=\left(a_0^2-a_3^2\right)\left(-d-{\displaystyle \frac{m_1}{k_1}}-{\displaystyle \frac{m_2}{k_2}}+r_{\max }\right)\hfill \\ & & X_{02}=a_1(a_3-a_0)\left(-{\tau }_{1}d-{\displaystyle \frac{{\tau }_1{m}_2}{k_2}}+r_{\max }-{\tau }_1{\gamma }_1\right)\hfill \\ & & X_{03}=a_1(a_3-a_0)\left(-{\tau }_{2}d-{\displaystyle \frac{{\tau }_2{m}_1}{k_1}}+r_{\max }-{\tau }_2{\gamma }_2\right)\hfill \\ & & x_1={\displaystyle \frac{K(X_{11}+X_{12}+X_{13})}{r_{\max }(a_0-a_3)(a_0^2-2a_1{a}_2+a_0{a}_3)}}\hfill \\ & & X_{11}=a_2(a_3-a_0)\left(-d-{\displaystyle \frac{m_1}{k_1}}-{\displaystyle \frac{m_2}{k_2}}+r_{\max }\right)\hfill \\ & & X_{12}=(a_0^2-a_1{a}_2)\left(-{\tau }_{1}d-{\displaystyle \frac{{\tau }_1{m}_2}{k_2}}+r_{\max }-{\tau }_1{\gamma }_1\right)\hfill \\ & & X_{13}=(a_1{a}_2-a_0{a}_3)\left(-{\tau }_{2}d-{\displaystyle \frac{{\tau }_2{m}_1}{k_1}}+r_{\max }-{\tau }_2{\gamma }_2\right)\hfill \\ & & x_2={\displaystyle \frac{K(X_{21}+X_{22}+X_{23})}{r_{\max }(a_0-a_3)(a_0^2-2a_1{a}_2+a_0{a}_3)}}\hfill \\ & & X_{21}=a_2(a_3-a_0)\left(-d-{\displaystyle \frac{m_1}{k_1}}-{\displaystyle \frac{m_2}{k_2}}+r_{\max }\right)\hfill \\ & & X_{22}=(a_1{a}_2-a_0{a}_3)\left(-{\tau }_{1}d-{\displaystyle \frac{{\tau }_1{m}_2}{k_2}}+r_{\max }-{\tau }_1{\gamma }_1\right)\hfill \\ & & X_{23}=(a_0^2-a_1{a}_2)\left(-{\tau }_{2}d-{\displaystyle \frac{{\tau }_2{m}_1}{k_1}}+r_{\max }-{\tau }_2{\gamma }_2\right)\hfill \\ & & -g_i{r}_{\max }{\sigma }_i\left(1-{\displaystyle \frac{{\alpha }_{i0}{x}_0+{\alpha }_{i1}{x}_1+{\alpha }_{i2}{x}_2}{K}}\right)+b_i{\omega }_i+{\lambda }_{i1}-{\lambda }_{i2}=0 \mathrm{for} i=1,2\hfill \\ & & u_i{\lambda }_{i1}=0 \mathrm{for} i=1,2\hfill \\ & & u_i{\lambda }_{i2}-{\lambda }_{i2}=0 \mathrm{for} i=1,2\hfill \\ & & {\eta }_i=g_i{u}_i \mathrm{for} i=1,2\hfill \\ & & {\tau }_i=e^{{\eta }_i} \mathrm{for} i=1,2\hfill \\ & & k_i{\gamma }_i+b{u}_i{\gamma }_i-m_i=0 \mathrm{for} i=1,2\hfill \\ & & {\sigma }_i{\tau }_i=1 \mathrm{for} i=1,2\hfill \\ & & {\beta }_i-b_i^2{u}_i^2-2k_i{b}_i{u}_i-k_i^2=0 \mathrm{for} i=1,2\hfill \\ & & {\omega }_i{\beta }_i-m_i=0 \mathrm{for} i=1,2\hfill \\ & & m_i\ge 0 \mathrm{for} i=1,2\hfill \\ & & x_i\ge 0 \mathrm{for} i=1,2\hfill \\ & & 0\le u_i\le 1 \mathrm{for} i=1,2,3\hfill \\ & & 0\le {\eta }_i\le g_i \mathrm{for} i=1,2\hfill \\ & & 0\le {\tau }_i\le e^{g_i} \mathrm{for} i=1,2\hfill \\ & & {\gamma }_i\ge 0 \mathrm{for} i=1,2\hfill \\ & & {\sigma }_i\ge 0 \mathrm{for} i=1,2\hfill \\ & & {\beta }_i\ge 0 \mathrm{for} i=1,2\hfill \\ & & {\omega }_i\ge 0 \mathrm{for} i=1,2\hfill \\ & & {\lambda }_{ij}\ge 0 \mathrm{for} i=1,2 \mathrm{and} j=1,2\hfill \end{array}$$

This formulation addresses the limitations of the prior approach. It ensures that all quantities are biologically relevant by imposing nonnegativity constraints on corresponding variables. It allows $m_i$ to take on arbitrary nonnegative values, not a small set of discretized values. It involves solving a single optimization problem instead of ${\displaystyle \frac{1}{h^2}}$ separate optimization problems. It uses KKT conditions to ensure that the values of $u_i$ that maximize $G_i$ are found regardless of whether they are interior or boundary solutions. And the approach guarantees finding a global optimum since that is guaranteed by Gurobi’s nonconvex MIQCP solver.

5. Experiments

We ran experiments with both approaches on a problem instance using the same parameter values as the prior approach [2], which are provided in

Table 2. Parameter values used in experiments.

Parameter	Value
$r_{\max }$	0.45
$g_1$	0.5
$g_2$	0.5
$a_0$	1
$a_1$	0.15
$a_2$	0.9
$a_3$	0.9
K	10,000
d	0.01
$k_1$	5
$k_2$	5
$b_1$	10
$b_2$	10
$Q^{\max }$	1
$w_1$	0.5
$w_2$	0.2
$r_1$	0.4
$r_2$	0.4
c	0.5

. All experiments were done on a single core of a laptop using Windows 11. Experiments with the prior approach used Matlab version 24.1.0.2689473 (R2024a) Update 6 [3], and experiments with the new approach were done using Gurobi version 11.03 [4] with Java version 14.0.2. For the optimizations in the prior approach, Matlab’s function fminsearchbnd was called using parameters TolX = $1\times {10}^{-12}$, MaxFunEvals = $1000.$ The results are shown in

Table 3. Experimental results for both approaches.

	Prior Approach	New Approach
$m_1^{*}$	0.4	0.40837
$m_2^{*}$	0.5	0.46579
$u_1^{*}$	0.19015	0.21361
$u_2^{*}$	0.3123	0.28554
$x_0^{*}$	5634.3774	5749.8474
$x_1^{*}$	−360.2658	1.3366
$x_2^{*}$	1316.2683	950.5000
$Q^{*}$	0.59936	0.59780
Running time (seconds)	282.69	1.65

. We can see that the prior approach found a solution with a negative value for $x_1^{*},$ which is not biologically sensible. The prior approach took nearly five minutes while our new approach took less than two seconds.

6. Conclusions

We presented a new approach for computing Stackelberg equilibrium strategies in a Stackelberg evolutionary game dynamic model of cancer evolution previously studied. Our approach is based on solving a new quadratic program formulation. We noted several limitations of the approach used by prior work which are addressed by our approach. When we compared the approaches on a sample instance our approach ran significantly faster and the prior approach output a solution that is not biologically relevant. As more complex game-theoretic and optimization models are being formulated for problems in biology and cancer treatment in particular, it is important to develop efficient algorithms for accurately solving them. While we focused on one instantiation presented in prior work, our approach is applicable more generally to computing optimal strategies in Stackelberg evolutionary games.