A Markovian Mechanism of Proportional Resource Allocation in the Incentive Model as a Dynamic Stochastic Inverse Stackelberg Game

Abstract

This paper considers resource allocation among producers (agents) in the case where the Principal knows nothing about their cost functions while the agents have Markovian awareness about his/her strategies. We use a dynamic setup of the stochastic inverse Stackelberg game as the model. We suggest an algorithm for solving this game based on Q-learning. The associated Bellman equations contain functions of one variable for the Principal and also for the agents. The new results are illustrated by numerical examples.

1. Introduction

Stackelberg games date back to the monograph [1]. The original setup includes two players, Leader (Principal) and Follower (Agent). The Leader makes the first move by choosing his/her strategy and informing the Follower of it. After that, the Follower seeks for an optimal response by maximizing his/her payoff function. There are Stackelberg games, in which the Leader has a constant strategy, and inverse Stackelberg games, in which the Leader’s strategy is a function of the Follower’s actions (feedback control mechanism). Stackelberg games with several Leaders and/or Followers are considered later. Inverse Stackelberg games in the static and dynamic setups are discussed in the surveys [2,3].

Inverse Stackelberg games provide a mathematical formalization for the incentive problem. The Principal designs a feedback control mechanism stimulating the agents to choose actions that are most beneficial for him/her. In this paper, we suggest a method for solving such problems in the dynamic setup under incomplete information about the agents’ behavior.

Below the problem is stated so that the associated Bellman equations contain functions of one variable for the Principal and also for the agents. In addition, the agents become independent players as soon as they receive the information from the Principal. The latter chooses his/her behavior based on the response of the players. The agents make their decisions by predicting the Principal’s behavior. An elementary implementation of this principle is the Markov principle, i.e., at a time t the Principal uses the agents’ response at the time (t − 1) to design his/her behavior. In turn, the agents observe the behavioral history of the Principal in order to predict his/her choice at the time (t + 1). The equilibrium calculation method for the Stackelberg game relies on online learning (more specifically, the Q-learning procedure) and recursive statistical estimation. The Q-learning procedure is mostlyused for solving the statistical dynamic programming problem, with the calculation of the Q-function. As a rule, the Q-function depends on two variables, the phase and control variables; so, the Q-function is defined on the Cartesian product of finite sets. The method has slow convergence, and the rate of convergence essentially depends on the number of elements in the definitional domain of the arguments. Therefore, the Stackelberg game in one of the two setups considered below is stated so that the resulting Q-function depends on a single argument. In the other setup, the slow Q-learning procedure is replaced by a faster one-dimensional maximization algorithm for a concave function of one variable. In both setups, the agents involve recursive statistics. Thus, in comparison to the standard Q-learning procedure, the suggested algorithms are expected to guarantee a faster convergence to the equilibrium of this game.

The main contribution of this paper is as follows:

• We have developed fast converging algorithms to calculate the solution of the dynamic inverse Stackelberg game without sufficient information about the agents.
• The suggested algorithms can be considered as numerical methods for solving the corresponding static inverse Stackelberg game without sufficient information about the payoff functions of the agents.
• This game-theoretic model has been applied for optimal resource allocation among producers in the case of insufficient information about their cost functions.

The remainder of this paper is organized accordingly. In Section 2, we provide a survey of the existing publications on this subject. In Section 3, we discuss the static incentive problem as an inverse Stackelberg game and also describe its dynamic extension in two setups. In Section 4, we present the results of calculations using numerical examples for both setups of the game. Finally, in Section 5, we give some concluding remarks and outline for future research.

2. Related Work

Resource allocation mechanisms in the static setup are studied in contract theory [4] and also in control of organizational systems [5]. Here, the main concern of the investigators is to design strategy-proof mechanisms [6,7]. More specifically, by assumption, the Principal does not know the exact characteristics of the agents and the latter can use this fact for strategic manipulation (information distortion for their own benefit), see [8]. A possible way to eliminate manipulation was described in the book [5]; the author suggested a strategy-proof direct resource allocation mechanism. Such an approach relies on the hypothesis that the optimal control is obtained by solving the static inverse Stackelberg game in which the Principal knows the goals of all agents. The paper [9] considered the discrete-time incentive model with Markovian dynamics and discounted payoff function on the infinite planning horizon. As demonstrated here, the approximate Stackelberg solution can be found by solving an optimal control problem with the difference between the controller’s income and executor’s cost as the optimality criterion.

Static and dynamic inverse Stackelberg games were surveyed in [2,3,10]. Also, note [11,12,13,14,15] among the earlier publications. The paper [16] considered a game-theoretic setup of incentive problems. Linear-quadratic inverse Stackelberg games were studied in [17,18]. Applications of these models were described in [19,20,21,22]. The authors [23] provided a survey of hierarchical games with application to marketing.

In particular importance, previous research suggested no common methods for solving inverse Stackelberg games. Meanwhile, the paper [24] proved the theorem on the ε-optimal guaranteeing strategy in the static inverse Stackelberg game, which reduces the maximin problem with bound variables to nonlinear programming problems with independent variables; as a result, calculations were considerably simplified. In [25,26], this approach was extended to dynamic inverse Stackelberg games. The corresponding theorem actually reduces constrained maximin calculation over complex functional spaces to the calculation of multiple maximins over finite-dimensional spaces.

The authors [27] analyzed a dynamic modification of the proportional resource allocation mechanism. Their suggested approach to the game-theoretic modeling of resource allocation in the hierarchical Principal-agent system possesses the following features.

(1)

Resource dynamics are explicitly described as the phase variable depending on the Principal’s control. The control function can be non-differentiable; in this case, the dynamic equation is interpreted in terms of the Lebesgue-Stieltjes integral.

(2)

The Principal’s control has smooth variations, which is formalized using the Lipschitz property of the control function. This assumption seems natural for the majority of real organizational and economic systems.

(3)

The Principal allocates resources among the agents proportionally to their actions, which stimulates the latter to choose more intensive plans.

(4)

This hypothesis is used to develop a genetic algorithm for calculating the Principal’s optimal strategy with a non-uniform partition of the time interval.

Evolutionary modeling and genetic algorithms were described in the monographs [28,29]. The paper [30] presented a hybrid learning procedure for artificial neural networks. The authors [31] proposed a genetic algorithm for solving the Germeier game with one Follower and the control function that satisfies the Lipschitz condition. Genetic algorithms for solving Stackelberg games were also considered in [32,33].

This paper is focused on the case in which the Principal has insufficient information about the goals of all agents while the latter does not know the Principal’s strategy for the whole duration of the game, merely its history (without loss of generality, the awareness structure will be considered Markovian). The problem statement involves statistical estimation and reinforcement learning, including the Q-learning [34,35]. Reinforcement learning was used for calculating Stackelberg equilibria in [36,37]. Particularly, as noted in [36], this algorithm has instability in the case of several Followers, especially if the latter calculates the Nash equilibrium for solving their problems. In this paper, we suggest a Stackelberg game-based model, for which there exists a stable algorithm.

3. Model

3.1. Static Setup and Dynamic Generalization

Consider a single Principal and M agents controlled by him/her. The static incentive model as an inverse Stackelberg game has the form

$$\begin{array}{l}\psi \left(x\right)-{\displaystyle \sum _{i=1}^M{\varphi }_i(x,\gamma )}\to \max \\ {\varphi }_i\left(x,\gamma \right)-f_i\left(x\right)\to \max ,i=1,\dots ,M\end{array}$$

Here $\psi (x)$ denotes the Principal’s income, a concave increasing function such that $\psi (0)=0$; $f_i(x)$ is the cost of agent i, a convex increasing function such that $f_i(0)=0$; ${\varphi }_i(x,\gamma )$ gives the incentive of agent i, i.e., the compensation of his/her cost by the Principal; i = 1, …, M; finally, $\gamma$ indicates the Principal’s control strategy. Then the optimal incentive mechanism has the form [5]

$${\varphi }_i^{*}(x,\gamma )=\{\begin{array}{l}{f}_i(x_i^{*},x_{-i})\mathrm{if}{x}_i=x_i^{*},\\ 0\mathrm{otherwise}(i=1,\dots ,M),\end{array}$$

where $x^{*}\in Arg\underset{x}{\max }\left[\psi (x)-{\displaystyle \sum _{i=1}^M{f}_i(x)}\right]$.

However, this solution may be impossible to calculate directly due to insufficient information about the cost functions $f_i\left(x\right)$ that is available to the Principal. So, we will suggest a computational scheme based on solving a dynamic stochastic inverse Stackelberg game.

Assume the Principal stimulates the activity of all agents by allocating resources among them. Their activity is described by a vector sequence $x^M\left(t\right)={\left(x_i\left(t\right)\right)}_{i=1}^M,t=0,1,\dots$ At each time t, the Principal chooses a control strategy $\gamma \left(t\right)$, and each agent i obtains a corresponding part ${\varphi }_i\left(x_i\left(t-1\right),\gamma \left(t\right)\right),i=1,\dots ,M$, of the resource. This resource allocation mechanism is selected because the Principal does not know the cost of all agents; while choosing a local control strategy $\gamma \left(t\right)$, he/she may rely just on the available history $x^M\left(0\right),\dots ,x^M\left(t-1\right)$. This paper considers the case of Markovian processes: at each time t, the available information about the preceding value $x\left(t-1\right)$ is used only. The Principal cannot inform the agents about his/her control strategies for the whole duration of the game and the agents cannot calculate these strategies. Therefore, the agents (like the Principal) should be guided by the available information at the time t, i.e., by the sequence $\gamma \left(1\right),\dots ,\gamma \left(t\right)$. Below we consider the two setups of the dynamic inverse Stackelberg game as a resource allocation model.

3.2. Dynamic Setup 1

The first model consists in the following. The set of admissible influences on agents (the set of all Principal’s scenarios) is finite. The agents suppose that $\gamma \left(1\right),\dots ,\gamma \left(t\right)$ is a segment of a Markov sequence defined in the stochastic basis ${\mathrm{B}}^{\gamma }=\langle {\mathsf{\Omega }}^{\gamma },{\left[F^{\gamma }\left(t\right)\right]}_{t\ge 0},F^{\gamma }(\infty ),G^{\gamma }\rangle$, $F^{\gamma }\left(t\right)=\sigma \left(\gamma \left(1\right),\dots ,\gamma \left(t\right)\right)$, i.e., the minimal sigma-subalgebra supplemented with the events of zero probability. The agents solve their problems with the discounted payoff functions $E_{G_y^{\gamma }}{\displaystyle \sum _{t=1}^{\infty }{\beta }^t\left[{\varphi }_i\left(x_i\left(t-1\right),\gamma \left(t\right)\right)-f_i\left(x_i\left(t\right)\right)\right]}$, where $0<\beta <1$ and $E_{G_y^{\gamma }}(.)$ denotes the conditional expectation operator with the probability measure induced by the sequence $\gamma$ given $\gamma \left(0\right)=y$.

Consider the problem of agent i:

$$E_{Q_y^{\gamma }}{\displaystyle \sum _{t=1}^{\infty }{\beta }^t\left[{\varphi }_i\left(x_i\left(t-1\right),\gamma \left(t\right)\right)-f_i\left(x_i\left(t\right)\right)\right]}\to \underset{}{\max }$$

This problem is solved by dynamic programming; the associated Bellman functions satisfy the equations

$$V_i\left(x,y\right)=\underset{z}{\max }\left[{\varphi }_i\left(x,y\right)-f_i\left(z\right)+\beta {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{V}_i\left(z,u\right)q^{\gamma }\left(du/y\right)}\right].$$

(1)

In addition, $x_i\left(0\right)=x,\gamma \left(0\right)=y$. The measure $q^{\gamma }\left(du/y\right)$ in the integrand expression is the transition core of the Markov sequence $\gamma$. Write the Bellman function in the form $V_i\left(x,y\right)={\varphi }_i\left(x,y\right)+W_i\left(y\right)$. As a result, we obtain the following Fredholm integral equation of the second kind in the unknown function $W_i\left(y\right)$:

$$W_i\left(y\right)=\beta {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{W}_i\left(u\right)q^{\gamma }\left(du/y\right)}+\underset{z}{\max }\left[-f_i\left(z\right)+\beta {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}\varphi \left(z,u\right)q^{\gamma }\left(du/y\right)}\right].$$

(2)

So, the existence of a unique solution for the Bellman equation is equivalent to the existence of a unique solution for the Fredholm integral equation of the second kind [38]. Consider the space $B\left(-\infty ,\infty \right)$ of all bounded functions with the norm $\underset{x}{\sup }\left|f\left(x\right)\right|$ and also the operator $Gf={\displaystyle \underset{-\infty }{\overset{\infty }{\int }}f\left(u\right)q^{\gamma }\left(du/y\right)}$ over this space. Obviously, $\Vert G\Vert \le 1$. If the function ${\varphi }_i\left(y\right)=$ $\underset{z}{\max }\left[-f_i\left(z\right)+\beta {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{\varphi }_i\left(z,u\right)q^{\gamma }\left(du/y\right)}\right]$ is bounded above, then Equation (2) has a unique solution. For the function ${\varphi }_i\left(y\right)$ to be bounded, a sufficient condition is that the functions ${\varphi }_i\left(z,u\right)-f_i\left(z\right)$ are bounded above for all u. Moreover, if the functions $f_i\left(x\right)$ are convex and ${\varphi }_i\left(z,u\right)$ are convex in the variable z for any u, then there exists a unique solution of the problem $z_i^{*}\left(y\right)=\arg \underset{z}{\max }\left[-f_i\left(z\right)+\beta {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}\varphi \left(z,u\right)q^{\gamma }\left(du/y\right)}\right]$.

The optimal activity of the agents satisfies the equality

$$x_i^{*}\left(t\right)=z_i^{*}\left(\gamma \left(t\right)\right).$$

(3)

What is of fundamental importance, the optimal behavior of all agents in this model depends on the Principal’s local control strategy $\gamma \left(t\right)$ only––in no way on the preceding values of the sequence. Therefore, the Principal’s problem is to calculate

$$\underset{\gamma \left(t\right)}{\max }\left[\psi \left({\displaystyle \sum _{i=1}^M{z}_i^{*}\left(\gamma \left(t\right)\right)}\right)-{\displaystyle \sum _{i=1}^M\varphi \left(x_i\left(t-1\right),\gamma \left(t\right)\right)}\right].$$

(4)

The Principal does not know the relationship (3), while the agents do not know the transition core of the Markov sequence $\gamma$. In our case, this core is defined by the transition probability matrix because a Markov chain has a finite set of admissible states. As mentioned earlier, the Principal and agents can observe the response of each other. So, the Principal learns by observing the response of agents to his/her actions while the agents learn by observing the Principal’s response to their actions. The problem becomes much easier in comparison with the general Q-learning procedure for solving the dynamic inverse Stackelberg games that wereconsidered in [36]: for the agents, it is required to approximate the transition probability matrix; for the Principal, to maximize the function of one variable. Statistical estimation of a transition probability matrix and maximization of a function of one variable are not so computationally intensive as calculation of the Q-functions for the Principal and agents. Let $\mathsf{\Gamma }=\left\{{\gamma }^1,\dots ,{\gamma }^r\right\}$ and denote by $P$ the transition probability matrix. The maximum likelihood estimate $P_t$ of this matrix is designed as follows. Consider a matrix sequence $G$ such that $G_t\left(i,j\right)=G_{t-1}\left(i,j\right)+I_{\left\{{\gamma }_{t-1}={\gamma }^i,{\gamma }_t={\gamma }^j\right\}}$, where $I_{\left\{S\right\}}$ means the indicator of a set $S$. The initial value is $G_0=0$. The maximum likelihood estimate $P_t$ has the form $P_t\left(i,j\right)=\frac{G_t\left(i,j\right)}{{\displaystyle \sum _k{G}_t\left(i,k\right)}}{I}_{\left\{{\displaystyle \sum _k{G}_t\left(i,k\right)}\ne 0\right\}}$. Let $\gamma \left(t\right)={\gamma }^j$. Then $\underset{z}{\max }\left[-f_i\left(z\right)+\beta {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{\varphi }_i\left(z,u\right)q^{\gamma }\left(du/y\right)}\right]$ can be written as the approximate problem

$$\underset{z}{\max }\left[-f_i\left(z\right)+\beta {\displaystyle \sum _k{P}_t\left(j,k\right){\varphi }_i\left(z,{\gamma }^k\right)}\right].$$

(5)

If the function $F_t^i\left(z\right)=-f_i\left(z\right)+\beta {\displaystyle \sum _k{P}_t\left(j,k\right){\varphi }_i\left(z,{\gamma }^k\right)}$ is concave and bounded above, then problem (5) has the unique solution $x_i^{*}\left(t\right)=\arg \max F_t^i\left(z\right)$. Then the Principal observes $R\left({\gamma }^j\right)$, i.e., the response of agents to the control strategy ${\gamma }^j$: $R\left({\gamma }^j\right)=\psi \left({\displaystyle \sum _{i=1}^M{x}_i^{*}\left(t\right)}\right)-{\displaystyle \sum _{i=1}^M{\varphi }_i\left(x_i\left(t-1\right),{\gamma }^j\right)}$. After that, the Principal modifies the Q-function using the reinforcement learning algorithm

$$Q_{t+1}^{}\left({\gamma }^j\right)=Q_t^{}\left({\gamma }^j\right)+h_t\left(R\left({\gamma }^j\right)-Q_t^{}\left({\gamma }^j\right)\right),$$

(6)

and chooses the next control strategy for the agents based on the probability distribution $P_{t+1}^l=\left\{p_{t+1}^l\left({\gamma }^1\right),\dots ,p_{t+1}^l\left({\gamma }^r\right)\right\}$. The probabilities can be calculated by the Boltzmann scheme, which is used in annealing [36]. This is a random search algorithm for maximization (minimization) of generally non-differentiable functions $f\left(x\right)$. The idea consists in comparing the current solution $x_t$ with a randomly chosen one $y_t$ in a small neighborhood of the former. Transition to the randomly chosen solution occurs if $f\left(y_t\right)>f\left(x_t\right)$ ($f\left(y_t\right)<f\left(x_t\right)$). If this condition fails, then transition to a next random value $y_t$ is performed with the probability $p_t=\exp \left(-\frac{f\left(x_t\right)-f\left(y_t\right)}{T_t}\right)$ $\left(p_t=\exp \left(\frac{f\left(x_t\right)-f\left(y_t\right)}{T_t}\right)\right)$. Therefore, it is natural to choose the next value $\gamma$ with the probability

$$p_{t+1}^l\left({\gamma }^j\right)=\exp \left(Q_{t+1}^l\left({\gamma }^j\right)/T_{t+1}\right)/{\displaystyle \sum _k\exp \left(Q_{t+1}^l\left({\gamma }^k\right)/T_{t+1}\right)}.$$

(7)

In formula (7) and also above, the parameter T—“temperature”—adjusts the degree of randomness for control strategies. This parameter is decreasing from a given maximal value to a given minimal value, e.g., $T_{t+1}=\delta T_t,0<\delta <1$. The initial condition is $Q_0^l\equiv 0$. The values $h_t$ regulating the amplitude of variations are supposed to satisfy the standard conditions ${\displaystyle \sum _{t=1}^{\infty }{h}_t}=\infty ,{\displaystyle \sum _{t=1}^{\infty }{h}_t^2}<\infty$. They guarantee the almost sure convergence to the optimal function $Q$ that is close to R if each element of the set $\mathsf{\Gamma }$ appears an infinite number of times in the course of learning.

Thus, the suggested algorithm consists of the following iterative operations:

• calculation of the maximum likelihood estimate for the transition probability matrix;
• calculation of the next value of the Q-function.

Once again, we underline an important advantage of the suggested algorithm: the Q-function depends on a single argument only. The first operation is to calculate the maximum likelihood estimate instead of a next value of the Q-function, which guarantees faster convergence of the algorithm against its counterparts in which the Q-function depends on two arguments and a next value of Q is calculated at the first and second stages. Also, note that the maximum likelihood estimates are consistent and asymptotically efficient, i.e., they all use available information and obeys the normal distribution in asymptotics.

3.3. Dynamic Setup 2

In the second model of this game, the agents assume that the Principal makes “no sudden moves” in control choice (see Postulate 2 from [27]). So, they describe the Principal’s behavior by

$$\gamma \left(t\right)=\gamma \left(t-1\right)\left(1+{\epsilon }_t\right).$$

(8)

The sequence ${\epsilon }_t$ consists of independent identically distributed random elements with the mass concentrated near the origin. For the second model, $\underset{z}{\max }\left[-f_i\left(z\right)+\beta {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{\varphi }_i\left(z,u\right)q^{\gamma }\left(du/y\right)}\right]$ can be written as $\underset{z}{\max }\left[-f_i\left(z\right)+\beta E_{\epsilon }\varphi \left(z,y\left(1+\epsilon \right)\right)\right]$. Consider the second term and apply the Taylor expansion up to the second order inclusive, following the standard approach of stochastic calculus (e.g., the Ito formula). As a result, we obtain the optimization problem

$$\underset{z}{\max }\left[-f_i\left(z\right)+\beta \left({\varphi }_i\left(z,y\right)+\frac{\partial {\varphi }_i\left(z,y\right)}{\partial y}y{E}_{\epsilon }\epsilon +\frac{1}{2}\frac{{\partial }^2{\varphi }_i\left(z,y\right)}{\partial y^2}{y}^2{E}_{\epsilon }{\epsilon }^2\right)\right],$$

(9)

provided that the second derivative exists. If for all $y$ the goal functions of the agents are concave and bounded, then there exist unique solutions $z_i^{*}\left(y\right)$ for the agents’ problems (9). The sample means $a_t=\frac{1}{t}{\displaystyle \sum _{i=1}^t\frac{\Delta \gamma \left(i\right)}{{\gamma }_{i-1}}},b_t=\frac{1}{t}{\displaystyle \sum _{i=1}^t{\left(\frac{\Delta \gamma \left(i\right)}{{\gamma }_{i-1}}\right)}^2}$ for the moments of distribution of $\gamma$ are consistent estimates for the moments in the right-hand side of Formula (9). Therefore, the agents solve the problems $\underset{z}{\max }\left[-f_i\left(z\right)+\beta \left({\varphi }_i\left(z,\gamma \left(t\right)\right)+\frac{\partial {\varphi }_i\left(z,\gamma \left(t\right)\right)}{\partial y}\gamma \left(t\right)a_t+\frac{1}{2}\frac{{\partial }^2{\varphi }_i\left(z,\gamma \left(t\right)\right)}{\partial y^2}\gamma {\left(t\right)}^2{b}_t\right)\right]$, while the Principal can use any maximization method of a concave function of one variable without derivatives. If the function $R\left(x\right)$ is concave and bounded above, then the consistent sample moments and the convergent one-dimensional search procedure guarantee that this algorithm converges to the equilibrium of the Stackelberg game in probability.

Thus, the learning procedure for the second model consists of the following iterative operations:

• calculation of the estimates for the first and second moments of distribution;
• calculation of the next approximation $\gamma$.

In comparison with the classical Q-learning [34,35], this approach does not calculate the Q-function and is based on estimating the first and second moments of distribution and one-dimensional maximization of a function of one variable, which forms an obvious advantage. In comparison with the previous algorithm (see Section 3.2), it does not need the preliminary analysis of the problem to find the Principal’s behavioral scenarios (the set $\mathsf{\Gamma }$).

4. Examples and Numerical Calculations

Consider an example in which $\psi \left(x\right)=\sqrt{x}$, ${\varphi }_i\left(x,y\right)=xy$, and $f_i\left(x\right)={\mu }_i{x}^2$. For the first model, problem (5) takes the form: $\underset{z}{\max }\left[-{\mu }_i{z}^2+\beta z{\displaystyle \sum _k{P}_t\left(j,k\right){\gamma }^k}\right]$, with the solution $x_i^{*}\left(t\right)=\frac{1}{2{\mu }_i}\beta {\displaystyle \sum _k{P}_t\left(j,k\right){\gamma }^k}$. The response is $R\left({\gamma }^j\right)=\sqrt{{\displaystyle \sum _{i=1}^M{x}_i^{*}\left(t\right)}}-{\gamma }^j{\displaystyle \sum _{i=1}^M{x}_i^{*}\left(t-1\right)}$.

For the second model, problem (9) takes the form $\underset{z}{\max }\left[-{\mu }_i{z}^2+\beta z\left(\gamma \left(t\right)+a_t\right)\right]$, with the solution $x_i^{*}\left(\gamma \left(t\right)\right)=\frac{\beta \left(\gamma \left(t\right)+a_t\right)}{2{\mu }_i}$, where $a_t=\frac{1}{t}{\displaystyle \sum _{i=1}^t\frac{\Delta \gamma \left(i\right)}{\gamma \left(i-1\right)}}$. The response $R\left(\gamma \left(t\right)\right)$ is the same as above.

For this problem, the Stackelberg equilibrium satisfies the equalities $x_i^{*}=\frac{\beta \gamma }{2{\mu }_i},{\gamma }^{*}={\left(\frac{1}{2\sqrt{2\beta {\displaystyle \sum _{i=1}^M\frac{1}{{\mu }_i}}}}\right)}^{2/3}$.

Choose the following numerical parameters for trial calculations:

Then the equilibrium values are

$${\gamma }^{*}=0.4524029361,x_1^{*}=0.2035813212,x_2^{*}=0.1017906606.$$

Consider two examples as follows.

In the first example, the set $\mathsf{\Gamma }=\left\{0.4,0.45,0.5\right\}$ contains a value close to the equilibrium in position 2. Our calculations were performed in Maple. The algorithm yielded the following results. The calculated values of the Q-function are $Q=\mathrm{vector}(0.02789054257,0.4073228841,0.09271767315)$. The maximal value of the Q-function is 0.4073228841 (position 2), which corresponds to the value $\gamma =0.45$ from the set $\mathsf{\Gamma }$. The calculated transition probability matrix has the form $P=\mathrm{matrix}\left(\right[0,3/4,1/4],[1/91,87/91,3/91],[3/4,1/4,0\left]\right)$. The maximal element of this matrix, which is close to 1, stands at the junction of column 2 and row 2. In other words, the most probable prefix of the chain is 0.45, 0.45, …, which also corresponds to the value $\gamma =0.45$ from the set $\mathsf{\Gamma }$. The calculated values $x_1=0.2029945055,x_2=0.1014972528$ are close to the equilibrium.

In the second example, the set $\mathsf{\Gamma }=\left\{0.2,0.4,0.6\right\}$ has no values close to the equilibrium. The calculated values of the Q-function are $Q=\mathrm{vector}(0.1254077349,0.4028884358,0.05489325521)$. The maximal value of the Q-function is $0.4028884358$ (position 2), which corresponds to the value $\gamma =0.4$ from the set $\mathsf{\Gamma }$. The calculated transition probability matrix has the form $P=\mathrm{matrix}\left(\right[1/5,3/5,1/5],[1/90,43/45,1/30],[3/4,1/4,0\left]\right)$. The maximal element of this matrix, which is close to 1, also stands at the junction of column 2 and row 2. The calculated values $x_1=0.1820000000,x_2=0.09100000000$ slightly vary from the equilibrium.

The second model. The results yielded by the algorithm for the second model are presented in

Table 1. The results yielded by the algorithm.

$M$	2
$\beta$	0.9
${\mu }_1$	1
${\mu }_2$	2

and

Table 2. The results of numerical calculations.

$\mathit{\gamma }$	${\mathit{x}}_1$	${\mathit{x}}_2$	$\mathit{R}$
0.35	0.1837500000	0.09187500000	0.2625000000
0.36	0.1778142858	0.08890714290	0.4172257999
0.37	0.1788773809	0.08943869045	0.4193054183
0.38	0.1816893340	0.09084466700	0.4200877844
0.39	0.1852001900	0.09260009500	0.4207793681
0.40	0.1890599915	0.09452999575	0.4214115632
0.41	0.1931187782	0.09655938910	0.4219456888
0.42	0.1973015878	0.09865079390	0.4223502262
0.43	0.2015667888	0.1007833944	0.4226042619
0.44	0.2058894150	0.1029447075	0.4226943805
0.45	0.2102535881	0.1051267940	0.4226120220
0.445	0.2070930676	0.1035465338	0.4170062493
0.4475	0.2078141892	0.1039070946	0.4193088334
0.45	0.2085934568	0.1042967284	0.4190913001
0.44875	0.2075715326	0.1037857663	0.4175845248
0.449375	0.2075256146	0.1037628073	0.4180160874

.

This table has the same notations as before. The obtained results indicate that fast convergence and additional comments are unnecessary.

The second algorithm seems to be preferable if the admissible set $\mathsf{\Gamma }$ has no values close to the equilibrium. In our example, the second algorithm demonstrated a faster convergence to the equilibrium than the first. However, for the set $\mathsf{\Gamma }$ containing a value close to the equilibrium, the first algorithm yielded more accurate results. At the same time, the second algorithm had a higher accuracy rate for the set $\mathsf{\Gamma }$ without such values.

5. Conclusions and Future Work

Proportional allocation is the most natural mechanism to distribute resources, which has been approved by the practical control of organizational systems. For this mechanism, the problem of strategy-proofness (protection against manipulation) comes at the forefront because the agents are interested in overrating their real resource demands unknown to the Principal. The static proportional resource allocation mechanism was studied in control of organizational systems (see [5]); a modification of this mechanism that guarantees strategy-proofness was also designed there.

In this paper, we have suggested a dynamic proportional resource allocation mechanism based on learning. We have constructed two stochastic models (setups) of the dynamic inverse Stackelberg game, each guaranteeing the existence of stationary equilibrium. The first model involved the ideology of a finite set of the Principal’s behavioral scenarios while the second relied on the natural limits of all Principal’s actions. Both models have been illustrated using numerical examples. Each model is associated with an algorithm to find equilibrium in the inverse Stackelberg game.

The experimental results allow us to formulate the following hypothesis. The developed algorithms for solving the dynamic stochastic inverse Stackelberg game can be also used for solving the corresponding static inverse Stackelberg game with insufficient information about the cost functions of all agents. This hypothesis still needs deeper analysis, which will be the subject of futureresearch.