Learning Deceptive Tactics for Defense and Attack in Bayesian–Markov Stackelberg Security Games

Abstract

In this paper, we address the challenges posed by limited knowledge in security games by proposing a novel system grounded in Bayesian–Markov Stackelberg security games (SSGs). These SSGs involve multiple defenders and attackers and serve as a framework for managing incomplete information effectively. To tackle the complexity inherent in these games, we introduce an iterative proximal-gradient approach to compute the Bayesian Equilibrium, which captures the optimal strategies of both defenders and attackers. This method enables us to navigate the intricacies of the game dynamics, even when the specifics of the Markov games are unknown. Moreover, our research emphasizes the importance of Bayesian approaches in solving the reinforcement learning (RL) algorithm, particularly in addressing the exploration–exploitation trade-off. By leveraging Bayesian techniques, we aim to minimize the expected total discounted costs, thus optimizing decision-making in the security domain. In pursuit of effective security game implementation, we propose a novel random walk approach tailored to fulfill the requirements of the scenario. This innovative methodology enhances the adaptability and responsiveness of defenders and attackers, thereby improving overall security outcomes. To validate the efficacy of our proposed strategy, we provide a numerical example that demonstrates its benefits in practice. Through this example, we showcase how our approach can effectively address the challenges posed by limited knowledge, leading to more robust and efficient security solutions. Overall, our paper contributes to advancing the understanding and implementation of security strategies in scenarios characterized by incomplete information. By combining Bayesian and Markov Stackelberg games, reinforcement learning algorithms, and innovative random walk techniques, we offer a comprehensive framework for enhancing security measures in real-world applications.

1. Introduction

1.1. Brief Review

In safeguarding vital infrastructures, strategically allocating limited security resources is crucial. The primary objective is to prevent attacks from unpredictable assailants targeting critical assets. However, in selecting defense techniques, the balance between defenders’ private information disclosure and attackers’ observational capabilities must be carefully considered. This equilibrium ensures effective protection against potential threats while minimizing vulnerabilities arising from asymmetrical information.

Game theory offers a powerful mathematical framework for optimizing decision-making in strategic scenarios, which is particularly evident in SSGs tackling high-complexity problems. SSGs encapsulate situations where defenders (leaders) and attackers (followers) strategically engage in sequential decision-making to achieve their objectives. In this model, defenders are represented by the leaders, while attackers are represented by the followers, highlighting the strategic interplay between protective and offensive roles in the system. Multiplayer game theory presents significant complexity and challenges in finding solutions due to the interaction of multiple agents with varying strategies and objectives. The dynamic nature and interdependencies of players’ actions create a highly intricate environment, requiring sophisticated approaches to model, analyze, and achieve equilibrium in these games. Initially, defenders commit to a strategy, which attackers then observe before determining their own course of action [1,2]. Defenders employ random techniques to safeguard potential targets, anticipating attackers’ objectives within the game’s dynamics. Meanwhile, attackers adopt best-reply strategies while cognizant of these defensive tactics. Mathematically, the defenders’ advantage lies in revealing their random plans, yet this overlooks potential knowledge loss due to incomplete observations. For instance, when attackers disguise themselves as civilians to infiltrate a target, defenders may struggle to ascertain the available security resources accurately. In such scenarios, revealing complete information isn’t always advantageous for defenders. Instead, strategic disclosure of incomplete information can be more beneficial [3,4]. This approach acknowledges the complexity of real-world security dilemmas, where optimal decision-making hinges on dynamically balancing information disclosure and strategic concealment. By leveraging game theory’s mathematical rigor, SSGs provide a framework for navigating these strategic challenges, ultimately enhancing security decision-making in complex and dynamic environments.

1.2. Related Work

Various game-theoretical frameworks have been employed to model the interaction between defenders and attackers in security scenarios, with Bayesian Stackelberg games emerging as a particularly successful solution [5]. Conitzer and Sandholm [6] adapted the Bayesian Stackelberg game into a normal-form representation using Harsanyi’s approach [5], determining game equilibrium by evaluating each follower’s plan to ascertain if it constitutes an optimal reaction. Paruchuri et al. [7] proposed a mixed-integer linear programming (MILP) approach to solve Bayesian Stackelberg games, leveraging optimization techniques to identify optimal strategies for both defenders and attackers. Jain et al. [8] introduced a strategy combining hierarchical decomposition with branch and bound techniques, aiming to efficiently explore the solution space and achieve equilibrium in the game. Yin and Tambe [9] suggested a hybrid technique merging best-first search with heuristic branching rules to solve Bayesian Stackelberg games effectively. This approach enhances the computational efficiency of finding equilibrium strategies by guiding the search process through the solution space. Overall, these methodologies contribute to advancing our understanding of security game dynamics and provide practical solutions for effectively addressing security challenges in real-world scenarios, highlighting the versatility and adaptability of game-theoretical approaches in security analysis and decision-making.

For further exploration, information can be found in Wilczynski’s comprehensive survey [10]. Sayed Ahmed [11] introduced a deception-based Stackelberg game anti-jamming mechanism. Gan et al. delved into security games involving uncoordinated defenders cooperating to protect targets, with each defender optimizing their resource allocation selfishly for utility maximization [12]. These studies underscore the breadth of applications and innovative approaches within the realm of SSGs, showcasing their relevance in modern security paradigms.

SSGs find diverse applications in everyday scenarios, contributing significantly to the enhancement of security measures across various domains. Wilczynski [10] provided a comprehensive survey highlighting the breadth of SSG applications. Basilico et al. [13] determined the minimum number of robots required to patrol a given environment, compute optimal patrolling strategies across various coordination dimensions, and experimentally evaluate the proposed techniques, demonstrating their effectiveness in ensuring comprehensive coverage and the efficient use of robotic resources. Clempner [14] proposed a security model implemented with a temporal difference method, and it incorporates prior information to effectively address security issues. By leveraging continuous learning and adapting to evolving threats, this model aims to enhance the robustness and resilience of security systems. Albarran et al. [15] applied SSGs to distribute security resources across airport terminals, leveraging partially observed Markov game settings to address complex security challenges. In urban settings, Trejo et al. [16] employed reinforcement learning within SSG frameworks to adapt attacker and defender strategies, improving security measures at geographically dispersed shopping malls. Solis et al. [17] extended SSG applications to maritime security, utilizing a multiplayer ship differential SSG to model pursuit–evasion scenarios in continuous time. Alcantara et al. [18] incorporated topographical information into SSGs to generate realistic patrol plans for defenders in major cities, enhancing target identification and security deployment strategies. Sayed Ahmed [11] proposed a deception-based SSG anti-jamming solution, addressing vulnerabilities in communication systems. Gan et al. [12] explored SSGs where uncoordinated defenders collaborate to protect targets, with each defender optimizing their resource allocation selfishly. Clempner and Poznyak [19] introduced an attacker–defender SSG system, leveraging ergodic Markov models and non-decreasing Lyapunov-like functions to represent solutions. Rahman and Oh [20] investigated online patrolling robot route-planning tasks using SSG frameworks, enhancing security surveillance capabilities. Wang et al. [21] developed the DeDOL method based on deep reinforcement learning within SSG models, offering advanced solutions for security optimization. Li et al. [22] proposed a Bayesian Stackelberg Markov game for adversarial federated learning, utilizing meta-RL-based pre-training and adaptation to combat diverse attacks, achieving robust, adaptive, and efficient federated learning defense strategies. Sengupta and Kambhampati [23] framed adversarial federated learning as a Bayesian Stackelberg Markov game, using RL-based pre-training and meta-RL defenses to combat adaptive attacks, achieving robust and efficient federated learning defense strategies. Shukla et al. [24] presented a cybersecurity game for networked control systems, where an attacker disrupts communication and a defender protects key nodes. A cost-based Stackelberg equilibrium and robust defense method optimize security. Genetic algorithms enhance strategies for large power systems.

The versatility and effectiveness of SSGs highlight their importance in enhancing security strategies and protecting critical assets across various contexts. Whether applied to scheduling security patrols at airports, optimizing resource allocation in urban settings, or addressing vulnerabilities in communication systems, SSGs offer valuable insights and solutions. By integrating game-theoretical frameworks with real-world security challenges, SSGs enable proactive and adaptive approaches to security management, ultimately contributing to safer environments and mitigating potential threats effectively.

1.3. Main Results

Improving the existing security framework involves addressing the challenges posed by incomplete information. This is achieved through several key approaches. Firstly, an iterative proximal-gradient approach is proposed to compute the Bayesian Stackelberg equilibrium, enabling effective decision-making despite uncertainty. Additionally, a Bayesian reinforcement learning (BRL) algorithm is introduced, allowing for adaptive strategies that learn from past experiences and observations. Furthermore, a random walk approach is developed to implement SSGs, offering a dynamic and flexible method for responding to evolving threats. By integrating these techniques, the security framework gains the ability to adapt and respond effectively in the face of incomplete information, thereby enhancing overall resilience and effectiveness in mitigating security risks. These methods enable decision-makers to make more informed choices and optimize security strategies in complex and uncertain environments. A numerical example is used to provide useful recommendations for defenders’ resource allocation against attackers.

1.4. Organization of the Paper

The paper follows a structured layout, with Section 2 providing preliminary information. In this section, foundational concepts and background knowledge relevant to the study are presented to establish a basis for subsequent discussions. Section 3 elaborates on the problems under consideration and introduces a Bayesian Stackelberg game model, which serves as the theoretical framework for analyzing strategic interactions between defenders and attackers. This section outlines the key components of the model and discusses its applicability to the specific security scenario under investigation. In Section 4, the paper proposes a learning algorithm tailored to the context of the Bayesian Stackelberg game, aiming to optimize decision-making strategies for both defenders and attackers. The algorithm is designed to adapt and improve over time based on feedback and observed outcomes. Section 5 introduces a novel random walk model, offering an alternative approach to addressing security challenges within the Bayesian Stackelberg game framework. This model presents a new perspective on strategic decision-making in dynamic security environments. A numerical example illustrating the application of the proposed methodologies is provided in Section 6, demonstrating their effectiveness and practical relevance in real-world scenarios. Finally, Section 7 concludes the paper by summarizing key findings, drawing conclusions based on the analysis conducted, and discussing potential avenues for future research and development in the field of SSGs.

2. Preliminaries

In a discrete-time and finite-horizon framework, we consider an environment characterized by private and independent values, as described in prior works [3,25,26]. In this setup, players denoted by $l\in \mathcal{L}$, where $\mathcal{L}=1,2,\dots ,n$, receive rewards in discrete time periods $t\in T$, with $T\subseteq \mathbb{N}$. These rewards are contingent upon the current physical allocation $a_t^l\in A^l$ and the type ${\Theta }^l\in {\Theta }^l$ of the player. Here, $A^l$ represents the feasible set of allocations available to player l at time t, while ${\Theta }^l$ denotes the set of possible types for player l. The reward obtained by player l at time t is determined by their chosen allocation and their specific type. This framework captures the dynamics of decision-making in scenarios where players have private information and their actions impact the outcomes over a finite time horizon. Such models find applications in various domains, including economics, game theory, and decision science, where understanding the interplay between private information and strategic decision-making is crucial.

In period t, the type vector is denoted by ${\theta }_t=({\theta }_t^1,\dots ,{\theta }_t^n)\in \Theta$, where $\Theta ={\times }_{l\in \mathcal{L}}{\Theta }^l$. The set of feasible allocations in period t may be contingent on the vector of past allocations $a_t=(a_t^1,\dots ,a_t^n)\in A$, where $A={\times }_{l\in \mathcal{L}}{A}^l$. We denote ${\Theta }^{-l}$ as ${\times }_{i\in \mathcal{L},i\ne l}{\Theta }^i$, and $\Delta \left(A\right)$ as the set of all probability distributions over A. Additionally, we assume that $A^l$ and ${\Theta }^l$ are finite sets, ensuring a manageable and well-defined problem space. This setup enables the modeling of dynamic interactions and decision-making processes among multiple players with diverse types and feasible allocation options.

Each player l has a common prior distribution $P^l\left({\theta }_0^l\right)$. The type ${\Theta }^l$ and action $a_t^l$ determine a probability distribution for the variables ${\theta }_{t+1}^l$ on ${\Theta }^l$, which is denoted by $p^l({\theta }_{t+1}^l|{\theta }_t^l{a}_t^l)$. This distribution captures the probabilistic relationship between a player’s current type and action and their subsequent type in the next period, facilitating the modeling of dynamic decision-making processes within the game.

A Markov chain is described by the transition matrix $p^l\left({\theta }_{t+1}^l|{\theta }_t^l{a}_t^l\right)$ and the common prior distribution $P^l\left({\theta }_0^l\right)$ where $P^l\left({\Theta }^l\right)\in \Delta {\Theta }^l$ and $\Delta {\Theta }^l$ establishes the probability distributions over ${\Theta }^l$. We assume that each chain $(P^l,p({\theta }_{t+1}^l|{\theta }_t^l,a_t^l))$ is ergodic, ensuring that the system will eventually reach a steady state regardless of its initial conditions.

The asymmetric information is determined by the private observation of ${\theta }_t^l$. We determine that by the priors distributions $P^l\left({\theta }_0^l\right)$ and the transition matrices $p^l({\theta }_{t+1}^l|{\theta }_t^l{a}_t^l)$ for each player l, the information of player l, given by ${\theta }_{t+1}^l$, does not depend on ${\theta }_t^i,$ for $i\ne l$. Following that, participants transmitted $m_t^l$ messages at the same time, and the message profile was made public $m_t^l$.

A (behavioral) strategy ${\sigma }^l(m_t^l|{\theta }_t^l)$ for player l is mapping ${\sigma }^l:M^l\times {\Theta }^l\to \Delta \left(M^l\right)$, which represents the likelihood with which a player l believes that a message $m_t^l$ is of type ${\Theta }^l$. The (behavioral) strategy set is given by

$$\begin{array}{c}{\mathcal{S}}_{adm}^l=\left\{{\sigma }^l(m_t^l|{\theta }_t^l)\ge 0\left|{\displaystyle \sum _{m_t^l\in {\Theta }^l}}{\sigma }^l(m_t^l|{\theta }_t^l)=1,{\theta }_t^l\in {\Theta }^l\right.\right\}.\end{array}$$

A strategy is a sequence $\left\{{\pi }^l(a_t^l|m_t^l)\right\}$ such that, for each period t, ${\pi }^l(a_t^l|m_t^l)$ is a stochastic kernel on A given history $H_n$. The set of all the admissible strategies is

$${\Pi }_{adm}^l=\left\{{\pi }^l(a_t^l|m_t^l)\ge 0\left|\sum _{a_t^l\in A^l}{\pi }^l(a_t^l|m_t^l)=1,m_t^l\in M^l\right.\right\}$$

Remark 1.

A strategy π is typically defined as a policy or plan of action for a player that depends only on the information they possess. In this case, π depends solely on the player’s own message, and this suggests that π is a strategy based on private information or the part of the information that is most relevant to the player. The assumption here might be that each player has access to their own private message but does not directly depend on the messages of other players in their strategy formulation. This can simplify the analysis in certain games. A strategy σ, on the other hand, often refers to a more general form of strategy that could depend on all available information, such as public messages or even the outcomes of communication between players. σ involves communication, and it might be labeled as a communication strategy, where players decide how to share or process information.

Let us introduce the cost function $v^l\left(a_t^l{\theta }_t^l{m}_t^l\right)$, which characterizes the losses incurred by players l when they take action $a_t^l$ based on the message $m_t^l$ generated under type ${\theta }_t^l$.

The average cost $[0,T]$ of the player l is

$$\begin{array}{c}{V}_T^l(\pi ,\sigma )={\displaystyle \sum _{t\in T}}{\displaystyle \sum _{{\theta }_t^l\in {\Theta }^l}}{\displaystyle \sum _{m_t^l\in M^l}}{\displaystyle \sum _{a_t^l\in A^l}}{v}^l\left(a_t^l{\theta }_t^l{m}_t^l\right)p^l({\theta }_{t+1}^l|{\theta }_t^l{a}_t^l){\displaystyle \prod _{i\in \mathcal{L}}}{\pi }^i(a_t^i|m_t^i){\sigma }^i(m_t^i|{\theta }_t^i)P^i\left({\theta }_t^i\right)=\\ {\displaystyle \sum _{t\in T}}{\displaystyle \sum _{{\theta }_t^l\in {\Theta }^l}}{\displaystyle \sum _{m_t^l\in M^l}}{\displaystyle \sum _{a_t^l\in A^l}}{W}^l\left(a_t^l{\theta }_t^l{m}_t^l\right){\displaystyle \prod _{i\in \mathcal{L}}}{\pi }^i(a_t^i|m_t^i){\sigma }^i(m_t^i|{\theta }_t^i)P^i\left({\theta }_t^i\right).\end{array}$$

where

$$W^l\left(a_t^l{\theta }_t^l{m}_t^l\right)=v^l\left(a_t^l{\theta }_t^l{m}_t^l\right)p^l({\theta }_{t+1}^l|{\theta }_t^l{a}_t^l).$$

We assume that players know their payoffs. The strategy ${\sigma }^l(m_t^l|{\theta }_t^l)$ minimizes the weighted cost function $V^l(\pi ,\sigma )$, realizing the rule, which is given by

$$\begin{array}{c}({\pi }^{\ast },{\sigma }^{\ast })\in Arg\underset{\sigma \in {\Pi }_{adm}}{min}\underset{\sigma \in {\mathcal{S}}_{adm}}{min}{\displaystyle \sum _{l\in \mathcal{L}}}{V}^l(\pi ,\sigma ).\end{array}$$

(1)

The policy ${\pi }^{\ast }$ and the strategy ${\sigma }^{\ast }$ is said to satisfy the Bayesian–Nash equilibrium if, for all $\pi \in {\Pi }_{adm}$ and $\sigma \in {\mathcal{S}}_{adm}$, the following condition holds:

$$V^l({\pi }^{\ast },{\sigma }^{\ast })\le V^l({\pi }^{\ast },{\sigma }^l,{\sigma }^{-l\ast }).$$

(2)

where ${\sigma }^{-l}=$$\left({\sigma }^1,\dots ,{\sigma }^{l-1},{\sigma }^{l+1},\dots ,{\sigma }^n\right)$.

Let us introduce the auxiliary $\xi$ variable as follows:

$${\xi }^l\left({\theta }_t^l{m}_t^l{a}_t^l\right):=\pi (a_t^l|m_t^l){\sigma }^l(m_t^l|{\theta }_t^l)P^l\left({\theta }_t^l\right).$$

(3)

such that

$$\begin{array}{c}{\Xi }_{adm}^l:=\left\{{\xi }^l\left({\theta }_t^l{m}_t^l{a}_t^l\right)\left|{\displaystyle \sum _{m_t^l\in M^l}}{\displaystyle \sum _{{\theta }_t^l\in {\Theta }^l}}{\displaystyle \sum _{a_t^l\in A^l}}{\xi }^l\left({\theta }_t^l{m}_t^l{a}_t^l\right)=1,\right.{\displaystyle \sum _{m_t^l\in M^l}}{\displaystyle \sum _{a_t^l\in A^l}}{\xi }^l\left({\theta }_t^l{m}_t^l{a}_t^l\right)=P^l\left({\theta }_t^l\right)>0,\right.\\ \left.{\displaystyle \sum _{m_t^l\in M^l}}{\displaystyle \sum _{a_t^l\in A^l}}{\displaystyle \sum _{{\theta }_t^l\in {\Theta }^l}}[{\delta }_{{\theta }_t^l,{\theta }_{t+1}^l}-p^l({\theta }_{t+1}^l|{\theta }_t^l{a}_t^l)]{\xi }^l\left({\theta }_t^l{m}_t^l{a}_t^l\right)=0,{\theta }_{t+1}^l\in {\Theta }^l\right\}.\end{array}$$

(4)

where ${\delta }_{{\theta }_t^l,{\theta }_{t+1}^l}$ is Kroneker’s symbol. It should be noted that the following relations hold:

$$\begin{array}{cc}{\displaystyle \sum _{m_t^l\in M^l}}\sigma (m_t|{\theta }_t)=1,& {\displaystyle \sum _{{\theta }_t^l\in {\Theta }^l}}{P}^l\left({\theta }_t^l\right)=1.\end{array}$$

We obtain that ${\xi }^l\in {\Delta }^l$, where

$$\begin{array}{c}{\Delta }^l:=\left\{{\xi }^l\left({\theta }_t^l{m}_t^l{a}_t^l\right)\left|{\displaystyle \sum _{m_t^l\in M^l}}{\displaystyle \sum _{{\theta }_t^l\in {\Theta }^l}}{\displaystyle \sum _{a_t^l\in A^l}}{\xi }^l\left({\theta }_t^l{m}_t^l{a}_t^l\right)=1,\right.{\displaystyle \sum _{m_t^l\in M^l}}{\displaystyle \sum _{a_t^l\in A^l}}{\xi }^l\left({\theta }_t^l{m}_t^l{a}_t^l\right)=P^l\left({\theta }_t^l\right)>0\right\}.\end{array}$$

(5)

3. Security Game Model

In the game involving multiple players, we consider two distinct groups: the defenders, denoted by $L=1,\dots ,l$, and the attackers, denoted by $F=1,\dots ,f$. Utilizing conventional game-theoretic notation, we denote the player index set as $\mathcal{I}$. The joint action profile of all participants, excluding agent q, is represented as $x^{-q}={\left(x^h\right)}_{h\in \mathcal{I}/q}$. In the context of our Stackelberg game, $\mathcal{I}=L\cup F$, where players are indexed by $\iota =\overline{1,l}$ for the defenders and $r=\overline{1,f}$ for the attackers. This indexing scheme allows for clear differentiation between the defenders and attackers, facilitating the analysis of strategic interactions and decision-making dynamics within the game framework.

Let us say the defenders’ strategies are denoted by $x^{\iota }\in X^{\iota }$, where X is a convex and compact set and

$$\begin{array}{c}{x}^{\iota }:=col\left({\xi }_{ik}^{\iota }\right),X^{\iota }:={\Xi }_{adm}^{\iota },X:={\displaystyle \underset{\iota =1}{\overset{l}{\otimes }}}{X}^{\iota }\end{array}$$

such that $col$ is a column-converting operator for the matrix ${\xi }_{ik}^{\iota }$. The joint strategy of the players is denoted by $x={(x^1,\dots ,x^n)}^{\top }\in X$, while the complement strategy of the players is denoted by $\widehat{x}=x^{-\iota }$. Here,

$$\begin{array}{c}{x}^{-\iota }:={\left(x^1,\dots ,x^{\iota -1},x^{\iota +1},\dots ,x^l\right)}^{\top }\in X^{-\iota }:={\displaystyle \underset{h=1,h\ne \iota }{\overset{l}{\otimes }}}{X}^{\iota }\end{array}$$

such that $x=(x^{\iota },x^{\widehat{\iota }})$.

Similarly, if we let $y^r\in Y^r$$\left(r=\overline{1,f}\right)$ represent the attackers’ strategies and Y be a convex and compact set where

$$\begin{array}{c}{y}^r:=col\left({\xi }_{ik)}^r\right),Y^r:={\Xi }_{adm}^r\left(r=\overline{1,f}\right),Y:={\displaystyle \underset{r=1}{\overset{f}{\otimes }}}{y}^r\end{array}$$

is denoted by $y=(y^1,\dots ,y^f)\in Y:={\otimes }_{r=1}^f{Y}^r$, then the joint strategy of the attackers and $y^{\widehat{r}}=y^{-r}$ are strategies of the complement of the players adjoint to $y^r$, i.e.,

$$\begin{array}{c}{y}^{-r}:={\left(y^1,\dots ,y^{r-1},y^{r+1},\dots ,y^f\right)}^{\top }\in Y^{-r}:={\displaystyle \underset{r=1,r\ne f}{\overset{f}{\otimes }}}{Y}^r\end{array}$$

such that $y=(y^r,y^{\widehat{r}})$.

In the scenario we investigate, defenders and attackers engage in a Nash game within the framework of a simultaneous play game that is restricted to a Stackelberg game. In simultaneous play games, the Nash equilibrium serves as the notion of equilibrium. Here, each player independently selects their strategy, aiming to maximize their own utility given the strategies chosen by other players. A Nash equilibrium is reached when no player can unilaterally deviate from their strategy to achieve a better outcome. In contrast, hierarchical play games, such as Stackelberg games, involve sequential decision-making, where one player (the defender) commits to a strategy first, and the other player (the attacker) observes this strategy before determining their own. The equilibrium concept in hierarchical play games differs from the Nash equilibrium, as it involves optimizing strategies in anticipation of the actions of other players in the sequence of play. Formalizing the Stackelberg game within this framework involves defining the roles of defenders and attackers, specifying their strategies, and analyzing the equilibrium outcomes under the sequential decision-making structure.

Let $W=X\cup Y$. If a player $q\in \mathcal{I}$ has a cost function ${\phi }^q\left(w^q,w^{-q}\right)$, then we have the following:

Definition 1.

The joint strategy $w^{\ast }\in W$ is a Nash equilibrium if, for each $q\in \mathcal{I}$,

$${\phi }^q\left(w^{\ast }\right)\le {\phi }^q\left(w^q,w^{-q\ast }\right),\forall w^q\in W^q.$$

In the game progression, defenders forecast attackers behavior by playing non-cooperatively, anticipating attackers’ actions at a Stackelberg equilibrium. This strategic anticipation enables defenders to make informed decisions when committing to their strategies, considering the likely responses of attackers within the framework of the Nash equilibrium. By strategically forecasting attackers’ behavior, defenders aim to optimize their own outcomes in the game. To achieve the game’s aim, defenders must first identify a strategy $x^{\ast }=\left(x^{1\ast },\dots ,x^{l\ast }\right)\in X$ that is satisfying for any admissible $x^{\iota }\in X^{\iota }$ and any $\iota =\overline{1,l}$.

$$\begin{array}{c}\Lambda \left(x\right):={\displaystyle \sum _{\iota =1}^l}\left[\left({\displaystyle \underset{x^{\iota }\in X^{\iota }}{min}}{\phi }^{\iota }\left(x^{\iota },x^{-\iota }\right)\right)-{\phi }^{\iota }\left(x^{\iota },x^{-\iota }\right)\right]\end{array}.$$

(6)

The cost function of the defenders $\iota$ who play the strategy $x^{\iota }\in X^{\iota }$ and the complement of the defenders who play the strategy $x^{-\iota }\in X^{-\iota }$ is ${\phi }^{\iota }\left(x^{\iota },x^{-\iota }\right)$.

Taking the utopia point as

$${\overline{x}}^{\iota }:=\underset{x^{\iota }\in X^{\iota }}{arg\; min}{\phi }^{\iota }\left(x^{\iota },x^{-\iota }\right)$$

(7)

then one can describe Equation (6) in the following manner:

$$\Lambda \left(x\right):=\sum _{\iota =1}^l\left[{\phi }^{\iota }\left({\overline{x}}^{\iota },x^{-\iota }\right)-{\phi }^{\iota }\left(x^{\iota },x^{-\iota }\right)\right].$$

(8)

The functions ${\phi }^{\iota }\left(x^{\iota },x^{-\iota }\right)$$\left(\iota =\overline{1,l}\right)$ are supposed to be convex in all their arguments.

The function $\Lambda \left(x\right)$ fulfills Nash’s condition

$$\begin{array}{c}{\displaystyle \underset{x\in X}{max}}g\left(x\right)={\displaystyle \sum _{\iota =1}^l}\left[{\phi }^{\iota }\left({\overline{x}}^{\iota },x^{-\iota }\right)-{\phi }^{\iota }\left(x^{\iota },x^{-\iota }\right)\right]\le 0\end{array}$$

for any $x^{\iota }\in x^{\iota }$ and all $\iota =\overline{1,l}$

A strategy $x^{\ast }\in X$ is a Nash equilibrium if

$$x^{\ast }\in Arg\underset{x\in X}{min}\left\{\Lambda \left(x\right)\right\}.$$

If $\Lambda \left(x\right)$ is strictly convex, then

$$x^{\ast }=arg\underset{x\in X}{min}\left\{\Lambda \left(x\right)\right\}.$$

In addition, in this process, the attackers attempt to reach some of the Nash equilibria and try to find a joint strategy $y^{\ast }=\left(y^{1\ast },\dots ,y^{f\ast }\right)$∈Y that satisfies any admissible $y^r\in y^r$ and any $r=\overline{1,f}$

$$\begin{array}{c}\Psi \left(y\right):={\displaystyle \sum _{r=1}^f}\left[\left({\displaystyle \underset{y^r\in Y^r}{min}}{\phi }_r\left(y^r,y^{-r}\right)\right)-{\phi }^r\left(y^r,y^{-r}\right)\right].\end{array}$$

The cost function of the attackers m who play the strategy $y^r\in y^r$ and the complement of the defenders who play the strategy $y^{-r}\in y^{-r}$ is ${\phi }^r\left(y^r,y^{-r}\right)$.

Bearing in mind the utopia point

$${\overline{y}}^r:=\underset{y^r\in Y^r}{arg\; min}{\phi }^r\left(y^r,y^{-r}\right)$$

it is possible to rewrite Equation (6) as follows:

$$\Psi \left(y\right):=\sum _{r=1}^f\left({\phi }^r\left({\overline{v}}^r,y^{-r}\right)-{\phi }^r\left(y^r,y^{-r}\right)\right).$$

The functions ${\phi }^r\left(y^r,y^{-r}\right)$$\left(r=\overline{1,f}\right)$ are supposed to be convex in all their arguments.

The function $\Psi \left(y\right)$ satisfies the Nash condition

$$\begin{array}{c}{\displaystyle \underset{y^r\in y^r}{max}}w\left(y\right)={\displaystyle \sum _{r=1}^f}\left({\phi }^r\left({\overline{y}}^r,y^{-r}\right)-{\phi }^r\left(y^r,y^{-r}\right)\right)\le 0\end{array}$$

for any $y^r\in y^r$ and all $r=\overline{1,f}$.

The defenders’ goal is to find a solution to the optimization challenge given by the following definition:

Definition 2.

A Stackelberg game is a game with l defenders and f attackers; it is called a Stackelberg–Nash game if the defenders strive to solve the issue presented by

$$\underset{x\in X}{min}\{\Lambda \left(x|y\right)|\underset{v\in Y}{y\in Argmin}\Psi \left(v|x\right)\}$$

and the attackers try to solve the problem

$$\underset{y\in Y}{min}\Psi \left(y|x\right)$$

such that

$$\Lambda \left(x|y\right):=\sum _{\iota =1}^l\left({\phi }^{\iota }\left({\overline{x}}^{\iota },x^{-\iota }|y\right)-{\phi }^l\left(x^{\iota },x^{-\iota }|y\right)\right)$$

and

$$\Psi \left(y|x\right):=\sum _{r=1}^f\left({\phi }^r\left({\overline{y}}^r,y^{-r}|x\right)-{\phi }^r\left(y^r,y^{-r}|x\right)\right).$$

Then, the equilibrium notion in games is the Nash equilibrium in simultaneous play games and the Stackelberg equilibrium in hierarchical play games.

Definition 3.

(Stackelberg equilibrium) In a game with l defenders, the strategy $x^{\ast }\in X$ is said to be a Stackelberg–Nash equilibrium strategy for the defenders if

$$\underset{x\in \rho \left(x^{\ast }\right)}{max}\Lambda \left(x^{\ast }|y\right)\le \underset{x\in \rho \left(x\right)}{max}\Lambda \left(x|y\right)$$

such that

$$\rho \left(x\right)=\{y\in Y|\Psi \left(y|x\right)\le \Psi \left(v|x\right)\forall v\in Y\}$$

is the best reply set of the attackers.

The definition of Stackelberg equilibrium given above can be redefined for the set of the attackers when the set $\rho \left(x\right)$ is substituted with the set of Nash equilibria, considering that the defenders play the strategy x and then the attackers’ best reply is a Nash equilibrium.

The general format iterative version $\left(n=0,1,\dots \right)$ of the proximal-gradient method for computing the Stackelberg equilibrium is as follows:

1. The first half-step (prediction):

$${\widehat{v}}_n=arg\underset{\tilde{w}\in X\times Y}{min}\left\{\frac{1}{2}{\parallel \tilde{w}-\tilde{v}}_n{\parallel }^2+\gamma \Psi {(\tilde{w},\tilde{v}}_n)\right\}$$

(9)

2. The second (basic) half-step:

$${\tilde{v}}_{n+1}=arg\underset{\tilde{w}\in X\times Y}{min}\left\{{\displaystyle \frac{1}{2}}{\parallel \tilde{w}-{\tilde{v}}_n\parallel }^2+\gamma \Psi (\tilde{w},{\widehat{v}}_n)\right\}$$

(10)

4. Learning

Reinforcement learning (RL) addresses the challenge of learning optimal actions in an unknown Markov setting through interaction [27]. Players aim to devise strategies that minimize their expected costs. In a discrete-time setting, at each time step t, players observe a cost $v^i$, where i denotes the player. The overarching objective is to minimize the average cost V over time, guiding decision-making towards more favorable outcomes. By iteratively adjusting actions based on observed costs and environmental feedback, RL algorithms seek to discover optimal strategies that lead to the most advantageous long-term outcomes in dynamic and uncertain environments.

We consider RL problems where the underlying environment is a Bayesian–Markov decision process. Specifically, in the Stackelberg game, players indexed by $\iota =\overline{1,l}$ are the defenders and players indexed by $r=\overline{1,f}$ are the attackers. We will develop the results in general for a player $i\in \mathcal{I}$ ($\mathcal{I}=L\cup F$) and we specify defenders and attackers when it will be necessary. We propose a setting based on experiences, which is calculated by adding the number $\chi$ of experiences given as follows: let $h_t\in {\mathbb{H}}_t$ ($t\in T$) be the history at time t and $({\theta }_t,a_t)\in \Theta \times A$. Let, for each $({\theta }_{t+1},{\theta }_t,a_t)\in \Theta \times \Theta \times A$,

$${\chi }^i({\tilde{\theta }}_{t+1}^i{\tilde{\theta }}_t^i{a}_t^i)=\sum _{t\in T}\mathbb{1}({\tilde{\theta }}_{t+1}^i{\tilde{\theta }}_t^i{a}_t^i)=\mathbb{E}\left\{\mathbb{1}({\tilde{\theta }}_{t+1}^i)\mathbb{1}({\tilde{\theta }}_t^i)\mathbb{1}(a_t^i)\right\}$$

denote the experimentally observed absolute average number of transitions from type ${\widehat{\theta }}_t^i$ applying action $a_t^i$, We obtain the normally distributed and asymptotically unbiased maximum likelihood estimator given by $p^i\left({\tilde{\theta }}_{t+1}^i|{\tilde{\theta }}_t^i{a}_t^i\right)$

$${\tilde{p}}^i\left({\tilde{\theta }}_{t+1}^i|{\tilde{\theta }}_t^i{a}_t^i\right)={\displaystyle \frac{{\chi }^i({\tilde{\theta }}_{t+1}^i{\tilde{\theta }}_t^i{a}_t^i)}{{\chi }^i({\tilde{\theta }}_t^i{a}_t^i)}}.$$

Therefore, the cost of player $l\in \mathcal{L}$ is

$${\tilde{v}}^i\left({\tilde{\theta }}_{t+1}^i{\tilde{\theta }}_t^i{a}_t^i\right)={\displaystyle \frac{{\sum }_{t\in T}{\alpha }^i({\tilde{\theta }}_{t+1}^i{\tilde{\theta }}_t^i{a}_t^i){\chi }^i({\tilde{\theta }}_{t+1}^i{\tilde{\theta }}_t^i{a}_t^i)}{{\chi }^i({\tilde{\theta }}_{t+1}^i{\tilde{\theta }}_t^i{a}_t^i)}}$$

(11)

where

$${\alpha }^i({\tilde{\theta }}_{t+1}^i{\tilde{\theta }}_t^i{a}_t^i)=v^i({\tilde{\theta }}_{t+1}^i{\tilde{\theta }}_t^i{a}_t^i)+{\alpha }^{i}r$$

and ${\alpha }^i\le u^i$ such that r randomly takes the values between $-1$ or 1.

We assume that the det ${\mathcal{P}}^i({\theta }_t^i|m_t^i)$ $\ne 0$ where $\Sigma ={\left[{\mathcal{P}}^i({\theta }_t^i|m_t^i)\right]}^{-1}$ exists (the inverse of ${\mathcal{P}}^i({\theta }_t^i|m_t^i)$). We suggest a framework from experiences that is calculated counting the number $\chi$ of unobserved experiences recursively as follows:

$$\begin{array}{c}{\chi }^i\left(m_t^{\prime i}{m}_t^i{a}_t^i\right)={\displaystyle \sum _{t\in T}}\mathbb{1}(m_t^{\prime i}|m_t^i{a}_t^i)=\mathrm{E}\left\{\mathbb{1}\left(m_t^{\prime i}\right)\mathbb{1}\left(m_t^i\right)\mathbb{1}\left(a_t^i\right)\right\}\end{array}$$

where ${\chi }^i\left(m_t^i{a}_t^i\right)$ is the number of visits in the state $m_t^i$ and ${\chi }^i\left(m_t^{\prime i}{m}_t^i{a}_t^i\right)$ is the total number of times that the system changes from $m_t^{\prime i}$ to $m_t^i$ applying $a_t^i$. We obtain that ${\chi }^i\left(m_t^i{a}_t^i\right)={\displaystyle \sum _{m_t^{\prime i}\in M_t^i}}{\chi }^i\left(m_t^{\prime l}{m}_t^i{a}_t^i\right)$.

The estimated transition matrix ${\tilde{p}}^i({\tilde{\theta }}_{t+1}|{\tilde{\theta }}_t{x}_t)$ is given by

$$\begin{array}{c}{\tilde{p}}^i\left({\tilde{\theta }}_{t+1}^i|{\tilde{\theta }}_t^i{a}_t^i\right)={\displaystyle \frac{{\displaystyle \sum _{m_t^i\in M^i}}{\displaystyle \sum _{m_t^{\prime i}\in M^i}}{\Sigma }^i(m_t^{\prime i}{\tilde{\theta }}_t^i){\Sigma }^i(m_t^i{\tilde{\theta }}_{t+1}^i){\chi }^i(m_t^{\prime i}{m}_t^i{a}_t^i)}{{\displaystyle \sum _{{\widehat{\theta }}_{t+1}^i\in {\Theta }_t^i}}\left[{\displaystyle \sum _{m_t^i\in M^i}}{\displaystyle \sum _{m_t^{\prime i}\in M^i}}{\Sigma }^i(m_t^{\prime i}{\tilde{\theta }}_t^i){\Sigma }^i(m_t^i{\tilde{\theta }}_{t+1}^i){\chi }^i(m_t^{\prime i}{m}_t^i{a}_t^i)\right]}}.\end{array}$$

(12)

In order to recover the variables of interest, we calculate

$$\begin{array}{c}{\sigma }^{\iota \ast }(m_t^{\iota }|{\theta }_t^{\iota })={\displaystyle \frac{{\displaystyle \sum _{a_t^{\iota }\in A^{\iota }}}{\xi }^{\iota \ast }\left({\theta }_t^{\iota }{m}_t^{\iota }{a}_t^{\iota }\right)}{{\displaystyle \sum _{m_t^{\iota }\in M^{\iota }}}{\displaystyle \sum _{a_t^{\iota }\in A^{\iota }}}{\xi }^{\iota \ast }\left({\theta }_t^{\iota }{m}_t^i{a}_t^{\iota }\right)}}.\end{array}$$

(13)

$$\begin{array}{c}{\pi }^{\iota \ast }(a_t^{\iota }|m_t^{\iota })={\displaystyle \frac{1}{\left|{\Theta }^{\iota }\right|}}{\displaystyle \sum _{{\theta }_t^{\iota }\in {\Theta }^{\iota }}}{\displaystyle \frac{{\xi }^{\iota }\left({\theta }_t^{\iota }{m}_t^{\iota }{a}_t^{\iota }\right)}{{\displaystyle \sum _{a_t^{\iota }\in A^{\iota }}}{\xi }^{\iota }\left({\theta }_t^{\iota }{m}_t^{\iota }{a}_t^{\iota }\right)}}.\end{array}$$

(14)

$$\begin{array}{c}{P}^{\iota \ast }\left({\theta }_t^{\iota }\right)={\displaystyle \sum _{m_t^{\iota }\in M^{\iota }}}{\displaystyle \sum _{a_t^{\iota }\in A^{\iota }}}{\xi }^{\iota \ast }\left({\theta }_t^{\iota }{m}_t^{\iota }{a}_t^{\iota }\right)>0.\end{array}$$

(15)

$$\begin{array}{c}{\sigma }^{r\ast }(m_t^r|{\theta }_t^r)={\displaystyle \frac{{\displaystyle \sum _{a_t^r\in A^r}}{\xi }^{r\ast }\left({\theta }_t^r{m}_t^r{a}_t^r\right)}{{\displaystyle \sum _{m_t^r\in M^r}}{\displaystyle \sum _{a_t^r\in A^r}}{\xi }^{r\ast }\left({\theta }_t^r{m}_t^j{a}_t^r\right)}}.\end{array}$$

(16)

$$\begin{array}{c}{\pi }^{r\ast }(a_t^r|m_t^r)={\displaystyle \frac{1}{\left|{\Theta }^r\right|}}{\displaystyle \sum _{{\theta }_t^r\in {\Theta }^r}}{\displaystyle \frac{{\xi }^r\left({\theta }_t^r{m}_t^r{a}_t^r\right)}{{\displaystyle \sum _{a_t^r\in A^r}}{\xi }^r\left({\theta }_t^r{m}_t^r{a}_t^r\right)}}.\end{array}$$

(17)

$$\begin{array}{c}{P}^{j\ast }\left({\theta }_t^r\right)={\displaystyle \sum _{m_t^r\in M^r}}{\displaystyle \sum _{a_t^r\in A^r}}{\xi }^{r\ast }\left({\theta }_t^r{m}_t^j{a}_t^r\right)>0.\end{array}$$

(18)

The BRL algorithm for the SSG is outlined in Algorithm 1 and Algorithm 2. At time $t=0$, for each defender $\iota =1,\cdots ,l$ and each attacker $r=1,\cdots ,f$, we initialize the parameters. Specifically, the initial belief states for both the defender and the attackers are represented as ${\tilde{\theta }}_0^{\iota }\sim P^{\iota }\left({\theta }_0^{\iota }\right)$ and ${\tilde{\theta }}_0^r\sim P^r\left({\theta }_0^r\right)$, where $P^{\iota }$ and $P^r$ are probability distributions over the possible states ${\theta }_0$ of the defender and attackers, respectively. These beliefs are updated throughout the game as players take action and observe the outcomes.

Algorithm 1: BRL algorithm for the SSG

$\iota =\overline{1,l}$ and $r=\overline{1,f}$. $t=1$. ${\tilde{\theta }}_0^{\iota }\sim P^{\iota }\left({\theta }_0^{\iota }\right)$ and ${\tilde{\theta }}_0^r\sim P^r\left({\theta }_0^r\right)$. ${\tilde{p}}^{\iota }=p^{\iota }$ and ${\tilde{p}}^r=p^r$. $\epsilon >0$. $e_t^{\iota }\leftarrow$ Error(${\widehat{p}}_{t-1}^{\iota }$,${\widehat{p}}_t^{\iota }$) and $e_t^r\leftarrow$ Error(${\widehat{p}}_{t-1}^r$,${\widehat{p}}_t^r$) while ($\epsilon >e_t^{\iota }$ and $\epsilon >e_t^r$) (i) $x^{\ast }\leftarrow \left\{{\displaystyle \underset{x\in X}{min}}\Lambda \left(x|y\right)|\underset{\alpha \in Y}{y\in arg\; min}\Psi \left(\alpha |x\right)\right\}$ and $y^{\ast }\leftarrow \underset{y\in Y}{min}\Psi \left(y|x\right).$ (ii) recover ${\pi }^{\ast }\leftarrow$ Equations (14)–(17) and ${\sigma }^{\ast }\leftarrow$ Equations (13)–(16) (iii) $m_t^{\iota }\sim {\sigma }^{\iota \ast }(m_t^{\iota }|{\tilde{\theta }}_t^{\iota })$ and $m_t^r\sim {\sigma }^{r\ast }(m_t^r|{\tilde{\theta }}_t^r)$. (iv) $a_t^{\iota }\sim {\pi }^{\iota \ast }(a_t^{\iota }|m_t^{\iota })$ and $a_t^r\sim {\pi }^{r\ast }(a_t^r|m_t^r)$. (v) ${\tilde{\theta }}_{t+1}^{\iota }\sim {\widehat{p}}^{\iota }({\theta }_{t+1}^{\iota }|{\theta }_t^{\iota }{a}_t^{\iota })$ and ${\tilde{\theta }}_{t+1}^r\sim {\widehat{p}}^r({\theta }_{t+1}^r|{\theta }_t^r{a}_t^r)$. (vi) ${\chi }^{\iota }\left({\tilde{\theta }}_{t+1}^i\tilde{\theta }{a}_t^{\iota }\right)+=1$, ${\chi }^{\iota }\left({\tilde{\theta }}^{\iota }{a}_t^{\iota }\right)+=1$ and ${\chi }^r\left({\tilde{\theta }}_{t+1}^i\tilde{\theta }{a}_t^r\right)+=1$, ${\chi }^r\left({\tilde{\theta }}^r{a}_t^r\right)+=1$. (vii) ${\tilde{v}}^{\iota }\left({\tilde{\theta }}_{t+1}^{\iota }{\tilde{\theta }}_t^{\iota }{a}_t^{\iota }\right)\leftarrow$ Equation (11) and ${\tilde{v}}^r\left({\tilde{\theta }}_{t+1}^r{\tilde{\theta }}_t^r{a}_t^r\right)\leftarrow$ Equation (11). (viii) ${\tilde{p}}^{\iota }\left({\tilde{\theta }}_{t+1}^{\iota }|{\tilde{\theta }}_t^{\iota }{a}_t^{\iota }\right)\leftarrow$ Equation (12) and ${\tilde{p}}^r\left({\tilde{\theta }}_{t+1}^r|{\tilde{\theta }}_t^r{a}_t^r\right)\leftarrow$ Equation (12). (ix) $e_t^{\iota }\leftarrow$ Error(${\widehat{p}}_{t-1}^{\iota }$,${\widehat{p}}_t^{\iota }$) and $e_t^r\leftarrow$ Error(${\widehat{p}}_{t-1}^r$,${\widehat{p}}_t^r$). (x) ${\tilde{\theta }}_t^{\iota }={\tilde{\theta }}_{t+1}^{\iota }$, ${\tilde{\theta }}_t^r={\tilde{\theta }}_{t+1}^r$, $p^{\iota }={\widehat{p}}^{\iota }$, $p^r={\widehat{p}}^r$, $v^{\iota }={\tilde{v}}^{\iota }$, $v^r={\tilde{v}}^r$ and $t=t+1$. end while

Algorithm 2: Computation of the error

function Error (${\widehat{p}}_{t-1}$,${\widehat{p}}_t$) $e={\sum }_{a_t\in A}tr\left({({\widehat{p}}_{t-1}-{\widehat{p}}_t)}^{⊺}({\widehat{p}}_{t-1}-{\widehat{p}}_t)\right)$. return e. end function

The transition probabilities for each player’s states are defined as ${\tilde{p}}^{\iota }=p^{\iota }({\tilde{\theta }}_1^{\iota }|{\tilde{\theta }}_0^{\iota }{a}_0^{\iota })$ and ${\tilde{p}}^r=p^r({\tilde{\theta }}_1^r|{\tilde{\theta }}_0^r{a}_0^r)$, where ${\tilde{p}}^{\iota }$ and ${\tilde{p}}^r$ represent the likelihood of transitioning from one state to another given the actions $a_0^{\iota }$ and $a_0^r$ taken by the defenders and the attackers. These transition probabilities are essential for predicting the future states of the game based on current strategies.

An important aspect of the algorithm is to account for estimation errors in the parameters of the model. We denote the allowable error in the estimated parameters by $\epsilon >0$, which sets a threshold for how much deviation is acceptable. During each step of the algorithm, the actual error $e_t^{\iota }$ for the defenders and $e_t^r$ for the attackers is computed to track the accuracy of the estimated parameters.

At each time step t, the BRL algorithm computes the optimal strategy ${\sigma }^{\ast }(m_t^l|{\Theta }^l)$ and the corresponding policy ${\pi }^{\ast }(a_t^l|m_t^l)$ for both the defenders and the attackers. The strategy ${\sigma }^{\ast }$ determines the probability distribution over the possible messages $m_t^l$ given the state ${\Theta }^l$, while the policy ${\pi }^{\ast }$ dictates the probability of taking a particular action $a_t^l$ given the message $m_t^l$. These strategies and policies are central to the Bayesian Stackelberg equilibrium, where players optimize their decisions based on their beliefs about the other players’ actions.

For each step, random messages $m_t^{\iota }$ and $m_t^r$ are drawn from the respective optimal strategies ${\sigma }^{\iota \ast }(m_t^{\iota }|{\tilde{\theta }}_t^{\iota })$ and ${\sigma }^{r\ast }(m_t^r|{\tilde{\theta }}_t^r)$. Based on these messages, actions are chosen randomly according to the policies ${\pi }^{\iota \ast }(a_t^{\iota }|m_t^{\iota })$ for the defenders and ${\pi }^{r\ast }(a_t^r|m_t^r)$ for the attackers.

Once the actions are selected and executed, the game transitions to the next state. The updated states ${\tilde{\theta }}_{t+1}^{\iota }$ and ${\tilde{\theta }}_{t+1}^r$ are determined by the transition probabilities ${\widehat{p}}^{\iota }({\theta }_{t+1}^{\iota }|{\theta }_t^{\iota }{a}_t^{\iota })$ and ${\widehat{p}}^r({\theta }_{t+1}^r|{\theta }_t^r{a}_t^r)$, respectively.

After transitioning to the new states, we update the counts ${\chi }^{\iota }({\tilde{\theta }}_{t+1}^{\iota },{\tilde{\theta }}_t^{\iota },a_t^{\iota })$ and ${\chi }^r({\tilde{\theta }}_{t+1}^r{\tilde{\theta }}_t^r{a}_t^r)$, which track how frequently each state–action pair is encountered. These counts are critical for refining the players’ estimates of the value functions ${\tilde{v}}^{\iota }({\tilde{\theta }}_{t+1}^{\iota }{\tilde{\theta }}_t^{\iota }{a}_t^{\iota })$ and ${\tilde{v}}^r({\tilde{\theta }}_{t+1}^r{\tilde{\theta }}_t^r{a}_t^r)$. These value functions represent the expected reward for each player given their current and next states and the actions taken.

As the game progresses, the algorithm updates the transition probabilities ${\tilde{p}}^{\iota }({\tilde{\theta }}_{t+1}^{\iota }|{\tilde{\theta }}_t^{\iota }{a}_t^{\iota })$ and ${\tilde{p}}^r({\tilde{\theta }}_{t+1}^r|{\tilde{\theta }}_t^r{a}_t^r)$ based on observed outcomes. At each step, the mean-square errors $e_t^{\iota }$ and $e_t^r$ are calculated to measure the difference between the estimated and actual outcomes. The algorithm continues to iterate as long as these errors exceed the predefined threshold $\epsilon$, ensuring that the estimates converge to an acceptable level of accuracy.

Once the errors $e_t^{\iota }$ and $e_t^r$ fall below the threshold $\epsilon$, the resulting strategy profile ${\sigma }^{\ast }\left(m_t\right|{\theta }_t)$ and policy ${\pi }^{\ast }\left(a_t\right|m_t)$ form a Bayesian Stackelberg equilibrium. This equilibrium represents the optimal solution for the game, where the defenders and attackers have maximized their respective utilities given the available information.

5. Random Walk Algorithm

Random walks are basic models in probability theory with deep mathematical features and a wide range of applications in SSGs [16,28]. The long-term asymptotic behavior of the defenders to catch the attackers is a key question for these models.

The random walk process for the SSG is described by Algorithm 3 as follows. We will look at a game in which four players compete: defenders and attackers. Both sides have the option of using a randomized approach. The defenders and attackers can both move to any state at the same time. The defenders’ goal is to catch the attackers in the fewest possible steps. The attackers, on the other hand, choose a strategy that maximizes the amount of time it takes for the defenders to catch the attackers. If one of the defenders moves to a state where an attacker is, they are said to be caught. The game is over when the defenders have caught the attackers. We refer to a random walk as a discrete-time Markov process $(X_t,t\ge 0)$ in a finite-type space $\Theta$. The random walk is assumed to be time-homogeneous. This means that the $X_{t+1}$ distribution given $(X_0,\dots ,X_t)$ is solely dependent on $X_t$ and not on time t. To ensure that the random walk does not become ’stuck’ in some place of the state space, we also require some type of irreducibility: the set $\Theta$ is finite.

Given a type ${\theta }_t$, a message $m_t$ is chosen randomly by the defenders and the attackers from the behavior strategies $m_t^{\iota }\sim {\sigma }^{\iota \ast }\left(m_t^{\iota }\right|{\theta }_t^{\iota })$ and $m_t^r\sim {\sigma }^{r\ast }\left(m_t^r\right|{\theta }_t^r)$. They then choose actions $a_t^{\iota }\sim {\pi }^{\iota \ast }\left(a_t^{\iota }\right|m_t^{\iota })$ and $a^r\sim {\pi }^{r\ast }\left(a_t^r\right|m_t^r)$. To minimize and maximize the chance of damage, the transition matrix $p^{\iota }\left({\theta }_{t+1}^{\iota }|{\theta }_t^{\iota }{a}_t^{\iota }\right)$ and $p^r\left({\theta }_{t+1}^r|{\theta }_t^r{a}_t^r\right)$ are used to select the next state in the process. The defenders and attackers aim to finish the SSG the next time ($t+1$). The process continues until the fulfillment of the SSG meets the game-over or capture statuses provided by the following:

$$\sum _{\iota \in L}\sum _{r\in F}\sum _{{\theta }_{t+1}^{\iota }\in {\Theta }_t^{\iota }}\sum _{{\theta }_{t+1}^r\in {\Theta }_t^r}\mathbb{1}\left({\theta }_{t+1}^{\iota }\right)\mathbb{1}\left({\theta }_{t+1}^r\right)$$

(19)

Algorithm 3: Random walk for the SSG

1. Choose randomly an initial type for the defenders ${\theta }_0^{\iota }\sim P^{\iota }\left({\theta }_0^{\iota }\right)$ and attackers ${\theta }_0^r\sim P^r\left({\theta }_0^r\right)$. 2. Let ${\pi }^{\iota \ast }\left(a_t^{\iota }\right|m_t^{\iota })$ and ${\pi }^{r\ast }\left(a_t^r\right|m_t^r)$ be the resulting policies and, ${\sigma }_{m_t|{\theta }_t}^{\iota \ast }$ and ${\sigma }_{m_t|{\theta }_t}^{r\ast }$ the resulting strategies. 3. For each attacker r and defender $\iota$ Do 4. Choose a type ${\theta }_t^{\iota }\sim {\sigma }_{m_t|{\theta }_t}^{\iota \ast }$ and ${\theta }_t^r\sim {\sigma }_{r|i}^{\iota \ast }$ 5. Choose actions $a_t^{\iota }\sim {\pi }^{\iota \ast }\left(a_t^{\iota }\right|m_t^{\iota })$ and $a^r\sim {\pi }^{r\ast }\left(a_t^r\right|m_t^r)$. 6. From $p^{\iota }\left({\theta }_{t+1}^{\iota }|{\theta }_t^{\iota }{a}_t^{\iota }\right)$ select ${\theta }_{t+1}^{\iota }$ considering $p^{\iota }\left(·|{\theta }_t^{\iota }{a}_t^{\iota }\right)$ and from $p^r\left({\theta }_{t+1}^r|{\theta }_t^r{a}_t^r\right)$ select ${\theta }_{t+1}^r$ considering $p^r\left(·|{\theta }_t^r{a}_t^r\right)$. 8. Update the original values by including ${\theta }_{t+1}^{\iota }$ and ${\theta }_{t+1}^r$ to the random walk process. 9. Repeat steps 3–8 until the status expressed in Equation (19) is met.

6. Numerical Example

We will employ three players to simulate the SSG: one defender ($\iota =1$) and two attackers ($r=1,2$). The primary objective of the defender is to minimize or halt the damage inflicted by the attackers, while the attackers aim to maximize the expected damage they can cause to a set of targets. This dynamic reflects a typical security scenario, where an agent (the defender) must safeguard resources or locations, while adversarial players (attackers) try to exploit weaknesses.

The game is structured in a sequential manner and is commonly modeled as a Stackelberg game. The defender acts as the leader, committing to a strategy first, which is observed by the attackers. Once the defender’s strategy is known, the attackers, acting as followers, select their strategies in response. This turn-based nature highlights the strategic advantage of the defender, as their actions guide the attackers’ responses.

Each player in the game has a set of possible states and actions. In this particular setup, we assume that each player has $\theta =4$ states, representing different possible situations or configurations in which they can find themselves during the game. Additionally, each player has $a=2$ possible actions they can take in any given state, representing the limited yet strategic decisions available at each step of the game.

The key variables that drive the analysis of the SSG are the policies ($\pi$), behavior strategies ($\sigma$), and distribution vectors (P) for both the defender and attackers. The policy ($\pi$) defines the overall plan of action for each player, essentially specifying the probabilities with which they select each available action in a given state. Behavior strategies ($\sigma$) represent the specific actions taken by players as they progress through the game, reflecting how their strategies evolve over time. The distribution vectors (P) capture the likelihood of players being in different states throughout the course of the game.

These variables will be recovered analytically, meaning that through mathematical derivations and strategic analysis, we will determine the optimal policies, strategies, and distributions for both the defender and the attackers. This will allow us to predict the outcomes of the game under various scenarios, providing insight into how effective the defender’s strategy is in mitigating the attackers’ impact and how the attackers can best exploit potential vulnerabilities.

The generated policies are supplied by using the proposed approach as follows:

$$\begin{array}{c}{\pi }^{1\ast }(a^1|m^1)=\left[\begin{array}{cc}0.3925& 0.6075\\ 0.3925& 0.6075\\ 0.3925& 0.6075\\ 0.3961& 0.6039\end{array}\right]\end{array}\begin{array}{c}{\pi }^{2\ast }(a^2|m^2)=\left[\begin{array}{cc}0.4748& 0.5252\\ 0.4748& 0.5252\\ 0.4748& 0.5252\\ 0.2492& 0.7508\end{array}\right]\end{array}\begin{array}{c}{\pi }^{3\ast }(a^3|m^3)=\left[\begin{array}{cc}0.7903& 0.2097\\ 0.7904& 0.2096\\ 0.7904& 0.2096\\ 0.6700& 0.3300\end{array}\right]\end{array}$$

The defender’s resultant (behavior) strategies are provided by

$$\begin{array}{c}{\sigma }^{1\ast }(m^1|{\theta }^1)=\left[\begin{array}{cccc}0.3204& 0.3205& 0.3204& 0.0387\\ 0.0223& 0.0223& 0.0223& 0.9332\\ 0.2253& 0.2253& 0.2253& 0.3240\\ 0.2500& 0.2500& 0.2500& 0.2500\end{array}\right]\end{array}$$

and the (behavior) strategies of the attackers are

$$\begin{array}{c}{\sigma }^{2\ast }(m^2|{\theta }^2)=\left[\begin{array}{cccc}0.0919& 0.0919& 0.0919& 0.7243\\ 0.0573& 0.0573& 0.0573& 0.8280\\ 0.2493& 0.2495& 0.2494& 0.2518\\ 0.2499& 0.2501& 0.2500& 0.2500\end{array}\right]\end{array}\begin{array}{c}{\sigma }^{3\ast }(m^3|{\theta }^3)=\left[\begin{array}{cccc}0.0287& 0.0287& 0.0287& 0.9139\\ 0.2501& 0.2503& 0.2501& 0.2495\\ 0.2491& 0.2494& 0.2498& 0.2517\\ 0.2498& 0.2498& 0.2500& 0.2504\end{array}\right]\end{array}$$

The distribution vectors are as follows:

$$\begin{array}{ccc}{P}^{1\ast }\left({\theta }^1\right)=\left[\begin{array}{c}0.4657\\ 0.4492\\ 0.0448\\ 0.0403\end{array}\right]& P^{2\ast }\left({\theta }^2\right)=\left[\begin{array}{c}0.1097\\ 0.1752\\ 0.5679\\ 0.1472\end{array}\right]& P^{3\ast }\left({\theta }^3\right)=\left[\begin{array}{c}0.3492\\ 0.2401\\ 0.1065\\ 0.3041\end{array}\right]\end{array}$$

The algorithm ensures the convergence of the strategies shown in Figure 1, Figure 2 and Figure 3, optimizing decision-making and adapting efficiently to dynamic threats in real-time environments. Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 demonstrate the convergence of the error in the reinforcement learning algorithm, illustrating its efficiency and effectiveness in optimizing strategies and adapting to dynamic environments over time. By integrating prior information and utilizing a continuous-time approach, the model enhances its robustness, ensuring that the security system remains resilient against evolving threats. These visualizations confirm the model’s capability to maintain optimal performance and adaptiveness.

The primary goal of patrol scheduling is to efficiently allocate security teams to safeguard fixed targets, taking into account limited workforce resources. To address this challenge, Algorithm 3 is utilized to illustrate the realization of an SSG, where target visitations are determined based on the resulting game strategy, denoted as ${\pi }_{k|m}$. This algorithm aids in the planning and deployment of patrols to maximize the protection of critical assets.

Figure 10 provides a visual representation of an instance of the SSG, showcasing the outcomes of engagements between attackers and defenders. In this particular scenario, attacker 2 is apprehended at state 1 by defender 1 after 10 time steps, indicating the successful interception of the threat. Additionally, attacker 2 is captured at state 1, this time after 12 time steps by defender 1. These outcomes demonstrate the effectiveness of the patrol planning process in thwarting attacks and protecting the designated targets. By employing Algorithm 2 and visualizing the SSG outcomes in Figure 1, security planners can gain insights into the effectiveness of their patrol scheduling strategies. This approach allows for the optimization of patrol routes and deployment strategies, ultimately enhancing the security posture despite workforce limitations.

In the alternate realization depicted in Figure 11, a different sequence of events unfolds within the SSG. Here, attacker 2 faces a swift apprehension at state 1 by defender 1 after three time steps. Simultaneously, attacker 1 is intercepted at state 1, albeit after a longer duration of nine time steps, by defender 1. Despite the varying durations and defender involvement, both attackers are ultimately captured. The prompt resolution of attacker 2 by defender 1 underscores the importance of timely responses in security operations. This rapid intervention effectively neutralizes the threat before it can progress further. Meanwhile, the prolonged engagement leading to the capture of attacker 1 highlights the complexities and challenges inherent in security patrolling and target defense.

With the apprehension of both attackers, the game reaches its conclusion. The successful outcomes achieved by the defenders validate the effectiveness of the patrol scheduling strategies implemented. Such realizations provide valuable insights for refining future security tactics, emphasizing the importance of adaptability and resource allocation in mitigating security risks effectively.

7. Conclusions

This research introduces a novel mathematical framework that enhances security measures by integrating optimization techniques to mitigate existing risks while addressing real-time threats. The framework is based on a Bayesian–Markov Stackelberg game model, designed to capture adversarial interactions in security scenarios with incomplete information, where players have limited knowledge of each other’s strategies.

A key feature of this approach is the use of an optimization strategy grounded in the proximal-gradient method, which significantly improves computational efficiency compared to traditional Bayesian–Markov Stackelberg solutions. By leveraging this method, the framework effectively reduces security risks associated with persistent threats. Notably, it incorporates a unique reinforcement learning algorithm that derives rewards from a prior Bayesian distribution. This contribution is particularly significant, as it integrates structured historical data into decision-making, enabling more adaptive and informed responses to evolving security threats.

The framework’s practical effectiveness is demonstrated through numerical examples, showcasing its ability to utilize past information to guide present decisions. These results provide both theoretical insights and empirical evidence of its applicability in real-world security scenarios. Overall, this research presents a comprehensive security optimization approach that bridges the gap between mitigating existing risks and countering ongoing threats in real time. By combining advanced optimization techniques with an innovative reinforcement learning paradigm, this offers a promising avenue for strengthening security in complex, dynamic environments.

Looking ahead, several challenges remain. A key technical endeavor is the implementation of a game-theoretic approach using novel reinforcement learning methods, where capture conditions depend on states and incomplete information, enhancing the realism of patrolling schedules. Additionally, applying this methodology in fortification games and conducting controlled trials to evaluate player responses to game-theoretic scheduling in real-world settings represent significant challenges. These trials can yield valuable insights into the practical implications of game theory for strategic decision-making and scheduling in uncertain environments, ultimately ensuring more robust and adaptable security measures.