A Bayesian Stackelberg Game Approach to Remote State Estimation Under SINR-Based DoS Attacks with Incomplete Information

Abstract

With limited energy constraints, the issue of transmission and interference strategies have received considerable critical attention in cyber–physical security. In this paper, for remote state estimation under signal-to-interference-plus-noise ratio-based denial-of-service (DoS) attacks, the Stackelberg game between the sensor and the attacker is investigated. To balance estimation performance and energy consumption, the two players determine the transmission power and interference power sequentially under an incomplete information structure where the sensor does not know the fading channel gain of the attacker exactly. The schedule problem over the infinite-time horizon is first formulated as a Markov decision process with finite state and action spaces. Then, a Bayesian Stackelberg game (BSG) is constructed by incorporating the probability information of the channel interference gain. Based on the definition of best-response, the solution of the BSG is presented and the existence of the Stackelberg equilibrium is proven. Furthermore, a Stackelberg Q-learning algorithm is used to obtain the optimal strategies for the two players. Numerical results demonstrate the effectiveness of the proposed game method when the sensor is unable to access an attacker’s channel gain information.

1. Introduction

Cyber–physical systems (CPSs), which integrate computational capabilities with physical processes, have found extensive application across a range of industrial operations and critical infrastructure [1]. However, unreliable wireless communication networks render these systems highly susceptible to malicious cyber attacks, such as false-data injection (FDI), eavesdropping, and denial-of-service (DoS) attacks, which pose severe threats to their operational security and reliability [2].

In CPSs, security against malicious attacks, especially DoS attacks, has become a critical concern [3,4]. Game theory has emerged as a widespread and powerful analytical tool for modeling such attacker–defender interactions [5]. However, most existing game-theoretic studies are built upon the Nash equilibrium concept, which assumes simultaneous decision-making among players [6]. This assumption does not adequately capture the hierarchical structure often present in security scenarios, where a defender typically acts first and an attacker responds accordingly. To address this, Stackelberg games have been introduced, providing a more realistic framework for sequential decision-making in layered security strategies [7,8]. Furthermore, the majority of existing Stackelberg models rely on the assumption of complete information, where all players have perfect knowledge of each other’s parameters and payoffs. This is often impractical in real-world communication environments because some critical attributes such as channel conditions, interference gains, or attacker capabilities may not be fully accessible to the defender. The lack of such information significantly affects the strategic design and performance of defensive measures. Although some recent studies have begun to address information asymmetry, their modeling of uncertainty remains limited. For instance, most of these works focus on single-type uncertainty or assume that the attacker’s channel gain follows a simple bivariate distribution [9]. Such simplifications fail to capture the practical situation where the attacker’s fading channel gain may follow a multi-type probability distribution.

Motivated by these gaps, this paper investigates remote state estimation under SINR-based DoS attacks within a Bayesian Stackelberg game framework. It is worth noting that our work focuses on the strategic response and resource allocation following the detection or presumption of an attack, rather than on the attack detection mechanisms themselves [10]. While Bayesian Stackelberg models have been studied for general network security, this work distinctively integrates this framework into the SINR-based remote state estimation to handle multi-type channel gain uncertainty and further develops a model-free Stackelberg Q-learning algorithm to derive optimal strategies under this specific information structure. In the incomplete information setting, probabilistic distribution information is incorporated to model the uncertainty of the attacker’s fading channel gain, and a sequential power allocation strategy is developed to balance estimation accuracy and energy efficiency. The proposed approach not only extends current Stackelberg-based security analysis to more realistic communication scenarios but also offers implementable learning-based strategies for deriving equilibrium policies under information asymmetry. The main contributions of this paper are summarized as follows:

• We study the strategic interaction between a sensor and a DoS attacker under energy constraints in an SINR-based remote state estimation system with unknown channel gain information. The sequential decision-making process is formulated as a finite-state and finite-action MDP, capturing the dynamic and uncertain nature of the environment.
• To address the incomplete information regarding the attacker’s channel gain, we model the interaction as a BSG, where the attacker is treated as having multiple possible types. Within this framework, we define the best-response strategies, prove the existence of an SE, and analytically derive the SE strategies for both players.
• A Stackelberg Q-learning algorithm is used to enable both players to learn their optimal strategies without prior knowledge of the opponent’s payoff structure. This model-free approach ensures adaptability and practicality in real-world settings. Finally, numerical simulations validate the effectiveness and robustness of the proposed BSG-based method, demonstrating its clear advantage over conventional Nash-based approaches under information asymmetry.

The remainder of this article is organized as follows. Section 2 reviews related work on secure state estimation under cyber-attacks. Section 3 establishes the system model for remote state estimation under SINR-based DoS attacks with incomplete channel gain information. Section 4 formulates the sequential decision-making process as a two-player Markov Decision Process (MDP). Building upon the MDP, Section 5 constructs the Bayesian Stackelberg Game (BSG) framework and analyzes the Stackelberg equilibrium. Section 6 designs a Stackelberg Q-learning algorithm to compute the optimal strategies for both players. Section 7 provides simulation results to verify the effectiveness of the proposed approach. Finally, conclusions are drawn in Section 8.

2. Related Work

Accurate and secure state estimation serves as the cornerstone of reliable decision-making in CPSs. However, the widespread dependence on wireless networks for data transmission exposes the estimation process to various cyber attacks, including FDI, eavesdropping, and DoS attacks. These attacks compromise data integrity, availability, and confidentiality [11]. These attacks can deliberately degrade estimation accuracy by corrupting, intercepting, or blocking data transmissions, which, in turn, destabilize system operation and threaten physical security [12,13].

DoS attacks, especially those targeting the signal-to-interference-and-noise ratio (SINR) of wireless channels, are the most disruptive to remote state estimation [4,14,15]. The sensor, as the defender, aims to enhance the system performance with less transmission power. On the contrary, the goal of the DoS attacker is to reduce estimation accuracy while consuming more communication resources. To describe such adversarial dynamics between attackers and defenders, game theory has emerged as a powerful analytical tool for the interactive decision-making process [16,17,18,19]. With energy constraints for both the sensor and the attacker, a Nash equilibrium (NE) of a zero-sum game was proven in [20] to be the optimal strategies for both sides. Then, the authors in [21] proposed a a Markov game framework under the SINR-based DoS attacks and applied a modified Nash Q-learning algorithm to obtain the optimal solutions. For multi-channel networks, a two-player zero-sum stochastic game was formulated to design the mixed strategies for the channel selection of the sensor and DoS attacker [22].

It is worth pointing out that most works on attack–defense games adopt NE as the game solution based on the assumption that the defender and the attacker choose their actions simultaneously, which is not applicable for situations where the players deploy their strategies sequentially. Consequently, the Stackelberg game, with its explicit “leader–follower” structure, has been a more appropriate and realistic framework for capturing the hierarchical interactions [23]. For the SINR-based fading wireless network, a Stackelberg equilibrium (SE) within a two-player nonzero-sum Markov game framework was constructed in [24] to obtain the transmission and interference strategies for the sensor and attacker, respectively. By taking the Stackelberg game method, the linear quadratic Gaussian (LQG) control strategy was analyzed in [25,26] for FDI attacks and DoS attacks, respectively. For distributed Kalman filtering, a Stackelberg game-based distributed reinforcement learning algorithm was developed to produce joint optimal strategies based on local observation information [27]. Within the Stackelberg game framework, an optimal stealthy robust attack method was designed based on Wasserstein ambiguity sets [28].

The aforementioned Stackelberg-based studies rely on the assumption of complete information, which rarely holds in practical wireless communication scenarios. For players in games, some crucial environmental information and other’s key attributes are often difficult to obtain, such as energy budgets, acknowledgment (ACK) information, and channel gains of the wireless network. An anti-jamming Bayesian Stackelberg game was proposed in [29], where the uncertainties of the channel state information and transmission cost information were incorporated. For the problem of multi-channel power schedule SINR-based DoS attacks, the SE was studied under two types of incomplete information, where the existence of the attacker and the total power of the attacker were, respectively, unknown to the defender [9]. For a multi-hop network under DoS attacks, the defender had no access to the available energy of the attacker; then, a Bayesian Stackelberg game was implemented and a Stackelberg Q-learning algorithm was presented to obtain the SE [30]. Furthermore, a stochastic Bayesian game was formulated in [31], where the ACK information from the remote estimator to the sensor was hidden from the attacker. While game theory provides a robust framework for strategic analysis, it is also worth noting that formal methods have been extensively explored for the verification and analysis of CPSs under attacks [32]. In contrast to these verification-based approaches, this paper emphasizes the adaptive decision-making process in scenarios where the sensor faces incomplete information about an attacker’s characteristics.

3. Problem Formulation

Notations: ${\mathbb{N}}_0$ denotes the set of nonnegative integers. ${\mathbb{R}}^n$ and ${\mathbb{R}}^{m\times n}$ are the n-dimensional Euclidean space and set of all $m\times n$ matrices, respectively. ${\mathbb{S}}_{⪰0}^n$ and ${\mathbb{S}}_{\succ 0}^n$ represent the sets of $n\times n$ real symmetric positive semi-definite and positive definite matrices, respectively. If $X\in {\mathbb{S}}_{⪰0}^n$, we simply write $X\ge 0$ ($X>0$ if $X\in {\mathbb{S}}_{\succ 0}^n$). Notation $X<Y$ means that the matrix $X-Y$ is negative definite. $Pr(·)$ is the probability of a random event. $\mathrm{E}[·]$ and $\mathrm{Cov}[·]$ stand for the expectation and covariance of a random vector, respectively. For functions g, h with appropriate domains, $g\circ h(·)$ represents the function composition $g\left(h\right(·\left)\right)$, with $h^n(·)\triangleq h\left(h^{n-1}(·)\right)$.

In this section, we introduce the remote state estimation system under DoS attack depicted in Figure 1. The local sensor estimates the system states by the Kalman filter and then transmits the estimates to the remote estimator through the signal-to-noise ratio (SNR)-based network. However, the network is unreliable and may be attacked by a DoS attacker.

3.1. System and Sensor Model

Consider the discrete-time linear time invariant system:

$$\begin{array}{cc}\hfill x_{k+1}& =A{x}_k+w_k,\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill y_k& =C{x}_k+v_k,\hfill \end{array}$$

(2)

where $x_k\in {\mathbb{R}}^n$ is the system state and $y_k\in {\mathbb{R}}^m$ is the measurement output. The process noise $w_k\in {\mathbb{R}}^n$ and the measurement noise $v_k\in {\mathbb{R}}^m$ are assumed to be independent white Gaussian noises with covariance matrices $Q\in {\mathbb{S}}_{\succ 0}^n$ and $R\in {\mathbb{S}}_{\succ 0}^m$, respectively. The initial state $x_0$ is zero-mean Gaussian with covariance $P_0^{-}\in {\mathbb{S}}_{\succ 0}^n$, and independent of $w_k$ and $v_k$ for all $k\in {\mathbb{N}}_0$. The pair $(A,C)$ is detectable.

The smart sensor runs a local Kalman filter to estimate the system states based on the the measurement set up to the current time k, that is, $Y_k=\{y_1,y_2,\dots ,y_k\}$. The local minimum mean-squared error (MMSE) estimate of the system state $x_k$ and the corresponding estimation error are, respectively, defined as

$$\begin{array}{cc}\hfill {\widehat{x}}_k^s& =\mathrm{E}\left[x_k\right|Y_k],\hfill \\ \hfill P_k^s& =\mathrm{E}\left[(x_k-{\widehat{x}}_k^s){(x_k-{\widehat{x}}_k^s)}^{\top }\right|Y_k].\hfill \end{array}$$

We define the the Lyapunov and Riccati operators h and g: ${\mathbb{S}}_{⪰0}^n\to {\mathbb{S}}_{⪰0}^n$ as follows:

$$\begin{array}{cc}\hfill h\left(X\right)& \triangleq AX{A}^{\top }+Q,\hfill \\ \hfill g\left(X\right)& \triangleq X-X{C}^{\top }{\left[CX{C}^{\top }+R\right]}^{-1}CX.\hfill \end{array}$$

The estimation error covariance of the Kalman filter converges to a unique value irrespective of the initial condition. For simplicity, it is assumed that the local Kalman filter has reached the steady state, and we let

$$P_0^s=\overline{P},$$

(3)

where $\overline{P}$ is the steady-state error covariance given by the unique positive semidefinite solution of $g\circ h\left(X\right)=X$.

3.2. Communication Channel and Attack Model with Incomplete Information

After obtaining the state estimate $x_k^s$, the sensor transmits the estimate to the remote estimator in the form of a data packet. However, random data packet drops may occur owing to channel fading and interference. Here, we assume that the sensor communicates with the remote estimator over an additive white Gaussian noise (AWGN) network to model this scenario. The packet dropout probability at time k is described by $SN{R}_k={\displaystyle \frac{\alpha {\lambda }_k}{{\sigma }_k}}$, where $\alpha >0$ is the fading channel gain of the sensor, ${\lambda }_k$ is the transmission power of the sensor at time k, and ${\sigma }_k$ is the additive white Gaussian noise power.

Considering a DoS attacker against the channel, the SNR is revised as the following SINR:

$$SIN{R}_k={\displaystyle \frac{\alpha {\lambda }_k}{{\sigma }_k+\beta {\delta }_k}},$$

(4)

where $\beta >0$ is the fading channel gain of the attacker; ${\delta }_k$ is the transmission power of the attacker at time k.

The transmission of $x_k^s$ between the sensor and the remote estimator can be characterized by a binary random process $\left\{{\gamma }_k\right\}$:

$${\gamma }_k=\left\{\begin{array}{cc}1,\hfill & \mathrm{if} x_k^s\mathrm{is} \mathrm{received} \mathrm{successfully},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

Then, the packet arrival rate ${\mu }_k$ is given as

$${\mu }_k\triangleq Pr({\gamma }_k=1)=1-2Q\left(\sqrt{\kappa SIN{R}_k}\right),$$

(5)

where $\kappa >0$ is a parameter; $Q\left(x\right)={\displaystyle \frac{1}{\sqrt{2\pi }}}{\int }_x^{\infty }exp(-{\displaystyle \frac{u^2}{2}})du$ represents the standard Q-function. It should be noted that the scalar function $Q(·)$ here is distinct from the system noise covariance matrix Q defined in the system models (1) and (2). It is seen that the SINR is not only dependent on the sensor’s transmission power but is also influenced by the interference power generated by the DoS attacker. Obviously, the larger SINR leads to the lower packet loss rate and better estimation performance.

In practice, the channel state information is not perfect. The uncertainties of the channel gain information need to be considered. Suppose that the attacker interferes with the transmission under channel gain ${\beta }_j$ with probability $\eta \left({\beta }_j\right)$, where $j\in \mathcal{N}=\{1,2,\dots ,H\}$ and ${\sum }_{j=1}^H\eta \left({\beta }_j\right)=1,H\ge 2$. Then, the SINR with the channel interference gain ${\beta }_j$ is defined as

$$SIN{R}_k^j={\displaystyle \frac{\alpha {\lambda }_k}{{\sigma }_k+{\beta }_j{\delta }_k^j}}.$$

(6)

Correspondingly, the packet arrival rate under ${\mathrm{SIN}\mathrm{R}}_k^j$ is given as

$${\mu }_k^j\triangleq Pr({\gamma }_k=1)=1-2Q\left(\sqrt{\kappa SIN{R}_k^j}\right).$$

(7)

This multi-type probabilistic model captures a realistic scenario where the sensor, as a defender, cannot precisely identify the attacker’s equipment or instantaneous channel condition but can estimate a probability distribution over a few distinct levels of threat severity. In this case, the sensor does not know the channel interference gain of the attacker but possesses the probability distribution of $\beta$. Therefore, with different levels of channel interference gain, we can think that there exists H type attackers in the environment.

Remark 1.

In practice, the acquisition of channel information often has a non-symmetric characteristic. An attacker can frequently intercept the sensor’s open pilot signals or reference transmissions. Then, the attacker can accurately estimate the sensor’s transmission channel gain by processing these known sequences [33,34]. When launching an active attack, the attacker employs non-cooperative, noise-like, or protocol-aware jamming signals, which are deliberately designed to be unpredictable [35]. Thus, it is difficult for the sensor to accurately estimate the interference channel gain from the attacker.

3.3. Remote State Estimation

The MMSE estimate of $x_k$ and the estimation error covariance at the remote estimator are denoted as ${\widehat{x}}_k$ and $P_k$, respectively. According to whether the date is received successfully, the state estimation is given by

$${\widehat{x}}_k=\left\{\begin{array}{cc}{\widehat{x}}_k^s,\hfill & {\gamma }_k=1,\hfill \\ A{x}_{k-1},\hfill & {\gamma }_k=0.\hfill \end{array}\right.$$

(8)

As a result, the error covariance $P_k$ is computed as

$$P_k=\left\{\begin{array}{cc}\overline{P},\hfill & {\gamma }_k=1,\hfill \\ h\left(P_{k-1}\right),\hfill & {\gamma }_k=0.\hfill \end{array}\right.$$

(9)

Assume that the initial value of the error covariance at the remote estimator also starts from $\overline{P}$, i.e., $P_0=\overline{P}$. The remote estimator will send ACKs to the sensor to indicate whether it has received the estimate at each time. Hence, the sensor can also calculate $P_k$ by (9). Note that $P_k$ takes values from the infinitely set $\{\overline{P},h\left(\overline{P}\right),h^2\left(\overline{P}\right),\dots \}$.

3.4. Strategy and Objective Function

Considering that both the sensor and the attacker have power constraints, we assume that the transmission power and attack power take values from the finite sets. Specifically, the transmission power ${\lambda }_k$ belongs to a finite set with level $l_d$, which is denoted as $\mathrm{\Lambda }=\{{\lambda }_1,\dots ,{\lambda }_{l_d}\}$. The interference power ${\delta }_k^j$ belongs to a finite set with level $l_a^j$, which is denoted as ${\mathrm{\Delta }}^j=\{{\delta }_1^j,\dots ,{\delta }_{l_a^j}^j\}$. This discretization models the practical constraints of digital power amplifiers and energy-limited devices and is a common simplification in power control problems.

The strategies of the sensor and the j-th type attacker at time k are ${\lambda }_k\in \mathrm{\Lambda }$ and ${\delta }_k^j\in {\mathrm{\Delta }}^j$ for all $k\in {\mathbb{N}}_0$, respectively. Then, the strategy of the attacker at time k is denoted as

$${\delta }_k=\{{\delta }_k^1,{\delta }_k^2,\dots ,{\delta }_k^H\}.$$

From (6) and (7), different types of attacker lead to different packet arrival rates and further affect the expected estimation error. The trace of the expected estimation error covariance under the attacker’s channel gain ${\beta }_j$ is given as

$$f_k^j\triangleq \mathrm{E}\left[\mathrm{tr}\left(P_k^j\right)\right]=\mathrm{tr}\left({\mu }_k^j\overline{P}+(1-{\mu }_k^j)h\left(P_{k-1}\right)\right).$$

(10)

The sensor aims to optimize the expected estimation error while reducing the total transmission cost. In contrast, the goal of the attacker is to deteriorate the estimation performance of the remote estimator and consume as much transmission energy of the sensor as possible while minimizing the total attack cost. Therefore, the one-step rewards of the sensor and the attacker with channel gain ${\beta }_j$ are, respectively, given as

$$\begin{array}{cc}\hfill r_{k,d}({\lambda }_k,{\delta }_k)& =-\sum _{j=1}^H\eta \left({\beta }_j\right)f_k^j-C_d{\lambda }_k,\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill r_{k,a}^j({\lambda }_k,{\delta }_k^j)& =f_k^j+C_d{\lambda }_k-C_a{\delta }_k^j,\hfill \end{array}$$

(12)

where $C_d$ and $C_a$ are the costs per unit power for the sensor and DoS attacker, respectively. Then, the corresponding payoff functions over the infinite time horizon for the sensor and the j-th type attacker are, respectively, given as follows:

$$\begin{array}{cc}\hfill J_d({\lambda }_k,{\delta }_k)& =\sum _{k=1}^{\infty }{\rho }_{k-1}{r}_{k,d}({\lambda }_k,{\delta }_k),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill J_a^j({\lambda }_k,{\delta }_k^j)& =\sum _{k=1}^{\infty }{\rho }_{k-1}{r}_{k,a}^j({\lambda }_k,{\delta }_k^j),\hfill \end{array}$$

(14)

where $\rho \in (0,1)$ is the discount factor.

For the considered system under attack, the sensor as the defender first decides its transmission power level and then the DoS attacker chooses the interference power level based on the current situation. For such sequentially interactive decision-making processes with incomplete information, we use the Bayesian Stackelberg game framework to analyze the optimal strategies for the sensor and attacker.

Remark 2.

Existing studies on SINR-based attacks against state estimation predominantly design transmission and interference strategies under the assumption of complete channel information [20,21,22,26]. This idealization ignores the reality that channel knowledge is incomplete and asymmetric in actual wireless networks, leading to multi-modal players and their decision-making spaces. Therefore, investigating the resulting hierarchical game is both highly meaningful and challenging.

4. MDP Formulation

In this section, a Markov decision process is first formulated to describe the dynamics of the state estimation at the remote estimator. Obviously, the scenario involves two players: the sensor and DoS attacker. The sensor does not know the attacker’s channel gain but has knowledge of the probability distribution information. At each time, the sensor and attacker take action based on the current process state and the information that they have previously collected. Then, they respectively receive the rewards, and the process moves to the next state according to the transition probability.

Taking into account the decision-making interaction between the sensor and the attacker, the state estimation process at the remote estimator can be formulated as a two-player MDP model, which consists of the following five essential elements:

(1) Desion epoch: Let T denote the infinite discrete set of decision epochs, i.e., $T=\{1,2,\dots \}$.

(2) State: According to (9), the estimation error covariance can be alternatively written as $P_k=h^{{\tau }_k}\left(\overline{P}\right)$, where ${\tau }_k=\{1,2,\dots \}$ is the holding time at time k, namely, the time interval between the current time k and the latest moment of the receiving packet. Then, the set of the possible estimation error covariance $P_k$ can be represented as ${\mathcal{Z}}_k=\{\overline{P},h\left(\overline{P}\right),\dots ,h^k\left(\overline{P}\right)\}$. Therefore, the state at time k is defined as the holding time of the time $k-1$, i.e., $s_k={\tau }_{k-1}$. Intuitively, this state $s_k$ represents the time interval elapsed since the last successful packet reception. According to the update Formula (9), it directly determines the current estimation error covariance at the remote estimator. Due to the low probability of a large holding time, the final state K is used to represent all states with ${\tau }_i\ge K$. Therefore, the state space is $\mathcal{S}=\{0,1,\dots ,K\}$.

(3) Action: At each time, based on the state $s_k$, the sensor and the j-th type attacker choose the actions ${\lambda }_k$ and ${\delta }_k^j$ from the action spaces ${\mathcal{A}}_d=\mathrm{\Lambda }$ and ${\mathcal{A}}_a^j={\mathrm{\Delta }}^j$, respectively.

(4) Transmission probability: Given the action ${\lambda }_k$ of the sensor and the action ${\delta }_k^j$ of the j-th type attacker, the probability that the state transmits from $s_k$ to $s_{k+1}$ is given as $\mu$.

$$Pr\left(s_{k+1}\right|s_k,{\lambda }_k,{\delta }_k^j)=\left\{\begin{array}{cc}{\mu }_k^j,\hfill & s_{k+1}=0,\hfill \\ 1-{\mu }_k^j,\hfill & s_{k+1}=s_k+1,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(15)

(5) Reward: The one-stage reward functions for the sensor and the j-th type attacker are, respectively, given as

$$\begin{array}{cc}\hfill & r_{k,d}(s_k,{\lambda }_k,{\delta }_k)\hfill \\ \hfill =& -\sum _{j=1}^H\eta \left({\beta }_j\right)\mathrm{tr}[{\mu }_k^j\overline{P}+(1-{\mu }_k^j)h^{s_k+1}\left(\overline{P}\right)]-C_d{\lambda }_k,\hfill \end{array}$$

(16)

and

$$\begin{array}{cc}\hfill & r_{k,a}^j(s_k,{\lambda }_k,{\delta }_k^j)\hfill \\ \hfill =& \mathrm{tr}[{\mu }_k^j\overline{P}+(1-{\mu }_k^j)h^{s_k+1}\left(\overline{P}\right)]+C_d{\lambda }_k-C_a{\delta }_k^j.\hfill \end{array}$$

(17)

Note that as the one-stage reward functions are time-invariant and stationary and can also be denoted as $r_d(s_k,{\lambda }_k,{\delta }_k)$ and $r_a^j(s_k,{\lambda }_k,{\delta }_k^j)$.

The sensor’s strategy ${\pi }_d\left(s\right)={\lambda }_k\left(s\right)$ means that the sensor takes action ${\lambda }_k$ under state $s_k$. Similarly, the strategy of the j-th type attacker under state s is denoted as ${\pi }_a^j\left(s\right)={\delta }_k^j\left(s\right)$. Then, the strategies of the sensor and the j-th type attacker are, respectively, written as

$$\begin{array}{cc}\hfill {\pi }_d& =\left\{{\pi }_d\left(s\right)\right|\forall s\in \mathcal{S}\}\in {\mathrm{\Pi }}_d,\hfill \\ \hfill {\pi }_a^j& =\left\{{\pi }_a^j\left(s\right)\right|\forall s\in \mathcal{S}\}\in {\mathrm{\Pi }}_a^j,\hfill \end{array}$$

where ${\mathrm{\Pi }}_d$ and ${\mathrm{\Pi }}_d^i$ are the set of all stationary and deterministic policies for the sensor and j-th type attacker, respectively. Furthermore, the strategy of the attacker is denoted as

$${\pi }_a=[{\pi }_a^1,{\pi }_a^2,\dots ,{\pi }_d^H]\in {\mathrm{\Pi }}_a$$

where ${\mathrm{\Pi }}_a$ is the set of all stationary and deterministic policies for the attacker. Then, the payoff functions of the sensor and j-th type attacker are, respectively, given as

$$\begin{array}{cc}\hfill J_d(s,{\pi }_d,{\pi }_a)& =\sum _{k=1}^{\infty }{\rho }^{k-1}{r}_d(s_k,{\lambda }_k,{\delta }_k),\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill J_a^j(s,{\pi }_d,{\pi }_a^j)& =\sum _{k=1}^{\infty }{\rho }^{k-1}{r}_a^j(s_k,{\lambda }_k,{\delta }_k^j).\hfill \end{array}$$

(19)

The goal of both the sensor and the attacker is to seek the optimal strategy to maximize the payoff function.

5. Bayesian Stackelberg Game

Based on the above MDP framework, a BSG with two players was investigated for designing the optimal strategies for both the sensor and the attacker. In this hierarchical game, the sensor, as the leader, first makes its action according to the state and declares its strategy. It is noted that the sensor knows the reaction from the attacker in a Stackelberg game. Then, the attacker, as the follower, takes action based on the acquired channel gain and the transmission power of the sensor. The payoff functions of the sensor and j-th type attacker are given as (18) and (19), respectively. The SE of the BSG is analyzed and the corresponding optimal strategies for both players are provided. The best response for each side is first defined as follows.

Definition 1.

The best response is that a player takes an action that optimizes its own payoff while taking into account other players’ actions. Specifically, the best responses for the sensor and j-th type attacker are given as

$$\begin{array}{cc}\hfill {\varphi }_d^{*}\left({\pi }_a\right)& =arg\underset{{\pi }_d\in {\mathrm{\Pi }}_d}{max}{J}_d(s,{\pi }_d,{\pi }_a),\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill {\varphi }_a^{j,\ast }\left({\pi }_d\right)& =arg\underset{{\pi }_a^j\in {\mathrm{\Pi }}_a^j}{max}{J}_a^j(s,{\pi }_d,{\pi }_a^j), j\in \mathcal{N}.\hfill \end{array}$$

(21)

Then, the best responses for the attacker are denoted as

$${\varphi }_a^{*}\left({\pi }_d\right)=[{\varphi }_a^{1,\ast }\left({\pi }_d\right),{\varphi }_a^{2,\ast }\left({\pi }_d\right),\dots ,{\varphi }_a^{H,\ast }\left({\pi }_d\right)].$$

(22)

The best-response set of the j-th type attacker to the sensor’s strategy ${\pi }_d$ is denoted as ${\mathcal{R}}_a^j\left({\pi }_d\right)$ and ${\mathcal{R}}_a\left({\pi }_d\right)=[{\mathcal{R}}_a^1\left({\pi }_d\right),\dots ,{\mathcal{R}}_a^H\left({\pi }_d\right)]$ is the best-response set of the attacker to strategy ${\pi }_d$.

Lemma 1.

For the proposed BSG with two players, the Stackelberg equilibrium ${\pi }_d^{SE}$ of the sensor satisfies

$$J_d(s,{\pi }_d^{SE},{\varphi }_a^{*}\left({\pi }_d^{SE}\right))\ge J_d(s,{\pi }_d,{\varphi }_a^{*}\left({\pi }_d\right)), \forall {\pi }_d\in {\mathrm{\Pi }}_d.$$

(23)

where ${\varphi }_a^{*}\left({\pi }_d^{SE}\right)$ is expressed as (22).

Proof. 

In the BSG, the channel gain and the strategy of the sensor can be obtained and imposed on the attacker; then, each type of attacker takes the best response to maximize its own payoff function. The sensor chooses the optimal strategy by taking account into the follower’s best response to maximize its own payoff functions, which completes the proof. □

Based on Lemma 1, the solution to the proposed Stackelberg game with incomplete channel gain information is given by the following theorem.

Theorem 1.

The solution to the proposed BSG is given by first solving

$${\pi }_d^{SE}={\varphi }_d^{*}\left({\varphi }_a^{*}\left({\pi }_d^{SE}\right)\right),$$

(24)

and then calculating

$$\begin{array}{c}\hfill {\pi }_a^{SE}=[{\pi }_a^{1,SE},{\pi }_a^{2,SE},\dots ,{\pi }_a^{H,SE}],\end{array}$$

(25)

where

$${\pi }_a^{j,SE}={\varphi }_a^{j,\ast }\left({\pi }_d^{SE}\right),j\in \mathcal{N}.$$

(26)

Therefore, the SE of this game is $({\pi }_d^{SE},{\pi }_a^{SE})\in {\mathrm{\Pi }}_d\times {\mathrm{\Pi }}_a$.

Proof. 

In the proposed BSG, the sensor as the leader and the DoS attacker as the follower make decisions sequentially. Given the sensor’s strategy ${\pi }_d$, the j-th type attacker will take the best response according to ${\pi }_a^j={\varphi }_a^{j,\ast }\left({\pi }_d\right)$. The sensor knows the reaction from the attacker; hence, it will choose the optimal strategy to maximize its own payoff function while taking account into the attacker’s strategy, that is,

$$\begin{array}{cc}\hfill {\pi }_d^{SE}& =arg\underset{{\pi }_d\in {\mathrm{\Pi }}_d}{max}{J}_d(s,{\pi }_d,{\varphi }_a^{*}\left({\pi }_d^{SE}\right))\hfill \\ \hfill & ={\varphi }_d^{*}\left({\varphi }_a^{*}\left({\pi }_d^{SE}\right)\right).\hfill \end{array}$$

After obtaining the sensor’s strategy ${\pi }_d^{SE}$, the j-th type attacker will choose the optimal strategy ${\pi }_a^{j,SE}$ by calculating

$${\pi }_a^{j,SE}={\varphi }_a^j\left({\pi }_d^{SE}\right),j\in \mathcal{N}$$

which completes the proof. □

Remark 3.

This theorem provides a constructive method to find the SE for the proposed BSG. It confirms that within the finite strategy spaces considered, the sensor can determine its optimal strategy by anticipating and incorporating the attacker’s best response, which, in turn, is computed based on the sensor’s declared action. This fixed-point characterization forms the theoretical foundation for the learning algorithm developed in Section 6.

Proposition 1.

There exists at least one SE in the BSG.

Proof. 

Given the sensor’s strategy ${\pi }_d$, the j-th type attacker will choose the strategy from the set ${\pi }_a^j$. Given the final state K, the numbers of possible strategies for the sensor and the j-th type attacker are $l_d^{K+1}$ and $l_a^{j,K+1}$, respectively. The j-th type attacker’s strategy space ${\pi }_a^j$ is finite, its optimal response ${\varphi }_a^{i,\ast }\left({\pi }_d\right)$ always exists for the fixed sensor’s strategy ${\pi }_d$. Moreover, the finite strategy space implies the existence of an equilibrium strategy for the sensor by (24). Therefore, there always exists an SE for the proposed BSG. □

Remark 4.

This conclusion substantiates the existence and characterization of the SE in our Bayesian game setting. It implies that despite the incomplete information regarding the attacker’s type, an optimal hierarchical strategy profile exists and can be sought through iterative best-response dynamics.

6. Reinforcement Learning

As mentioned previously, the sensor is unable to access to the SINR of the AWGN channel. In this section, model-free reinforcement learning, i.e., Q-learning is introduced to find the SE. Then, a Stackelberg Q-learning algorithm for the two-player BSG is used to obtain the optimal policy for both the sensor and the attacker.

The optimal payoff functions are given as

$$\begin{array}{cc}\hfill J_d^{*}\left(s\right)& =\underset{{\pi }_d\in {\mathrm{\Pi }}_d}{max}\underset{{\pi }_a\in {\mathcal{R}}_a\left({\pi }_d\right)}{min}{Q}_d^{*}(s,\lambda ,\delta ),\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill J_a^{j,\ast }\left(s\right)& =\underset{{\pi }_a^j\in {\mathcal{R}}_a^j\left({\pi }_d^{SE}\right)}{max}{Q}_a^{j,\ast }(s,{\lambda }^{SE},{\delta }^j),\hfill \end{array}$$

(28)

where the optimal state-action-value functions (Q-functions) for the sensor and the j-th type attacker are, respectively, defined as follows:

$$\begin{array}{cc}\hfill Q_d^{*}(s,\lambda ,\delta )& =r_d(s,\lambda ,\delta )+\rho \sum _{s^{\prime }\in \mathcal{S}}\left(\sum _{j=1}^H\eta \left({\beta }_j\right)Pr\left(s^{\prime }\right|s,\lambda ,{\delta }^j)\right)J_d^{*}\left(s^{\prime }\right),\hfill \\ \hfill Q_a^{j,\ast }(s,\lambda ,{\delta }^j)& =r_a^j(s,\lambda ,{\delta }^j)+\rho \sum _{s^{\prime }\in \mathcal{S}}Pr\left(s^{\prime }\right|s,\lambda ,{\delta }^j)J_a^{j,\ast }\left(s^{\prime }\right).\hfill \end{array}$$

(29)

Then, the SE strategy is obtained by

$$\begin{array}{cc}\hfill {\pi }_d\left(s\right)& =arg\underset{{\pi }_d\in {\mathrm{\Pi }}_d}{max}\underset{{\pi }_a\in {\mathcal{R}}_a\left({\pi }_d\right)}{min}{Q}_d^{*}(s,\lambda ,\delta ),\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill {\pi }_a^j\left(s\right)& =arg\underset{{\pi }_a^j\in {\mathcal{R}}_a^j\left({\pi }_d^{SE}\right)}{max}{Q}_a^{j,\ast }(s,{\lambda }^{SE},{\delta }^j).\hfill \end{array}$$

(31)

Considering the multi-type channel interference gain of the attacker, the Q-value functions of the sensor and the attacker are recursively updated by SE presented in Theorem 1:

$$\begin{array}{cc}\hfill & Q_{k+1,d}(s,\lambda ,\delta )\hfill \\ \hfill =& (1-{\alpha }_k)Q_{k,d}(s,\lambda ,\delta )+{\alpha }_k{\mathrm{\Psi }}_k\left(Q_{k,d}(s,\lambda ,\delta )\right),\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill & Q_{k+1,a}^j(s,\lambda ,{\delta }^j)\hfill \\ \hfill =& (1-{\alpha }_k)Q_{k,a}^j(s,\lambda ,{\delta }^j)+{\alpha }_k{\mathrm{\Psi }}_k\left(Q_{k,a}^j(s,\lambda ,{\delta }^j)\right),\hfill \end{array}$$

(33)

where

$$\begin{array}{cc}\hfill {\mathrm{\Psi }}_k\left(Q_{k,d}(s,\lambda ,\delta )\right) & =r_d(s,\lambda ,\delta )+\rho \sum _{s^{\prime }\in \mathcal{S}}(\sum _{j=1}^H\eta \left({\beta }_j\right)Pr\left(s^{\prime }\right|s,\lambda ,{\delta }^j))stackelberg{Q}_{k,d}(s^{\prime },{\lambda }^{\prime },{\delta }^{\prime }),\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill {\mathrm{\Psi }}_k\left(Q_{k,a}^j(s,\lambda ,{\delta }^j)\right) & =r_a^j(s,\lambda ,{\delta }^j)+\rho \sum _{s^{\prime }\in \mathcal{S}}Pr\left(s^{\prime }\right|s,\lambda ,{\delta }^j)stackelberg{Q}_{k,a}^j(s^{\prime },{\lambda }^{\prime },{\delta }^{j{,}^{\prime }})\hfill \end{array}$$

(35)

with the learning rate ${\alpha }_k\in [0,1)$, $stackelberg{Q}_{k,d}(s^{\prime },{\lambda }^{\prime },{\delta }^{\prime })$ and $stackelberg{Q}_{k,a}^j(s^{\prime },{\lambda }^{\prime },{\delta }^{j{,}^{\prime }})$ are the Q-values of of the SE solutions at state $s^{\prime }$ for the sensor and the j-th type attacker, respectively.

In the BSG, the sensor and the attacker acts as the leader and the follower, respectively; hence, the update of Stackelberg Q-value is also hierarchically updated. To find the maximum Q-value of the attacker for the state $s^{\prime }$, the attacker chooses the optimal action from the best response (26). Based on the observation of the sensor’s action, the optimal action that the j-th type attacker takes in response to the sensor’s action ${\lambda }_k$ is determined as

$${\varphi }_a^{j,\ast }\left({\lambda }^{\prime }\right)=arg\underset{{\delta }^j}{max}{Q}_{k,a}^j(s^{\prime },{\lambda }^{\prime },{\delta }^j).$$

(36)

Then the optimal action of the attacker is given as

$${\varphi }_a^{*}\left({\lambda }^{\prime }\right)=[{\varphi }_a^{1,\ast }\left({\lambda }^{\prime }\right),\dots ,{\varphi }_a^{H,\ast }\left({\lambda }^{\prime }\right)].$$

(37)

Based on the BSG framework, the optimal action of the sensor is given by

$${\lambda }^{*}=arg\underset{{\lambda }^{\prime }}{max}{Q}_{k,d}(s^{\prime },{\lambda }^{\prime },{\varphi }_a^{*}\left({\lambda }^{*}\right)),$$

(38)

Consequently, we compute $stackelberg{Q}_{k,d}(s^{\prime },{\lambda }^{\prime },{\delta }^{\prime })$ and $stackelberg{Q}_{k,a}^j(s^{\prime },{\lambda }^{\prime },{\delta }^{j{,}^{\prime }})$ in the Q-functions update as

$$\begin{array}{cc}\hfill stackelberg{Q}_{k,d}(s^{\prime },{\lambda }^{\prime },{\delta }^{\prime }) & =Q_{k,d}(s^{\prime },{\lambda }^{*},{\varphi }_a^{*}\left({\lambda }^{*}\right))\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill stackelberg{Q}_{k,a}^j(s^{\prime },\lambda ,{\delta }^{j{,}^{\prime }}) & =Q_{k,a}^j(s^{\prime },{\lambda }^{*},{\varphi }_a^{j,\ast }\left({\lambda }^{*}\right))\hfill \end{array}$$

(40)

As a result, the Bayesian Stackelberg Q-learning Algorithm for the incomplete channel gain is presented in Algorithm 1. Key implementation details are as follows: The Q-values for both players are initialized to zero, a common practice that does not bias the asymptotic convergence. This initialization does not affect the final optimal policy learned, ensuring the algorithm’s robustness to initial conditions. An $ϵ$-greedy exploration strategy is employed, where, with probability $ϵ$, the actions are chosen randomly, and with probability $1-ϵ$, they are chosen as the SE actions based on current Q-values. The learning rate ${\alpha }_k$ for updating the Q-tables follows a schedule that ensures the Robbins-Monro conditions are met (as required by Theorem 2), typically by decaying with the number of visits to each state-action pair. The specific forms of ${\alpha }_k$ and $ϵ$ used in our simulations are provided in Section 7.

Algorithm 1 Bayesian Stackelberg Q-learning Algorithm with Incomplete Channel Gain Information

Input: System parameter matrices A, C, Q, R, ${\sigma }^2$    1:    2: Initialization: Initialize $Q_{0,d}(s,\lambda ,\delta )$ and $Q_{0,a}^j(s,\lambda ,{\delta }^j)$ for all $j\in \mathcal{N}$, $s\in \mathcal{S}$, $\lambda \in \mathrm{\Lambda }$, and ${\delta }^j\in {\mathrm{\Delta }}^j$; initial state $s_0$; Total iterations T; exploration probability $\epsilon$. Set iteration counter $k=1$;    3: while $k<T$do    4:       Randomize a number $\zeta \in [0,1]$;    5:       if $\zeta \in [\epsilon ,1]$ then    6:           Take actions $\lambda$ and ${\delta }^j$ for $j\in \mathcal{N}$ based on (36)–(38);    7:       else    8:           Select random actions $\lambda$ and ${\delta }^j$ for $j\in \mathcal{N}$;    9:       end if  10:       Obtain rewards $r_d(s,\lambda ,\delta )$ and $r_d(s,\lambda ,\delta )$;  11:       Update $Q_{k+1,d}(s,\lambda ,\delta )$ and $Q_{k+1,a}^j;(s,\lambda ,{\delta }^j)$, and move to next state;  12:       $k=k+1$;  13: end while Output: The optimal strategies and the optimal payoff functions for the sensor and the attacker

Remark 5.

The per-iteration computational cost of the Stackelberg Q-learning algorithm (Algorithm 1) is linear in the product of the state space size $\left|\mathcal{S}\right|$ and the action space sizes $\left|\mathrm{\Lambda }\right|\times |{\mathrm{\Delta }}^j|$, as it requires updating Q-tables for all state–action pairs encountered. This makes it efficient for the moderately sized problems presented here. For systems with significantly larger state or action spaces, the well-known curse of dimensionality would necessitate the use of function approximation (e.g., deep Q-networks), which is a promising direction for future work, as noted in Section 8.

The convergence of Algorithm 1 to the optimal Q-functions is guaranteed under the conditions specified in Theorem 2. The learning rate and exploration schedules described above are designed to satisfy these conditions, in particular the Robbins–Monro requirements for the learning rate.

Theorem 2

([36]). The Bayesian Stackelberg Q-learning sequences described in (32) and (33) will converge to the optimal values if the following assumptions hold:

1. 

The recursive processes (32) and (33) converge to $Q_d{(s,\lambda ,\delta )}^{*}$ and $Q_a{(s,\lambda ,\delta )}^{*}$ with probability 1, respectively.

2. 

There exists a number $a\in (0,1)$ and a sequence ${\varsigma }_k\ge 0$ converging to zero with probability 1, such that

$$\begin{array}{cc}\hfill \parallel {\mathrm{\Psi }}_k\left(Q_d\right)-\mathrm{\Psi }\left(Q_d^{*}\right)\parallel & \le a\parallel Q_d-Q_d^{*}\parallel +{\varsigma }_k,\forall Q_d\in \mathbf{Q}\hfill \\ \hfill \parallel {\mathrm{\Psi }}_k\left(Q_a^j\right)-\mathrm{\Psi }\left(Q_a^{j,\ast }\right)\parallel & \le a\parallel Q_a^j-Q_a^{j,\ast }\parallel +{\varsigma }_k,\forall Q_a\in \mathbf{Q}\hfill \end{array}$$

where $\mathbf{Q}$ is the Q-value space.

3. 

The learning rate ${\alpha }_k$ satisfies that ${\alpha }_k\in [0,1)$, and ${\sum }_{k=1}^n{\alpha }_k$ converges to infinity uniformly as $n\to \infty$.

Remark 6.

Condition 1 assumes that the recursive updates for both players converge with probability 1, meaning the learning process is inherently stable and reaches a fixed point. This is approximated by implementing ε-greedy exploration in Algorithm 1 and running the training for a large number of steps. Condition 2 ensures that the algorithm is driven toward a unique fixed point while noise vanishes by a contraction operator with a decaying perturbation. The learning rate in Condition 3 must satisfy the Robbins–Monro conditions for stochastic approximation. In the simulation section, the learning rate ${\alpha }_k$ is designed to be a nonzero decreasing function of time step k and the current state and actions.

7. Simulation Results

In this section, a numerical example is provided to verify the performance of the proposed stochastic attack strategy. The following system parameter are considered:

$$\begin{array}{cc}\hfill A& =\left[\begin{array}{cc}1& 0.5\\ 0& 1.05\end{array}\right], C=\left[\begin{array}{cc}1& 0\end{array}\right], Q=0.5\ast I_2, R=0.5.\hfill \end{array}$$

Then the steady-state error covariance $\overline{P}$ is $\overline{P}=[0.3802,0.2840;0.2840,1.6894]$. The AWGN power and the modulation parameter are ${\sigma }_k=0.5$ and $\kappa =0.2$, respectively. The fading channel gain and the action space of the sensor are given as $\alpha =0.7$ and ${\mathcal{A}}_d=\{1,2,3\}$. Assume that the fading channel gains of the DoS attacker are $[0.5,0.8]$ with probability distribution $[0.8,0.2]$. This setting creates a representative asymmetric information scenario in which the sensor needs to design a strategy against an attacker that is most likely to have a moderate interference capability but must also account for a significant possibility of facing a more potent attacker. Correspondingly, the action spaces for the two types of the attacker are ${\mathcal{A}}_a^1=\{4,5,6\}$ and ${\mathcal{A}}_a^2=\{1,2,3\}$, respectively. The unit power costs for the sensor and the attacker are $C_d=1$ and $C_a=2$, respectively. Set the finial state to $K=4$, and the state space is given as $\mathcal{S}=\{0,1,2,3,4\}$, which means the estimation error takes the value from $\{\overline{P},h\left(\overline{P}\right),\dots ,h^4\left(\overline{P}\right)\}$. The discount factor for payoff functions is $\rho =0.9$.

In the Bayesian Stackelberg Q-learning algorithm, the learning rate and the initial exploring rate are set as $\alpha =10/[15+\mathrm{count}(s,\lambda ,\delta \left)\right]$ and ${ϵ}_0=0.98$, respectively, where $\mathrm{count}(s,\lambda ,\delta )$ is the number of the occurrence of state–action pair $(s,\lambda ,\delta )$. This design ensures that the learning rate satisfies the Robbins–Monro conditions, a standard requirement for the convergence of stochastic approximation algorithms like Q-learning. The initial values are set as $Q_{0,d}(s,\lambda ,\delta )=0$ and $Q_{0,a}^j(s,\lambda ,{\delta }^j)=0$ for all $j\in \mathcal{N}$.

After 100,000 learning steps, the Q function values in different states converge. The equilibrium payoffs to the sensor and attacker for states $s=\overline{P},\dots ,h^4\left(\overline{P}\right)$ are given as follows:

$$\begin{array}{cc}\hfill J_d^{*}\left(s\right)& =[-131.9945,-141.6237,-149.8064,-155.1217,-155.1217],\hfill \\ \hfill J_a^{1,\ast }\left(s\right)& =[81.5995,87.9412,95.6919,101.0546,101.0546],\hfill \\ \hfill J_a^{2,\ast }\left(s\right)& =[97.7494,103.6945,111.4222,116.5348,116.5348].\hfill \end{array}$$

The corresponding optimal transmission and interference strategies are given as follows:

$$\begin{array}{cc}\hfill {\pi }_d^{*}\left(s\right)& =[1,3,3,3,3],\hfill \\ \hfill {\pi }_a^{1,\ast }\left(s\right)& =[4,4,5,6,6],\hfill \\ \hfill {\pi }_a^{2,\ast }\left(s\right)& =[1,3,3,3,3].\hfill \end{array}$$

The resulting Stackelberg equilibrium strategies are explicitly state-dependent, quantitatively showing how the sensor optimally increases transmission power as the estimation error grows. Finally, the equilibrium payoffs are quantitatively characterized, showing the cost–performance trade-off across states. The optimal strategies under the BSG confirm the intuition that a relatively small error covariance at the remote estimator enables the sensor to use less power for transmission, also leading to less interference power for the attacker. For a larger estimation error covariance, the sensor is inclined to choose a high power level to increase the packet arrival rate and, therefore, improve estimation performance. In this case, the attacker also increases the interference power accordingly.

The algorithm converges to distinct, stable Q-values for different state–action pairs. Due to the high dimension of the Q-values $Q_k(s,\lambda ,\delta )$, next, we present the learning process for the initial state $s=\overline{P}$ as an illustrative example. When the sensor takes the action $\lambda =1$, the corresponding Q-functions converge, as shown in

Table 1. Converged Q-values $Q_d(s,\lambda ,\delta )$ of the sensor for state $s=\overline{P}$ and sensor action $\lambda =1$. The columns represent the joint attacker actions $\delta =({\delta }^1,{\delta }^2)$.

$\mathit{\delta }$	${\mathit{Q}}_{\mathit{d}}(\mathit{s},\mathit{\lambda },\mathit{\delta })$
$(4,1)$	$-131.5542$
$(5,1)$	$-131.6878$
$(6,1)$	$-131.7834$
$(4,2)$	$-131.6996$
$(5,2)$	$-131.8332$
$(6,2)$	$-131.9288$
$(4,3)$	$-131.7653$
$(5,3)$	$-131.8989$
$(6,3)$	$-131.9945$

.

Table 2. Converged Q-values for both attacker types at state $s=\overline{P}$. Each row corresponds to a sensor action $\lambda$, with columns showing the Q-values for attacker type 1 ($Q_d^1$ under action ${\delta }^1$) and type 2 ($Q_d^2$ under action ${\delta }^2$).

$\mathit{\lambda }$	${\mathit{\delta }}^1$	${\mathit{\delta }}^2$	${\mathit{Q}}_{\mathit{a}}^1(\mathit{s},\mathit{\lambda },{\mathit{\delta }}^1)$	${\mathit{Q}}_{\mathit{a}}^2(\mathit{s},\mathit{\lambda },{\mathit{\delta }}^2)$
1	4	1	$81.5995$	$97.7494$
1	5	2	$80.6239$	$96.0405$
1	6	3	$79.6724$	$94.4824$
2	4	1	$80.7636$	$97.4259$
2	5	2	$79.9470$	$96.3323$
2	6	3	$79.1444$	$95.2768$
3	4	1	$79.8811$	$96.7319$
3	5	2	$79.1792$	$95.9321$
3	6	3	$78.4862$	$95.1471$

gives all converged Q-values of the attacker. We can see that the optimal strategy for the attacker is $\delta =(4,1)$ under state $s=\overline{P}$.

Figure 2, Figure 3 and Figure 4 depict the learning processes for the optimal strategies of the sensor, the first type attacker, and the second type attacker for state $\overline{P}$, respectively. In the figures, different colored lines respectively represent the iterative Q-values under different actions of the sensor and attacker. These figures collectively depict the convergence of Q-functions for all possible action combinations at state $\overline{P}$. The high density of lines illustrates the learning algorithm’s exploration across the entire strategy space. While individual lines are not labeled for clarity, their collective trend towards stabilization after around 5000 iterations is evident, confirming the convergence of the learning process. The precise converged Q-values that underpin the optimal strategies are detailed in Table 1 and Table 2.

Next, we consider the situation that the attacker can launch attacks without any cost constraints, that is, the cost of the interference power can be ignored. In the case of $C_a=0.01$, the equilibrium payoffs to the sensor and attacker for states $s=\overline{P},\dots ,h^4\left(\overline{P}\right)$ are given as follows:

$$\begin{array}{cc}\hfill J_d^{*}\left(s\right)& =[-138.7763,-148.5242,-156.6659,-161.9812,-161.9812],\hfill \\ \hfill J_a^{1,\ast }\left(s\right)& =[189.0304,196.9475,205.1795,210.5454,210.5454],\hfill \\ \hfill J_a^{2,\ast }\left(s\right)& =[155.1126,162.1956,169.9786,175.0912,175.0912].\hfill \end{array}$$

The corresponding optimal transmission and interference strategies are given as follows:

$$\begin{array}{cc}\hfill {\pi }_d^{*}\left(s\right)& =[1,3,3,3,3],\hfill \\ \hfill {\pi }_a^{1,\ast }\left(s\right)& =[6,6,6,6,6],\hfill \\ \hfill {\pi }_a^{2,\ast }\left(s\right)& =[3,3,3,3,3].\hfill \end{array}$$

The optimal strategies confirm that when the cost of interference power $C_a$ is negligible, both types of attackers consistently select the highest available power level in their respective action sets. This behavior aligns perfectly with game-theoretic intuition: as the marginal cost of interference diminishes, the rational objective for the attacker shifts overwhelmingly towards maximizing the degradation of the estimation performance, leading to the selection of maximum jamming power. This result underscores the critical role of cost parameters in shaping the equilibrium of the security game.

The learning processes for the optimal strategies of the sensor, the first type attacker, and the second type attacker for state $\overline{P}$ are shown in Figure 5, Figure 6 and Figure 7. Similar to the previous case, the convergence trends for all action combinations are shown. The eventual stabilization of all trajectories validates the algorithm’s effectiveness even in this complex setting. Notably, as seen in Figure 5, the Q-values of the sensor exhibit significant fluctuation and converge more slowly compared to those of the attackers in Figure 6 and Figure 7. This can be attributed to the information asymmetry inherent in the problem: the sensor lacks precise knowledge of the attacker’s channel gain and must learn an optimal strategy based only on the probabilistic distribution over attacker types. This uncertainty inherently increases the exploration burden and complexity of the learning process for the leader.

Therefore, the numerical results validate the algorithm’s effectiveness by demonstrating its capability to solve the proposed BSG model: It is seen that the algorithm achieves convergence to a stable equilibrium where both players’ strategies are mutually optimal in the sequential sense, as theorized in Section 5 and Section 6. The logical adaptation of strategies to different states and costs further confirms that the learned policies align with game-theoretic intuition, providing strong empirical support for the proposed framework.

8. Conclusions

This paper investigated the Bayesian Stackelberg game for remote state estimation under SINR-based DoS attacks. The two players sequentially decide their transmission and interference powers under incomplete information. Specifically, the sensor lacks exact knowledge of the attacker’s fading channel gain. The optimization problem over an infinite-time horizon is first formulated as an MDP with finite state and action spaces. By taking advantage of the probabilistic information about the channel interference gain, a BSG is modeled to describe the iterative decision-making process between the sensor and the various types of attackers. Based on the solution to the BSG, a Stackelberg Q-learning algorithm is used to obtain the optimal strategies of the two players. Numerical results validate the effectiveness of the proposed game-theoretic approach in the case of uncertain channel gains. The presented framework operates under core assumptions of a known attacker type distribution and discrete action spaces, which define its current scope but also present opportunities for future generalization. Future work also includes analyzing games where both players have incomplete information and extending the framework to larger-scale systems via function approximation (e.g., deep reinforcement learning) to mitigate the curse of dimensionality.