Security State Estimation for Cyber-Physical Systems against DoS Attacks via Reinforcement Learning and Game Theory

Abstract

This paper addressed the optimal policy selection problem of attacker and sensor in cyber-physical systems (CPSs) under denial of service (DoS) attacks. Since the sensor and the attacker have opposite goals, a two-player zero-sum game is introduced to describe the game between the sensor and the attacker, and the Nash equilibrium strategies are studied to obtain the optimal actions. In order to effectively evaluate and quantify the gains, a reinforcement learning algorithm is proposed to dynamically adjust the corresponding strategies. Furthermore, security state estimation is introduced to evaluate the impact of offensive and defensive strategies on CPSs. In the algorithm, the $\epsilon$-greedy policy is improved to make optimal choices based on sufficient learning, achieving a balance of exploration and exploitation. It is worth noting that the channel reliability factor is considered in order to study CPSs with multiple reasons for packet loss. The reinforcement learning algorithm is designed in two scenarios: reliable channel (that is, the reason for packet loss is only DoS attacks) and unreliable channel (the reason for packet loss is not entirely from DoS attacks). The simulation results of the two scenarios show that the proposed reinforcement learning algorithm can quickly converge to the Nash equilibrium policies of both sides, proving the availability and effectiveness of the algorithm.

1. Introduction

With the close interaction and integration between computational and physical resources, cyber-physical systems (CPSs) have emerged and gained widespread attention. Due to the application of 3C (computation, communication, control) technologies, CPSs can perform real-time sensing and remote control [1]. Therefore, CPSs are widely applied in critical infrastructure control, aerospace systems, military systems, etc. [2]. In CPSs, wireless sensors are widely used due to their flexibility, power saving and easy scalability [3]. However, while improving communication efficiency, the transmission of measurement data over wireless networks poses security issues such as significant damage to industrial systems [4]. For example, Sberbank, Russia’s largest bank, was hit by the largest DDoS attack ever with peak traffic of 450 GB/s in May 2022, posing a huge threat to its cybersecurity. The frequent occurrence of such malicious cyber attacks has led many scholars and experts to pay great attention to security issues in CPSs. Hence, CPSs under cyber attacks are studied and countermeasures are proposed to ensure the security of the systems [5,6].

In terms of specific attack types, cyber attacks encountered on CPSs are mainly divided into DoS attacks [7,8], spoofing attacks [9,10] and injection attacks [11,12,13]. Among them, DoS attacks have been paid more attention as one of the most frequent and easy cyber attacks to implement. They mainly prevent remote estimators from receiving and processing sensor data properly by interfering with the communication channel. The existing modeling approaches for the DoS attack problem are mainly divided into attack constraint modeling and stochastic modeling. The former is a modeling approach that constrains the duration and switches of DoS attacks. For example, Refs. [14,15] puts constraints on the frequency and duration of DoS attacks and focuses on finding the maximum attack frequency and maximum attack duration while maintaining system performance. The latter is a stochastic modeling approach related to Bernoulli distribution and Markov chain. This paper conducts a study on the latter modeling approach, where DoS attacks occur in accordance with Bernoulli distribution. Furthermore, the attack frequency is random and the attack duration is the sampling interval.

For CPSs under DoS attacks, numerous scholars have carried out research on state estimation based on systematic measurement sequences and prior models, with methods such as Kalman filtering, see [16,17,18,19] and references therein. From the attacker’s perspective, most of the existing literature focuses on the DoS attack scheduling strategies. In [17], the problem of optimal scheduling of DoS attacks under energy-limited conditions was investigated based on signal-to-interference-plus-noise ratio (SINR) of the channel. Some literature, on the other hand, stand from the defender’s perspective and consider the stability of the system under DoS attacks, or propose DoS attack detection methods [18,19,20]. Among them, a method to protect state privacy by maximizing the state estimation error of eavesdroppers for energy-constrained sensors was studied in [20]. Furthermore, the PMU data of the bus was used by Hasnat in [18] to perform state estimation of the attacked components of the system under different DoS attack strengths, so as to attenuate the impact brought by the attacks. Different from the above literature, this paper not only uses the state estimation method to evaluate the status of CPSs under DoS attacks, but also introduces game theory to study the optimal strategies of both attackers and defenders.

In CPSs, some scholars study the relationship between DoS attackers and system defenders in the system, and regard the confrontation between the two as a two-player game [21,22]. In some papers, the interactive game is further formulated as a Markov decision process, by optimizing the behavior strategy of the attacker or defender to meet the needs of its interaction with the environment [23]. The game of attackers and defenders is transformed into a static Bayesian game to obtain the optimal strategy for both sides in [24]. Furthermore, some papers considered how to reach a Nash equilibrium policy under the game between attackers and defenders, that is, neither player unilaterally changes his strategy under the Nash equilibrium to improve his own reward when the other keeps theirs unchanged [25,26,27]. The interactive game process between the sensor and the attacker as a Markov game framework and employs an improved Nash learning algorithm in order to obtain a Nash equilibrium for the two sides was investigated in [26]. Since the reward gained by the attacker comes entirely from the loss of the defender, ref. [28] treated the game between the attacker and the defender as a zero-sum matrix game and designs a time-difference (TD) learning-based algorithm to obtain the optimal attack strategy. Based on the above discussions, a two-player zero-sum deterministic game is introduced in this paper to describe the interactive decision process between the attacker and the sensor. Since the existing static game methods cannot fully satisfy the demands of real-time state update in CPSs, this paper adopts the linear programming method to obtain the Nash equilibrium strategy of both sides as the optimal strategy.

With the development of artificial intelligence, reinforcement learning methods have attracted much attention, focusing on how agents learn optimal strategies by interacting with unknown environments [29]. In recent research, reinforcement learning methods have been used to solve the game problem between attackers and system defenders in CPSs [30,31]. Numerous studies about reinforcement learning algorithms were carried out in different scenarios. In [32], reinforcement learning is classified into model-based and model-free approaches, and a model-free reinforcement learning method is designed to solve the attacker’s security-aware planning. Furthermore, the game between attackers and sensors from the open-loop case and closed-loop case is studied in [33], and centralized and distributed reinforcement learning methods are proposed to solve the Nash equilibrium of the two sides. Furthermore, some limitations of CPSs make it difficult to gather collective information and perform state estimation efficiently. For instance, sensor devices are often small in size and carry energy-constrained batteries, which are hard to replace in some situations. Moreover, bandwidth resources are also limited, which result in network congestion or packet loss with the increase of the number of sensor nodes. It is necessary to optimize the utilization of system resources by reducing the needless consumption of system resources. Up to now, little attention has been paid to the optimal strategy obtained by reinforcement learning from the perspective of channel reliability and resource utilization, which is also one of the motivations of this paper.

In summary, this paper investigates the problem of the remote security state estimation problem of CPSs with DoS attacks, where a reinforcement learning algorithm to achieve the Nash equilibrium policies for the two-player game between both the attacker and the sensor. This paper develops from two scenarios: reliable channel and unreliable channel, where the reason for packet loss in reliable channel transmission can only be DoS attacks, while unreliable channel transmission may lose packets due to other reasons. The contributions of the paper are as follows: (i) This paper introduces security state estimation into existing reinforcement learning methods to evaluate the impact of the policies of attackers and defenders on the state estimation; (ii) A two-player zero-sum game is introduced to describe the game between the sensor and the attacker, and the Nash equilibrium strategies are studied to obtain the optimal actions. Besides, resource constraints for the sensor and the attacker are considered in the game; (iii) Reinforcement learning algorithms are designed to enable sensors and attackers dynamically learn and adjust policies in the interaction, where $\epsilon$-greedy policy is improved to achieve a balance of exploration and exploitation; (iv) Considering the influence of channel reliability on CPSs, the reinforcement learning algorithm is studied in two scenarios: reliable channels and unreliable channels, the packet loss probability of the two scenarios is compared.

The rest of this paper is organized as follows. Section 2 formulates the system model and introduced some preparatory knowledge of a two-player zero-sum game and the Q-learning algorithm. In Section 3, the state estimation algorithm based on Kalman filter is designed, and the influence of the DoS attack on state estimation is described. Reinforcement learning algorithms for reliable and unreliable channels are designed in Section 4 and Section 5, respectively. The simulation results of two cases in Section 6 illustrate the effectiveness and efficiency of the reinforcement learning algorithm. Section 7 draws the conclusion and discusses the future direction.

2. Preliminaries

Section 2 provides a theoretical basis for this paper which can better explain the methods used in the paper, so that the reader better understands the content. The section formulates the system model and introduces some preparatory knowledge of the two-player zero-sum game and Q-learning algorithm. In the part of the system model, the principle of remote security estimation of a wireless channel under DoS attack is described in detail. The preparatory knowledge of the two-player zero-sum game includes the definition of the two-player zero-sum game and the definition of the pure strategy Nash equilibrium and mixed strategy Nash equilibrium. The preparatory knowledge of the Q-learning algorithm include the introduction of the Markov decision process and the iteration of the one-step Q-learning method according to the Bellman optimal equation.

2.1. System Model

Consider a CPS with a sensor, attacker and remote estimator as presented in Figure 1; a DoS attack may occur on the wireless communication channel between the sensor and the remote state estimator. At time k, the expression of the linear system under DoS attacks can be given by:

$$\begin{array}{c}\left\{\begin{array}{c}x(k+1)=Ax\left(k\right)+Bw\left(k\right)\hfill \\ y\left(k\right)=Hx\left(k\right)+v\left(k\right),\hfill \end{array}\right.\end{array}$$

(1)

where $k\in \mathbb{Z}$ denotes the discrete time step. $x\left(k\right)\in {\mathbb{R}}^{d_x}$ refers to the state vector of the system, $y\left(k\right)\in {\mathbb{R}}^{d_y}$ is the sensor measurement vector. $w\left(k\right)\in {\mathbb{R}}^{d_w}$ and $v\left(k\right)\in {\mathbb{R}}^{d_v}$ represent the process and measurement noises with zero mean, and their covariance matrices are $Q\left(k\right)$ and $R\left(k\right)$, respectively. A, B, and H are coefficient matrices with corresponding dimensions.

2.2. Two-Player Zero-Sum Game

In this paper, a two-player zero-sum deterministic game is considered to describe the interaction between the sensor and the attacker. The definition of the zero-sum game is as follows [34]:

Definition 1.

The basic game model that meets the following three conditions is called the zero-sum matrix game: (i) The number of players in the game is two, namely $N=\{n_1,n_2\}$; (ii) Each player’s strategy space is a finite set. That is, the strategy spaces ${\Pi }_1=\{{\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_m\}$ and ${\Pi }_2=\{{\beta }_1,{\beta }_2,\cdots ,{\beta }_n\}$ are finite sets, and any strategy ${\alpha }_i$ and ${\beta }_j$ can be taken out from the strategy spaces ${\Pi }_1$ and ${\Pi }_2$ to form a strategy combination $({\alpha }_i,{\beta }_j)$; (iii) For any strategy combination $({\alpha }_i,{\beta }_j)$, the payoffs of the players are $U_1({\alpha }_i,{\beta }_j)$ and $U_2({\alpha }_i,{\beta }_j)$ respectively, and $U_1({\alpha }_i,{\beta }_j)+U_2({\alpha }_i,{\beta }_j)=0$.

In CPSs under attacks, there are only two players in the game: the sensor and the attacker. The two players choose their strategies in a limited set of policies, that is, the sensor can choose a secure or insecure transmission channel and the attacker can choose to attack or not. The goal of the sensor is to minimize the state estimation error and maximize the attack cost, while the attacker has the opposite goal. Since the basic conditions of a zero-sum game are satisfied, and there is no randomness in the action, the game can be formulated as a zero-sum deterministic game.

The Nash equilibrium policy of the zero-sum deterministic game is a joint policy, in which each player considers the behavior of the other player and makes the best response. Nash equilibrium is divided into pure strategy Nash equilibrium and mixed strategy Nash equilibrium, which are defined as follows.

Definition 2.

For the two-player zero-sum game, a pure strategy Nash equilibrium is a joint policy that each player sticking to only one choice that is most beneficial to him, which is a combination of actions $(a_{*}^1,a_{*}^2)$ such that:

$$\begin{array}{c}\begin{array}{cc}{r}^1\left(a_{*}^1,a_{*}^2\right)\ge r^1\left(a^1,a_{*}^2\right)\hfill & forall{a}^1\in A^1\hfill \\ r^2\left(a_{*}^1,a_{*}^2\right)\ge r^2\left(a_{*}^1,a^2\right)\hfill & forall{a}^2\in A^2,\hfill \end{array}\end{array}$$

where $r^1$ and $r^2$ correspond to the rewards of the two players, respectively, and $A^1$ and $A^2$ are the action sets.

Definition 3.

A mixed strategy Nash equilibrium of two-player zero-sum game allow players to choose actions probabilistically to give the best response to the action of the other player. It is an action combination represented by a vector $({\omega }_{*}^1,{\omega }_{*}^2)$ such that:

$$\begin{array}{c}\begin{array}{cc}{\omega }_{*}^1{R}^1{\omega }_{*}^2\ge {\omega }^1{R}^1{\omega }_{*}^2\hfill & forall{\omega }^1\in \Omega \left(A^1\right)\hfill \\ {\omega }_{*}^1{R}^2{\omega }_{*}^2\ge {\omega }_{*}^1{R}^2{\omega }^2\hfill & forall{\omega }^2\in \Omega \left(A^2\right),\hfill \end{array}\end{array}$$

where ${\omega }_{*}^1$ and ${\omega }_{*}^2$ are non-negative vectors whose entries sum to 1, $\Omega \left(A^k\right)$ is the set of probability distributions over the action space $A^k$, $R^1$ and $R^2$ are the payoff matrices of two players.

In 1950, Nash proved that there is at least one Nash equilibrium in any finite time-horizon game [35]. Therefore, we assume that there exist Nash equilibrium policies $\{{\pi }_1^{*},{\pi }_2^{*}\}$ for all state $s\in S$, where ${\pi }_1^{*}$ is the policy for sensor and ${\pi }_2^{*}$ is the policy for the attacker.

2.3. Q-Learning Algorithm

As an important branch of artificial intelligence, reinforcement learning is a mathematical framework that helps learners to interact with an unknown environment in order to achieve their goals. Markov decision process (MDP) is one of the theoretical bases of reinforcement learning, which refers to the sequential decision-making problem in completely observable random environment with the Markov transition model and rewards [36]. The Markov decision process consists of five tuples, which are defined as follows: $MDP::=\langle \mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\rho \rangle$ [37]. In this defining equation, $\mathcal{S}$ and $\mathcal{A}$ represent the discrete state space and discrete action space, respectively. The agent’s reward function $\mathcal{R}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ indicates the reward that an agent can get by choosing an action in a certain state. The state transition function $\mathcal{P}:\mathcal{S}\times \mathcal{A}\to [0,1]$ denotes the probability that the agent chooses an action and enters the next state. $\rho$ is the discount factor between 0 and 1.

Q-learning is an essential reinforcement learning approach, which is an off policy learning algorithm based on the Markov decision process. It was put forward by Watkins [38] in 1989, and is currently a widely used reinforcement learning approach. The Q-learning algorithm enables the agent to select the optimal strategy and maximize the reward by evaluating the current system state and actions without considering the external environment model.

Assume that both the set of state S and the set of action A are finite sets. Then the optimal action cost function $Q(s,a)$ can be expressed as a table with m rows and n columns, where m is the number of states and n is the number of actions. While the system is in state $s_k$, the decision formula of time k is:

$$\begin{array}{c}{a}_k=\underset{a\in A}{argmax}{Q}_{🟉}(s_k,a).\end{array}$$

(2)

According to the Bellman optimal equation in [39], the iteration of the one-step Q-learning method is derived as:

$$\begin{array}{c}\begin{array}{cc}\hfill Q\left(s_k,a_k\right)& \leftarrow Q\left(s_k,a_k\right)+\alpha [r_{k+1}\hfill \\ & +\rho \underset{a}{max}Q(s_{k+1},a)\left.-Q\left(s_k,a_k\right)\right],\hfill \end{array}\end{array}$$

(3)

where $\alpha$ is the learning rate, Q-value function is the learned state-action value function.

3. Problem Statement

3.1. State Estimation Based on Kalman Filter

A local Kalman filter is employed to complete the state estimation of recursive updating of the system state. At each time k, the minimum mean-squared error (MMSE) estimate $\widehat{x}\left(k\right)$ of the state vector $x\left(k\right)$ is obtained according to the measurement data by running the Kalman filter. Then, $\widehat{x}\left(k\right)$ is sent to the remote estimator by the sensor. The MMSE estimate $\widehat{x}\left(k\right)$ of $x\left(k\right)$ is denoted by:

$$\begin{array}{c}\widehat{x}\left(k\right)=\mathbf{E}\left[x\left(k\right)\mid y\left(1\right),\dots ,y\left(k\right)\right]\end{array}$$

(4)

with its corresponding estimation error covariance

$$\begin{array}{c}\begin{array}{c}P\left(k\right)=\mathbf{E}\left[\left(x\left(k\right)-\widehat{x}\left(k\right)\right){\left(x\left(k\right)-\widehat{x}\left(k\right)\right)}^T\right.\\ \left.\mid y\left(1\right),\dots ,y\left(k\right)\right].\end{array}\end{array}$$

(5)

In accordance with the Kalman filter equations, $\widehat{x}\left(k\right)$ and $P\left(k\right)$ are updated recursively. For simplicity, the Lyapunov and Riccati operators h and $\tilde{g}$ is defined as:

$$\begin{array}{c}\begin{array}{c}h\left(X\right)\triangleq AX{A}^T+Q\hfill \\ \tilde{g}\left(X\right)\triangleq X-X{C}^T{\left[CX{C}^T+R\right]}^{-1}CX.\hfill \end{array}\end{array}$$

(6)

Then the recursive updating equation of the Kalman filter can be expressed as:

$$\begin{array}{c}\begin{array}{c}\widehat{x}(k\mid k-1)=A\widehat{x}(k-1)\hfill \\ P(k\mid k-1)=h\left(P(k-1)\right)\hfill \\ K\left(k\right)=P(k\mid k-1)C^T{\left[CP(k\mid k-1)C^T+R\right]}^{-1}\hfill \\ \widehat{x}\left(k\right)=\widehat{x}(k\mid k-1)+K\left(k\right)\left(y\left(k\right)-C\widehat{x}(k\mid k-1)\right)\hfill \\ P\left(k\right)=g\left(P(k\mid k-1)\right).\hfill \end{array}\end{array}$$

(7)

Due to the stabilizability and detectability assumptions, the estimation error covariance $P\left(k\right)$ converges to a unique fixed point $\overline{P}\left(k\right)$ of $h\circ g$ at an exponential rate [40]. In order to simplify the subsequent reinforcement learning problem, we assume that the Kalman filter has already converged to the steady state, i.e., for all $k\ge 1,P\left(k\right)=\overline{P}$.

3.2. DoS Attack Model

According to the system model shown in Figure 1, a DoS attack can occur in the channel between the sensor and the remote estimator. The likelihood of DoS attacks varies depending on the choice of the sensor and the attacker. For the sensor, there are two choices of sending data, which can be denoted as $a^s=0$ and $a^s=1$. The former notation means that the sensor sends packets at a cost of 0, but is vulnerable to DoS attacks; the latter notation means that the sensor spends extra cost $c_s$ to avoid attacks. On the other hand, the DoS attacker can choose to attack or not and the attack will induce additional constant cost $c_a$.

Denote $a_k^s\in \{0,1\}$ and $a_k^a\in \{0,1\}$ as the decision variable of the sensor and the attacker at time k. The sensor are more prone to DoS attacks when $a_k^s=0$, while it transmits in a safer way when $a_k^s=1$. In addition, the DoS attacker chooses to attack at time k when $a_k^a=1$, and no attack happens when $a_k^a=0$.

In CPSs, packet loss may occur due to DoS attacks or other reasons such as channel congestion. To indicate the arrival of packets and to determine whether packet loss has occurred at time k, we define the arrival indicator ${\delta }_k$ as:

$$\begin{array}{c}{\delta }_k=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{packet}\mathrm{arrives}\mathrm{at}\mathrm{the}\mathrm{remote}\mathrm{estimator}\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(8)

Furthermore, the packet loss probability ${\xi }_k\in (0,1)$ is defined to describe the proportion of packets lost in data transmission to the total packets sent. When the action combination of the sensor and the attacker is easy to induce attacks, the overall performance of the network will decline, and the packet loss probability ${\xi }_k$ will be high.

Packet loss is not only related to the choice of sensors and attackers, but also to the reliability of the communication channel. We will illustrate the impact of channel reliability on the packet loss probability by showing the occurrence of attacks under reliable and unreliable channels in Section 4 and Section 5, respectively.

Remark 1.

In this paper, DoS attacks are modeled as stochastic dropouts, and the filtering process is similar to that of data loss. However, there are some differences between the two processes. Similar to transmission delay, channel fading, etc., data loss and intermittent observation are intrinsic phenomena in the network. They are passive representations of network unreliability and instability. On the contrary, this paper pays more attention to the game of the attackers and defenders and their Nash equilibrium points, which is a dynamic game process of active attack and active defense. From passive description to active attack confrontation, this paper combines game theory and reinforcement learning methods to evaluate and dynamically adjust the dynamic game process between attackers and defenders.

3.3. Remote Estimation

At the remote estimator, the estimation process of $\widehat{x}\left(k\right)$ can be described as follows: if the estimator receives the packet successfully, it synchronizes the received data packet to obtain the estimation. Otherwise, the estimator updates the estimation based on the optimal estimation obtained at the previous time step, i.e.,

$$\begin{array}{c}\overline{x}\left(k\right)=\left\{\begin{array}{cc}\widehat{x}\left(k\right),\hfill & {\delta }_k=1\hfill \\ A\overline{x}(k-1),\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(9)

with the corresponding estimation error covariance

$$\begin{array}{c}P\left(k\right)=\left\{\begin{array}{cc}\overline{P},\hfill & {\delta }_k=1\hfill \\ h\left(P\right(k-1\left)\right),\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(10)

In order to simplify the error covariance $P\left(k\right)$, the indicator $\tau$ is defined as:

$$\begin{array}{c}{\tau }_k\triangleq k-\underset{0\le l\le k}{max}\left\{l:{\delta }_l=1\right\},\end{array}$$

(11)

which indicates the time interval from the last time l when packet was received to the current time k. The iteration of ${\tau }_k$ can be expressed as:

$$\begin{array}{c}{\tau }_k=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{\delta }_k=1\hfill \\ {\tau }_{k-1}+1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(12)

We make the following assumption that at the beginning of the transmission, the estimator can receive the packet ${\widehat{x}}_0$ successfully, that is, ${\delta }_0=1$. Based on (10) and (11), it is easy to obtain the estimation error covariance at the remote estimator as $P\left(k\right)=h^{{\tau }_k}\left(\overline{P}\right)$.

In the game between sensors and attackers, there are differences in the information they can obtain. The sensor knows the system model parameters, so $P\left(k\right)$ is available at time k. Therefore, the sensor’s action selection is based on the following cost minimization principle: when the safe transmission cost $c_s$ is less than the cost caused by packet loss, i.e., $c_s\le Tr\left(P\left(k\right)\right)-Tr\left(\overline{P}\right)$, the sensor can only choose $a_k^s=1$. However, for the attacker, the system model parameters are not available, so there is no principle for its action selection.

In accordance with the above principle, the set of values of ${\tau }_k$ and $P\left(k\right)$ are finite sets. Due to ${\tau }_k$, take values from ${\mathbb{Z}}_k\triangleq \{0,1,2,\dots ,k\}$, $P\left(k\right)$ take values from $\mathbb{P}\triangleq \{\overline{P},h\left(\overline{P}\right),h^2\left(\overline{P}\right),$$\dots ,h^n\left(\overline{P}\right),\}$. Notice that the current state $P\left(k\right)$ depends only on the last state $P(k-1)$ and $\delta$ from (8). Hence, the sequence of random states $P\left(k\right)$ forms a Markov chain, and the transition process is depicted in Figure 2. With the actions of the two agents $a_k^s$ and $a_k^a$, the transition can be described by a simple transition probability matrix

$$\begin{array}{c}T=\left(\begin{array}{cccc}1-{\xi }_0\hfill & {\xi }_0\hfill & & \\ 1-{\xi }_1\hfill & & {\xi }_1\hfill & \\ 1-{\xi }_2\hfill & & & {\xi }_2\hfill \\ ⋮\hfill & & & & \ddots \hfill \end{array}\right),\end{array}$$

where the element $T(i,j)$ represents the transition probability from state $s_k=h^{i-1}\left(\overline{P}\right)$ to state $s_{k+1}=h^{j-1}\left(\overline{P}\right)$. Based on Markov chains in Figure 2, states can only be transferred from $h^i\left(\overline{P}\right)$ to the next adjacent state $h^{i+1}\left(\overline{P}\right)$ or the initial state $\overline{P})$. Hence, except for all $T(i,1)$ and $T(i,i+1)$, the other default elements are 0.

4. Reinforcement Learning for Reliable Channel

In CPSs, reliability is an important indicator of system performance [41]. The measurement data from the sensor are transmitted over a wireless channel, and packets transmitted over a reliable channel will not be corrupted or lost by timeout, and vice versa for an unreliable channel. Assuming an ideal state in which the packets are transmitted in a reliable channel, there is no congestion and timeout, and packets are lost only because of DoS attacks. According to the description of Section 3.2, the packet loss probability for a reliable channel can be expressed as:

$$\begin{array}{c}{\xi }_k=\left\{\begin{array}{cc}100\%,\hfill & \mathrm{if}{a}_k^s=0,a_k^a=1\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(13)

That is, when the sensor chooses to transmit insecurely and the attacker chooses to perform a DoS attack, the packet loss probability is 100% and in other cases the packet loss probability is 0.

According to the above assumptions and the existing reinforcement learning framework, an MDP is established to describe the interactive decision-making problem between sensor and attacker. The elements of MDP can be described as follows:

State: We denote the finite set of states in the reinforcement learning problem by $S\triangleq \{s_1,s_2,\cdots ,s_n\}$. The state of the system is represented by the estimation error covariance of the estimator, i.e., $s_k=P\left(k\right)$, where $P\left(k\right)\in \mathbb{P}$. At time k, the state of the system is affected by the state $s_{k-1}$ and actions $a_{k-1}^s$ and $a_{k-1}^a$ of sensor and attacker at time $k-1$.

Action: In the zero-sum deterministic game, the sensor needs to choose whether to spend cost $c_s$ for secure transmission. Besides, the DoS attacker will choose to implement the DoS attack or not. It should be noted that the action selection of sensor and attacker are independent, that is to say, the actions of one player will not affect the other player. According to the description in the DoS attack model, the decision variable of the sensor is $a_k^s\in \{0,1\}$ and the decision variable of the attacker is $a_k^a\in \{0,1\}$. Hence, the sensor and the attacker both have two strategies at state $s_k$. There are four combinations of sensor and attacker actions at each time k. Thus, we denote A as the set of actions of $a_k=(a_k^s,a_k^a)$ and A has four elements.

State transition: As mentioned above, the estimation of the remote estimator is closely related to whether the packet loss occurs, and the same is true for the state transition. In reliable channel, DoS attack occurs when and only when the sensor chooses the insecure transmission and the attacker launches DoS attack i.e., $a_k^s=0,a_k^a=1$, resulting in packet loss. Hence, the state of time $k+1$ is determined by the action combination of the sensor and the attacker, and can be obtained according to (10) as follows:

$$\begin{array}{c}{s}_{k+1}=P(k+1)=\left\{\begin{array}{cc}h\left(P\left(k\right)\right),\hfill & \mathrm{if}{a}_k^s=0,a_k^a=1\hfill \\ \overline{P},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(14)

Reward function: A reward function is defined to evaluate the payoff of both the sensor and the attacker during the game. Under the cost setting in Section 3.2, the reward of the system depends on the state of the system, as well as the cost and strategy of the attacker and the sensor. The sensor’s goal is to minimize the reward function and the attacker’s is the opposite. At time k, the immediate reward $r_k$ can be calculated as:

$$\begin{array}{c}{r}_k=Tr\left(s_k\right)+c_s{a}_k^s-c_a{a}_k^a.\end{array}$$

(15)

Discount factor: In the reinforcement learning problem, since we prefer to focus more on the current reward rather than the future reward, a discount factor is set to reduce the impact of future rewards on the current state. The discount factor is a parameter between 0 and 1, with a time-based penalty to achieve better performance of the algorithm. By setting the discount factor, the farther the future is, the greater the discount is given to the reward, which makes the algorithm converge faster.

For the two-player zero-sum game between the sensor and the attacker, this paper proposes a Q-learning-based game algorithm to find the Nash equilibrium policies. The algorithm is divided into the following steps. Firstly, the states, actions and the game matrix based on Q values are initialized. Secondly, at each time k, the sensor and the attacker choose actions according to the game matrix by employing the $\epsilon$-greedy strategy. Thirdly, the current reward as well as the next state $s_{k+1}$ are obtained based on the current state $s_k$ and the combination of actions $a_k=(a_k^s,a_k^a)$. Then, the Q-value matrix is adjusted and then the game process is carried out for the next moment. Finally, the converged Q matrix is obtained and the Nash equilibrium between the sensor and the attacker is observed. The algorithm can be described explicitly as follows:

Step 1: Input the system parameters, initialize the system state, action and Q-value matrix. Under the principles set in Section 3.3, given the cost and error covariance matrix, then the system can be determined to have n states, assuming that the initial state $s_1=\overline{P}$. Since the system has n states, and each state has four combinations of sensor and attacker actions, so the Q-value matrix with n rows and four columns is initialized. The initial value of the Q-value matrix is set to m, where m satisfies $m\ge {max}_{k\ge 1}\left\{r_k\right\}/(1-\rho )$, then the monotonically non-increasing property of $\tilde{Q}(s,a^s,a^a)$ is guaranteed.

Step 2: At each time k, the sensor and the attacker select the action with the $\epsilon -greedy$ strategy. According to this strategy, the sensor selects the action randomly with a probability of $\epsilon$ and the optimal action with a probability of $1-\epsilon$.

Remark 2.

At the beginning of the iteration, the value of ε is set to be large, that is, the action selection has great randomness. As the algorithm iterates, the value of ε decreases gradually until the set minimum value is reached. The core idea of this method is to strike a balance between exploration and exploitation. Setting a large ε at the beginning of the iteration allows sensors and attackers to choose actions relatively randomly to learn the rewards for each action combination, which is called exploration. At the end of the iteration, sensors and attackers have observed some data, and the value of ε is very small, so they can choose the action that obtains the highest reward based on the existing data, which is exploitation. In this way, better actions are selected in the case of sufficient data collection, achieving a balance between exploration and exploitation.

The optimal action is obtained by calculating the Nash equilibrium using the linear programming method in Lemma 1.

Lemma 1.

Let the value of the matrix game be $v>0$. The optimal strategy of the sensor and the attacker is equivalent to the linear programming problem as follows:

$$\begin{array}{c}\begin{array}{cc}& maxZ=v\hfill \\ & \begin{array}{c}s.t.\sum _i{a}_{ij}{x}_i\ge v,j=1,2,\cdots ,n\hfill \\ \sum _i{x}_i=1\hfill \\ x_i\ge 0,i=1,2,\cdots ,m\hfill \end{array}\hfill \\ & minZ=v\hfill \\ & \begin{array}{c}s.t.\sum _j{a}_{ij}{y}_j\le v,i=1,2,\cdots ,m\hfill \\ \sum _j{y}_j=1\hfill \\ y_j\ge 0,j=1,2,\cdots ,n.\hfill \end{array}\hfill \end{array}\end{array}$$

The probability distribution of the optimal strategy can be obtained by solving the linear programming problem.

According to the solution of the linear programming, the Nash equilibrium policies of the sensors and attackers can be obtained, and the two players implement their optimal actions depending on this equilibrium respectively.

Step 3: With the current state $s_k$ and the combination of actions $a_k=(a_k^s,a_k^a)$ of sensor and attacker, the reward $r_k$ can be calculated by (15). Meanwhile, the next state is obtained according to (14). Note that when $s_k=h^n\left(\overline{P}\right)$, in accordance with the Markov chain shown in Figure 2, the next state is determined, namely $s_{k+1}=\overline{P}$.

Step 4: We use $\tilde{Q}(s,a^s,a^a)$ to denote the Q-value function under the state $s_k$ and action $a_k=(a_k^s,a_k^a)$. In order to update the Q-value matrix, the Q-value function is calculated according to the following iteration rules:

$$\begin{array}{c}\begin{array}{cc}\hfill {\tilde{Q}}_{k+1}& (s,a^s,a^a)=\left(1-{\alpha }_k\right){\tilde{Q}}_k(s,a^s,a^a)\hfill \\ & +{\alpha }_k\left(r_k+\rho \underset{a_{k+1}^a}{max}\underset{a_{k+1}^s}{min}{\tilde{Q}}_k(s,a^s,a^a)\right),\hfill \end{array}\end{array}$$

(16)

where $\alpha$ is the learning rate in $(0,1)$, which determines the extent of learning the results of new attempts. $\rho$ is the discount factor, $r_k$ is the immediate reward, and the Q-value matrix is obtained by the maxmin operation.

Step 5: Determine whether the termination condition is satisfied. If the termination condition is met, the algorithm terminates; otherwise k = k + 1, and go back to step 2.

Step 6: After the loop terminates, the converged Q-value matrix ${\tilde{Q}}^{*}(s,a^s,a^a)$ is obtained and the optimal policy based on Nash equilibrium ${\pi }_1^{*}$ and ${\pi }_2^{*}$ can be obtained for each state, where ${\pi }_1^{*}$ is the optimal policy of the sensor and ${\pi }_2^{*}$ is the optimal policy of the attacker.

The Q-learning algorithm for reliable channel is presented in Algorithm 1.

Algorithm 1 Q-learning Algorithm for Reliable channel

Input  :The parameters of the system A, C; the steady-state error covariance $\overline{P}$; cost $c_s$ and $c_a$; learning rate $\alpha$, discount factor $\rho$ and exploration rate $\epsilon$.

Output :Optimal Q-value matrix ${\tilde{Q}}^{*}(s,a^s,a^a)$, Nash equilibrium ${\pi }_1^{*}$ and ${\pi }_2^{*}$.

Initialize: Set initial state $s_1=0$, initialize Q-value matrix with m for all s and $a=(a^s,a^a)$, set $k=1$.

1: while $∥{\tilde{Q}}_{k+1}(s_{,}{a}^s,a^a)-{\tilde{Q}}_k(s_{,}{a}^s,a^a)∥<\eta$ do

2:  if $rand<\epsilon$ then

3:   Choose actions randomly;

4:  else

5:   Find the optimal actions obtained by linear programming method.

6:  end if

7:  Observe the reward $r_k$ by (15).

8:  Observe the next state $s_{k+1}$ according to (14).

9:  Update the Q-value matrix by (16).

10:  $s_k⟵s_{k+1}$

11:  $k⟵k+1$

12: end while

13: Return Q-value matrix for ${\tilde{Q}}^{*}(s,a^s,a^a)$.

14: Observe the Nash equilibrium ${\pi }_1^{*}$ and ${\pi }_2^{*}$.

5. Reinforcement Learning for Unreliable Channel

In practical CPSs, the channels over which the packets are transmitted is usually unreliable channels. In this scenario, packet loss can occur due to different reasons besides DoS attacks, including signal degradation, channel fading and channel congestion. Whereas the occurrence of packet loss is related to the choice of sensors and attackers, the packet loss probability of an unreliable channel can be described as follows:

$$\begin{array}{c}{\xi }_k=\left\{\begin{array}{cc}{p}_1,\hfill & \mathrm{if}{a}_k^s=0,a_k^a=0\hfill \\ p_2,\hfill & \mathrm{if}{a}_k^s=0,a_k^a=1\hfill \\ p_3,\hfill & \mathrm{if}{a}_k^s=1,a_k^a=0\hfill \\ p_4,\hfill & \mathrm{if}{a}_k^s=1,a_k^a=1.\hfill \end{array}\right.\end{array}$$

(17)

That is, when the sensor chooses insecure transmission and the attacker chooses not to attack, i.e., $a_k^s=0,a_k^a=0$, packets may be lost due to other reasons such as channel congestion, so the packet loss probability is $p_1$. Similarly, the packet loss probability under other action combinations can be obtained.

Remark 3.

In practical CPSs, there usually exists a relationship that $p_2>p_4>p_1>p_3$, the main reasons are as follows. Firstly, the sensor choosing insecure transmission namely $a_k^s=0$ and the attacker choosing to attack namely $a_k^a=1$, will both cause channel insecurity and increase the packet loss probability. Secondly, DoS attacks are the main cause of packet loss, thus their impact on the packet loss probability is greater than other causes such as channel congestion.

An MDP is set up to depict the interactive process for sensor and attacker under the framework of an unreliable channel. In the five tuples $\langle \mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\rho \rangle$ of MDP, the state, action and discount factor are the same as in the reliable channel; however, the state transition and reward function are different, which can be described as follows.

State transition: When data packets are transmitted in an unreliable channel, the state transition is not only based on whether a DoS attack occurs, but also on the packet loss probability. Under the action combination of the sensor and the attacker, the packet loss probability ${\xi }_k$ can be obtained according to (17). The data packet is lost with a probability of ${\xi }_k$ and is not lost with a probability of $1-{\xi }_k$, the corresponding arrival indicators are ${\delta }_k=0$ and ${\delta }_k=1$ respectively. Hence, the state transition occurs accordingly, which can be described as:

$$\begin{array}{c}{s}_{k+1}=P(k+1)=\left\{\begin{array}{cc}h\left(P\left(k\right)\right),\hfill & {\delta }_k=0\hfill \\ \overline{P},\hfill & {\delta }_k=1.\hfill \end{array}\right.\end{array}$$

(18)

Reward function: Since the packet loss probability changes, the reward of the system at state k has to change accordingly. If $s_k$ and $a_k$ are state and action, then the immediate reward $r_k$ at time k can be obtained as:

$$\begin{array}{c}{r}_k=Tr\left(\mathbf{E}\left[P\left(k\right)\right]\right)+c_s\ast a_k^s-c_a\ast a_k^a,\end{array}$$

(19)

where $\mathbf{E}\left[P\right(k\left)\right]$ represents the average expectation of the remote estimation error covariance $P\left(k\right)$, which is obtained by:

$$\begin{array}{c}\mathbf{E}\left[P(k+1)\right]=(1-{\xi }_k)\ast \overline{P}+{\xi }_k\ast h\left(P\left(k\right)\right).\end{array}$$

(20)

Remark 4.

The algorithm for unreliable channels has the following differences from the algorithm for reliable channels. First of all, the packet loss probability ξ under different combinations of actions needs to be entered first in the algorithm for unreliable channels. Second, at each time k, after the sensor and the attacker select actions according to the $\epsilon -greedy$ strategy, an additional step is needed to obtain the packet loss probability based on the action combinations. Then, in the calculation of the reward, it is necessary to use the reward that combines the error covariance expectation in Equation (19). Finally, in the observation of the next state, Equation (18) is also used to obtain $s_{k+1}$.

The Q-learning algorithm for the unreliable channel is presented in Algorithm 2.

Algorithm 2 Q-learning Algorithm for Unreliable channel

Input  :The parameters of the system A, C; the steady-state error covariance $\overline{P}$; cost $c_s$ and $c_a$; packet loss probability ${\xi }_k$ in each action combination; learning rate $\alpha$, discount factor $\rho$ and exploration rate $\epsilon$.

Output :Optimal Q-value matrix ${\tilde{Q}}^{*}(s,a^s,a^a)$, Nash equilibrium ${\pi }_1^{*}$ and ${\pi }_2^{*}$.

Initialize: Set initial state $s_1=0$, initialize Q-value matrix with m for all s and $a=(a^s,a^a)$, set $k=1$.

1: while $∥{\tilde{Q}}_{k+1}(s_{,}{a}^s,a^a)-{\tilde{Q}}_k(s_{,}{a}^s,a^a)∥<\eta$

2:  if $rand<\epsilon$ then

3:   Choose actions randomly;

4:  else

5:   Find the optimal actions obtained by linear programming method.

6:  end if

7:  According to the actions of sensors and attackers $a_k^s,a_k^a$, the packet loss probability ${\xi }_k$ is obtained by (17).

8:  Observe the reward $r_k$ by (19).

9:  Observe the next state $s_{k+1}$ according to (18).

10:  Update the Q-value matrix by (16).

11:  $s_k⟵s_{k+1}$

12:  $k⟵k+1$

13: end while

14: Return Q-value matrix for ${\tilde{Q}}^{*}(s,a^s,a^a)$.

15: Observe the Nash equilibrium ${\pi }_1^{*}$ and ${\pi }_2^{*}$.

Remark 5.

Whether the proposed algorithm is suitable for extending to DDoS attacks is also investigated. DDoS attacks are distributed denial of service attacks, which combines multiple computers as an attack platform to achieve the purpose of hindering the normal service of the computer or network. The attack–defense game problem under DDoS attacks adds a many-to-one dimension to the problem under DoS attacks. That is to say, DDoS attacks combine multiple attack sources to simultaneously attack a single sensor, and the attack cost and attack intensity of different attack sources may be different. For attackers, multiple attack sources need to be coordinated to minimize attack costs, while for system defenders, the impact of multi-source attacks needs to be minimized. In the future, we will focus on the impact of DDoS attacks on the attack and defense decisions of single-sensor systems and multi-sensor systems. One of our future directions is to coordinate multiple attack sources and sensors so that both attackers and defenders can make optimal decisions.

6. Simulations and Experiments

Two examples are given in this section to illustrate the effectiveness of reinforcement learning algorithms for reliable and unreliable channels, respectively.

6.1. Case 1: Simulation Example for Reliable Channel

Consider a CPS with parameters of system model shown as follows.

$$\begin{array}{c}A=\left[\begin{array}{cc}1& 0.5\\ 0& 1\end{array}\right],C={\left[\begin{array}{c}1\hfill \\ 0\hfill \end{array}\right]}^T,\\ Q=\left[\begin{array}{cc}0.8& 0\\ 0& 0.8\end{array}\right],R=0.8.\end{array}$$

The Kalman filter running converges to a steady state, and the steady state error covariance is obtained as:

$$\begin{array}{c}\overline{P}=\left[\begin{array}{cc}0.6\hfill & 0.4\hfill \\ 0.4\hfill & 2.4\hfill \end{array}\right],\end{array}$$

where trace $Tr\left(\overline{P}\right)$ is 3.

The cost for the sensor to choose secure transmission is set to $c_s=6.7$ and the attacker’s attack cost is set to $c_a=8$. In the game between the sensor and the attacker, the sensor knows the parameters of the system, so it can choose whether to adopt the costly secure transmission according to the situation. Based on the principle set in Section 3.3, the sensor can only choose secure transmission when its cost is less than the cost of continued packet loss. Due to $Tr\left(h^2\left(\overline{P}\right)\right)<c_s+Tr\left(\overline{P}\right)<Tr\left(h^3\left(\overline{P}\right)\right)$, it is easily learned that the sensor can only choose secure transmission after two consecutive moments of packet loss, namely $s_k=Tr\left(h^2\left(\overline{P}\right)\right)$. Therefore, the system state is a finite set of $S=\{\overline{P},h\left(\overline{P}\right),h^2\left(\overline{P}\right)\}$. Due to the system settings, when $s_k=Tr\left(h^2\left(\overline{P}\right)\right)$, it is bound to return to $s_k=\overline{P}$, which provides additional security for the system.

In this case, we set the learning rate and discount factor as 0.9 and 0.6, respectively. The Kalman filter as well as the Q-learning algorithm are run for 5000 iterations. During the first 500 iterations, the value of $\epsilon$ is set to slowly decrease from 1 to fully explore the reward for each action combination. In the subsequent iterations, $\epsilon$ is set to 0.1 unchanged, that is, the optimal action is selected with a probability of 0.9, and an action is randomly selected with a probability of 0.1. The optimal actions are obtained by the Nash equilibrium with the linear programming method.

The variation of $Tr\left(P\right(k\left)\right)$ is plotted, as shown in Figure 3. From this figure, we can observe the occurrence of packet loss and the change of state during the learning process, where $Tr\left(P\right(k\left)\right)=3$, $Tr\left(P\right(k\left)\right)=5.6$ and $Tr\left(P\right(k\left)\right)=9.6$ represent $s_k=\overline{P}$, $s_k=h\left(\overline{P}\right)$ and $s_k=h^2\left(\overline{P}\right)$, respectively. In the first 500 iterations, the selection of actions is random, so the frequency of packet loss is high. In the subsequent iterations, the sensor and the attacker select the optimal actions for the current situation, and it can be shown that the maximum number of consecutive packet losses is 2.

The learning process of $\tilde{Q}(s,a^s,a^a)$ at state $s=\{\overline{P},h\left(\overline{P}\right),h^2\left(\overline{P}\right)\}$ is shown in Figure 4. According to Figure 4, it can be concluded that the elements of the Q-value matrix decrease and converge to ${\tilde{Q}}^{*}(s,a^s,a^a)$ by the Q-learning algorithm, and the Q-value matrix obtained by convergence is shown in

Table 1. ${\tilde{Q}}^{*}(s,a^s,a^a)$ for reliable channel.

State-Action	$\mathit{a}=(0,0)$	$\mathit{a}=(0,1)$	$\mathit{a}=(1,0)$	$\mathit{a}=(1,1)$
$s=\overline{P}$	7.500	2.100	14.300	6.300
$s=h\left(\overline{P}\right)$	7.500	10.190	14.300	6.300
$s=h^2\left(\overline{P}\right)$	7.538	−0.499	14.301	7.237

. $s=\{\overline{P},h\left(\overline{P}\right),h^2\left(\overline{P}\right)\}$ represents three different states. $a=(a_k^s,a_k^a)=\{(0,0),(0,1),(1,0),(1,1)\}$ represents different action-pairs of the sensor and attacker, which has been defined in Section 3.2 and the MDPs in Section 4. The numeric values in each grid represent the convergence of $\tilde{Q}(s,a^s,a^a)$ for each state-action.

According to Nash equilibrium theory, Nash equilibrium points exist for any finite game. From the Q-value matrix, we know that there exist Nash equilibrium points of the game, which are shown in

Table 2. Nash equilibrium ${\pi }_1^{*}$ and ${\pi }_2^{*}$ for reliable channel.

State-${\mathit{\pi }}^{*}$	${\mathit{\pi }}_1^{*}$	${\mathit{\pi }}_2^{*}$
$s=\overline{P}$	$(1,0)$	$(1,0)$
$s=h\left(\overline{P}\right)$	$(0.748,0.252)$	$(0.364,0.636)$
$s=h^2\left(\overline{P}\right)$	$(1,0)$	$(1,0)$

${\mathit{\pi }}_1^{*}=\left(Pr\right(a^s=0),Pr(a^s=1\left)\right),{\mathit{\pi }}_2^{*}=\left(Pr\right(a^a=0),Pr(a^a=1\left)\right)$.

. The solution is conducted with the linear programming method to obtain the mixed strategy Nash equilibrium of the sensor and the attacker, denoted by ${\pi }_1^{*}$ and ${\pi }_2^{*}$, respectively. For example, at state $s=h\left(\overline{P}\right)$, the mixed strategy Nash equilibria for the sensor and the attacker are (0.748,0.252) and (0.364,0.636), respectively. This implies that the sensor chooses insecure transmission with probability 0.748 and secure transmission with probability 0.252; meanwhile, the attacker chooses no attack with probability 0.364 and attack with probability 0.636.

After obtaining the optimal policy for the sensor and the attacker under the DoS attack of the reliable channel, we can calculate the probability of packet loss under a single attack and two consecutive attacks as an evaluation criterion for the algorithm. It can be known that the probability of two consecutive packet losses is 0.04%, and the probability of single packet loss is 2.44%.

To assess the computational costs of the proposed algorithm, we computed that the running time of the algorithm is 0.277 s in the reliable channel.

On the whole, the reinforcement learning algorithm can quickly converge to the Nash equilibrium policies and the probability of packet loss of two consecutive attacks is less than that of a single attack.

6.2. Case 2: Simulation Example for Unreliable Channel

Consider the same system as in Case 1, where data are transmitted over an unreliable channel, i.e., packet loss may occur due to a variety of reasons. Setting the packet loss rate as $p=\{p_1,p_2,p_3,p_4\}=\{0.01,1,0.001,0.1\}$. With this setting, the packet loss rate in different states can be expressed as:

$$\begin{array}{c}{\xi }_k=\left\{\begin{array}{cc}1\%,\hfill & \mathrm{if}{a}_k^s=0,a_k^a=0\hfill \\ 100\%,\hfill & \mathrm{if}{a}_k^s=0,a_k^a=1\hfill \\ 0.1\%,\hfill & \mathrm{if}{a}_k^s=1,a_k^a=0\hfill \\ 10\%,\hfill & \mathrm{if}{a}_k^s=1,a_k^a=1.\hfill \end{array}\right.\end{array}$$

The variation of $Tr\left(P\right(k\left)\right)$ is plotted, as shown in Figure 5. It can be seen that the frequency of data packet loss in the unreliable channel is greater than that in the reliable channel. Furthermore, in 501–5000 iterations, the maximum number of consecutive packet losses is 2 as well.

The learning process of $\tilde{Q}(s,a^s,a^a)$ at state $s=\{\overline{P},h\left(\overline{P}\right),h^2\left(\overline{P}\right)\}$ is shown in Figure 6. Q-value matrix converges at about 500 steps. The convergence curve fluctuates slightly between 500 and 1300 steps which is because the existence of $ϵ$ preserves a certain probability of random exploration. After 1300 steps, the wave disappears. According to Figure 6, it can be concluded that the elements of the Q-value matrix decrease and converge to ${\tilde{Q}}^{*}(s,a^s,a^a)$ by the Q-learning algorithm, and the Q-value matrix obtained by convergence is shown in

Table 3. ${\tilde{Q}}^{*}(s,a^s,a^a)$ for unreliable channel.

State-Action	$\mathit{a}=(0,0)$	$\mathit{a}=(0,1)$	$\mathit{a}=(1,0)$	$\mathit{a}=(1,1)$
$s=\overline{P}$	7.565	2.163	13.344	5.859
$s=h\left(\overline{P}\right)$	7.605	9.620	13.352	6.659
$s=h^2\left(\overline{P}\right)$	7.682	12.820	13.364	7.819

.

There also exist Nash equilibrium points of the game based on game theory, which are shown in

Table 4. Nash equilibrium ${\pi }_1^{*}$ and ${\pi }_2^{*}$ for unreliable channel.

State-${\mathit{\pi }}^{*}$	${\mathit{\pi }}_1^{*}$	${\mathit{\pi }}_2^{*}$
$s=\overline{P}$	$(1,0)$	$(1,0)$
$s=h\left(\overline{P}\right)$	$(0.769,0.231)$	$(0.340,0.660)$
$s=h^2\left(\overline{P}\right)$	$(0.519,0.481)$	$(0.468,0.532)$

${\mathit{\pi }}_1^{*}=\left(Pr\right(a^s=0),Pr(a^s=1\left)\right),{\mathit{\pi }}_2^{*}=\left(Pr\right(a^a=0),Pr(a^a=1\left)\right)$.

. The solution is conducted with the linear programming method to obtain the mixed strategy Nash equilibrium of the sensor and the attacker, denoted by ${\pi }_1^{*}$ and ${\pi }_2^{*}$, respectively. It can be seen that, under channel unreliability conditions, sensors are more focused on secure transmission so as to reduce the probability of packet loss.

Likewise, after obtaining the optimal policy for the sensor and the attacker, the probability of packet loss under a single attack and two consecutive attacks are calculated as an evaluation criterion for the algorithm. The probability of two consecutive packet losses is 0.42%, and the probability of single packet loss is 3.76%. Compared with the results of case 1, it can be seen that the probability of packet loss in the unreliable channel is significantly higher than that in the reliable channel.

In the unreliable channel, we also evaluate the computational cost of the proposed algorithm, and the running time of the algorithm is 0.489 s.

In conclusion, the reinforcement learning algorithm can also quickly converge to the Nash equilibrium policies under the unreliable channel. The probability of losing packets in two consecutive attacks is less than that in a single attack as well and the probability of packet loss increases compared with the results of case 1.

7. Conclusions

This paper studies the security state estimation and optimal policy selection algorithm of attackers and sensors of CPSs under DoS attacks. A two-player zero-sum deterministic game is employed to describe the interactive decision process between the sensor and the attacker. A Kalman filter and reinforcement learning algorithm are introduced to evaluate policy influence and dynamically adjust strategies. Furthermore, the paper studies how to reach the Nash equilibrium of both sides to guide their decision-making choices. In consideration of the different packet loss rates under different channels in practical application, this paper proposes a reinforcement learning method based on Q-learning in two scenarios: reliable channel and unreliable channel. Simulation results of reliable and unreliable channels show that the Q-value matrix can converge in a finite number of steps to obtain Nash equilibrium policies for both sides, which proves the effectiveness of the algorithm. The convergence process, Nash equilibrium strategy and packet loss probability under reliable and unreliable channels are elaborated and analyzed, which can guide the selection of attack and defense strategies in practical CPSs. Moreover, based on the consideration of the limited bandwidth and energy of the system, this paper optimizes the resource allocation. Specifically, the attacker and the sensor make the best response to the other’s behavior by the algorithm proposed in this paper. In this way, unnecessary resource consumption caused by confrontation can be reduced, thus achieving the optimization of system resource utilization. Future work includes extending reinforcement learning algorithms under reliable and unreliable channels to multi-sensor single-attacker or multi-sensor multi-attacker security problems. In addition, the model expansion under different attacks will also be our future consideration, such as how to coordinate multiple attack sources and sensors under DDoS attacks to make optimal decisions.