Secure Change Control for Supply Chain Systems via Dynamic Event Triggered Using Reinforcement Learning under DoS Attacks

Abstract

In this paper, a distributed secure change control scheme for supply chain systems is presented under denial-of-service (DoS) attacks. To eliminate the effect of DoS attacks on supply chain systems, a secure change compensation is designed. A distributed policy iteration method is established to approximate the coupled Hamilton–Jacobi–Isaacs (HJI) equations. Based on the established reinforce–critic–actor (RCA) structure using reinforcement learning (RL), the reinforced signals, performance indicators, and disturbance input are proposed to update the traditional time-triggered mechanism, and the control input is proposed to update the dynamic event-triggered mechanism (DETM). Stability is guaranteed based on the Lyapunov method under secure change control. The simulation results for supply chain systems show the effectiveness of the secure change control scheme and verify the results.

1. Introduction

The production and manufacturing of enterprises are closely connected, thus forming a multi-connected network supply chain system, which is usually a complex network system composed of manufacturers, distributors, and retailers [1]. The control of the supply chain production inventory system has always been an important task of enterprise management. The supply chain production inventory system is designed by the traditional single-level sub-chain system, but for large enterprises, the production inventory system is a network system composed of multi-level sub-chains, which is more in line with the research on modern supply chain systems. Therefore, multi-agent is widely used in research on supply chain systems [2,3,4]. The control theory has been widely used in supply chain management. In [5,6], the supply chain is controlled by synovial control. In [7], the dynamic supply chain is designed via fuzzy robust control. In [8,9], the multi-agent supply chain is tracked and controlled at a fixed time. In addition, distributed model predictive control has been applied to supply chain inventory management [10,11]. However, the modern supply chain inventory system is faced with many challenges, such as the difficulty in accurately obtaining system dynamics information and accurately constructing system structure, etc. Therefore, there is an urgent need to develop data-driven methods that rely only on data instead of models. In addition, unexpected network attacks will seriously damage the normal production operation of enterprises and damage the security of the system. How to develop a security scheme to deal with major network events, and realizing the security control of the supply chain production and inventory system is an important means to prevent enterprises from suffering heavy economic losses.

From the perspective of control research on supply chain production inventory systems, firstly, based on the precise mathematical dynamics model of the system [12], all designs are based on the static mechanism model and [13] matrix inequalities are used to optimize and solve the controller, but this method relies on the precise dynamics information of the system. Secondly, with the rapid development of information technology, the manufacturing industry is developing rapidly towards intelligent production and manufacturing [14]. From the perspective of the agent, the inventory–production–modeling integrated architecture of the supply chain system is presented in [15]. Intelligent manufacturing is composed of interconnected enterprises, machines, and human and physical systems through the basic network of the industrial Internet. This enables comprehensive sensing, dynamic transmission, and real-time analysis of industrial data and then intelligent control and scientific decision-making to improve the efficient allocation of manufacturing resources. During the operation of supply chain systems, a large amount of input and process data are generated, which are recorded and kept by the system equipment for the subsequent data-driven design [16]. The operation and execution of the current large-scale intelligent industrial system and machinery and equipment ensure the smooth circulation of the large-scale industrial Internet. The accurate transmission of the Internet equipment determines the normal operation of the machine equipment, and the machine equipment and the industrial Internet transmission information medium are usually connected through the data; for large enterprises, the network structure is more complex, data transmission is extremely large and dense, and the transmission equipment requirements are higher. Therefore, determining how to save transmission data resources for enterprises and reduce the pressure of communication load is an important problem to be solved in the supply chain system, and it is also an important task of this study.

With the development of big data, artificial intelligence, and digital twins, modern supply chain systems have become intelligent systems integrating production equipment, robots, and the industrial Internet [17]. The intelligent system of a large-scale network will be subject to unexpected external events, which will lead to supply chain disruption. Therefore, change control in the face of emergencies has become an important issue in supply chain design. The authors of [18] developed a recovery control algorithm for supply chain disruptions. For a non-linear supply chain system, in [19], faced with the influence of demand disturbance and unexpected events, the feasible solution of the Takagi–Sugeno fuzzy system is given by using linear matrix inequality. However, network security events will lead to supply chain system transmission interruption; these security events will directly attack the information data of the system and destroy the security operation of the whole supply chain system, resulting in major security accidents. Therefore, the problem of security change control has been paid more and more attention by enterprises. It has become an important task of the inventory control of supply chain systems to design security change plans for the system and ensure that the system can respond to emergencies quickly and in a timely manner. At present, secure change control has been widely explored and studied in the field of multi-agent. For aperiodic persistent network attacks [20], an estimator is established to compensate for the state under network attacks, and a double-ended event-triggered mechanism is proposed to ensure the consistency of system security. A new adaptive dynamic event-triggered mechanism was proposed and the maximum duration of a network attack was deduced to achieve security consensus in [21]. Model-free adaptive control is applied to the secure control as an effective data-driven method. For an aperiodic network attack, the compensation scheme under network attack is given by [22]. For periodic network attacks, an emergency compensation scheme was proposed at the time of network attacks and an observer was used to estimate the output in [23]. Adaptive dynamic programming (ADP) has been widely developed and studied for solving optimal controllers. RL has been utilized to tackle the optimal control problem for multi-agent systems, such as tracking control [24,25]. After this, refs. [26,27] both unitized event-triggered ADP to address optimal control problems. Although the adaptive dynamic programming technique can be used to approach the optimal controller in the above paper, the external disturbance is not considered enough. For the supply chain system, the external uncertain demand is usually unknown, and it will gradually amplify along the reverse direction of the product side of the system, which is the bullwhip effect. Therefore, weakening the adverse impact of the bullwhip effect on the supply chain system has become another important task in the design of this system. Effective methods to weaken the bullwhip effect are proposed in [28,29]. However, ADP is rarely applied to the design of supply chain systems, and a large number of studies have found that the dynamic event triggering method can reduce the communication load pressure and save the communication load between supply chains. Therefore, we studied the supply chain inventory control based on the combination of adaptive dynamic planning technology and the dynamic event triggering mechanism.

The previous works focused on supply chain systems under DoS attacks can be used to detect and mitigate the impact through Machine Learning (ML), Deep Learning (DL), and reinforcement learning. In the attack detection for supply chain systems, leverage evolutionary and DL approaches to detect cyber-attacks in a cloud-based Supply Chain Management environment are proposed in [30]. A Machine Learning approach for network anomaly detection and constructing data-driven models to detect distributed DoS attacks on industry is presented in [31]. A federated learning-based efficient detection model named DFF-SC4N is addressed in [32] to identify intrusions from supply chain 4.0 networks. In the prediction detection for supply chain systems, Logistic Regression, Decision Tree, Naïve Bayes, and Random Forest classification algorithms are considered to learn a dataset for performance accuracies and threat predictions based on the CSC resilience design principles in [33]. However, RL is the only ML technique that can learn without any dataset. It is considered that supply chain systems only have initial permission or arbitrary data, so secure control for supply chain systems can be achieved.

Inspired by the large amount of research mentioned above, we carried out the following work. Firstly, the problem of the bullwhip effect caused by uncertain market demand is considered, and the idea of a zero-sum differential game is introduced into the supply chain system. Secondly, a goal-heuristic dynamic programming adaptive reinforcement learning method combined with the dynamic event triggering mechanism is designed. The dynamic event-triggering mechanism is then compared with a static event-triggering scheme [34,35]. The dynamic triggering scheme is adopted to further reduce the number of triggers. Finally, due to the packet loss caused by DoS attacks, an emergency compensation scheme is designed to realize the security change control of the supply chain systems, and a Lyapunov proof based on emergency compensation under DoS attacks is given. The simulation results fully demonstrate the effectiveness of the proposed method. The contributions of this paper are as follows:

(1)

For the supply chain production inventory system, the production input and uncertain demand are regarded as two sides of a zero-sum game. Based on the HJI equation, the RCA network online learning structure is established.

(2)

On this basis, a dynamic event-triggered mechanism is proposed, and appropriate internal parameters of the dynamic event-triggered mechanism are selected to reduce the number of iterations of the neural network, so as to realize the dynamic event triggering tracking control of each sub-chain of the supply chain to the main chain.

(3)

A secure change control scheme under DoS attack is proposed to ensure the normal operation of the supply chain production inventory system. The simulation analysis carried out proves that the proposed security change scheme can achieve effective change control.

The structure of the rest of this paper is organized as follows. Section 2 provides some preliminary knowledge. In Section 3, the dynamic event triggering mechanism and stability analysis are given. The learning structure of the neural network is presented in Section 4, the proposed method is verified by simulation, and Section 5 provides the summary and future research direction of this paper.

2. Preliminaries

2.1. Algebraic Graph Theory

In this paper, we consider a communication topology $\mathcal{G}=(\mathcal{V},ℰ,\mathcal{A})$ consisting of a vertex set $\mathcal{V}=\left\{v_1,v_2,\cdots ,v_N\right\}$. $ℰ\in \mathcal{V}\times \mathcal{V}$ is an edge set, which indicates that $v_i$ can obtain state information directly from $v_j$. $\mathcal{A}=\left[a_{ij}\right]$ is a weighted adjacency matrix with element $a_{ij}$, $a_{ij}>0$ if and only if $\left(v_i,v_j\right)\in ℰ$, and $a_{ij}=0$ otherwise. The in-degree matrix defines a diagonal matrix $\mathcal{D}=\mathrm{diag}\left\{d_1,d_2,\cdots ,d_n\right\}$ with $d_i\in {\displaystyle \sum _{j\in N_i}{a}_{ij}}$, and the Laplacian matrix $L$ can be defined as $L=\mathcal{D}-\mathcal{A}$. The connected matrix of the leader is defined as a diagonal matrix $ℬ=diag\left\{b_1,b_2,\cdots ,b_{\mathrm{n}}\right\}$ where $b_i>0$ if $v_i$ can receive the information leader and $b_i=0$ otherwise.

2.2. Problem Formulation

We consider a supply chain system consisting of $N$ subchains and a chain leader. The dynamics are described as:

$$\{\begin{array}{l}{x}_i\left(k+1\right)=A{x}_i\left(k\right)+B{u}_i\left(k\right)+D{\widehat{\omega }}_i\left(k\right)\hfill \\ y_i\left(k\right)=C{x}_i\left(k\right)\hfill \end{array}$$

(1)

where $i=1,2,\cdots ,N$, $x_i\left(k\right)\in R^n$ is the production inventory status for the subchain $i$ at $k$. $u_i\left(k\right)\in R^n$ is the productivity for the subchain $i$ at $k$. ${\widehat{\omega }}_i\in R^n$ is the market demand for subchain $i$ at $k$, and $d_i\in R^n$ is the constant market. The production inventory, productivity, and market can be regarded as the state variable, the control input, and external disturbances for control theory. $A$, $B$, and $D$ represent the unknown system matrix.

The chain leader is the tracking target of the other subchains. The dynamics of the chain leader are described as:

$$x_0\left(k+1\right)=A{x}_0\left(k\right)+D{d}_i$$

(2)

where $x_0\left(k\right)\in R^n$ is the production inventory status for the chain leader at $k$.

Definition 1.

The design goal of the supply chain production inventory system is to design a distributed minimum control strategy $u_i\left(k\right)$ and maximum disturbance strategy ${\omega }_i\left(k\right)$, so that the inventory status of all subchains can follow the inventory status of the chain leader $x_i\left(0\right)$, that is:

$$\underset{k\to \infty }{\lim }\Vert x_i\left(k\right)-x_0\left(k\right)\Vert =0$$

(3)

Definition 2.

For the supply chain production inventory system, there exists a bullwhip suppression parameter, which makes the following bullwhip effect suppression conditions valid, that is:

$${\displaystyle \sum _{k=0}^{\infty }{e}_i^{{\rm T}}\left(k\right)}{Q}_{ii}{e}_i\left(k\right)+{\displaystyle \sum _{k=0}^{\infty }{u}_i^{{\rm T}}\left(k\right)}{R}_{ii}{u}_i\left(k\right)\le {\gamma }^2{\displaystyle \sum _{k=0}^{\infty }{\omega }_i^{{\rm T}}\left(k\right)T_{ii}{\omega }_i\left(k\right)}$$

(4)

Assumption 1.

The directed communication topology contains a spanning tree with the root node.

Assumption 2.

There exists a positive constant $m$ such that:

$$\Vert f_i\left(e_i\left(k\right),u_i\left(k_s^i\right),{\omega }_i\left(k\right)\right)\Vert \le m\Vert e_i\left(k\right)\Vert +m\Vert {ϵ}_i\left(k\right)\Vert$$

(5)

Lemma 1.

([24]). According to Assumption 1, $L+ℬ$ is a positive definite matrix (non-singular). Then, the consensus error is bounded by:

$$\Vert \xi \left(k\right)\Vert \le \Vert e\left(k\right)\Vert /{\lambda }_{\min }\left(L+ℬ\right)$$

(6)

where ${\lambda }_{\min }\left(L+ℬ\right)$ is the minimum singular value of $\left(L+ℬ\right)$.

The local neighborhood error of subchain $i$ is defined as:

$$e_i\left(k\right)={\displaystyle \sum _{j\in N_i}{a}_{ij}\left(x_i\left(k\right)-x_j\left(k\right)\right)}+b_i\left(x_i\left(k\right)-x_0\left(k\right)\right)$$

(7)

The global consensus error vector is given by:

$$e\left(k\right)=\left(\left(L+ℬ\right)\otimes I_n\right)\left(x\left(k\right)-{\overline{x}}_0\left(k\right)\right)$$

(8)

where $e\left(k\right)={\left[e_1^{{\rm T}}\left(k\right),e_2^{{\rm T}}\left(k\right),\dots ,e_N^{{\rm T}}\left(k\right)\right]}^{{\rm T}}\in R^{Nn}$, $x\left(k\right)={\left[x_1^{{\rm T}}\left(k\right),x_2^{{\rm T}}\left(k\right),\dots ,x_N^{{\rm T}}\left(k\right)\right]}^{{\rm T}}\in R^{Nn}$, ${\overline{x}}_0\left(k\right)=\left(1\otimes I_n\right)x_0\left(k\right)\in R^{Nn}$.

Then, the global synchronization error vector is written as:

$$\xi \left(k\right)=x\left(k\right)-{\overline{x}}_0\left(k\right)$$

(9)

where $\xi \left(k\right)={\left[{\xi }_1^{{\rm T}}\left(k\right),{\xi }_2^{{\rm T}}\left(k\right),\dots ,{\xi }_N^{{\rm T}}\left(k\right)\right]}^{{\rm T}}\in R^{Nn}$.

The dynamic of the local neighborhood error for subchain $i$ is obtained as:

$$\begin{array}{l}{e}_i\left(k+1\right)=A{e}_i\left(k\right)+\left(d_i+b_i\right)B{u}_i\left(k\right)-{\displaystyle \sum _{j\in N_i}B{u}_j\left(k\right)}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\left(d_i+b_i\right)D{\omega }_i\left(k\right)-{\displaystyle \sum _{j=1}^N{a}_{ij}D{\omega }_j\left(k\right)}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=f_i\left(e_i\left(k\right),u_i\left(k\right),{\omega }_i\left(k\right)\right)\end{array}$$

(10)

3. Results

3.1. The Secure Change Consensus Control Scheme

The purpose of DoS attacks is to decrease the supply chain systems’ performance by blocking the useful information transmitted between the sensor and the controller. DoS attacks cause packet dropouts in communication channels, resulting in production equipment being unable to operate normally. The supply chain systems cannot be designed according to the objective control scheme. The structure of the secure change control for supply chain systems is shown in Figure 1. The data packets received by the controller can be transformed into the following form:

$${\tilde{e}}_{di}\left(k\right)={\alpha }_i\left(k\right)e_i\left(k\right)$$

(11)

where ${\alpha }_i\left(k\right)$ represents whether the DoS attacks are successful in the communication channels. If the DoS attacks are successful, ${\alpha }_i\left(k\right)=1$; otherwise, ${\alpha }_i\left(k\right)=0$. The probability of DoS attacks conforms to a Bernoulli distribution, $P\left\{{\alpha }_i\left(k\right)=1\right\}={\beta }_i$,$P\left\{{\alpha }_i\left(k\right)=0\right\}=1-{\beta }_i$.

To eliminate the effects of DoS attacks, the secure change scheme is designed as:

$${\tilde{e}}_{di}\left(k\right)=\left(1-{\alpha }_i\left(k\right)\right)e_i\left(k\right)+{\alpha }_i\left(k\right)e_i\left(k-1\right)$$

(12)

The internal dynamic variable ${\tilde{\theta }}_{di}\left(k\right)$ under the secure change satisfies

$${\tilde{\theta }}_{di}\left(k\right)=\left(1-{\alpha }_i\left(k\right)\right){\theta }_i\left(k\right)+{\alpha }_i\left(k\right){\theta }_i\left(k-1\right)$$

(13)

To improve the tracking performance of the supply chain system, the local internal performance signals can be described as:

$$\begin{array}{l}{P}_i\left(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)\\ ={\displaystyle \sum _{m=k}^{\infty }{\alpha }^{m-k}{r}_i\left(e_i\left(m\right),u_i\left(m\right),u_{-i}\left(m\right),{\omega }_i\left(m\right),{\omega }_{-i}\left(m\right)\right)}\end{array}$$

(14)

where $\alpha \in \left(0,1\right)$ is the discount factor, $u_{-i}\left(k\right)=\left\{u_j\left(k\right)|j\in N_i\right\}$ is the control input of the $i$ subchain’s neighbors, and ${\widehat{\omega }}_{-i}\left(k\right)=\left\{{\widehat{\omega }}_j\left(k\right)|j\in N_i\right\}$ is the disturbance input of the $i$ subchain’s neighbors. The external reinforcement signal is given by:

$$\begin{array}{l}{r}_i\left(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_i\left(k\right)\right)=e_i^{{\rm T}}\left(k\right)Q_{ii}{e}_i\left(k\right)+u_i^{{\rm T}}\left(k\right)R_{ii}{u}_i\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\displaystyle \sum _{j\in N_i}{u}_j^{{\rm T}}\left(k\right)R_{ij}{u}_j\left(k\right)}-{\gamma }^2{\omega }_i^{{\rm T}}\left(k\right)T_{ii}{\omega }_i\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-{\gamma }^2{\displaystyle \sum _{j\in N_i}{\omega }_j^{{\rm T}}\left(k\right)T_{ij}{\omega }_j\left(k\right)}\end{array}$$

(15)

where $R_{ii}>0$, $R_{ij}>0$, $T_{ii}>0$, $T_{ij}>0$ are all positive symmetric weighting matrices.

Then, the local performance function can be defined as:

$$\begin{array}{l}{J}_i\left(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)={\displaystyle \sum _{l=k}^{\infty }\{e_i^{{\rm T}}\left(k\right)Q_{ii}{e}_i\left(k\right)+u_i^{{\rm T}}\left(k\right)R_{ii}{u}_i\left(k\right)}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\displaystyle \sum _{j\in N_i}{u}_j^{{\rm T}}\left(k\right)R_{ij}{u}_j\left(k\right)}-{\gamma }^2{\displaystyle \sum _{j\in N_i}{\omega }_j^{{\rm T}}\left(k\right)T_{ij}{\omega }_j\left(k\right)}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-{\gamma }^2{\omega }_i^{{\rm T}}\left(k\right)T_{ii}{\omega }_i\left(k\right)\}\end{array}$$

(16)

Given the admissible control input $u_i$ and disturbance input ${\omega }_i$, we define the local value function $V_i\left(e_i\left(k\right)\right)$ as:

$$V_i\left(e_i\left(k\right)\right)={\displaystyle \sum _{t=k}^{\infty }{\eta }^{t-k}{P}_i\left(e_i\left(t\right),u_i\left(t\right),u_{-i}\left(t\right),{\omega }_i\left(t\right),{\omega }_{-j}\left(t\right)\right)}$$

(17)

where $\eta \in \left(0,1\right)$ is the discount factor.

According to Equation (16), the Bellman equation is given by:

$$V_i\left(e_i\left(k\right)\right)=P_i\left(u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)+\eta V_i\left(e_i\left(k+1\right)\right)$$

(18)

Based on the Bellman optimality principle, the optimal value function $V_i^{\ast }\left(e_i\left(k\right)\right)$ of subchain $i$ satisfies the following $HJI$ equation:

$$\begin{array}{l}{V}_i^{\ast }\left(e_i\left(k\right)\right)=\underset{u_i}{\min }\underset{{\omega }_i}{\max }\left\{P_i\left(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)+\eta V_i^{\ast }\left(e_i\left(k+1\right)\right)\right\}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\underset{{\omega }_i}{\max }\underset{u_i}{\min }\left\{P_i\left(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)+\eta V_i^{\ast }\left(e_i\left(k+1\right)\right)\right\}\end{array}$$

(19)

where the local internal reinforcement signals can be rewritten as:

$$\begin{array}{l}{P}_i\left(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)\\ =r_i\left(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)\\ +\alpha P_i\left(e_i\left(k+1\right),u_i\left(k+1\right),u_{-i}\left(k+1\right),{\omega }_i\left(k+1\right),{\omega }_{-i}\left(k+1\right)\right)\end{array}$$

(20)

Then, the optimal control pair can be expressed as:

$$u_i^{\ast }\left(k\right)=\underset{u_i}{\arg \min }\left\{P_i\left(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)+\eta V_i^{\ast }\left(e_i\left(k+1\right)\right)\right\}$$

(21)

$${\omega }_i^{\ast }\left(k\right)=\underset{{\omega }_i}{\arg \max }\left\{P_i\left(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)+\eta V_i^{\ast }\left(e_i\left(k+1\right)\right)\right\}$$

(22)

For subchain $i$, we denote ${\left\{k_s^i\right\}}_{s=0}^{\infty }$ as the incrementally triggering sequence.

The local neighbor error is rewritten by:

$$e_i\left(k\right)=e_i\left(k_s^i\right)$$

(23)

The control input of subchain $i$ is rewritten by:

$$u_i\left(k\right)=u_i\left(k_s^i\right),k\in \left[k_s^i,k_{s+1}^i\right)$$

(24)

Then, we define the error variable ${ϵ}_i\left(k\right)$ as:

$${ϵ}_i\left(k\right)=e_i\left(k_s^i\right)-e_i\left(k\right),k\in \left[k_s^i,k_{s+1}^i\right)$$

(25)

Once the event is triggered, ${ϵ}_i\left(k\right)=0$.

According to the $HJI$ equation under the dynamic event-triggered mechanism, the optimal control pair can be rewritten as:

$$\begin{array}{l}{V}_i^{\ast }\left(e_i\left(k\right)\right)=\underset{u_i}{\min }\underset{{\omega }_i}{\max }\left\{P_i\left(e_i\left(k\right),u_i\left(k_s^i\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)+\eta V_i^{\ast }\left(e_i\left(k+1\right)\right)\right\}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\underset{{\omega }_i}{\max }\underset{u_i}{\min }\left\{P_i\left(e_i\left(k\right),u_i\left(k_s^i\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)+\eta V_i^{\ast }\left(e_i\left(k+1\right)\right)\right\}\end{array}$$

(26)

The optimal control input under the dynamic event-triggered mechanism is rewritten as:

$$u_i^{\ast }\left(k\right)=\underset{u_i}{\arg \min }\left\{P_i\left(e_i\left(k\right),u_i\left(k_s^i\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)+\eta V_i^{\ast }\left(e_i\left(k+1\right)\right)\right\}$$

(27)

The optimal disturbance input under the traditional time-triggered mechanism is rewritten as:

$${\omega }_i^{\ast }\left(k\right)=\underset{{\omega }_i}{\arg \max }\left\{P_i\left(e_i\left(k\right),u_i\left(k_s^i\right),u_{-i}\left(k\right),{\omega }_i\left(k\right),{\omega }_{-i}\left(k\right)\right)+\eta V_i^{\ast }\left(e_i\left(k+1\right)\right)\right\}$$

(28)

3.2. Stability Analysis

For subchain $i$, the dynamic event-triggered mechanism is given by:

$$2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2\le \left(1-2m^2\right){\Vert e_i\left(k\right)\Vert }^2+{\rho }_i{\theta }_i\left(k\right)$$

(29)

where $0<{\rho }_i<1$, $0<m<\frac{\sqrt{2}}{2}$, and ${\theta }_i\left(k\right)$ satisfy:

$${\theta }_i\left(k+1\right)=\left(1-{\lambda }_i\right){\theta }_i\left(k\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e_i\left(k\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k\right)\Vert }^2\right)$$

(30)

where $0<\left(1-{\xi }_i\right){\rho }_i<{\lambda }_i<1$, $0<{\xi }_i<1$.

Lemma 2.

For the dynamic event-triggered mechanism, it satisfies:

$${\theta }_i\left(k\right)>0$$

(31)

Proof of Lemma 2.

According to (28) and (29), one has

$$\left(2m^2\right){\Vert {ϵ}_i\left(k\right)\Vert }^2-\left(1-2m^2\right){\Vert e_i\left(k\right)\Vert }^2\le {\rho }_i{\theta }_i\left(k\right)$$

(32)

Based on (29), one has

$${\theta }_i\left(k+1\right)\ge \left(1-{\lambda }_i-{\xi }_i{\rho }_i\right){\theta }_i\left(k\right)\ge ,\dots ,\ge {\left(1-{\lambda }_i-{\xi }_i{\rho }_i\right)}^{k+1}{\theta }_i\left(0\right)\ge {\left(1-{\rho }_i\right)}^{k+1}{\theta }_i\left(0\right)$$

(33)

It is clear that ${\theta }_i\left(0\right)>0$, $\left(1-{\lambda }_i-{\xi }_i{\rho }_i\right)>0$; therefore, ${\theta }_i\left(k\right)>0$. This completes the proof. □

Theorem 1.

Suppose that Assumption 1 and Assumption 2 hold. Supply chain systems (1) and (2) could achieve secure change consensus under the dynamic event-triggered mechanisms (28) and (29) under DoS attacks.

Proof of Theorem 1.

In order to guarantee the stability of the designed systems, consider the following Lyapunov function for $k\in \left[k_s^i,k_{s+1}^i\right)$

$$L_i\left(k\right)=L_{1i}\left(k\right)+L_{2i}\left(k\right)$$

(34)

where $L_{1i}\left(k\right)={\tilde{e}}_{di}^{{\rm T}}\left(k\right){\tilde{e}}_{di}\left(k\right)$, $L_{2i}\left(k\right)={\tilde{\theta }}_{di}\left(k\right)$. □

(1)

If DoS attacks do not occur at two continuous sampling times $k$ and times $k+1$, it is assumed that ${\alpha }_i\left(k\right)={\alpha }_i\left(k+1\right)=0$

The difference of $L_{1i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{1i}\left(k\right)=L_{1i}\left(k+1\right)-L_{1i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=e_i^{{\rm T}}\left(k+1\right)e_i\left(k+1\right)-e_i^{{\rm T}}\left(k\right)e_i\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left({\Vert e_i\left(k+1\right)\Vert }^2-{\Vert e_i\left(k\right)\Vert }^2\right)\end{array}$$

(35)

According to Assumption 2 and the Cauchy–Schwarz inequality equation, one has

$$\begin{array}{l}\Delta L_{1i}\left(k\right)=L_{1i}\left(k+1\right)-L_{1i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le {\Vert e_i\left(k+1\right)\Vert }^2-{\Vert e_i\left(k\right)\Vert }^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\le {\left(m\Vert e_i\left(k\right)\Vert +m\Vert {ϵ}_i\left(k\right)\Vert \right)}^2-{\Vert e_i\left(k\right)\Vert }^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\le 2m^2{\Vert e_i\left(k\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2-{\Vert e_i\left(k\right)\Vert }^2\end{array}$$

(36)

The difference of $L_{2i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{2i}\left(k\right)=L_{2i}\left(k+1\right)-L_{2i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}={\theta }_i\left(k+1\right)-{\theta }_i\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=-{\lambda }_i{\theta }_i\left(k\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e_i\left(k\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k\right)\Vert }^2\right)\end{array}$$

(37)

Then,

$$\begin{array}{l}\Delta L_i\left(k\right)=\Delta L_{1i}\left(k\right)+\Delta L_{2i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(1-{\xi }_i\right)\left(2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2-\left(1-2m^2\right){\Vert e_i\left(k\right)\Vert }^2\right)-{\lambda }_i{\theta }_i\left(k\right)\end{array}$$

(38)

According to the dynamic event-triggered mechanism, it can be rewritten as:

$$\begin{array}{l}\Delta L_i\left(k\right)=\Delta L_{1i}\left(k\right)+\Delta L_{2i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(1-{\xi }_i\right)\left(2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2-\left(1-2m^2\right){\Vert e_i\left(k\right)\Vert }^2\right)-{\lambda }_i{\theta }_i\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(1-{\xi }_i\right){\rho }_i{\theta }_i\left(k\right)-{\lambda }_i{\theta }_i\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left[\left(1-{\xi }_i\right){\rho }_i-{\lambda }_i\right]{\theta }_i\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}<0\end{array}$$

(39)

The secure consensus for the supply chain system is achieved.

(2)

If DoS attacks occur at sampling time $k$ and do not occur at sampling time $k+1$, it is assumed that ${\alpha }_i\left(k\right)=0$, ${\alpha }_i\left(k+1\right)=1$.

The difference of $L_{1i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{1i}\left(k\right)=e_{di}^{{\rm T}}\left(k+1\right)e_{di}\left(k+1\right)-e_{di}^{{\rm T}}\left(k\right)e_{di}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=e_i^{{\rm T}}\left(k\right)e_i\left(k\right)-e_i^{{\rm T}}\left(k\right)e_i\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=0\end{array}$$

(40)

The difference of $\Delta L_{2i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{2i}\left(k\right)={\theta }_{di}\left(k+1\right)-{\theta }_{di}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}={\theta }_i\left(k\right)-{\theta }_i\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=0\end{array}$$

(41)

The secure consensus for the supply chain systems is achieved.

(3)

If DoS attacks do not occur at sampling time $k$ and occur at sampling time $k+1$, it is assumed that ${\alpha }_i\left(k\right)=1$, ${\alpha }_i\left(k+1\right)=0$. We discuss two situations in the next section.

When the triggering mechanisms are satisfied at $k-1$, the difference of $\Delta L_{1i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{1i}\left(k\right)=e_i^{{\rm T}}\left(k+1\right)e_i^{{\rm T}}\left(k+1\right)-e_i^{{\rm T}}\left(k-1\right)e_i^{{\rm T}}\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le {\left(m\Vert e_i\left(k\right)\Vert +m\Vert {ϵ}_i\left(k\right)\Vert \right)}^2-{\Vert e_i\left(k-1\right)\Vert }^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\le 2m^2{\Vert e_i\left(k\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2-{\Vert e_i\left(k-1\right)\Vert }^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(2m^2-1\right){\Vert e_i\left(k\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2+{\Vert e_i\left(k\right)\Vert }^2-{\Vert e_i\left(k-1\right)\Vert }^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(2m^2-1\right){\Vert e_i\left(k\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2+\left(2m^2-1\right){\Vert e_i\left(k-1\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k-1\right)\Vert }^2\end{array}$$

(42)

The difference of $\Delta L_{2i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{2i}\left(k\right)=L_{2i}\left(k+1\right)-L_{2i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}={\theta }_i\left(k+1\right)-{\theta }_i\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}={\theta }_i\left(k+1\right)-{\theta }_i\left(k\right)+{\theta }_i\left(k\right)-{\theta }_i\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=-{\lambda }_i{\theta }_i\left(k\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e\left(k\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k\right)\Vert }^2\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}-{\lambda }_i{\theta }_i\left(k-1\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e\left(k-1\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k-1\right)\Vert }^2\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=-{\lambda }_i{\theta }_i\left(k\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e_i\left(k\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k\right)\Vert }^2\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}-{\lambda }_i{\theta }_i\left(k-1\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e\left(k-1\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k-1\right)\Vert }^2\right)\end{array}$$

(43)

Therefore, combining (42) and (43) can further give:

$$\begin{array}{l}\Delta L_i\left(k\right)=\Delta L_{1i}\left(k\right)+\Delta L_{2i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(\left(1-{\xi }_i\right){\rho }_i-{\lambda }_i\right){\theta }_i\left(k\right)+\left(\left(1-{\xi }_i\right){\rho }_i-{\lambda }_i\right){\theta }_i\left(k-1\right)\end{array}$$

(44)

According to $0<\left(1-{\xi }_i\right){\rho }_i<{\lambda }_i$ and ${\theta }_i\left(k\right)>0$, ${\theta }_i\left(k-1\right)>0$, one can obtain $\Delta L_i\left(k\right)<0$. The secure consensus for the supply chain system is achieved.

When the triggering mechanisms are dissatisfied at $k-1$, the difference of $\Delta L_{1i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{1i}\left(k\right)=e_i^{{\rm T}}\left(k+1\right)e_i^{{\rm T}}\left(k+1\right)-e_i^{{\rm T}}\left(k-1\right)e_i^{{\rm T}}\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le {\left(m\Vert e_i\left(k\right)\Vert +m\Vert {ϵ}_i\left(k\right)\Vert \right)}^2-{\Vert e_i\left(k-1\right)\Vert }^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\le 2m^2{\Vert e_i\left(k\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2-{\Vert e_i\left(k-1\right)\Vert }^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(2m^2-1\right){\Vert e_i\left(k\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2+{\Vert e_i\left(k\right)\Vert }^2-{\Vert e_i\left(k-1\right)\Vert }^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(2m^2-1\right){\Vert e_i\left(k\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2+\left(2m^2-1\right){\Vert e_i\left(k-1\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k-1\right)\Vert }^2\end{array}$$

(45)

The difference of $\Delta L_{2i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{2i}\left(k\right)=L_{2i}\left(k+1\right)-L_{2i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}={\theta }_i\left(k+1\right)-{\theta }_i\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}={\theta }_i\left(k+1\right)-{\theta }_i\left(k\right)+{\theta }_i\left(k\right)-{\theta }_i\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=-{\lambda }_i{\theta }_i\left(k\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e\left(k\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k\right)\Vert }^2\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}-{\lambda }_i{\theta }_i\left(k-1\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e\left(k-1\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k-1\right)\Vert }^2\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=-{\lambda }_i{\theta }_i\left(k\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e_i\left(k\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k\right)\Vert }^2\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}-{\lambda }_i{\theta }_i\left(k-1\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e\left(k-1\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k-1\right)\Vert }^2\right)\end{array}$$

(46)

It is noted that ${\Vert {ϵ}_i\left(k-1\right)\Vert }^2=0$; thus, the Lyapunov function $\Delta L_i\left(k\right)$ is calculated as:

$$\begin{array}{l}\Delta L_i\left(k\right)=\Delta L_{1i}\left(k\right)+\Delta L_{2i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(\left(1-{\xi }_i\right){\rho }_i-{\lambda }_i\right){\theta }_i\left(k\right)-{\lambda }_i{\theta }_i\left(k-1\right)+\left(1-{\xi }_i\right)\left(2m^2-1\right){\Vert e_i\left(k-1\right)\Vert }^2\end{array}$$

(47)

Since $0<\left(1-{\xi }_i\right){\rho }_i<{\lambda }_i$, ${\lambda }_i>0$, $0<{\xi }_i<1$, $0<m<\frac{\sqrt{2}}{2}$, we have $\Delta L_i\left(k\right)<0$, and the secure consensus control for supply chain systems could be achieved.

(4)

If DoS attacks occur at two continuous sampling times $k$ and $k+1$, it is assumed that ${\alpha }_i\left(k\right)={\alpha }_i\left(k+1\right)=1$. We also discuss two situations in the next section.

When the triggering mechanisms are satisfied at $k-1$, the difference of $\Delta L_{1i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{1i}\left(k\right)=e_i^{{\rm T}}\left(k\right)e_i\left(k\right)-e_i^{{\rm T}}\left(k-1\right)e_i\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(2m^2-1\right){\Vert e_i\left(k-1\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k\right)\Vert }^2\end{array}$$

(48)

The difference of $\Delta L_{2i}\left(k\right)$ can be calculated as follows

$$\begin{array}{l}\Delta L_{2i}\left(k\right)={\theta }_i\left(k\right)-{\theta }_i\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=-{\lambda }_i{\theta }_i\left(k-1\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e_i\left(k-1\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k-1\right)\Vert }^2\right)\end{array}$$

(49)

From (43) and (44), it follows that:

$$\begin{array}{l}\Delta L_i\left(k\right)=\Delta L_{1i}\left(k\right)+\Delta L_{2i}\left(k\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(\left(1-{\xi }_i\right){\rho }_i-{\lambda }_i\right){\theta }_i\left(k-1\right)\end{array}$$

(50)

Based on $0<\left(1-{\xi }_i\right){\rho }_i<{\lambda }_i$ and ${\theta }_i\left(k-1\right)>0$, it yields that $\Delta L_i\left(k\right)<0$. The secure consensus for the supply chain system is achieved. Thus, this completes the proof.

When the triggering mechanisms are dissatisfied at $k-1$, the difference of $\Delta L_{1i}\left(k\right)$ can be calculated as follows.

$$\begin{array}{l}\Delta L_{1i}\left(k\right)=e_i^{{\rm T}}\left(k\right)e_i\left(k\right)-e_i^{{\rm T}}\left(k-1\right)e_i\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\le \left(2m^2-1\right){\Vert e_i\left(k-1\right)\Vert }^2+2m^2{\Vert {ϵ}_i\left(k-1\right)\Vert }^2\end{array}$$

(51)

The difference of $\Delta L_{2i}\left(k\right)$ can be calculated as follows:

$$\begin{array}{l}\Delta L_{2i}\left(k\right)={\theta }_i\left(k\right)-{\theta }_i\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=-{\lambda }_i{\theta }_i\left(k-1\right)+{\xi }_i\left(\left(1-2m^2\right){\Vert e_i\left(k-1\right)\Vert }^2-\left(2m^2\right){\Vert {ϵ}_i\left(k-1\right)\Vert }^2\right)\end{array}$$

(52)

Since ${\Vert {ϵ}_i\left(k-1\right)\Vert }^2=0$, the difference of $\Delta L_i\left(k\right)$ can thus be calculated as follows:

$$\begin{array}{l}\Delta L_i\left(k\right)={\theta }_i\left(k\right)-{\theta }_i\left(k-1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=-{\lambda }_i{\theta }_i\left(k-1\right)-\left(1-{\xi }_i\right)\left(\left(1-2m^2\right){\Vert e_i\left(k-1\right)\Vert }^2\right)\end{array}$$

(53)

Based on ${\theta }_i\left(k-1\right)>0$, it yields that $\Delta L_i\left(k\right)<0$. The secure consensus for the supply chain systems is achieved. Thus, this completes the proof.

3.3. Neural Network-Based Dynamic Event-Triggered Mechanism under DoS Attacks

The RCA structure is shown in Figure 2. It can be seen that all training processes use our methods, and the learning process is introduced in the next section.

3.3.1. Reinforce NN Learning Network Secure Change Design

For subchain $i$, we define the local internal reinforcement approximates as:

$${\widehat{P}}_i\left({\tilde{Z}}_{gi}\left(k\right)\right)={\psi }_{gi}\left(W_{g2i}^{{\rm T}}\cdot {\psi }_{gi}\left(W_{g1i}^{{\rm T}}\cdot {\tilde{Z}}_{gi}\left(k\right)\right)\right)$$

(54)

where ${\tilde{Z}}_{gi}\left(k\right)$ is the reinforced network secure change; it consists of ${\tilde{e}}_{di}\left(k\right)$, $u_i\left(k_s^i\right)$, $u_{-i}\left(k_s^i\right)$, ${\omega }_i\left(k\right)$, and ${\omega }_{-i}\left(k\right)$. $W_{g1i}$ denotes the weights between the input layer and the hidden layer, and $W_{g2i}$ denotes the weights between the hidden layer and the output layer.

When training the reinforced NN, the error function of the reinforced NN is defined as:

$${\tilde{\delta }}_{gi}\left(k\right)=\alpha {\widehat{P}}_i\left({\tilde{Z}}_{gi}\left(k\right)\right)-{\widehat{P}}_i\left({\tilde{Z}}_{gi}\left(k-1\right)\right)+{\tilde{r}}_i\left(k-1\right)$$

(55)

The loss function is to minimize the following objective function:

$${\tilde{E}}_{gi}\left(k\right)=\frac{1}{2}{\tilde{\delta }}_{gi}^{{\rm T}}\left(k\right){\tilde{\delta }}_{gi}\left(k\right)$$

(56)

The weights $W_{g2i}$ updating the rules for subchain $i$ are expressed as:

$$W_{g2i}\left(k+1\right)=W_{g2i}\left(k\right)-{\mu }_{gi}\left(\frac{\partial {\tilde{E}}_{gi}}{\partial W_{g2i}}\right)$$

(57)

Based on the chain backpropagation rules, we derive:

$$\begin{array}{l}{W}_{g2i}\left(k+1\right)=W_{g2i}\left(k\right)-{\mu }_{gi}\left(\frac{\partial {\tilde{E}}_{gi}\left(k\right)}{\partial {\tilde{\delta }}_{gi}\left(k\right)}\frac{\partial {\tilde{\delta }}_{gi}\left(k\right)}{\partial {\widehat{P}}_i\left({\tilde{Z}}_{gi}\left(k\right)\right)}\frac{\partial {\widehat{P}}_i\left({\tilde{Z}}_{gi}\left(k\right)\right)}{\partial W_{g2i}\left(k\right)}\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=W_{g2i}\left(k\right)-\frac{1}{2}{\mu }_{gi}\alpha {\tilde{\delta }}_{gi}\left(k\right)\left(1-{\widehat{P}}_i^2\left({\tilde{Z}}_{gi}\left(k\right)\right)\right){\varphi }_{gi}\left(k\right)\end{array}$$

(58)

where $0<{\mu }_{gi}<1$ is the learning rate of the reinforced NN, ${\varphi }_{gi}\left(k\right)={\psi }_{gi}\left(W_{g1i}^{{\rm T}}\cdot {\tilde{Z}}_{gi}\left(k\right)\right)$.

3.3.2. Critic NN Learning Network Secure Change Design

For subchain $i$, we define the critic network approximates as:

$${\widehat{V}}_i\left({\tilde{Z}}_{ci}\left(k\right)\right)=W_{c2i}^{{\rm T}}{\psi }_{ci}\left(W_{c1i}^{{\rm T}}\cdot {\tilde{Z}}_{ci}\left(k\right)\right)$$

(59)

where ${\tilde{Z}}_{ci}\left(k\right)$ is the critic network secure change; it consists of ${\tilde{e}}_{di}\left(k\right)$, $u_i\left(k_s^i\right)$, $u_{-i}\left(k_s^i\right)$, ${\omega }_i\left(k\right)$, ${\omega }_{-i}\left(k\right)$, and ${\widehat{P}}_i\left({\tilde{Z}}_{gi}\left(k\right)\right)$. $W_{c1i}$ denotes the weights between the input layer and the hidden layer, and $W_{c2i}$ denotes the weights between the hidden layer and the output layer.

The error of the critic network is obtained:

$${\tilde{\delta }}_{gi}\left(k\right)=\alpha {\widehat{P}}_i\left({\tilde{Z}}_{gi}\left(k\right)\right)-{\widehat{P}}_i\left({\tilde{Z}}_{gi}\left(k-1\right)\right)+{\tilde{r}}_i\left(k-1\right)$$

(60)

The objective function of the critic network to be minimized is:

$${\tilde{E}}_{ci}\left(k\right)=\frac{1}{2}{\tilde{\delta }}_{ci}^{{\rm T}}\left(k\right){\tilde{\delta }}_{ci}\left(k\right)$$

(61)

The weights $W_{c2i}$ updating the rules for subchain $i$ are given by:

$$W_{c2i}\left(k+1\right)=W_{c2i}\left(k\right)-{\mu }_{ci}\left(\frac{\partial {\tilde{E}}_{ci}\left(k\right)}{\partial W_{c2i}\left(k\right)}\right)$$

(62)

Based on the chain backpropagation rules, we derive:

$$\begin{array}{l}{W}_{c2i}\left(k+1\right)=W_{c2i}\left(k\right)-{\mu }_{ci}\left(\frac{\partial {\tilde{E}}_{ci}\left(k\right)}{\partial {\tilde{\delta }}_{ci}\left(k\right)}\frac{\partial {\tilde{\delta }}_{ci}\left(k\right)}{\partial {\widehat{V}}_i\left({\tilde{Z}}_{ci}\left(k\right)\right)}\frac{\partial {\widehat{V}}_i\left({\tilde{Z}}_{ci}\left(k\right)\right)}{\partial W_{c2i}\left(k\right)}\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=W_{c2i}\left(k\right)-{\mu }_{ci}\eta {\tilde{\delta }}_{ci}\left(k\right){\psi }_{ci}\left(W_{c1i}^{{\rm T}}\cdot {\tilde{Z}}_{ci}\left(k\right)\right)\end{array}$$

(63)

where $0<{\mu }_{ci}<1$ is the learning rate of the reinforced NN.

3.3.3. Actor NN Learning Network Secure Change Design

For subchain $i$, we define the optimal control input under the dynamic event-triggered mechanism as:

$${\widehat{u}}_i\left(k\right)={\psi }_{ai}\left(W_{a2i}^{{\rm T}}\cdot {\psi }_{ai}\left(W_{a1i}^{{\rm T}}\cdot {\tilde{Z}}_{ai}\left(k\right)\right)\right)$$

(64)

where ${\tilde{Z}}_{ai}\left(k\right)$ is the actor network secure change for optimal control input; it consists of $e_{di}\left(k_s^i\right)$. $W_{a1i}$ denotes the weights between the input layer and the hidden layer, and $W_{a2i}$ denotes the weights between the hidden layer and the output layer.

The error of the actor network secure change for optimal control input is obtained:

$${\delta }_{ai}\left(k\right)={\widehat{V}}_i\left({\tilde{Z}}_{ci}\left(k\right)\right)-U_c$$

(65)

The objective function of the actor network to be minimized is:

$${\tilde{E}}_{ai}\left(k\right)=\frac{1}{2}{\tilde{\delta }}_{ai}^{{\rm T}}\left(k\right){\tilde{\delta }}_{ai}\left(k\right)$$

(66)

The weights $W_{c2i}$ updating the rules for subchain $i$ are expressed as:

$$W_{a2i}\left(k+1\right)=W_{a2i}\left(k\right)-{\mu }_{ai}\left(\frac{\partial {\tilde{E}}_{ai}\left(k\right)}{\partial W_{a2i}\left(k\right)}\right)$$

(67)

Based on the chain backpropagation rules, we derive:

$$\begin{array}{l}{W}_{a2i}\left(k+1\right)=W_{a2i}\left(k\right)-{\mu }_{ai}\left(\frac{\partial {\tilde{E}}_{ai}\left(k\right)}{\partial {\widehat{V}}_i\left({\tilde{Z}}_{ci}\left(k\right)\right)}\frac{\partial {\widehat{V}}_i\left({\tilde{Z}}_{ci}\left(k\right)\right)}{\partial {\widehat{u}}_i\left(k\right)}\frac{\partial {\widehat{u}}_i\left(k\right)}{\partial W_{a2i}\left(k\right)}\right)\\ =W_{a2i}\left(k\right)-\frac{1}{4}{\mu }_{ai}{\varphi }_{ai}\left(k\right)W_{c2i}^{{\rm T}}\left(k\right)C_1\left(k\right)\left(1-{\widehat{u}}_i^2\left(k\right)\right)\left[W_{c2i}^{{\rm T}}\left(k\right){\psi }_{ci}\left(W_{c1i}^{{\rm T}}\cdot {\tilde{Z}}_{ci}\left(k\right)\right)\right]\end{array}$$

(68)

where $0<{\mu }_{ai}<1$ is the learning rate of the actor network secure change for optimal control input, ${\varphi }_{ai}\left(k\right)={\psi }_{ai}\left(W_{a1i}^{{\rm T}}\cdot {\tilde{Z}}_{ai}\left(k\right)\right)$, $C_1\left(k\right)=\partial {\psi }_{ci}\left(W_{c1i}^{{\rm T}}\cdot {\tilde{Z}}_{ci}\left(k\right)\right)/\partial {\widehat{u}}_i\left(k\right)$.

For subchain $i$, we define the worst-case disturbance input under the traditional time-triggered mechanism as:

$${\widehat{\omega }}_i\left(k\right)={\psi }_{ai}\left(W_{d2i}^{{\rm T}}\cdot {\psi }_{ai}\left(W_{d1i}^{{\rm T}}\cdot {\tilde{Z}}_{di}\left(k\right)\right)\right)$$

(69)

where ${\tilde{Z}}_{di}\left(k\right)$ is the actor network secure change for the worst-case disturbance input; it consists of ${\tilde{e}}_{di}\left(k\right)$. $W_{d1i}$ denotes the weights between the input layer and the hidden layer, and $W_{d2i}$ denotes the weights between the hidden layer and the output layer.

The error function of the actor network secure change for the worst-case disturbance input and objective function are the same results as in (57) and (58).

The weights $W_{d2i}$ updating the rules for subchain $i$ are expressed as:

$$W_{d2i}\left(k+1\right)=W_{d2i}\left(k\right)-{\mu }_{di}\left(\frac{\partial {\tilde{E}}_{di}\left(k\right)}{\partial W_{d2i}\left(k\right)}\right)$$

(70)

Based on the chain backpropagation rules, we derive:

$$\begin{array}{l}{W}_{d2i}\left(k+1\right)=W_{d2i}\left(k\right)-{\mu }_{di}\left(\frac{\partial {\tilde{E}}_{ai}\left(k\right)}{\partial {\widehat{V}}_i\left({\tilde{Z}}_{ci}\left(k\right)\right)}\frac{\partial {\widehat{V}}_i\left({\tilde{Z}}_{ci}\left(k\right)\right)}{\partial {\widehat{\omega }}_i\left(k\right)}\frac{\partial {\widehat{\omega }}_i\left(k\right)}{\partial W_{d2i}\left(k\right)}\right)\\ =W_{d2i}\left(k\right)-\frac{1}{4}{\mu }_{di}{\varphi }_{di}\left(k\right)W_{c2i}^{{\rm T}}\left(k\right)C_2\left(k\right)\left(1-{\widehat{\omega }}_i^2\left(k\right)\right)\left[W_{c2i}^{{\rm T}}\left(k\right){\psi }_{ci}\left(W_{c1i}^{{\rm T}}\cdot {\tilde{Z}}_{ci}\left(k\right)\right)\right]\end{array}$$

(71)

where $0<{\mu }_{ai}<1$ is the learning rate of the actor network secure change for the worst-case disturbance input, ${\varphi }_{di}\left(k\right)={\psi }_{di}\left(W_{d1i}^{{\rm T}}\cdot {\tilde{Z}}_{di}\left(k\right)\right)$, $C_2\left(k\right)=\partial {\psi }_{ci}\left(W_{c1i}^{{\rm T}}\cdot {\tilde{Z}}_{ci}\left(k\right)\right)/\partial {\widehat{\omega }}_i\left(k\right)$.

4. Simulation

In this section, we consider a supply chain system to testify to the validity of the proposed results. We consider a supply chain system with four subchains and one chain leader. The system matrices are $A=\left[\begin{array}{cc}0.7& 0\\ 0& 0.8\end{array}\right]$, $B=\left[\begin{array}{cc}1& -1\\ 0& 1\end{array}\right]$, and $D=\left[\begin{array}{c}0\\ -1\end{array}\right]$. The topology of the communication network is illustrated in Figure 3. The four subchains are denoted 1 2 3 4 and one chain leader is 0.

We can obtain the edge matrix with $a_{21}=a_{31}=a_{42}=1$ and the pinning gain with $b_1=b_2=1$, $b_3=b_4=0$. The weight matrices are selected as $Q_{ii}=I_{2\times 2}$, $R_{21}=R_{31}=R_{42}=1$, $R_{11}=R_{22}=R_{33}=R_{44}=1$. We set the initial production inventory status as $x_0\left(0\right)={\left[1.5,1.5\right]}^{{\rm T}}$, $x_1\left(0\right)={\left[1,1\right]}^{{\rm T}}$, $x_2\left(0\right)={\left[1,1.2\right]}^{{\rm T}}$, $x_3\left(0\right)={\left[0.5,1.2\right]}^{{\rm T}}$, and $x_4\left(0\right)={\left[0.8,0.9\right]}^{{\rm T}}$. The initial productions are chosen as $u_1\left(0\right)={\left[0.3,0.1\right]}^{{\rm T}}$, $u_2\left(0\right)={\left[0.5,0.2\right]}^{{\rm T}}$, $u_3\left(0\right)={\left[0.1,0.3\right]}^{{\rm T}}$, and $u_4\left(0\right)={\left[0.2,0.4\right]}^{{\rm T}}$. The initial demand market is chosen as ${\omega }_i\left(0\right)=0.1$. The attenuation level of the bullwhip effect $\gamma$ is chosen as $\gamma =1$ and the constant demand market $d=0.1$.

In the training process, we select the discount factors $\alpha =\eta =0.9$ and the learning rates ${\mu }_{gi}={\mu }_{ci}={\mu }_{ai}={\mu }_{di}=0.04$. Next, we choose dynamic event-triggered parameters $m^2=0.1$, ${\lambda }_i=0.2$, ${\rho }_i=0.1$, and ${\xi }_i=0.3$ and the initial internal dynamic variable ${\theta }_1\left(0\right)=1$, ${\theta }_2\left(0\right)=2$, ${\theta }_3\left(0\right)=3$, and ${\theta }_4\left(0\right)=4$. The parameters of DoS attacks are chosen as ${\beta }_i={\beta }_2={\beta }_3={\beta }_4=0.5$. The initial weights are selected randomly in $\left(0,1\right)$.

The results are shown in Figure 4 and Figure 5. The production inventory status of the four subchains and chain leaders under the proposed secure change control scheme can be seen.

The weight curves of the reinforced neural network, the critic neural network, and the actor neural network are shown in Figure 6, Figure 7, Figure 8 and Figure 9. It can be observed that the weights are convergent at $k=5$ a day. Due to the supply chain system still operating using the RCA structure under DoS attacks, the subchains cannot track the supply leaders. Thus, in Figure 4 and Figure 5, it can be seen that the production inventory status of all subchains and supply leaders demonstrates that synchronization is achieved around $k=60$ a day under DoS attacks.

Figure 10 and Figure 11 depict that the inventory status errors eventually converge to zero around $k=60$ a day. It can be seen that the internal dynamic variable ${\theta }_i$ is always positive and convergent in Figure 12. The trigger instants of the four subchains are shown in Figure 13. The trajectories of triggering errors ${\Vert {\epsilon }_i\left(k\right)\Vert }^2$ along with the dynamic event-triggering threshold are shown in Figure 14. The instants under DoS attacks can be seen in Figure 15a,b.

5. Conclusions

In this paper, supply chain systems are provided by a new data-driven method based on the established RCA structure. The problems of unknown demand and the dynamic model can be solved by this method under Dos attacks. The secure change control problem for supply chain systems under DoS attacks is solved under the dynamic event-triggered mechanism. The proposed method requires no system model information, only the inventory status, production input, and disturbance input. Firstly, to alleviate the influence of DoS attacks, a structure of secure change control for supply chain systems is designed. A secure change mechanism is used to store the latest received data packets based on the structure. Then, a dynamic event-triggered mechanism is proposed for supply chain systems using RL. In addition, an RCA structure of secure change control for supply chain systems is provided. The dynamic event-triggered mechanism is applied to reduce the number of production input updates. The stability proof is provided by using the Lyapunov function under DoS attacks. Finally, the simulation results verify that the subchains can be tracked by the chain leaders using an RCA structure under DoS attacks. The weight curves of the network are eventually convergent.

It is worth noting that the proposed method can be applied not only to linear supply chain systems to achieve secure change tracking control but also to non-linear supply chain systems. However, considering that actual supply chain systems’ DoS attacks are usually aperiodic and unpredictable, supply chain systems’ aperiodic denial of service attacks under network attack events may be developed and studied in future work.