An Anti-Jamming Hierarchical Optimization Approach in Relay Communication System via Stackelberg Game

Abstract

In this paper, we study joint relay selection and the power control optimization problem in an anti-jamming relay communication system. Considering the hierarchical competitive relationship between a user and jammer, we formulate the anti-jamming problem as a Stackelberg game. From the perspective of game, the user selects relay and power strategy firstly which acts as the leader, while the jammer chooses power strategy then that acts as follower. Moreover, we prove the existence of Stackelberg equilibrium. Based on the Q-learning algorithm and multi-armed bandit method, a hierarchical joint optimization algorithm is proposed. Simulation results show the user’s strategy selection probability and the jammer’s regret. We compare the user’s and jammer’s utility under the proposed algorithm with a random selection algorithm to verify the algorithm’s superiority. Moreover, the influence of feedback error and eavesdropping error on utility is analyzed.

1. Introduction

The anti-jamming problem has always been a hot topic in wireless communication domain. As a major threat to reliable communication, the jammer can launch noise-like signals to legitimate receiver to decrease communication quality. In order to cope with malicious jamming attacks, a lot of anti-jamming works have been studied that focused on power control [1,2,3,4,5,6,7,8,9,10], frequency selection [11,12,13,14,15,16,17,18,19,20], and spatial domain [21,22,23,24,25], and many effective anti-jamming technologies were proposed. In this paper, we investigate the anti-jamming hierarchical optimization problem in relay communication system.

The power adjustment is considered as an effective method to respond to the jammer’s attacks directly. In [1], the authors made receivers more robust with adaptive arrays utilizing power inversion algorithm in jamming environment. In [2], based on the competitive relationship between user and jammer, a Stackelberg game was proposed to model the anti-jamming communication hierarchical optimization problem in the presence of smart jammer. Considering it is hard for the user and jammer to obtain the accurate power and channel state information, the authors analyzed the anti-jamming performance with observation error in [3,4,5]. In [6], the authors studied the communication problem between in WBANs via Stackelberg frame facing jamming attacks. In [7], the authors investigated cross-layer anti-jamming joint optimization problem based on Q-learning algorithm. In [8], the authors investigated discrete power strategy optimization problem with imperfect information. In [9], the power allocation problem was studied in cognitive radio communication networks within different game frames. Taking the perspective of Stackelberg game, the authors combed the anti-jamming technologies under different scenarios in [10], moreover, an anti-jamming decision-making framework was also proposed based on the adversarial characteristics between user and jammer, incomplete information and so on.

In the domain of relay communication [26,27,28,29,30,31,32,33,34,35,36,37,38,39], the anti-jamming problem has also been widely investigated. In [27], the authors studied communication frequency selection optimization problem in unpredictable jamming environment. In [28], the authors formulated the relay selection and discrete power optimization problem under power constraint as a potential game, in which there existed more than one pure strategy Nash equilibrium (NE) and the optimal strategy profile was obtained. In [29], the cooperative relaying technology was investigated in vehicular networks. In [30], the authors studied the ongoing power control optimization problem via Stackelberg game and obtained the closed-form solutions of optimal power strategy. In [31], the authors decreased algorithm complexity and obtained the optimal joint strategy in a distributed wireless communication networks. In [32], the authors focused on game-theoretical approaches in cognitive communication and networking systems, in which a stochastic anti-jamming game was proposed to design the optimal adaptive defense strategies against cognitive malicious attackers. In [33], the authors designed an opportunistic wireless communication setting and modelled the asset selling problem as a game theoretic variant in the completely observable and the partially observable cases respectively.

With the increasing complexity of communication scenarios and jamming modes, in [34,35], the authors studied the optimal power allocation problem in friendly jamming and Gaussian noise jamming environments to increase secrecy sum rate in untrusted relay communication networks. In [36], the authors studied anti-jamming power optimization problem in multi-user relay communication system, a three-layer Stackelberg game was proposed and Stackelberg equilibrium (SE) was also obtained based on duality optimization theory. In [37], the authors studied the problem of resource assigning in a single carrier communication system in the presence of a jammer, a Bayesian jamming game framework was proposed and the Nash strategy was also compared to the Stackelberg strategy to verify its sensibility. In [38], the authors studied a new anti-jamming problem of unknown nodes in a peer-to-peer network attacked by a random jammer or an intelligent one. In [39], considering the idealized case and potential energy constraint, the authors investigated the jamming attacks optimization problem that the jammer could control the probability of jamming and transmission range to cause maximal damage to wireless network. The existing works [34,35,36,37,38,39] investigated the anti-jamming problems from the perspectives of adaptive defense strategies, potential energy constraint, jamming modes, ideal and non-ideal cases in different communication scenarios to perfect the study. Game theoretical approaches, such as stochastic game and Stackelberg game, are just methods which can help us analyze the anti-jamming problems effectively.

The main difference between this paper and the above anti-jamming works [29,30,31,32,33,34,35,36,37,38,39] is that, in the presence of an intelligent jammer, we investigate the relay selection problem under a power–energy constraint with imperfect information. Moreover, considering the existing power optimization methods in anti-jamming domain, there are still some problems needing to be solved. Firstly, it is hard for the user and jammer to get the math utilities and channel state information. Secondly, there exists error when a user measures the communication quality or a jammer measures the jamming effect. Thirdly, the optimizations of power strategy and relay selection need to be considered simultaneously.

In this paper, we investigate the joint relay selection and power control optimization problem in an anti-jamming relay communication scenario. Considering the competitive relationship between the user and jammer, we formulate the anti-jamming problem as a Stackelberg game, which is an effective tool to model the hierarchical optimization problem. In the game frame, the user acts as leader to select relay and power strategy firstly, and the jammer acts as follower then to choose power strategy. Moreover, the proof of existence of SE is given. Based on Q-learning algorithm and multi-armed bandit (MAB) method, a hierarchical joint optimization algorithm is proposed. Simulation results show the user’s strategy selection probability and the jammer’s regret. We give utility comparison under the proposed algorithm and random selection algorithm with feedback error and eavesdropping error. In the last, we summarize the main contributions of this paper in the following:

• In relay communication scenario, considering the competitive relationship between the user and jammer, we formulate the anti-jamming joint optimization problem as a Stackelberg game, in which the user acts as leader and jammer acts as follower.
• We prove the existence of SE and propose a hierarchical joint optimization algorithm based on Q-learning and MAB method via Stackelberg game frame.
• Simulation results show the user’s strategy selection probability and jammer’s regret. Utility under the proposed algorithm is compared with random selection algorithm to verify the algorithm’s superiority. Moreover, the influence of feedback error and eavesdropping error on utility is analyzed.

The main differences from our previous work [36] are summarized as: (i) There exist multiple relays in the anti-jamming system which needs to realize the optimization of relay selection and power control simultaneously. (ii) The feedback error and eavesdropping error are introduced because it is hard for the the user and jammer to get accurate feedback information. (iii) The user and jammer do not need to know the power and channel fading information which is ignored in [36].

In the rest of this paper, we establish the anti-jamming model in the proposed relay communication scenario, and give the relative problem formulations in Section 2. In Section 3, we formulate the anti-jamming joint optimization problem as a Stackelberg game and prove the existence of SE. Moreover, a hierarchical joint optimization algorithm is proposed. Simulation results show the user’s strategy selection probability and the jammer’s regret in Section 4, besides, utility comparison under different algorithms with feedback error and eavesdropping error is also given. Finally, we draw a conclusion in Section 5.

2. System Model and Problem Formulation

2.1. System Model

We assume an anti-jamming relay communication scheme consisted of a base station (BS), a user, a jammer and a relay group, as shown in Figure 1. The user collects nearby information and sends it to the BS. Considering the serious channel fading of wireless communication, the user cannot transmit messages to BS directly, which communicates with BS through a relay group. In phase 1, the user transmits messages to the relay group that contains its collected information and instruction which relay is acquired to assist the user to forward messages to BS. In phase 2, the relay adopts apply and forward (AF) mode, and retransmits the messages received from the user to BS. The user and relay use the same channel. The jammer sends malicious noise-like signals to decrease communication quality. Moreover, both the user and jammer have ability to change power to achieve a better communication or jamming effect. Note that for simplicity, we consider relay’s power remains unchanged, because relay as the fixed station on the ground, the power constraint is not obvious compared to the user.

We consider the user and jammer update their respective strategies with different time scales. On the one hand, jammer updates its power strategy every epoch. On the other hand, in order to improve anti-jamming ability, the user updates its joint relay and power strategy quicker than jammer to adjust timely and obtain the optimal strategy. As shown in Figure 2, each epoch is divided as T time slots, and at the end of each time slot, BS feeds the signal-on-interference-plus-noise ratio (SINR) information of current time slot back to the user as the communication reward, which is also eavesdropped by jammer [40,41,42] to measure the jamming effect at the current epoch.

In the formulated model, we assume that the user has M power levels and its power set is $\mathcal{P}=\left[P_1,P_2,\dots ,P_M\right]$. Jammer has L power levels and its power set is $\mathcal{J}=\left[J_1,J_2,\dots ,J_L\right]$. The relay group contains N relays and the relay set is $\mathcal{R}=\left[R_1,R_2,\dots ,R_N\right]$. The $n\mathrm{th}$ relay’s power is a fixed value and defined as $Q_n$, and all relays’ power set is $\mathcal{Q}=\left[Q_1,Q_2,\dots ,Q_N\right]$. The distance between the user, jammer, BS and the $n\mathrm{th}$ relay are $d_{u,r_n},d_{j,r_n},d_{B,r_n}$, respectively. The distance between BS and jammer is $d_{B,j}$. For convenience, some used notations are listed in

Table 1. Summation of used notation.

$\mathcal{P},\mathcal{J}$	Power set of the user and jammer
$\mathcal{R},\mathcal{Q}$	Relay set and all relays’ power set
$P,J$	Transmitting power of the user and jammer, i.e., $P\in \mathcal{P},J\in \mathcal{J}$
${\alpha }_{u,r_n},{\beta }_{j,r_n}$	The channel gain between the user, jammer and the $n\mathrm{th}$ relay
${\alpha }_{B,r_n},{\beta }_{B,j}$	The channel gain between the $n\mathrm{th}$ relay, jammer and BS
$N_{r_n},N_B$	The back noise power at the $n\mathrm{th}$ relay and BS
$G_n$	The $n\mathrm{th}$ relay’s amplification factor
$X_u,X_j,X_{r_n}$	The user’s, jammer’s and $n\mathrm{th}$ relay’s transmitted signal
$Y_{r_n},Y_B$	The $n\mathrm{th}$ relay’s and BS’s received signal
$C_u$, $C_j$	The transmitting cost of the user and jammer
${\epsilon }_1,{\epsilon }_2$	Feedback error and eavesdropping error
$\gamma$	The SINR received at BS
${\gamma }^{\prime },{\gamma }^{″}$	The feedback SINR received by the user and eavesdropped by jammer

.

2.2. Problem Formulation

In the anti-jamming system, the communication process is divided into two phases. In phase 1, the user transmits messages and selects one relay to help forward messages to BS. When the jammer senses communication signals between the user and relay, it releases jamming signals immediately to destroy normal communication. Let P and J denote the user’s and jammer’s power, where $P\in \mathcal{P}$ and $J\in \mathcal{J}$. Inspired by [43], we define ${\alpha }_{u,r_n}=d_{u,r_n}^{-\delta }$ and ${\beta }_{j,r_n}=d_{j,r_n}^{-\delta }$ as the channel gain between the user, jammer and the $n\mathrm{th}$ relay, respectively, among which $\delta$ is the path-loss factor. The backward noise signal at the $n\mathrm{th}$ relay is $m_{r_n}$. $X_u$ and $X_j$ denote the user’s and jammer’s transmitted signal, respectively. $Y_{r_n}$ denotes the $n\mathrm{th}$ relay’s received signal from the user and jammer, which is defined as follows:

$$\begin{array}{c}\hfill Y_{r_n}=\sqrt{P}{\alpha }_{u,r_n}{X}_u+\sqrt{J}{\beta }_{j,r_n}{X}_j+m_{r_n}.\end{array}$$

(1)

In phase 2, the relay adopts amplifies and forwards (AF) mode. Considering the $n\mathrm{th}$ relay is selected to help the user forward messages to BS, its amplification factor is expressed as follows:

$$\begin{array}{c}\hfill G_n=\sqrt{\frac{Q_n}{P{\alpha }_{u,r_n}^2+J{\beta }_{j,r_n}^2+N_{r_n}}}.\end{array}$$

(2)

Let $X_{r_n}$ denote the $n\mathrm{th}$ relay’s retransmitted signal. We define the received signal at BS as $Y_B$, and it is expressed as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Y_B& ={\sqrt{Q}}_n{\alpha }_{B,r_n}{X}_{r_n}+\sqrt{J}{\beta }_{B,j}{X}_j+m_B\hfill \\ & =G_n\sqrt{P}{\alpha }_{B,r_n}{\alpha }_{u,r_n}{X}_u+G_n\sqrt{J}{\alpha }_{B,r_n}{\beta }_{j,r_n}{X}_j+G_n{\alpha }_{B,r_n}{m}_{r_n}+\sqrt{J}{\beta }_{B,j}{X}_j+m_B,\hfill \end{array}\end{array}$$

(3)

where ${\alpha }_{B,r_n}=d_{B,r_n}^{-\delta }$ and ${\beta }_{B,j}=d_{B,j}^{-\delta }$ denote the channel gain between the $n\mathrm{th}$ relay, jammer and BS, respectively. $m_B$ is the backward noise signal at BS.

Thus the SINR received at BS can be defined as follows:

$$\begin{array}{c}\hfill \gamma =\frac{G_n^{2}P{\alpha }_{B,r_n}^2{\alpha }_{u,r_n}^2}{G_n^{2}J{\alpha }_{B,r_n}^2{\beta }_{j,r_n}^2+G_n^2{\alpha }_{B,r_n}^2{N}_{r_n}+J{\beta }_{B,j}^2+N_B}.\end{array}$$

(4)

For the user, it obtains feedback SINR ${\gamma }^{\prime }$ at the end of each time slot. Considering there may exists feedback information error, we introduce feedback error ${\epsilon }_1=\left|\gamma -{\gamma }^{\prime }\right|/\phantom{\left|\gamma -{\gamma }^{\prime }\right|\gamma }\gamma$, which means the deviation degree of feedback SINR ${\gamma }^{\prime }$ with the actual SINR $\gamma$. Inspired by [2,3,4], considering the transmitting cost, we give the user’s utility function based on the received SINR ${\gamma }^{\prime }$ in the following:

$$\begin{array}{c}\hfill U={\gamma }^{\prime }-C_{u}P.\end{array}$$

(5)

For jammer, it eavesdrops the feedback SINR ${\gamma }^{\prime }$ from BS to the user at the end of each time slot and obtains eavesdropping result ${\gamma }^{″}$. Similarly, considering that it is hard for a jammer to eavesdrop ${\gamma }^{\prime }$ accurately, we introduce eavesdropping error ${\epsilon }_2=\left|\gamma -{\gamma }^{″}\right|/\phantom{\left|\gamma -{\gamma }^{″}\right|\gamma }\gamma$, which means the deviation degree of eavesdropped SINR ${\gamma }^{″}$ with the actual SINR $\gamma$. Similarly, considering the jamming cost, we give the jammer’s utility function based on the eavesdropped result ${\gamma }^{″}$ in the following:

$$\begin{array}{c}\hfill V=-{\gamma }^{″}-C_{j}J.\end{array}$$

(6)

3. The Joint Relay Selection and Power Control Optimization Method via Stackelberg Game

In this section, we formulate the anti-jamming joint optimization problem as a Stackelberg game firstly, which is an effective theoretical method to deal with the hierarchical confrontation relationship between the user and jammer. Then we prove the existence of Stackelberg equilibrium (SE). We propose a hierarchical joint optimization algorithm under the game frame in the last.

3.1. Stackelberg Game Model

In the anti-jamming relay communication, the user transmits messages firstly and selects a relay to help broadcast messages to base station, after having sensed the user’s signals, the jammer releases jamming signals immediately. Take a perspective of Stackelberg game, the user acts as leader and jammer acts as follower in the proposed scenario. The proposed game can be denoted as $\mathcal{G}=\left\{\mathcal{P},\mathcal{Q},\mathcal{J},U,V\right\}$, where $\mathcal{P}$ and $\mathcal{Q}$ denote the user’s power strategy space and relay selection strategy space respectively, $\mathcal{J}$ denotes jammer’s power strategy space, U and V denote the user’s and jammer’s utility, respectively.

Based on the utilities given in Section 2.2, both the user and jammer aim to maximize their utilities to get a better communication effect or jamming effect. For the jammer, given a user’s joint strategy $P\in \mathcal{P},Q_n\in \mathcal{Q}$, it makes the optimal power strategy to destroy normal communication, and the optimization problem is expressed as:

$$\begin{array}{c}\hfill {\displaystyle \underset{J}{max}}V(P,Q_n,J).\end{array}$$

(7)

For the user, it makes a joint relay selection and power control strategy to guarantee anti-jamming communication quality, and the user’s optimization problem can be expressed as follows:

$$\begin{array}{c}\hfill {\displaystyle \underset{P,Q_n}{max}}U(P,Q_n,J).\end{array}$$

(8)

Based on the analysis above, the joint optimization problem can be solved by hierarchical decision-making method via Stackelberg game frame, which is shown as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\displaystyle \underset{P,Q_n}{max}}U(P,Q_n,J)\hfill \\ \mathrm{subject}to:P\in \mathcal{P},Q_n\in \mathcal{Q}\hfill \\ Theoptimalsolution:\left(J^{*}\right)\hfill \\ \left\{\begin{array}{c}{\displaystyle \underset{J}{max}}V(P,Q_n,J)\hfill \\ \mathrm{subject}to:J\in \mathcal{J}\hfill \end{array}\right.\hfill \end{array}\right.\end{array}$$

(9)

3.2. Existence of Stackelberg Equilibrium

In the proposed scenario, we consider that the user adopts mixed strategy to fool the jammer due to the randomness of strategy, which can increase anti-jamming performance effectively. Let $\mathrm{q}$ denotes the user’s mixed strategy, i.e., the probability distribution of the user’s optional relay selections and power strategies. Motivated by [4,8], we define the Stackelberg equilibrium (SE) and give the proof of existence of SE in the following.

Definition 1.

If no player can improve the utility by deviating its optimal strategy unilaterally, the policy profile $(q^{*},J^{*})$ constitutes the SE, which satisfies the following conditions:

$$\begin{array}{c}\hfill U({\mathrm{q}}^{*},J^{*})\ge U(\mathrm{q},J^{*}),\end{array}$$

(10)

$$\begin{array}{c}\hfill V({\mathrm{q}}^{*},J^{*})\ge V({\mathrm{q}}^{*},J).\end{array}$$

(11)

Lemma 1.

There exists a user’s stationary strategy and a smart jammer’s stationary strategy, which constitute a SE [8].

Proof. 

Inspired by [44,45], every finite strategy game has a mixed strategy equilibrium [46], which means there exists a SE in the formulated game.

For the jammer, it aims to maximize the utility and makes strategy based on best-response:

$$\begin{array}{c}\hfill J^{*}={\displaystyle \underset{J}{argmax}}V(\mathrm{q},J).\end{array}$$

(12)

Having known the jammer’s optimal strategy, the user’s optimal strategy can be obtained as follows:

$$\begin{array}{c}\hfill {\mathrm{q}}^{*}={\displaystyle \underset{\mathrm{q}}{argmax}}U(\mathrm{q},J^{*}\left(\mathrm{q}\right)).\end{array}$$

(13)

Based on the analysis above, the policy profile $({\mathrm{q}}^{*},{\mathrm{J}}^{*})$ constitutes the SE. □

3.3. Hierarchical Joint Optimization Algorithm

In this section, we propose a hierarchical joint optimization algorithm, which realizes the user’s strategy optimization based on Q-learning algorithm [47] and jammer’s strategy optimization based on multi armed bandit (MAB) method [48,49,50], respectively.

For the user, its mixed power and relay selection strategy at the $t\mathrm{th}$ time slot is expressed as $\mathrm{q}\left(t\right)=\left\{q_{1,1}\left(t\right),q_{1,2}\left(t\right),\dots ,q_{1,N}\left(t\right),\dots ,q_{m,n}\left(t\right),\dots ,q_{M,1}\left(t\right),q_{M,2}\left(t\right),\dots ,q_{M,N}\left(t\right)\right\}$, where $q_{m,n}\left(t\right)$ denotes the probability to select the $n\mathrm{th}$ relay and power $P_m$ at the $t\mathrm{th}$ time slot. Then we update the user’s Q value as follows:

$$\begin{array}{c}\hfill Q_{m,n}(t+1)=(1-{\kappa }^t)Q_{m,n}\left(t\right)+{\kappa }^{t}r\left(t\right),\end{array}$$

(14)

where ${\kappa }^t$ denotes the user’s learning rate at the $t\mathrm{th}$ time slot, which satisfies ${\sum }_{t=0}^{\infty }{\kappa }^t=\infty ,{\sum }_{t=0}^{\infty }{\left({\kappa }^t\right)}^2<\infty$. $r\left(t\right)$ denotes the user’s reward, i.e., its utility U at the $t\mathrm{th}$ time slot. Based on the analysis above, the user’s mixed strategy is updated as follows:

$$\begin{array}{c}\hfill q_{m,n}(t+1)=\frac{exp\left[q_{m,n}\left(t\right)/\phantom{q_{m,n}\left(t\right){\tau }_0}{\tau }_0\right]}{{\displaystyle \sum _{1\le m\le M,1\le n\le N}}exp\left[q_{m,n}\left(t\right)/\phantom{q_{m,n}\left(t\right){\tau }_0}{\tau }_0\right]},\end{array}$$

(15)

where ${\tau }_0$ controls the tradeoff of exploration-exploitation.

For the jammer, we formulate the finite strategy optimization as a MAB problem, and consider each power strategy as a arm to select, i.e., the jamming power strategy $J_l$ is considered as the $l\mathrm{th}$ arm. $J\left(k\right)$ denotes the jammer’s power strategy at the $k\mathrm{th}$ epoch. Thus the times of power $J_l$ has been selected in the past K epochs is defined as:

$$\begin{array}{c}\hfill A_l\left(K\right)={\displaystyle \sum _{k=1}^K}\delta (J_l,J\left(k\right)),\end{array}$$

(16)

where $\delta (x,y)$ is the Kronecker delta function and it can be expressed as:

$$\delta (x,y)=\left\{\begin{array}{cc}1\hfill & x=y\hfill \\ 0\hfill & x\ne y\hfill \end{array}\right.$$

(17)

Thus, the jammer’s $l\mathrm{th}$ arm’s statistical average reward ${\mu }_l$ in the past K epochs is defined as:

$${\mu }_l=\frac{{\displaystyle \sum _{k=1}^K}V\left(k\right)\delta (J_l,J\left(k\right))}{A_l\left(K\right)}.$$

(18)

$B\left(K\right)$ denotes the jammer’s total utility in the past K epochs and is expressed as:

$$B\left(K\right)={\displaystyle \sum _{k=1}^K}V\left(k\right).$$

(19)

Inspired by [48,49,50], we define jammer’s regret in the past K epochs, which is an important index to represent the loss of utility so far because of the failure to select the optimal jamming power strategy, as shown in the following:

$$\begin{array}{cc}\hfill E\left[R\left(K\right)\right]& =E\left[B^{*}\left(K\right)\right]-E\left[B\left(K\right)\right]\hfill \\ & ={\displaystyle \sum _{l\ne l^{*}}}(V_{l^{*}}-V_l)E\left[A_l\left(K\right)\right],\hfill \end{array}$$

(20)

where $l^{*}$ denotes the estimated optimal arm of jammer based on the historical statistical average reward. $B^{*}\left(K\right)$ denotes jammer’s accumulative utility it could get if jammer had always chosen $l^{*}$ in the past K epochs.

In the MAB problem, UCB1 [48,49,50] is an effective policy to realize the optimization of power strategy. Adopting the UCB1 method, jammer updates its strategy in the $(K+1)\mathrm{th}$ epoch based on the following condition:

$$J(K+1)={\displaystyle \underset{l\in \left[1,2,\dots ,L\right]}{argmax}}\left({\mu }_l+\sqrt{\frac{2lnK}{A_l\left(K\right)}}\right).$$

(21)

According to the analysis above, based on the Q-learning algorithm and UCB1 method, both the user’s and jammer’s optimal strategy can be obtained through hierarchical joint optimization algorithm, which is shown in Algorithm 1.

Algorithm 1: Hierarchical joint optimization algorithm (HJOA).

Initialization:

The number T of time slots in one epoch, the number $k_{max}$ of all epochs.

User selects power P randomly, $t=1$, $k=0$.

Outer Iteration:

1. Jammer selects the optimal arm $J(k+1)$ according to Equation (21).

Inner Iteration:

(1) In the $t\mathrm{th}$ time slot, the user selects its joint relay and power strategy according to $\mathrm{q}\left(t\right)$.

(2) Obtain the user’s utility $U\left(t\right)$ through the feedback of SINR at the end of current time slot according to Equation (5).

(3) Update the user’s Q values according to Equation (14).

(4) Update the user’s mixed strategy $\mathrm{q}\left(t\right)$ according to Equation (15).

(5) Time slot $t=t+1$.

(6) Return to (1) until $t>T$.

End Inner Iteration.

2. Obtain jammer’s utility $V\left(k\right)$ through eavesdropping the feedback of SINR in the $k\mathrm{th}$ epoch according to Equation (6).

3. Update $A_l\left(k\right),{\mu }_l$ and $B\left(k\right)$ in turn.

4. Epoch $k=k+1$, $t=1$.

5. Return to 1 until $k>k_{max}$.

End Outer Iteration.

4. Simulation Results and Discussions

In this section, the simulation parameters and system location setting are given, and then we present some necessary simulation results and give brief discussions. In Section 4.1, we give the user’s power strategy probability and relay selection probability respectively. Moreover, the jammer’s regret is also given. In Section 4.2, we analyze the influence of feedback error and eavesdropping error, and we also compare the user’s and jammer’s utility under the proposed hierarchical joint optimization algorithm (HJOA) with random selection algorithm to verify the algorithm’s superiority.

In the simulation, we assume the user’s and jammer’s discrete power sets are $\mathcal{P}=\mathcal{J}=[0.5\mathrm{W},1\mathrm{W},1.5\mathrm{W},2\mathrm{W},2.5\mathrm{W}]$, and their transmission costs are $C_u=C_j=0.1$. The relay set is $\mathcal{R}=[R_1,R_2,R_3,R_4]$ and different relay has the same power $Q=2\mathrm{W}$. The number of time slots in one epoch $T=100$. Path-loss factor $\gamma =2$.

As shown in Figure 3, there exist a BS, a user, a jammer and a relay group in the investigated scenario. The coordinates of BS and the user are (0 km, 10 km) and (10 km, 0 km). The jammer is located in (10 km, 10 km). The relay group consisted of four relays, which were located in (2.5 km, 2.5 km), (5 km, 2.5 km), (5 km, 5 km), (7.5 km, 5 km) respectively. The red arrow is the jamming signal and blue arrow is the communication signal.

4.1. User’s Strategy Selection Probability and Jammer’s Regret

Figure 4 shows the user’s power selection probability in one epoch, and it converges to a stationary mixed strategy after about 70 time slots. All power strategies have possibilities to be selected which can increase the randomness of strategy and fool jammer effectively. It is easy to find that the user tends to choose power strategy $P_3$ and tends not to choose $P_1$ and $P_2$, that is because a higher transmission power can guarantee the SINR at BS. However, an exorbitant power causes the increase of power cost, so the possibility of $P_3$ is higher than $P_4$ and $P_5$, which realizes the tradeoff of anti-jamming effect and power cost at the same time.

Figure 5 shows the user’s relay selection probability in one epoch, and it converges to a stationary mixed strategy after about 50 time slots. We can find the selection probabilities of $R_1,R_2,R_3,R_4$ are about 0.247, 0.355, 0.204, 0.194 respectively when they have achieved convergence. That is because though $R_3$ is closer to BS and the user, but the distance between $R_3$ and the jammer is also shorter. So in order to minimize the jamming influence of the jammer, the user tends to choose $R_2$ and $R_1$ even they are further from BS and the user compared to $R_3$ and $R_4$. Considering the influence of distance on channel fading, the user tends to select $R_2$ compared to $R_1$.

Figure 6 shows the jammer’s regret in 1000 epochs, which denotes the loss of payoff due to the fact that the optimal strategy is not always chosen during the decision-making process. We can find the jammer’s regret grows nearly logarithmically, i.e., it grows quickly at the start and grows slowly subsequently, which means the loss caused by the wrong choice of the optimal strategy is smaller and smaller. However, regret still grows because jammer selects other non-optimal strategies occasionally to avoid missing a future potential optimal strategy, which realizes the exploration and exploitation simultaneously.

4.2. Utility Comparison under Different Algorithms with Feedback Error and Eavesdropping Error

In one epoch, we average the user’s utility every 20 time slots and obtain the comparison of the user’s utility under the proposed HJOA algorithm and random selection algorithm, as shown in Figure 7. Moreover, we analyzed the influence of feedback error on utility. Under the HJOA algorithm, we can find the user’s utility grows firstly and remains stable then, that is because the user has gradually obtained the optimal strategy with the iterations of algorithm. When the user can receive correct feedback, i.e., ${\epsilon }_1=0$, we can find the user’s utility under the HJOA algorithm reaches the maximum value and is improved compared with random selection algorithm. The greater feedback error ${\epsilon }_1$ is, the larger deviation between the user’s received feedback and actual value is, which causes the decrease of utility.

In Figure 8, we average the jammer’s utility every 20 epochs and obtain the comparison of the user’s utility under the proposed algorithm and random selection algorithm. We analyze the influence of eavesdropping error on utility. Under the proposed the HJOA algorithm, we can find that the jammer’s utility grows quickly firstly and grows slower then, that is because the jammer has gradually obtained the optimal strategy. When the jammer can eavesdrop the correct feedback, i.e., ${\epsilon }_2=0$, its utility reaches the maximum value and is higher than utility under random selection algorithm. The greater feedback error ${\epsilon }_2$ is, the lower the jammer’s utility is because the existence of ${\epsilon }_2$ causes the jammer cannot eavesdrop the user’s feedback information accurately and influences its decision-making.

5. Conclusions

In this paper, we studied the joint relay selection and power control optimization problem in an anti-jamming relay communication system via Stackelberg game, in which a user acted as the leader and a jammer acted as follower. Based on the Q-learning algorithm and multi-armed bandit method, a hierarchical joint optimization algorithm was proposed. Simulation results showed the user’s strategy selection probability and jammer’s regret. Moreover, we analyzed the influence of feedback error and eavesdropping error, and compared the user’s and jammer’s utilities under the proposed algorithm with random selection algorithm to verify the algorithm’s superiority. In the future, we will consider the dynamic change of the user’s and jammer’s position in the multi-user scenario to improve the anti-jamming performance.