An Intelligent Access Channel Algorithm Based on Distributed Double Q Learning

Abstract

Aiming at the problem of mutual interference between nodes and external malicious jamming in wireless communication networks, this paper proposes an intelligent communication anti-jamming algorithm based on distributed double Q-learning. First of all, the proposed algorithm anti-jamming problem is modeled as a time-frequency two-dimensional optimization problem. The decoupled relationship between subframe and channel is used. Each node uses two Q-learning. According to the automatically received feedback signal, users choose their subframe and channel to avoid malicious external jamming and mutual interference between users. It maximizes the sum of all user throughput. Simulation results show that the proposed algorithm can effectively shorten the convergence time and improve the performance of the system.

1. Introduction

With the explosive growth of wireless communication devices, wireless communication presents networked development [1,2]. Channel access is one of the key problems to be solved in wireless communication networks in order to prevent interference caused by multiple users using the same channel at the same time [3]. In addition, due to the openness of electromagnetic wave propagation space [4], wireless communication networks are susceptible to external malicious jamming [5,6]; as an important threat to the security of wireless communication, it mainly affects and destroys the detection of useful signals by electromagnetic radiation signals, which seriously degrades the system performance. Existing communication anti-jamming measures are mainly extended spectrum communication technology [7], which can effectively deal with conventional jamming such as multi-tone and blocking. With the application of artificial intelligence [8] to the field of communication jamming, the jamming modes are characterized by diversity, dynamics and intelligence [9,10]. The conventional extended spectrum communication can no longer ensure the reliability of communication. In the face of a more complex and harsher electromagnetic environment, how to use more “clever” algorithms to avoid mutual interference and external malicious jamming, to ensure reliable communication between users, is a wireless communication network problem to be solved urgently.

In order to solve the problem of mutual interference between different users, the traditional approach is to formulate access criteria for users through access protocol [11]. However, the current protocols do not consider the existence of external jamming. Considering future more complex wireless network scenarios, an artificial intelligence algorithm is a worthwhile idea. Machine learning is an important kind of artificial intelligence algorithm, which can be divided into supervised learning, unsupervised learning, semi-supervised learning and reinforcement learning according to learning methods [12,13]. The paper mainly studies how to use reinforcement learning [14] to solve the communication anti-jamming problem. The main idea of reinforcement learning is that in the face of an unknown environment, the agent can take actions, conduct trial and error experiments, interact with the environment, constantly learn from the environment feedback and update and improve its action strategy, so as to learn the optimal strategy and make the return value of the agent reach the optimal value. Multi-agent reinforcement learning algorithms can be divided into centralized Q-learning [15] and distributed Q-learning [16] according to whether they depend on the central controller. Centralized Q-learning is aimed at cooperative reinforcement learning scenarios with a central controller, which is the learning subject. Distributed Q-learning is aimed at the non-centralized cooperative reinforcement learning scenario, where all communication nodes are learning subjects. Reference [17] proposes a multi-user cooperative anti-jamming algorithm, but the convergence effect is slow, which is difficult to cope with the real-time demand for decision making under the battlefield. Reference [18] designs a Markov game framework and proposes a multi-user joint anti-jamming algorithm. The algorithm can shorten convergence time, but only for a few users. Many users will lead to a dimensional disaster problem, resulting in difficult convergence. Reference [19] considers the influence of distance on mutual disturbance and proposes a multi-agent Q-learning algorithm based on distance metrics. References [17,18,19] are all centralized Q-learning. Centralized Q-learning requires centralized learning and distributed execution. It requires the central controller to perform all the operations. When there are many users, the response speed will slow down. Reference [20] proposes a multi-user reinforcement learning algorithm, but each user adopts the single-user reinforcement learning method [21] without considering the cooperation and interference between users. The convergence speed is slow. In reference [22], a novel collaborative distributed Q-learning mechanism is designed for resource-constrained communication devices, so that they can find idle subframes for their transmission and take the occupancy order of subframes as the global competition cost. This algorithm only considers the time domain subframes, without considering the frequency domain channel. Reference [23] presents a distributed Q-learning based on the decoupling (DQL) scheme. Each device separately uses two Q-learning mechanisms to find a spare subframe and code book (a collection of all the available frequency hopping codes). The scheme has strong scalability and can expand the number of users to hundreds, but it does not consider the external disturbance problem. References [20,21,22,23] are all distributed Q-learning algorithms. As each user conducts distributed computing and distributed executing, and with moderate hardware performance requirements, it displays strong extendibility and processing capacity.

Distributed Q-learning was applied to multi-user scenarios anti-interference decision making. The paper puts forward a distributed double Q-learning algorithm. Each communication node uses two Q-learning. According to the automatically received feedback signal, users choose the optimal subframe and channel at the same time to avoid malicious external jamming and mutual interference between the users and improve the network overall throughput and reduce the dimension of action set. To improve the convergence of the algorithm. The main contributions of this paper are as follows:

• A distributed double Q-learning communication anti-jamming algorithm (DDQL) is proposed, which can effectively avoid the interference between users and external malicious jamming. The algorithm reduces the dimension of action set and shortens the convergence time of the algorithm.
• It is proved that the channel subframe binary optimization problem can be equivalently transformed into two unary optimization problems.

The rest of the article is organized as follows. The system model is presented in Section 2, while the problem is formulated. Section 3 introduces the anti-jamming algorithm of intelligent communication based on distributed double Q-learning, while the simulation results and analyses are given in Section 4. The final part is the summary.

2. System Model and Problem Formulation

2.1. System Model

In order to facilitate the research, this paper makes the following assumptions:

• As shown in Figure 1, we consider a wireless communication network. There are $N$ users, one base station, and one jammer. The number of channels available is $M$. The set of channels available is $C=\left[C_1,\dots ,C_m,\dots C_M\right]$. Each user distributed computing and distributed execution, and hardware performance requirements are not high with good scalability and strong processing power.
• A MAC frame consists of $K$ subframes, and each subframe consists of $B$ time slots. Therefore, the time slot of a MAC frame is $N_{slot}=KB$. Each subframe is followed by a feedback signal from the base station. The content of the feedback signal is the current subframe and the current channel occupancy information, so that all users can make full use of the global resource occupancy information.
• The jamming mode of the jammer is multi-channel intelligent blocking interference [23]. The jammer can quickly sense the channel occupied by the user and select $Z\left(Z<M\right)$ channels that have been occupied for the longest time in the previous multiple subframes for jamming. The jamming channel set can be defined as $J=\{J_1,\cdots ,J_{\mathrm{z}},\cdots ,J_Z\}$.
• It is assumed that the subframe and time slot of the system are synchronized, and the subframe is the basic transmission unit in this paper. Each user has $L$ packets to be transmitted. Each time slot can successfully transmit one packet.

2.2. Problem Formulation

In this section, we first formulate a throughput optimization problem to maximize the sum of throughput of all users by selecting the optimal subframes and signal malicious jamming information on the interactive jammer userbase station channels for each user. We propose a collaborative decoupling distributed double Q-learning algorithm to solve this problem.

Due to the interference between users and the malicious jamming of the external jammer, the communication between users will fail. We introduce a binary variable ${\xi }_{n,m,k}\in \left\{0,1\right\}$. The binary variable represents the result of subframe-channel selection for the $nth$ user. If $nth$ user selects the $mth$ channel in the $kth$ subframe, then ${\xi }_{n,m,k}=1$, and otherwise ${\xi }_{n,m,k}=0$. The number of users occupying the $mth$ channel in the $kth$ subframe is defined as $l_{m,k}={\displaystyle {\sum }_n{\xi }_{n,m,k}}$. The users’ set of successful transmission is $S\left(n^{\prime }\right)\triangleq \left\{n^{\prime }|{\displaystyle {\sum }_{v}1(l_{m,k}=1){\xi }_{n,m,k}=1}\right\}$, where $n^{\prime }$ is users that choose the idle channel; $1(·)$ indicates function. When the events of $(·)$ is true, the $1(·)=1$. If user $n$ occupies the $mth$ idle channel in the $kth$ subframe, then user $n$ can successfully transmit the packet to the base station. Assume that the base station can only successfully receive users in user set $S\left(n^{\prime }\right)$. Then, a binary variable ${\beta }_{n,m,k}$ is introduced to represent whether user $n$ can make the base station successfully receive the packet by selecting the $mth$ channel in the $kth$ subframe. If $n\in S\left(n^{\prime }\right)$, then ${\beta }_{n,m,k}=1$, otherwise, ${\beta }_{n,m,k}=0$. The goal of the paper is to maximize the sum of throughput of all users. In this paper, it is assumed that the throughput is the ratio of the number of successful user communications to the number of subframes. The optimization problem can be formulated as follows:

$$\underset{\left\{{\beta }_{n,m,k}\right\}}{\max }\hspace{1em}O={\displaystyle \sum _{n=1}^N{\beta }_{n,m,k}/K}$$

(1)

$$\begin{array}{l}\mathrm{s}.\mathrm{t}. {\beta }_{n,m,k}\in \left\{0,1\right\}\\ \forall n\in \left[1,2,\dots ,N\right]\\ \forall m\in \left[1,2,\dots ,M\right]\\ \forall k\in \left[1,2,\dots ,K\right]\end{array}$$

According to the literature [23], Equation (1) is a non-convex function, so it is usually difficult to directly obtain the optimal solution. In order to maximize the system throughput, the channel conflicts between users should be minimized and the channel without malicious jamming should be selected. For each user, an idle channel should be selected in a subframe. Ideally, each user selects a unique subframe and channel at a time to send its packets. The selected channel is not jammed by jammers. However, this ideal is almost impossible to achieve with completely random access. However, access control methods based on machine learning (ML) algorithms (such as DQL algorithm) are expected to solve the problem (1). In this paper, the solution of Equation (1) can be found automatically by the distributed DQL algorithm deployed on all users, so that each user can find the optimal selection result of available subframes and channels, and avoid the interference between users and the malicious jamming of external jammers as much as possible.

The process of joint channel and subframe selection is modeled as a two-element Markov problem, where each user is the learning subject. Its state space is the currently available channel subframe combination:

$$S_{k,m}=\left\{(k,m)\right\},k\in (1,2,\dots ,K),m\in (1,2,\dots ,M)$$

(2)

The action space of channel and subframe joint selection Markov is defined as the joint action of channel and subframe that can be selected:

$$A_{k,m}=\left\{(k,m)\right\},k\in (1,2,\dots ,K),m\in (1,2,\dots ,M)$$

(3)

The current reward function of user $n$:

$$R_n\left(k_n,m_n\right)=\{\begin{array}{ll}1& \left(k_n,m_n\right)\ne \left(k_j,m_j\right)\& \left(k_n,m_n\right)\ne \left(k_i,m_i\right)\left(i\in {\rm N}/n\right)\\ -1& \mathrm{otherwise}\end{array}$$

(4)

where, $\left(k_n,m_n\right)$ denotes the subframe and channel selected by user $n$. $\left(k_j,m_j\right)$ denotes the subframe and channel where the jamming is located. $\left(k_i,m_i\right)$ denotes the subframe and channel selected by other users except user $n$.

Furthermore, we model the channel and subframe selection process as two Markov decision processes (MDP), and equivalently prove that the two-element joint selection optimization problem of the subframe channel is equivalent to two unitary optimization problems of the subframe channel, and each access subframe and channel are, respectively, treated as the states of different MDP. In the Markov decision process, each agent constantly interacts with the environment, queries the Q table according to the current state of the environment, selects the optimal action, so as to obtain the return brought by the action, and then enters the next state.

The current states of the two Markov processes defining access subframe and channel are, respectively, each access subframe and channel, so the two state spaces are:

$$S_k=\left\{1,2,\dots ,K\right\}$$

(5)

$$S_m=\left\{1,2,\dots ,M\right\}$$

(6)

The current action spaces of the two Markov processes defining access subframe and channel are, respectively, each access subframe and channel, so the two action spaces are:

$$A_k=\left\{1,2,\dots ,K\right\}$$

(7)

$$A_m=\left\{1,2,\dots ,M\right\}$$

(8)

In the existing literature, the reward function $R_1(n,k)$ of user $n$ in the $kth$ time slot is called binary classical Q-learning, and its expression is as follows:

$$R_1(n,k)=\{\begin{array}{l}+1\hspace{1em} \mathrm{if} \mathrm{transmission} \mathrm{succeeds}\\ -1\hspace{1em} \mathrm{otherwise}\end{array}$$

(9)

We define the occupancy intensity of a subframe as the number of users that a user intends to transmit packets in the same subframe; that is $ZL=[Z{L}_1,Z{L}_2,\dots ,Z{L}_K]$, where $ZL$ is a one-dimensional vector of $1\times K$. On the basis of occupancy intensity, we define the Markov process reward function for accessing subframes as $R_1(n,k)$:

$$R_1(n,k)=\{\begin{array}{ll}+1& \mathrm{when} ZL(k)\le (M-\mathrm{Z})\\ -ZL(k)/K& \mathrm{when} ZL(k)>(M-Z)\end{array}$$

(10)

where, $R_1(n,k)$ is the reward function for the user $n$ to select a subframe, $n\in \left\{1,\dots ,N\right\}$, $k\in \left\{1,\dots ,K\right\}$, $m\in \left\{1,\dots ,M\right\}$. The number of channels jammed by the jammer is $Z$. $ZL(k)$ represents the user occupancy intensity of the $kth$ subframe. That is, the number of users who intend to transmit packets in the same subframe. If the user occupancy intensity in the $kth$ subframe is less than or equal to the number of all channels $M-Z$, then $R_1(n,k)$ equals 1, indicating that the transmission is successful; Otherwise, it equals $-ZL(k)/K$, indicating that the transmission fails.

The Markov process reward function of access channel is defined as $R_2(n,m)$:

$$R_2(n,m)=\{\begin{array}{l}+1\hspace{1em}\mathrm{when} n\in S\left(n^{\prime }\right)\\ -1\hspace{1em}\mathrm{when} n\notin S\left(n^{\prime }\right)\end{array}$$

(11)

where, $R_2(n,m)$ is the reward function for the user $n$ to select a channel, $k\in \left\{1,\dots ,K\right\}$,$m\in \left\{1,\dots ,M\right\}$. The set of users who occupy idle channel collection is $S\left(n^{\prime }\right)$. $n\in S\left(n^{\prime }\right)$ represents user $n$ select the $mth$ channel that is idle channel in the $kth$ subframe. There is no other user interference and jamming, otherwise the user $n$ in the $kth$ subframe select the $mth$ channel where other user or jammer occupying. It can lead to channel conflict or disturbance of communication signals, lead to the failure signals. If the $mth$ channel selected by user $n$ in the $kth$ subframe is an idle channel, then $R_2(n,m)$ is equal to 1, indicating that the transmission is successful; Otherwise, it is equal to −1, indicating that the transmission fails.

3. Anti-Jamming Algorithm of Intelligent Communication Based on Distributed Double Q-Learning

To solve the problem shown in Equation (1), we propose an anti-jamming algorithm based on distributed double Q-learning. Each user uses two similar Q-learning algorithms to select channel and sub-frame, respectively, according to the occupancy strength of channel and subframe. In the Markov decision process, each agent constantly interacts with the environment, queries the Q table according to the current state of the environment, selects the optimal action, so as to obtain the return brought by the action, and then enters the next state. As mentioned earlier, the goal of the proposed algorithm is to maximize the sum of the throughput of all users in the wireless communication network.

First, we prove that two unary Q learning algorithms and a binary Q learning algorithm (subframe is row, channel is column) are equivalent; that is, we prove whether the binary optimization problem of the joint selection of subframe and channel is equivalent to two unary optimization problems of subframe selection and channel selection. According to the optimization theory [24], the necessary and sufficient condition for equivalence is that the two independent variables are separable.

Prove that $Q_2(n,m)$ = $Q_n\left(k_n,m_n\right)$, $Q_1(n,k)$ is the Q value of the selected subframe, $Q_2(n,m)$ is the Q value of selected channel. $Q_n\left(k_n,m_n\right)$ is the Q value of selected subframe and channel together.

In Equation (4), $\left(k_n,m_n\right)\ne \left(k_j,m_j\right)\& \left(k_n,m_n\right)\ne \left(k_i,m_i\right)\left(i\in {\rm N}/n\right)$ indicates that the user $n$ does not suffer from inter-user interference or external malicious jamming. It fits the characteristics of the element in $S\left(n^{\prime }\right)$. The $mth$ channel selected by user $n$ in the $kth$ subframe is the idle channel.

Therefore, Equation (9) can be expressed as:

$$R_n\left(k_n,m_n\right)=\{\begin{array}{ll}1& n\in S\left(n^{\prime }\right)\\ -1& \mathrm{otherwise}\end{array}$$

(12)

$R_1(n,k)R_2(n,m)$ is:

$$R_1(n,k)R_2(n,m)=\{\begin{array}{ll}+1& \mathrm{when} n\in S\left(n^{\prime }\right)\\ -1& otherwise\end{array}$$

(13)

According to Equations (12) and (13):

$$R_1(n,k)R_2(n,m)=R_n\left(k_n,m_n\right)$$

(14)

According to the literature [14], the relationship between Q value and return function is as follows:

$$Q\left(s,a\right)=r\left(s^{\prime }|s,a\right)+\gamma {\displaystyle \sum _{s^{\prime }\in S}{P}_{s{s}^{\prime }}^a\underset{a\in A}{\max }}Q\left(s^{\prime },a^{\prime }\right)$$

(15)

where, $s^{\prime }$ is the state of the next subframe, and $P_{s{s}^{\prime }}^a$ is the state transition probability that is transferred from state $s$ to state $s^{\prime }$ after action $a$ is taken by state $s$. $r\left(s^{\prime }|s,a\right)$ represents the immediate reward transferred to state $s^{\prime }$ after action $a$ is taken by state $s$. The action taken in this paper only considers the current reward and does not consider the future reward. That is, the discount factor γ of the future reward is 0, so Equation (15) can be expressed as:

$$Q\left(s,a\right)=r\left(s^{\prime }|s,a\right)$$

(16)

According to Equations (14) and (16):

$$Q_1(n,k)Q_2(n,m)=Q_n\left(k_n,m_n\right)$$

(17)

As can be seen from Equation (17), $k$ and $m$ are separable, so the algorithm proposed in this paper is equivalent to the subframe and channel joint algorithm. Therefore, we transform a two-element optimization problem of joint selection of subframe channels into two unary optimization problems of the separate selection of the subframe and channel.

Each user needs to maintain two one-dimensional Q tables, namely user-subframe Q table and user-channel Q table. We denote the $t-th$ iteration Q value of user $n$ transmitting a packet in the $kth$ subframe as $Q_{1,t}\left(n,k\right)$, and the $t-th$ iteration Q value of user $n$ transmitting a packet in the $mth$ channel as $Q_{2,t}\left(n,m\right)$.

According to the literature [25], Q value can be updated as follows:

$$Q_{1,t+1}\left(n,k\right)=\left(1-\alpha \right)Q_{1,t}\left(n,k\right)+\alpha *R_1\left(n,k\right)$$

(18)

$$Q_{2,t+1}\left(n,m\right)=\left(1-\alpha \right)Q_{2,,t}\left(n,m\right)+\alpha *R_2\left(n,m\right)$$

(19)

where, α is the learning rate, and our proposed algorithm takes actions that only consider the current reward and do not consider the future reward. The discount factor of the future reward $\gamma$ is 0, because once the user successfully communicates, their needs are satisfied

The base station can obtain the channel collision status of users in each time slot through preamble detection [26]. The subframe occupancy intensity and the channel collision state can be broadcast to all users through the base station, reducing the interaction between users, signaling overhead and user edge computation complexity. The simplest way to implement the Q-learning algorithm is to use the $\epsilon$-greedy method of trade-off exploration; that is, each user chooses the action with the highest Q value with the probability of 1 − $\epsilon$, and randomly chooses an action in the action space with the probability of $\epsilon$. The optimal policy of this user is to execute his actions according to the $\epsilon$-greedy policy. The greedy policy selects the actions with the highest Q value in subframe-Q and channel-Q tables with a probability of 1 − $\epsilon$, namely:

$${\pi }_n^{*}\left(k,m\right)=\{\begin{array}{l}\underset{k,m}{\arg \max }\left\{Q_1\left(n,k\right),Q_2\left(n,m\right)\right\}\hspace{1em} \mathrm{Probability} \mathrm{of} 1-\epsilon \\ {\forall }_k,{\forall }_m\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \mathrm{Probability} \mathrm{of} \epsilon \end{array}$$

(20)

For the user $n$, the specific process of the proposed algorithm is shown in Algorithm 1. In the first iteration, the values of the two Q tables are all initialized to 0, and each user randomly selects any subframe and channel. After receiving the feedback signal from the base station, Equations (10) and (11) are used to calculate the reward $R_1(n,k)$ and $R_2(n,m)$ brought by the action selected by the user, and then Equations (18) and (19) are used to update the corresponding Q values in the time slot and channel table. After each iteration of the Q table, the user selects the channel corresponding to the maximum Q value in the channel-q table, and then selects the subframe corresponding to the maximum Q value in the subframe -Q table to send the packet. When the number of iterations does not reach the maximum number of iterations or the convergence result is not reached, the DDQL algorithm will continue to update iteratively.

The processing flow of user $n$ DDQL algorithm is shown in Algorithm 1. We first initialize the parameters, the learning factor $\alpha$, the maximum number of iterations T, and then set all Q values in the subframe-Q table and channel-Q table of each user to 0. In the first iteration ($t=1$), each user selects a channel in a random manner and sends a packet on a subframe selected in a random manner. Later, when each user feedback signal received from the base station, through the Equations (18) and (19), update the subframe Q table, respectively, and channel—Q value in the table, where each iteration after the update, the user choosing the Q value of the highest channel—Q value and the corresponding channel, and select in the table in a subframe—Q value of the highest in the corresponding subframe to transmit packets. Until the number of iterations T reaches the maximum number of iterations L or the system converges, the DDQL algorithm will continue to be executed until each user can obtain the best subframes and channels, thus obtaining the optimal solution to problem (1).

Algorithm1: The user $n$ cooperative distributed double Q learning algorithm

1: Initialization: $Q_{1,,t}\left(n,k\right)=0$, $Q_{2,,t}\left(n,m\right)=0$, $t=1$, $\forall k\in [1,\dots ,K]$, $\forall m\in \left[1,\dots ,M\right]$.

Set the learning factor $\alpha$ and the maximum number of iterations $T$.

2:  repeat

3:      for $\forall k,m$ do

4:         According to the base station feedback signal, users calculate their own reward $R_1(n,k)$ and $R_2(n,m)$ according to the Equations (10) and (11).

5:         Update $Q_{1,t+1}\left(n,k\right)$ and $Q_{2,,t+1}\left(n,m\right)$ according to the Equations (18) and (19).

6:         End for

7:      According to Equation (20), the optimal strategy is found.

8: until the number of iterations $t$ has reached $T$ or the convergence has been reached.

4. Simulation Results and Discussion

4.1. Parameter Settings

We set the parameters related to simulation in

Table 1. Settings of Model related parameter.

Parameters	Value
Communication users $N$	$10-150$
Available channels $M$ Subframes $K$ Slots $B$ Packet to be transmitted $L$ Number of blocked channels $Z$	$5$ $50$ $100$ $100$ $3$
Learning $\alpha$	$0.9$
Discount factor $\gamma$	$0$
Greedy index $\epsilon$ Number of iterations $T$	$1/\sqrt{t}$ $300$

as follows:

Based on the simulation setup, we set up the five channels and 50 subframes. The jammer occupies the three channels every time. Each subframe has 100 slots. The above assumes each time slot transmission of a packet and each user has a 100-packet transmission. Therefore, the user in a MAC frame only need to finish the communication to successfully transmit a subframe. The channel number is far less than the number of users. Users can also communicate.

In this paper, simulation experiments are carried out to verify the performance of the proposed algorithm (DDQL) under traditional interference linear frequency sweep interference and multi-channel intelligent blocking jamming. As shown in Figure 2a, linear frequency sweep jamming is a dynamic jamming that periodically scans the target channel according to the MAC frame by narrow-band strong power jamming signals. It is simple and efficient, and widely used in the actual battlefield environment. The number of simultaneous blocked channels of the parameter jammer channels $Z$ is 1. As shown in Figure 2b, the multi-channel jamming blocking channel is an upgrade of linear frequency sweep interference. The jammers can sense the channel used by the communication users in the last transmission subframe, so as to focus on interference in the next subframe, and take the MAC frame as periodic interference, and increase the number of jamming channels $Z$ to 3.

4.2. Analysis of Simulation

In this paper, simulation verification is carried out on the MATLAB platform. In order to highlight the superiority of the proposed algorithm, we compare it with the other two algorithms.

• Channel selection algorithm: based on Sense and random selection. Before selecting a channel, the user can sense whether there is interference in the channel. If no interference is detected in the current transmission channel, the user still transmits in the same channel; otherwise, the user randomly switches to other channels in the next action slot. Here, no information is exchanged between users.
• Binary independent Q learning algorithm (ILQ): each user executes the Q-learning algorithm independently only through the local learning results to make transmission decisions and never considers the decision of other users.

For multi-channel intelligent blocking jamming, we consider the impact of the number of users in the network on the normalized throughput of the network. We calculate the maximum access load $\rho =(M-Z)*K=100$ that can be borne by a frame. The maximum number of users can be allowed to access in a frame. We control the number of users within the range of 10–150, as shown in Figure 3. It can be seen that within the maximum access load of the system, the normalized throughput increases with the increase in the number of users. When the maximum access load exceeds; that is, the number of users exceeds 100, the normalized throughput decreases significantly, because the number of users increases. The interference between users due to the frequency conflict leads to the decrease in the overall network throughput.

To investigate the effect of learning factors $a$ on the convergence effect, we set the learning factors of the proposed algorithm DDQL and the independent Q-learning algorithm IQL to $\alpha =0.2$ and $\alpha =0.1$.As shown in Figure 4, in both algorithm DDQL and algorithm IQL, the learning factor $\alpha =0.2$ converges faster than $\alpha =0.1$. For the proposed algorithm DDQL, $\alpha =0.2$, it converges after about 50 iterations, and for $\alpha =0.1$, it converges after about 80 iterations. The independent Q-learning algorithm IQL, for $\alpha =0.2$, converges after approximately 170 iterations, and for $\alpha =0.1$, the algorithm converges after approximately 200 iterations.

We verify the convergence of the proposed algorithm through simulation. Figure 5 shows that the normalized throughput of the proposed DDQL can converge to the optimal value of 1 at the 50th iteration, which indicates that users can still communicate normally in the environment of multi-channel intelligent blocking jamming. Independent of binary Q-learning (IQL) at about 160 times convergence, convergence to 0.9, compared with the proposed DDQL gap in the convergence and reliability of the algorithm, occurs mainly because the classical Q-learning algorithm can interact with the environment, and trial and error to effectively avoid multi-channel jamming, but because each user is an independent subject, it does not interact with other users, so there is no way to avoid interference between users; the algorithm is based on perception and random channel effect is worse. Throughput has nothing to do with the number of iterations and convergence, the main reason being that users cannot perceive the transmission signals of other users; at the same time it will not be able to eliminate mutual interference due to the user frequency: random selection after the encountered jamming in channel transmission strategy cannot be efficient due to multichannel random disturbance. Therefore, it has the worst anti-interference effect and cannot guarantee the normal communication of users.

5. Conclusions

In this paper, in order to solve the problem of channel conflicts and external malicious jamming caused by random access in wireless communication networks, we derive a throughput optimization problem to find the optimal selection of subframes and channel policies for each user. In order to solve this optimization problem, a distributed double Q learning algorithm is proposed. Each user learns the subframe and channel selection according to the received congestion level using two Q-learning, respectively. Compared with the existing algorithms, the convergence performance and robustness of the proposed scheme are improved.