Inner External DQN LoRa SF Allocation Scheme for Complex Environments

Abstract

In recent years, with the development of Internet of Things technology, the demand for low-power wireless communication technology has been growing, giving rise to LoRa technology. A LoRa network mainly consists of terminal nodes, gateways, and LoRa network servers. As LoRa networks often deploy many terminal node devices for environmental sensing, the limited resources of LoRa technology, the explosive growth in the number of nodes, and the ever-changing complex environment pose unprecedented challenges for the performance of the LoRa network. Although some research has already addressed the challenges by allocating channels to the LoRa network, the impact of complex and changing environmental factors on the LoRa network has yet to be considered. Reasonable channel allocation should be tailored to the situation and should face different environments and network distribution conditions through continuous adaptive learning to obtain the corresponding allocation strategy. Secondly, most of the current research only focuses on the channel adjustment of the LoRa node itself. Still, it does not consider the indirect impact of the node’s allocation on the entire network. The Inner External DQN SF allocation method (IEDQN) proposed in this paper improves the packet reception rate of the whole system by using reinforcement learning methods for adaptive learning of the environment. It considers the impact on the entire network of the current node parameter configuration through nested reinforcement learning for further optimization to optimize the whole network’s performance. Finally, this paper evaluates the performance of IEDQN through simulation. The experimental results show that the IEDQN method optimizes network performance.

1. Introduction

Low-power wide area network (LPWAN) [1] technology has recently revolutionized Internet of Things access. However, since the 1980s, LPWAN has begun to sprout all over the world, and some technologies have developed into large-scale networks. Typical examples include the global low-rate data network DataTAC, Mobex, which originated in Europe, and AlarmNet, which was created for fire alarms. But as time progressed, the ultimate fate of these three technologies, similar to that of LPWAN, was to decline along with decrease in the commercial use of 2G networks [2]. In 2023, with global operators using 4G as the mainstream network and beginning to use 5G on a large scale, the retirement of 2G networks began to be implemented or put on the agenda. As the Internet of Things (IoT) market advances, especially with the increasing demand for wireless connectivity for low-rate, low-power consumption scenarios, LPWAN (low-power wide area network) technologies represented by LoRa and NB-IoT [3] are filling this gap.

LPWAN technology has the characteristics of long-distance transmission, low power consumption, and low maintenance. Compared with existing technologies such as WiFi, Bluetooth, and ZigBee, LPWAN can achieve comprehensive area coverage and can complete the entire low-cost Internet of Things at a low cost. Many types of LPWAN technologies are available on the market, among which the most prominent are Long Range, SigFox, MIOTY, Weightless, and Ingenu [4].

Table 1. Key performance indicators for different LPWAN technologies [5].

Parameter	LoRa/LoRaWAN	NB-IOT	SigFox
Range	<14 km	<22 km	<17 km
Frequency Spectrum	unlicensed	licensed	unlicensed
Signal Bandwidth	125 kHz	180 kHz	0.1 kHz
Data Rate	<10 kBbps	200 kBbps	10 Bps
Open Standard	yes	partial	partial
Deployment	widely	widely	widely

shows a performance indicator comparison [5] for various LPWAN technologies.

LoRa technology [6] is one of the proprietary LPWAN technologies based on chirp spread spectrum (CSS) [7] radio propagation, and it is a low-power broadband wireless communication technology. LoRa uses frequency expansion technology and forward error correction methods to operate at lower signal strengths, providing long-distance, low-power wireless connection capabilities over an extensive range. The LoRa network operates in an unlicensed frequency band and can network flexibly without relying on the network equipment of operators. Each gateway under its network can connect thousands of nodes, which is suitable for large-scale IoT deployment [8]. LoRa technology was launched by Semtech, and currently, what is generally referred to as LoRa technology mainly describes the physical layer. LoRaWAN is a communication protocol at the MAC layer. LoRaWAN is an open standard managed by the LoRa Alliance and defines a communication protocol for implementing wireless IoT connections in the LoRa network. However, because the LoRa physical layer technology is open, users can design a communication protocol suitable for real application scenarios based on specific requirements. Although the LoRa physical layer technology itself is open, it is not available in practice because the underlying chirp that implements LoRa modulation can only be obtained from Semtech [9]. Semtech provides some basic parameters that can be changed, such as the spreading factor (SF), transmission power (TP), coding rate (CR), bandwidth (BW), etc., to adapt to different LoRa networks. Among the parameters provided by Semtech, there is no doubt that SF is the most critical one. The core idea of LoRa modulation is to use a frequency-changing mode to modulate the baseband signal. The rate of chirp change is the so-called SF. The more significant the spread factor, the farther the transmission distance, but the cost is a reduction in data rate because a longer chirp has to be used to represent a symbol. In addition, different SF channels are orthogonal to each other, so transmitting data packets at different SFs simultaneously will not cause conflicts. Although in practice there is a certain degree of interference between different SFs, users can ensure that SF transmissions are orthogonal to each other by changing the remaining parameters; so in this study, it is assumed that different SFs represent different channels, and there is no collision between them.

Regarding the allocation of spreading factors, ref. [10] proposed an SF allocation technology on a LoRa network based on reinforcement learning, aiming to maximize the entire network’s throughput. This method uses the time difference reinforcement learning method to iterate and allocate channels to each node through the channel’s PCR. This method is finally compared with the uniform distribution method and the greedy algorithm to find that its effect is the best. Although this method allocates channels to each node, it does not consider the indirect impact of the node’s allocation on the entire network. The authors of [9] proposed a channel occupancy perception resource allocation method that focused on energy efficiency on a LoRa network using DQN and optimization assistance. This method first uses DQN to allocate SF to the channels collected from the terminal nodes and then solves a convex optimization problem to distribute power. This method is mainly used to solve the energy consumption problem of the system in order to minimize the energy consumption of the entire system, so the feedback factor of this DQN is energy consumption, and it does not take the system network performance as the first element. Currently, most SF allocation strategies are actively modified on the node side, which leads to improvement of the parameters on the node side but does not consider whether the entire network’s performance improves. In other words, the optimal strategy for each node is not equivalent to the optimal strategy for the whole system. Secondly, the performance optimization of the LoRa network is not only related to the parameters of each node but also depends on the complex external environment. Based on the above points, this topic will start from the complex environment and first establish a model to optimize network performance. To optimize the network performance, this article first proposes a random spreading factor allocation method (SF-random) that considers probabilities and uses a random method to perform each node’s SF allocation through the given empirical likelihood. On this basis, this article proposes the Inner External DQN SF allocation method. Through learning from the environment, the Inner DQN method can obtain the local optimal solution of each node and then use this as feedback to the External DQN to consider the impact of the local optimal solution on the optimal performance of the entire system for secondary optimization to make the performance of the whole network optimal. In addition, External DQN uses the properties of DQN to continuously give feedback to the four-tuple obtained from the interaction with the environment to the Target Q Network and Q Network for learning to predict the following behavior so that the performance of the entire system reaches the optimum in continuous iteration and learning. The main contributions of this research are as follows:

• To optimize system performance, this paper proposes a method of allocating parameters to nodes through the gateway, as opposed to the traditional method of directly changing node parameters.
• The method proposed in this paper first considers how to optimize each node; then it considers the impact of the optimal single node on the overall optimization and suggests the Inner External DQN SF allocation method.
• The current network performance optimization methods do not take into account the impact of the environment. This paper starts from the node-optimal Inner DQN method proposed at the beginning, pointing out that SF allocation can be carried out through continuous learning with the environment.
• This paper compares the proposed SF-random method using LoRaSIM simulation software and verifies its effectiveness.
• Through the method proposed in this paper, a set of exclusive node SF allocation schemes corresponding to each fixed scene can be established according to local conditions. For scenes with environmental changes, the parameters of the entire system of nodes can be updated through the gateway by intermittent updates without manual operation, thereby effectively improving the entire network’s performance.

The rest of this paper is organized as follows. Section 2 presents the system model and problem description. In Section 3, this paper proposes the SF-random method for complex environments, and on this basis, the Inner External DQN SF allocation method is proposed. Section 4 evaluates the performance of the proposed Inner External DQN SF allocation method in multiple aspects. Finally, Section 5 summarizes the entire paper.

2. System Model and Problem Description

2.1. System Model

As shown in Figure 1, this paper investigates the LoRa network link model in complex environments. The network is considered a star network [11] comprising a LoRa network server, gateway, and multiple end nodes. End nodes transmit data to the gateway via LoRa, and the gateway transmits data to the LoRa network server via Ethernet/4G/WiFi. This study considers M LoRa nodes $N=\left\{N^{\left(i\right)}\right|1\le i\le M\}$ propagating over six channels, each with orthogonal spreading factors (Equation (1)). As shown, nodes with the same channel assignment are marked with the same color.

$$SF=\left\{S{F}^{\left(i\right)},i\in \{1,2,\dots ,6\},S{F}^{\left(i\right)}\in \{7,8,\dots ,12\}\right\}$$

(1)

Considering the architecture of existing IoT networks, the gateway in this LoRa system model is primarily responsible for receiving packets sent by nodes and can modify the SF parameters of LoRa nodes via custom protocols. The spreading factor value assigned to each node is denoted as $A_n=S{F}^{\left(i\right)}$, and $S{F}_{\mathrm{flag}}^{\left(i\right)}$ indicates whether the spreading factor of node i has been assigned. For uplink data transmission, nodes send fixed-size packets of $L=10$ bytes to the gateway G every $T=10$ seconds following a Poisson distribution [12]. The uplink transmission delay for each node is denoted as $D^{\left(i\right)}$.

Under the interference of complex environments on LoRa link transmission, this paper specifically categorizes the impacts of complex environments into path loss and shadowing effects [13]. Shadowing effects are caused by obstacles between the transmitter and receiver, which attenuate signal power through absorption, reflection, scattering, and diffraction, potentially even blocking the signal. Since these factors are generally unknown in practice, they must be characterized by statistical models. The most commonly used model is the log-normal shadowing model, which assumes the ratio of transmitted power to received power $\psi =P_t/P_r$ is a log-normally distributed random variable, yielding Equation (2), where $\xi =\frac{10}{ln10}$, ${\mu }_{{\psi }_{dB}}$ is the path loss, and ${\sigma }_{{\psi }_{\mathrm{dB}}}$ is the standard deviation of ${\psi }_{\mathrm{dB}}$ in dB.

$$p\left(\psi \right)=\frac{\xi }{\sqrt{2\pi }{\sigma }_{{\psi }_{dB}}\psi }exp\left[-\frac{{\left(10{log}_{10}\psi -{\mu }_{{\psi }_{dB}}\right)}^2}{2{\sigma }_{{\psi }_{dB}}^2}\right],\psi >0.$$

(2)

In terms of path loss, it is generally believed that the path loss generated by a signal over the same propagation distance under identical conditions is the same. The long-distance characteristics of the channel follow a log-normal distribution and are commonly modeled by a log-normal distribution model. The path loss model is shown in Equation (3), where $PL\left(d\right)$ represents the path loss of the received signal at a distance $d\left(m\right)$, $PL\left(d_0\right)$ represents the path loss of the received signal at a reference distance $d_0$, n is the path loss exponent in a specific environment and describes the rate of loss, which increases with distance, and $X_{\sigma }$ is in dB, represents shadow fading, and is a log-normally distributed random variable [14].

$$PL\left(d\right)=PL\left(d_0\right)+10nlog\left(\frac{d}{d_0}\right)+X_{\sigma }$$

(3)

Considering the impact of the aforementioned environmental factors on data transmission, this paper uses $\alpha =\left\{{\alpha }_n\mid 1\le n\le M\right\}$ to represent the packet arrival rate of each node. Therefore, the number of packets received by the gateway from the nodes within a time period $\Delta T$ is shown in Equation (4).

$$P_{\mathrm{receive}}=\sum _{n=1}^M{\alpha }_n\Delta T$$

(4)

Since each node sends packets to the gateway every T seconds according to a Poisson distribution, the theoretical number of packets received by the gateway within $\Delta T$ without path loss is shown in Equation (5).

$$P_{\mathrm{Theoretical}}=\frac{\Delta T}{T}M$$

(5)

The proposed model aims to optimize system performance and introduces the package receive rate (PRR) as a performance metric; the PRR indicates the proportion of successfully received packets. The PRR is commonly used to measure the reliability of the packet transmission process by indicating the proportion of successfully received packets out of the total sent packets. Observation of changes in the PRR approximates changes in system performance. Given the actual packet reception number $P_{\mathrm{receive}}$ and the theoretical reception number $P_{\mathrm{Theoretical}}$, the PRR is calculated as shown in Equation (6).

$$PRR=\frac{P_{\mathrm{receive}}}{P_{\mathrm{Theoretical}}}$$

(6)

2.2. Problem Formulation

The objective of this paper is to maximize system performance [11], characterized by the packet receive rate [15], leading to the following equation:

$$\begin{array}{ccc}{max}_{G,N}\hfill & PRR\hfill & \\ \mathrm{s}.\mathrm{t}.\hfill & D^{\left(i\right)}<T\hfill & \forall i\hfill \\ \hfill & S{F}_{\mathrm{flag}}^{\left(i\right)}\in \{0,1\}\hfill & \forall i\hfill \\ \hfill & {\sum }_{i=1}^{M}S{F}_{\mathrm{flag}}^{\left(i\right)}=M\hfill & \hfill \end{array}$$

(7)

The first constraint of Equation (7) ensures that the uplink transmission delay for each node must be less than the specified data packet transmission interval. Second constraint specifies that each node can be assigned to only one channel. The third constraint ensures that every node is assigned to a channel.

This paper utilizes the PRR to evaluate the performance of the system. Based on this, the paper first introduces a reinforcement learning approach. It then proposes the SF-random method, which is based on random spreading factor allocation, and subsequently introduces the Inner DQN and External DQN methods to explain the proposed Inner External DQN SF allocation approach.

3. SF Allocation Model Based on Inner External DQNS

3.1. Deep Reinforcement Learning

Reinforcement learning (RL) [16,17] is a domain in machine learning that deals with sequential decision making. The Markov decision process [18,19] is a typical formulation for RL problems and consists of the tuple $(S,A,T,R,\epsilon )$, where S represents the state space, A represents the action space, $T\left(s,a,s^{\prime }\right)$ denotes the probability of transitioning from state s to state $s^{\prime }$ after taking action a, $R\left(s,a,s^{\prime }\right)$ is the reward function and returns the reward obtained by taking action a in state s to reach state $s^{\prime }$, and $\epsilon \in S$ indicates terminal states that prevent future actions or rewards.

An important aspect of RL is its enabling of the agent to learn optimal behaviors, which means it gradually modifies or acquires new behaviors and skills. Another crucial aspect is the trial-and-error experience, where the agent collects information through continuous interaction with the environment and learns to determine the next action based on this information. Generally, RL problems can be reduced to discrete-time stochastic control problems, where the agent interacts with the environment, as illustrated in Figure 2: starting from an initial state $s_0\in S$, it collects observations ${\omega }_0\in \mathrm{\Omega }$, and at each step t, the agent must take an action $a_t\in A$. This action results in one of three outcomes: (i) the agent receives a reward $r_t\in R$, (ii) the state transitions to $s_{t+1}\in S$, or (iii) the agent obtains the next observation ${\omega }_{t+l}\in \mathrm{\Omega }$ [20].

When neural networks are used to approximate the value function $\widehat{v}(s;\theta )$ or $\widehat{q}(s,a;\theta )$ in RL, it is termed deep reinforcement learning [21], where $\theta$ represents the weights in the deep neural network. DRL commonly employs gradient descent to update the weight parameters [22]. The Deep Q-Network proposed by Mnih and others has become the benchmark and building point for many deep reinforcement learning researches.

The Q function is a concept associated with MDPs, where $Q^{\pi }:S\times A\to \mathbb{R}$ defines the expected future discounted reward of taking action A in state S and then following policy $\mathrm{\Pi }$. According to the Bellman equation, the Q function under the optimal policy can be recursively expressed as:

$$Q^{*}(s,a)=\sum _{s^{\prime }\in S}T\left(s,a,s^{\prime }\right)\left[R\left(s,a,s^{\prime }\right)+\gamma \underset{a^{\prime }}{max}{Q}^{*}\left(s^{\prime },a^{\prime }\right)\right]$$

(8)

Here, $0\le \gamma \le 1$ is the discount factor and determines the value of immediate rewards relative to future rewards. Given $Q^{*}$ and the optimal policy ${\mathrm{\Pi }}^{*}$, the highest Q-value for the current state can be obtained by greedily selecting the action: ${\pi }^{*}\left(s\right)=arg{max}_a{Q}^{*}(s,a)$. In Q-learning, the agent starts with an arbitrary estimate $Q_0$ and updates its Q function estimate by taking arbitrary actions in the environment, as show in Equation (9); the agent observes the rewards of these actions and the subsequent state. The terms $s_t,a_t,r_t$ represent the state, action, and reward, respectively, at step t, and ${\alpha }_t$ is the step size parameter [23].

$$Q_{t+1}\left(s_t,a_t)=Q_t(s_t,a_t\right)+{\alpha }_t\left[r_{t+1}+\gamma \underset{a^{\prime }}{max}{Q}_t\left(s_{t+1},a^{\prime }\right)-Q_t\left(s_t,a_t\right)\right]$$

(9)

Deep Q-Learning (DQN) [24] is a variant of the classical Q-learning algorithm that uses deep convolutional neural networks [25] to approximate the Q function [26]. The deep convolutional architecture provides a general mechanism for estimating Q function values from historical experiences. DQN maintains a large set of recent experiences, where each experience is a tuple $\left(s,a,s^{\prime },r,T\right)$, representing the state s, the action a taken to reach state $s^{\prime }$, the reward r received, and T indicating if the state is terminal. After each step, the agent adds the experience to its memory and randomly samples mini-batches for Q function updates. Experience replay shuffles the data to eliminate correlations between experiences, making them independently and identically distributed, thereby reducing variance in parameter updates and enhancing convergence speed. DQN introduces target networks $Q\left(s,a;w^{-}\right)$ and evaluation networks $Q(s,a;w)$ [27], which have the same structure but different parameters. The evaluation network guides the agent’s behavior, while the target network is used to compute the TD target. Only the evaluation network’s weights are updated, and after a certain number of steps, the target network’s weights are updated to match the evaluation network’s, thereby stabilizing learning by keeping the target value relatively constant over several updates [28].

3.2. SF-Random

In traditional LoRa networks, each node’s spreading factor (SF) is allocated using a uniform distribution method. However, due to the characteristics of LoRa, as referenced in the Semtech SX1262 datasheet, the air transmission time for a node when all SF channels have signal coverage is given by:

$$\mathrm{ToA}=\frac{2^{SF}}{BW}*N_{\mathrm{symbol}}$$

(10)

where SF represents the spreading factor, BW is the bandwidth, ToA is the air time for LoRa transmission, and $N_{\mathrm{symbol}}$ is the number of symbols [29]. This indicates that air time is proportional to SF: as air time increases, the probability of a node occupying the channel increases, leading to a higher likelihood of collisions within the same channel. Therefore, using a uniform distribution for SF allocation is unsuitable under the SF channel coverage conditions described in this paper. As the allocation probability for a single SF channel increases, the number of nodes assigned to that channel also increases. For low-SF channels, although the data transmission rate per node is faster, the collision probability also increases due to the higher number of nodes. Algorithm 1 presents the steps for calculating the optimal SF allocation probability.

Algorithm 1 The formula for calculating the optimal probability for SF-random

1:    Input: The SF values (ranging from 7–12) and the transmission time interval D of each node. 2:    Output: Statistical table of SF allocation probabilities. 3:    Initialization: $S{F}_{flag}^{\left(i\right)}=0,i\in \{1,2,\cdots ,M\};S{F}_{\mathrm{count}}^{\left(i\right)}=0,i\in \{7,8,\cdots ,12\}$; $S{F}_{\mathrm{probability}}^{\left(i\right)}=1/6,i\in \{7,8,\cdots ,12\};{\mathrm{A}}_i=0,i\in \{1,2,\cdots ,M\}$; HashTable(). 4:    while 1 do 5:       for $i=1,2\dots$ M do 6:           ${\mathrm{A}}_i=random\left(S{F}_{\mathrm{probability}}^{\left(i\right)}\right)$ 7:       end for 8:       Calculating PRR 9:       HashTable.append $\left(S{F}_{\mathrm{probability}}^{\left(i\right)},\mathrm{PRR}\right)$ 10:     $S{F}_{\mathrm{probability}}^{\left(7\right)}+1.6\%;S{F}_{\mathrm{probability}}^{\left(8\right)}+0.2\%;S{F}_{\mathrm{probability}}^{\left(9\right)}+0\%;S{F}_{\mathrm{probability}}^{\left(10\right)}-0.5\%;$ $S{F}_{\mathrm{probability}}^{\left(11\right)}-0.6\%;S{F}_{\mathrm{probability}}^{\left(12\right)}-0.7\%$; 11:    ${\mathrm{A}}_i=0,i\in \{1,2\dots M\}$ 12:    if $S{F}_{\mathrm{probability}}^{\left(7\right)}>52\%$ then 13:        Output: max HashTable. 14:    end if 15: end while

During the initialization phase, each node is given the same SF allocation probability, and SF allocation begins according to $S{F}_{\mathrm{probability}}^{\left(i\right)}$. After each SF allocation, the corresponding counter increments by 1, and the allocation flag is set. When all nodes’ SF allocations are completed, the system’s PRR value is calculated and recorded in the hash table. By fine-tuning the SF allocation probability to meet the aforementioned distribution trend, this process is repeated until the allocation probability for SF7 reaches 52%, indicating the optimal probability calculation is complete. Reviewing the recorded hash table, the SF allocation probability for the maximum system PRR is obtained.

Figure 3 shows the variation of the PRR under different SF allocation probabilities. Observation reveals that the PRR peaks at 98.09% at the 17th iteration. Thereafter, as the SF allocation probability changes, collisions among nodes with the same SF increase, causing the PRR to gradually decline until the iterations stop.

Table 2. Optimal SF-random SF assignment probabilities.

SF	7	8	9	10	11	12
P	42.2667%	19.8667%	16.6667%	8.6667%	7.0667%	5.4667%

shows the optimal SF allocation ratio for the SF-random method, which significantly optimizes the overall system performance compared to the initial uniform distribution.

3.3. Inner External DQN SF Allocation Scheme

3.3.1. Inner System DQN

Considering the SF allocation for a single node, this paper proposes the Inner System DQN method to optimize the SF allocation for individual nodes through continuous interaction and learning with the environment. The primary objective of the Inner System DQN model is to maximize the signal-to-noise ratio for a single node.

During the optimization process, due to potential multipath effects and co-channel interference among different nodes, it is impractical to optimize the SNR through manual parameter adjustments. Therefore, using the DQN method to directly obtain state inputs from the environment and to output corresponding Q-value estimates for actions allows efficient end-to-end learning, ultimately maximizing the SNR. To clarify the relationship between the Inner System DQN and the subsequent External System DQN method, we define Inner_Ite as the number of one-step iterations required by the Inner System DQN, after which, the process transitions to the External System DQN. Figure 4 illustrates its detailed internal design. Next, we define the parameters for the Inner System DQN algorithm, including the state space, action space, and reward function.

State space: Given that the goal of the Inner System DQN is to maximize the node’s SNR, we represent the state space of the node as $S^{\left(i\right)}=SNR$. A higher SNR indicates a higher power ratio of the signal to noise, resulting in stronger signals, less noise, and improved transmission quality and overall system performance.

Action space: The Inner System DQN optimizes the current node’s parameters by modifying the SF. Since the SF range is typically from 7 to 12, the action space is also restricted to this range to ensure compatibility.

Reward function: The objective of the Inner DQN is to optimize the node’s SNR; thus, $r=SNR$ is used as the feedback. Through this feedback, the Inner System DQN interacts with the environment to maximize the cumulative reward in the long term.

$$a_t=\left\{\begin{array}{cc}random\hfill & \mathrm{with}\mathrm{probability}ϵ,\hfill \\ arg{max}_{a}Q\left(s_t,a\right)\hfill & \mathrm{with}\mathrm{probability}1-ϵ.\hfill \end{array}\right.$$

(11)

In the Inner System DQN, the agent needs to learn the optimal SF allocation strategy in an unknown environment. During the exploration phase, the agent attempts unknown actions to discover more information about the environment; during the exploitation phase, the agent selects the optimal actions based on the knowledge it has acquired. The Inner System DQN uses the $ϵ$-greedy strategy to balance exploration and exploitation [30]. Specifically, the $ϵ$-greedy strategy selects random actions with probability $ϵ$ and the best current action with probability $1-ϵ$, as shown in Equation (11). This enables the agent to try different actions during learning: avoiding local optima and gradually reducing the exploration frequency to more effectively utilize the learned knowledge.

This paper presents the pseudo-code steps of the Inner System DQN in Algorithm 2. Since the environment for each node differs, it is necessary to first determine the ID of the operating node. For this node, the Inner System DQN model continuously learns through interaction with the environment by using the $ϵ$-greedy strategy. During the model generation process, each step first determines the next action based on the current model predictions and the $ϵ$-greedy strategy; then it obtains the corresponding reward signal from the environment and saves these experiences through the experience replay mechanism. Subsequently, part of the saved experience data are extracted to update the model parameters, gradually improving model performance. In this way, the model learns to make accurate actions and continually increases its rewards.

Algorithm 2 Inner System DQN SF allocation algorithm

1:   Input: The SF values, which range from 7 to 12; the ID of the node for which SF optimization is performed; Inner_Ite value; Learning_Episodes value; Learning_EP_Steps value. 2:   Output: The optimal SF allocation value. 3:   Initialization: Step_count $=0;{\mathrm{A}}_i=0;$ Total $=0$ 4:   for $i=1,2\dots$ Learning_Episodes do 5:       for $j=1,2\dots$ Learning_EP_Steps do 6:           $\mathrm{a}=$ Based on $ϵ-\mathrm{greedy}$, select action InnerDQN.choose action() 7:           s, $r=$ Based on InnerDQN.step(a), obtain the state and the reward 8:           InnerDQN.store transition(s, a, r, s) 9:           The neural network is trained by randomly drawing samples from the experience replay buffer InnerDQN.learn() 10:        Restore the original state InnerDQN.render() 11:     end for 12: end for 13: for Step_count < Inner_Ite do 14:    $\mathrm{a}=$ Based on $ϵ-\mathrm{greedy}$, select action InnerDQN.choose action() 15:    $\mathrm{s},\mathrm{r}=$ Based on InnerDQN.step(a), obtain the state and the reward 16:    ${\mathrm{A}}_i=\mathrm{a}$ 17:    Total = Total $+\mathbf{r}$ 18:    Step_count = Step_count + 1 19: end for

The trained Inner System DQN model performs SF optimization iterations within the specified Inner_Ite: no longer training the convolutional neural network for Q-values but using the trained network for action selection. When Step_count reaches the preset Inner_Ite, the training ends and the optimal SF allocation is provided. Since the reward function for the Inner System DQN model is defined as the SNR, the neural network will eventually always choose the action that yields the maximum reward after multiple attempts, resulting in the Inner System DQN algorithm returning the optimal SF as the choice from the last action.

3.3.2. External System DQN

To maximize the overall system performance, this paper proposes the External System DQN method. This approach combines with the Inner System DQN to explore the impact of individual node parameter adjustments on the entire system, optimizing the system’s packet reception rate, which was the initial goal of this paper:

$$maxPRR$$

(12)

As shown in Figure 5, the internal block diagram of the External System DQN illustrates how this method optimizes action selection through interaction with the environment to maximize future cumulative rewards. Although the optimization process of DQN is theoretically non-convergent, the algorithm is considered to have converged when action selection becomes stable within a certain runtime, reaching the termination condition. To facilitate this, this paper introduces the External_Ite parameter, which describes the upper limit on the number of consecutive identical actions, thus providing an exit condition in order for the DQN to stop. Next, we define the parameters for the External System DQN, including the state space, action space, and reward function.

State space: The goal is to maximize the system’s PRR, so the state space of the entire system is represented by $s=PRR$. Changes in the PRR reflect the system’s performance and interference conditions in the wireless network. A sudden drop in the PRR might indicate that the SF allocation by the Inner System DQN for a single node has disrupted the entire system.

Action space: The External System DQN balances the impacts of the Inner System DQN on the system. Actions are described based on which node to modify to improve the PRR, with the action space $a=\{1\dots M\}$ representing the node ID to be modified. Once an action is selected, it is implemented by passing the terminal node ID to the Inner System DQN to allocate the optimal SF.

$$r=\left\{\begin{array}{cc}5\hfill & \Delta PRR>\mu \hfill \\ -1\hfill & -\mu <\Delta PRR<\mu \hfill \\ -2\hfill & \Delta PRR<-\mu \hfill \end{array}\right.$$

(13)

Reward function: To encourage the External System DQN to explore extensively for the optimal system SF allocation, the reward function is defined as shown in Equation (13), which provides different rewards based on changes in the PRR, where $\mu$ is a reference empirical value. Positive rewards guide the agent to frequently take beneficial actions for the system, while neutral rewards are negative, indicating that exploration is not desirable. Ultimately, the method will converge to continuous allocation for one node, indicating that the optimum has been reached.

This paper presents the steps of the External System DQN in Algorithm 3. For a given network of M nodes, External_Ite is set as the final decision threshold of the system, meaning the system is considered optimal if the same node is chosen consecutively for External_Ite times. Learning_Episodes represents the number of learning cycles in the system model establishment process, and Learning_EP_Steps represents the number of steps in each learning cycle. During the system model establishment, actions obtained through the External System DQN neural network are input to the Inner System DQN, and the feedback results of the state and reward are observed and stored in the experience pool. Then, part of the experience data is used to train the neural network to establish the system model. If External_Ite is not reached, the same steps are repeated to obtain actions through the neural network, iterating until convergence and outputting the optimal per-node SF allocation statistics table.

Algorithm 3 External System DQN SF allocation algorithm

1:   Input: The nodes in the range of $1-M$; External_Ite value; Learning_Episodes value; Learning_EP_Steps value. 2:   Output: The optimal per-node SF allocation statistics table. 3:   Initialization: Action_count $=0;{\mathrm{A}}_{\mathrm{old}}=0;$ Total $=0;$ 4:   for $i=1,2\dots$ Learning_Episodes do 5:       for $j=1,2\dots$ Learning_EP_Steps do 6:           $\mathrm{a}=$ Based on $ϵ-\mathrm{greedy}$, select action ExternalDQN.choose_action() 7:           s, $r=$ Based on Inner System DQN ExternalDQN.step(a), obtain the state and 8:           the reward 9:           ExternalDQN.store_transition(s, a, r, s_) 10:        The neural network is trained ExternalDQN.learn() by randomly drawing samples from the experience replay buffer 11:        Restore the original state ExternalDQN.render() 12:     end for 13: end for 14: for Action_count < External_Ite do 15:     $\mathrm{a}=$ Based on $ϵ-\mathrm{greedy}$, select action ExternalDQN.choose_action() 16:     s, r = Based on ExternalDQN.step(a), obtain the state and the reward 17:     Total = Total $+\mathbf{r}$ 18:     if $\mathrm{a}=={\mathrm{A}}_{\mathrm{old}}$ then 19:         Action_count = Action_count +1 20:     end if 21:     if $\mathrm{a}!={\mathrm{A}}_{\mathrm{old}}$ then 22:         QAction_count $=0$ 23:     end if 24:     ${\mathrm{A}}_{\mathrm{old}}=\mathrm{a}$ 25: end for

4. Simulation Results

4.1. Simulation of LoRa Transmission Environment

The experimental verification of the Inner and External DQN SF allocation method was conducted in the interactive programming environment of Jupyter Notebook [31] and utilizing the Python programming language for simulation. To obtain realistic simulation results, the study employed the widely used LoRa simulator. LoRaSim [32] is a discrete event simulator developed using the SimPy package in Python. It determines whether a collision has occurred between two packets, x and y, by considering parameters such as overlap $O(x,y)$, carrier frequency ’${\mathrm{C}}_{\mathrm{freq}}(\mathrm{x},\mathrm{y})$’, spreading factor ’${\mathrm{C}}_{\mathrm{sf}}$’, transmission power ’${\mathrm{C}}_{\mathrm{pwr}}(\mathrm{x},\mathrm{y})$’, and transmission delay ’${\mathrm{C}}_{\mathrm{cs}}(\mathrm{x},\mathrm{y})$’ [33].

Considering the impact of complex environments, this study modified the path loss method in LoRaSim using the specific calculations detailed in Equations (2) and (3).

Table 3. LoRaSim simulation model parameters.

Area size $\mathbf{D}\times \mathbf{D}$	$1.7\times 1.7\left[{\mathrm{k}\mathrm{m}}^2\right]$
Transmission power ${\mathit{P}}_{\mathit{t}\mathit{x}}$	$22\left[\mathrm{dBm}\right]$
Path loss index $\mathit{\gamma }$	2.08
Reference distance ${\mathbf{d}}_0$	$1.5\left[\mathrm{km}\right]$
Frequency band	$470\left[\mathrm{MHz}\right]$
Bandwidth $\mathbf{W}$	$500\left[\mathrm{kHz}\right]$
Standard deviation $\mathit{\sigma }$	$1\left[\mathrm{dB}\right]$
Noise figure ${\mathit{N}}_{\mathit{F}}$	10–20 [dB]
Initial proportion of $\mathbf{SF}$	7: $32.00\%;$
	8: $24.00\%;$
	9: $16.00\%;$
	10: $12.00\%;$
	11: $4.00\%;$
	12: $4.00\%;$
Coding rate $\mathbf{C}$	$4/5$
Number of LoRa nodes $\mathbf{M}$	100
Number of LoRa gateways $\mathbf{G}$	1
Size of the payload Bpl	$10\times 8$ [bit]
Transmission cycle of LoRa nodes	10,000 [ms]
Maximum protocol data units	8

lists the specific experimental parameters of the LoRaSim simulation model, with some parameters referencing the configuration of the ASR6500 LoRa chirp used in real-world environments. The transmission range in this study is 1.7 × 1.7 square kilometers, with all nodes set to a transmission power of 22 dBm, a bandwidth of 500 kHz, a coding rate of 4/5, and an SF range of 7–12. To validate the proposed SF allocation method, the number of nodes was set to 100, and the number of gateways was set to 1, reflecting a real IoT application scenario. Figure 6 shows the environmental model constructed using the LoRaSim simulation platform, with red indicating the gateway location and blue indicating the node locations.

4.2. Inner System DQN SF Allocation Simulation

In this section, we conduct simulation verification of the Inner System DQN SF allocation method. To achieve more precise SF allocation, we set the number of training iterations to 500, with each episode consisting of 200 steps. After each training step, the model is trained by evaluating both the neural network and the target neural network.

Table 4. Inner System DQN detailed parameters.

$\mathit{ϵ}$	0.7
Replay buffer size	2000
Batch size	32
Learning rate	0.1
Target network update frequency	20
Reward discount $\mathit{\gamma }$	0.9
Episodes	500
Episode steps	200

shows the detailed parameters of the Inner DQN.

To obtain the optimal SF allocation strategy, we first verify through simulation experiments the impact of the $ϵ$ value on action selection. As shown in Figure 7, the action changes with one-step increments as $ϵ$ varies from 0 to 1 with a step size of 0.1 under the same environment. In $ϵ$-greedy, $ϵ$ controls the balance between random actions and actions based on the current optimal policy during training. When $ϵ$ is 0, the agent selects actions entirely according to the current optimal policy, which is referred to as the ‘greedy policy’. When $ϵ$ is 1, the agent selects actions completely at random, which is known as the ‘random policy’. Between these extremes, the agent selects random actions with probability $ϵ$ and optimal actions with probability $1-ϵ$. Observational results show that as $ϵ$ increases, the agent’s exploration decreases, and when $ϵ$ reaches 1, the Inner System DQN stops exploring and only follows the previously learned optimal policy. To balance exploration and exploitation, we set $ϵ$ to 0.7, meaning the agent selects random actions with 70% probability and optimal actions with 30% probability. This setup encourages the agent to explore the environment and discover more action choices, avoid premature convergence to local optima, and gradually converge to the global optimal policy.

We used the LoRaSim simulation platform to verify the feasibility of the Inner System DQN method. During the training process, we considered environmental interference using the model to ensure that the DQN’s learning process also accounted for environmental factors. The model was constructed through iterative training, and a random node was selected for simulation. The node’s SNR variation with steps is shown in Figure 8. The results indicate that the model ultimately converges to the optimal point with an SF of 12 during exploration.

As shown in Figure 9, the histogram of the action distribution during the Inner DQN process provides statistical data. Initially, the number of nodes that are allocated different SFs are SF7: 35, SF8: 36, SF9: 16, SF10: 9, SF11: 1, and SF12: 3, with the test node’s initial SF set to 7. The goal of the Inner DQN is to maximize the SNR of a single node, and its action is to allocate the optimal SF to that node. Through preliminary model learning and rewards, it is observed that action selection involves an exploration process, which ultimately converges to SF12, further verifying the feasibility of the Inner DQN.

4.3. External System DQN SF Allocation Simulation

To maximize the PRR of the entire system, the External DQN uses the Inner DQN as its action and adjusts different actions based on the PRR feedback in order to achieve the optimal overall PRR.

Table 5. External System DQN detailed parameters.

Inner $\mathit{ϵ}$	0.7
Inner replay buffer size	2000
Inner batch size	32
Inner learning rate	0.1
Inner target network update frequency	20
Inner reward discount $\mathit{\gamma }$	0.9
Inner episodes	100
Inner episode steps	100
External $\mathit{ϵ}$	0.5
External replay buffer size	2000
External batch size	10
External learning rate	0.1
External target network update frequency	20
External episodes	100
External episode steps	50
Allocation steps	1000

presents the detailed simulation parameter configuration of the External DQN, which includes additional parameters for the External System DQN on top of those for the Inner System DQN. Considering the runtime during the model training phase, the training iterations for the Inner DQN model are set to 100, with 100 steps per episode, while the External DQN is trained for 100 iterations, with 50 steps per episode.

Figure 10 shows the linear graph for action selection using the Inner External DQN method. Since there are 100 nodes, the method sets the range of action selection to [1–100]. Selecting one of the nodes with action indicates that the Inner DQN is used to optimize the parameters of this node. Through simulation, it is found that as the steps progress, each node finally converges to a fixed node value. The above indicates that the current system has converged to a stable value. After selecting the remaining actions, the rewards will no longer increase.

Figure 11 illustrates the line graph of PRR variation when using the Inner External DQN method. Since this study considers different SF allocation methods for nodes in complex environments, it is believed that node parameter allocation should be tailored to different environments. Figure 11a provides a detailed view of node allocation changes in a random environment. In a complex environment, the initial PRR of the system is relatively high: reaching convergence after 300 one-step iterations, indicating that the External DQN method gradually approaches optimality during iterations. Figure 11b shows the overall PRR variation line graph for the External DQN in a random environment. In a complex environment, the initial PRR is not high, but the overall trend is upward progression until convergence.

Because the proposed Inner External DQN method iterates using the SF-random initial allocation strategy, it significantly improves the overall system PRR compared to the strategy based only on the SF-random allocation method.

5. Conclusions

This paper addresses the issue of SF parameter allocation for LoRa network nodes in complex environments. Initially, it introduces a complex environmental model for LoRa networks that considers the impact of environmental factors on data transmission and proposes the research objective of maximizing system performance. Based on this, the paper presents two methods: the SF-random method, which allocates spreading factors randomly, and the Inner External DQN SF allocation method. The Inner External DQN SF method builds upon SF-random by employing inner and outer DQN allocation strategies, thereby addressing the issue of local optima that arises from traditional methods that focus only on individual node optimization. Due to the Inner DQN method’s ability to achieve node optimization through environmental exploration, the proposed method demonstrates good adaptability in complex environments.

The proposed model is simulated using the LoRaSim simulation platform; we include experimental validation of important parameters such as the selection of $ϵ$ using the $ϵ$-greedy method. The results show significant performance improvements compared to methods that only consider node parameter optimization or ignore the impact of complex environments. This method provides a more optimal system channel parameter allocation scheme. Depending on the current environment, it can allocate the most suitable parameters to each node, providing good physical channel resources for subsequent operations. Future research can build on the current framework by modifying environmental impact factors to adapt to different application scenarios to target more detailed application environments.