MADDPG-Based Deployment Algorithm for 5G Network Slicing

Abstract

One of the core features of 5G networks is the ability to support multiple services on the same infrastructure, with network slicing being a key technology. However, existing network slicing architectures have limitations in efficiently handling slice requests with different requirements, particularly when addressing high-reliability and high-demand services, where many issues remain unresolved. For example, predicting whether actual physical resources can meet network slice request demands and achieving flexible, on-demand resource allocation for different types of slice requests are significant challenges. To address the need for more flexible and efficient service demands, this paper proposes a 5G network slicing deployment algorithm based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG). Firstly, a new 5G network slicing deployment system framework is established, which measures resources for three typical 5G network slicing scenarios (eMBB, mMTC, uRLLC) and processes different types of slice requests by predicting slice request traffic. Secondly, by adopting the multi-agent approach of MADDPG, the algorithm enhances cooperation between multiple service requests, decentralizes action selection for requests, and schedules resources separately for the three types of slice requests, thereby optimizing resource allocation. Finally, simulation results demonstrate that the proposed algorithm significantly outperforms existing algorithms in terms of resource efficiency and slice request acceptance rate, showcasing the advantages of multi-agent approaches in slice request handling.

1. Introduction

With the significant growth in mobile data traffic, the widespread application of 5G network technology in various services has become increasingly important. The International Telecommunication Union (ITU) defines three typical service scenarios in the literature: enhanced mobile broadband (eMBB), ultra-reliable low-latency communication (uRLLC), and massive machine-type communication (mMTC) [1]. A core characteristic of 5G networks is their ability to provide different services on the same infrastructure. According to literature sources [2,3,4,5], deploying network slicing using Network Function Virtualization (NFV) can realize the advantages of differentiated services. These Virtual Network Functions (VNFs) ensure platform flexibility and scalability while reducing operational costs. Studies [6,7,8,9,10,11,12] have investigated the use of NFV technology in network slicing deployment. By monitoring the resource status of VNFs and considering factors such as load and bandwidth, this optimization can reduce network latency and improve resource utilization.

The literature [13,14,15,16] analyzes the scenario of virtual network mapping, optimizing the mapping process for nodes and links and modeling the Virtual Network Embedding (VNE) problem to meet node and link constraints while optimizing physical resource allocation to enhance deployment success rates. The literature [17,18,19,20,21] explores different solutions to the resource allocation problem using Markov chains, big data analysis, and queuing theory. The literature [22,23,24] employs a near-optimal Integer Linear Programming (ILP) approach to address the network slicing deployment problem. However, this method is slow in handling large-scale problem instances and can only be used in offline scenarios. Due to the excellent performance of the Deep Deterministic Policy Gradient (DDPG) algorithm in solving problems within continuous action spaces, the authors of [25,26,27] use the DDPG algorithm for dynamic resource allocation to optimize network performance. However, these studies have some limitations when using heuristic and reinforcement learning algorithms to address the separation between slice requests and resource allocation. They may face challenges in handling resource scheduling for multiple correlated requests and in adopting more conservative criteria to meet high reliability and demanding service requirements.

In order to efficiently implement network slicing requests on physical networks and provide high reliability and quality of service, this paper proposes a 5G network slicing deployment algorithm based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG). The algorithm aims to optimize the deployment process for different types of slicing requests. The main contributions of this paper are as follows: (1) A new 5G network slicing deployment system framework is established, which measures resources for three typical 5G network slicing scenarios (eMBB, mMTC, uRLLC) and addresses different types of slicing requests by predicting the traffic of slice requests. (2) The proposed algorithm utilizes the concept of multi-agent deep reinforcement learning, enhancing cooperation among agents. The algorithm focuses on training the action selection for slice requests and performs resource scheduling for each slicing request separately to achieve optimal resource allocation rewards. Simulation results show that the proposed algorithm performs excellently in terms of resource utilization and the slice deployment success rate, demonstrating its effectiveness and advantages.

2. System Model

As shown in Figure 1, the following two modules have been created for the deployment framework of 5G network slicing: a prediction module and a scheduling module.

• At present, the network slicing request arrives at the prediction module, which determines the user request based on the traffic prediction, and then chooses whether to accept or reject the network slicing request (NSR) by combining the amount of resources of the scheduling module and the state of the physical network at the previous moment.
• If the prediction module accepts the NSR, it sends a notification to the scheduling module to schedule the resource and embed it in the physical network, or otherwise refuses.
• Upon receiving the notification, the scheduling module sends a notification to the physical network at the next moment to virtualize the deployment targets for the three different types of slices, as well as the provisioning of resources.

2.1. Physical Network

We perform unification of the various hardware facilities into physical nodes that provide physical resources. The underlying physical network consists of a network of virtual machines and can be represented by an assignment undirected graph $G_S=(N_S,L_S)$, where $N_S$ is denoted as the set of physical nodes, $L_S$ is the set of bi-directional physical links. Each physical node $n_S\in N_S$ can be characterized by a set of attributes $R^n$ that help to define whether a specific physical node can host a virtual node.

2.2. NSR

Three types of NSR based on 5G services are considered: eMBB, mMTC, and uRLLC. We denote each NSR as an undirected graph $G_V=(N_V,L_V)$ where $N_V$ denotes the set of virtual nodes and $L_V$ is the set of virtual links in the NSR. The requirements of each virtual node or virtual link are typically related to the same parameters of the physical nodes or links. The legend for the symbols is shown in

Table 1. Symbol description table.

Notation	Description
$G_S$	Physical network
$N_S$	Physical nodes
$L_S$	Physical links
$G_V$	Virtual network
$N_V$	Virtual nodes
$L_V$	Virtual links
$C_s\left(n_s\right)$	The resource capacity of the physical node
$C_v\left(n_v\right)$	The resource capacity of the virtual node
$B_i\left(l_i\right)$	The bandwidth of the physical link
$B_v\left(l_v\right)$	The bandwidth of the virtual link

below:

2.3. System Framework for 5G Network Slicing Deployment

2.3.1. User Traffic Characteristics

Traffic prediction is based on the aggregation of each accepted tenant. Tenants may request different network slices according to their specific service requirements, without specifying the set of physical cells where their users are located. In fact, a particular vertical tenant may only require a small number of cells to provide specific services for its users. For example, the automotive industry may only need cells that cover suburban roads. Our approach involves classifying traffic requests according to relevant service requirements and geographic locations, thus enabling separate predictions for each cell and each slice. In our analysis, we initially assume that traffic requests are uniformly distributed across the entire network.

We assume the following traffic model. We assume different classes of traffic based on specific SLAs, as shown in

Table 2. Table of network slicing traffic types.

k	T	Type—5QI
0	10 ms	GBR—65
1	100 ms	GBR—1
2	50 ms	Non-GBR—79
3	300 ms	Non-GBR—6
4	10 ms	Delay Critical GBR—83

[28]. Let the traffic volumes of tenant $i$ for traffic $k$ be modeled as a point process, ${\zeta }_i^{(k)}={\sum }_{t=0}^T{\delta }_t{\displaystyle {\sum }_{c\in \psi }{r}_{i,c}^{(k)}}$, where ${\delta }_t$ denotes the Dirac metric for sample $t$ and $\Psi$ is the set of cell. We express each user’s traffic requests $r_{i,c}^{(k)}\left(t\right)$ as the required resource metric, and for each user $i$ express the aggregate traffic as $r_i^{(k)}={\displaystyle {\sum }_{c\in \psi }{r}_{i,c}^{(k)}}$.

2.3.2. Traffic Forecasting

In our model, a fundamental key assumption is that traffic requests follow a cyclical pattern, which is essential for applying time series forecasting algorithms. Based on the assumed periodicity, traffic forecasting is conducted based on an observed time window $T_0$ and is represented by a vector, $r_i^{(k)}=(r_i^{(k)}(t-T_0)$, $r_i^{(k)}(t-T_0-1),\cdots ,r_i^{(k)}(t))$. Then, we fix a future time window $T_F$ that provides the predicted traffic for the time period $[t+1,t+T_F]$, ${\widehat{r}}_i^{(k)}=({\widehat{r}}_i^{(k)}(t+1),{\widehat{r}}_i^{(k)}(t+2)$, $\cdots ,{\widehat{r}}_i^{(k)}(t+T_F))$, through a forecast function $f$. As the observed time window $T_0$ extends and more information is collected, the accuracy of traffic predictions for each user’s slice in the future time window $T_F$ increases.

Under the above assumptions, the system exhibits periodic behavior, where $W_s$ represents a season that repeats over time. Within a single season, we assume that the process ${\zeta }_i^{(k)}$ is stationary and ergodic [29]:

$${\overline{\mu }}_i^{(k)}={\displaystyle \frac{1}{Z}}{\displaystyle \sum _{z=0}^{Z}X[z]=\frac{1}{T}{\displaystyle \sum _{t=0}^T{r}_{i,k}[t]}}$$

(1)

where ${\overline{\mu }}_i^k$ expresses the average traffic request for the $k\text{-th}$ type of traffic of the $i\text{-th}$ user, $Z$ represents the number of units, and $r_{i,k}\left[t\right]$ defined as the $k\text{-th}$ type of traffic request for the $i\text{-th}$ user at time $t$.

To this end, we use the Holt–Winters (HW) forecasting method to analyze and predict future traffic requests associated with specific network slices across all selected cells. The definition of the forecasting function $f_H$ is as follows:

$$\begin{array}{l}{f}_H=R^{T_0+1}\to R^{T_F}\\ r_i^{(k)}\to {\widehat{r}}_i^{(k)}\end{array}$$

(2)

We denote a specific predicted traffic request ${\widehat{r}}_i^{(k)}\left(t\right)$ by ${\widehat{r}}_i^{(k)}$. We use the additive version of the HW forecasting model, as the seasonal effect does not depend on the mean traffic level of the observed time window but is instead added based on values predicted through level and trend effects. Following the standard HW procedure and assuming a seasonality frequency (W) based on traffic characteristics, we can predict such requests using the level $s_t$, trend $b_t$, and seasonal $u_t$ factors, as follows:

$$\begin{array}{l}{\widehat{r}}_{i,t+T_F}^{(k)}=s_t+b_t{T}_F+u_{t+T_F-W(k+1)}\\ s_t=\alpha (r_{i,t}^{(k)}-u_{t-W})+(1-\alpha )(s_{t-1}+b_{t-1})\\ b_t=\beta (S_t-s_{t-1})+(1-\beta )b_{t-1}\\ u_t=\gamma (r_{i,t}^{(k)}-s_{t-1}-b_{t-1})+(1-\gamma )u_{t-W}\end{array}$$

(3)

where $k$ denotes the integer part of $(T_F-1)/W$ and the set of optimal HW parameters $\alpha$, $\beta$, and $\gamma$ can be obtained during a training period employing existing techniques. We focus on how prediction errors and inaccuracies affect our network slicing solutions. Inaccurate predicted traffic values may lead to incorrect embedding of network slices, which in turn may fail to adapt to the system’s capacity, resulting in degraded service quality. Therefore, the training prediction error is further defined as follows:

$$m_{i,t}^{(k)}=r_{i,t}^{(k)}-{\widehat{r}}_{i,t}^{(k)}=r_{i,t}^{(k)}-(s_{t-1}+b_{t-1}+u_{t-1})$$

(4)

which can be computed during the training of the prediction algorithm. For any predicted value at time y, where a prediction interval $[{\widehat{l}}_{i,y}^{(k,\eta )}, {\widehat{h}}_{i,y}^{(k,\eta )}]$ can be derived for which the error probability of future traffic requests is ${\eta }_i^{(k)}$ for that particular network slice request. Given that our process ${\zeta }_i^{(k)}$ is iterative, assuming an optimal set of hardware parameters, for any predicted value at time $z$, we can derive the prediction interval $[{\widehat{l}}_{i,y}^{(k,\eta )}, {\widehat{h}}_{i,y}^{(k,\eta )}]$ with a certain probability ${\eta }_i^{\left(k\right)}$ for future traffic requests specific to our network slice. Therefore, it is considered that

$$\mathrm{Pr}\{{\widehat{l}}_{i,y}^{(k,\eta )}\le {\widehat{r}}_{i,y}^{(k)}\le {\widehat{h}}_{i,y}^{(k,\eta )}\}={\eta }_i^{(k)},\forall y\in [t+1,t+T_F]$$

(5)

where ${\widehat{h}}_{i,y}^{(k,\eta )}\left(\mathrm{or}{\widehat{l}}_{i,y}^{(k,\eta )}\right)={\widehat{r}}_{i,y}^{(k)}+(-){\Omega }_{\eta }\sqrt{V_{ar}(m_{i,y}^{(k)})}$, ${\Omega }_{\eta }$ denotes the one-tailed value of a standard normal distribution, $Var(m_{i,y}^{(k)})\approx (1+(y-1)){\alpha }^2[1+y\beta +{\displaystyle \frac{y(2y-1)}{6}}{\beta }^2]){\sigma }_m^2$, ${\eta }_i^{(k)}$ expresses the probability, and ${\sigma }_m^2$ is the variance of the one-step training prediction error over the observed time window, i.e., ${\sigma }_m^2=V_{ar}(m_{i,t}^{(k)})$.

Due to the requirements imposed by traffic SLAs, we focus only on the upper bound of the prediction interval as it provides the “worst-case” of a forecasted traffic level. Following the above, best-effort traffic requests with no stringent requirements can tolerate a prediction with a longer time pace that results in imprecise values. This makes the upper bound ${\widehat{h}}_{i,y}^{(k,\eta )}$ very close to the future value $r_{i,y}^{(k,\eta )}$ regardless of the error probability ${\eta }_i^{(k)}$; the number of predicted $y$ values is finite. On the other hand, when considering the need to meet bit rate traffic requirements within a shorter period, the forecasting process becomes more complex and requires more predicted values $y$. Such traffic needs to be modeled with a higher probability of prediction error probability ${\eta }_i^{(k)}$.

According to the traffic categories defined in Table 1, the traffic category $k=0$ requires a shorter prediction range than the other traffic categories, and therefore further predictions are necessary, for which an upper bound on the prediction probability error per user for that traffic category is derived. We define the maximum gain between sliced requests and predicted traffic requests as ${\widehat{d}}_i^{(k)}=\underset{y\in T_F}{\max }(R_i^{(k)}-{\widehat{r}}_{i,y}^{(k)})$. Then, the prediction error probability is as follows:

$${\eta }_i^{(k=0)}:{\Omega }_{\eta }\sqrt{Var(m_{i,y}^{(k=0)})}={\widehat{d}}_i^{(k=0)}$$

(6)

As soon as the potential gain ${\widehat{d}}_i^{(k=0)}$ becomes very large, we cap the one-tailed value ${\Omega }_{\eta }$ to 3.49, resulting in ${\eta }_i^{(k=0)}$ = 99.9%. Conversely, for the best-effort traffic ($k$ = 4), we compute the forecasting error probability ${\eta }_i^{(k=\left|K\right|)}$ = 50%, due to its more relaxed service. For the other traffic classes $k$, intermediate forecasting error probabilities ${\eta }_i^{(k)}$ are calculated from (6) by deriving ${\widehat{d}}_i^{(k)}$ values from the upper and the lower bound values.

2.3.3. Scheduling Model

We define the network slicing request as ${\omega }_i^{(k)}=\{R,T,i,k\}$, where $R$ is the amount of resources required, $T$ is the duration of the slice, $i$ denotes the tenant, and $k$ is the traffic class. In general, the user requests are denoted as $R_i^{(k)}(T_i)$ and the traffic classes are predicted for a fixed time window $T_F$. We assume that the physical network resources are a rectangular box that represents a limited amount of resources. where the width corresponds to $T_F$ and the height corresponds to the total amount of available resources $\vartheta$. The cost of each network slice request $s_i$ is proportional to the amount of resources requested $R_i$. The allocation of resources is an NP-hard problem.

The amount of resources required per time $y$ based on the predicted traffic type $k$ (with a given prediction error probability ${\eta }_i^{(k)}$) is denoted by the set ${\widehat{R}}_{i,y}^{(k)}={\widehat{h}}_{i,y}^{(k,\eta )}$, and ${\widehat{r}}_{i,y}^{(k)}$ denotes the requests of user $i$ for category $k$. Five traffic categories shown in Table 1 are considered, each characterized by a time window $T^{(k)}$ that identifies the duration $[y, y+T^{(k)}]$ over which the category should see the metric, which is shorter for highly demanding traffic categories and longer for more moderate categories. The scheduling module will give the appropriate amount of resources ${\widehat{R}}_{i,y}^{(k)}$ to meet the traffic metric within this length.

To ensure that the scheduling module expects the network slice traffic level to be no lower than the predicted traffic boundary, $r_{i,y}^{(k)}>{\widehat{R}}_{i,y}^{(k)},\forall y\in T_i$. The key goal of the scheduling module is to minimize the consumed resources, i.e., minimize the cost, while guaranteeing the SLA of the traffic within the network slice by setting a time window $T_s$ that will encompass all classes of time windows $T^{(k)}$, $T_s\gg T^{(k)}$. The mathematical model of the scheduling module is established as follows:

$$\begin{array}{l}\min {\displaystyle \sum _{n\in T_s}(z_{i,n}^{(k)}+\xi P_{i,n}^{(k)})}\\ s.t.({\displaystyle \sum _{n=yk}^{yk+T^{(k)}}{z}_{i,n}^{(k)}})\ge r_{i,y}^{(k)}{x}_i^{(k)},\forall y\in [0,{\displaystyle \frac{T_i}{T^{(k)}}}-1]\\ {\displaystyle \sum z_{i,n}^{(k)}\le \vartheta +}{P}_{i,n}^{(k)},\forall n\in T\\ z_{i,n}^{(k)}\in R,\forall i\in N,n\in T,k\in \tau \end{array}$$

(7)

where $\vartheta$ is the total capacity of physical resources, $P_{i,n}^{(k)}$ is the penalty generated by not meeting the SLA of the user’s network slicing service used for feedback prediction, $\xi$ is a larger constant factor set to ensure the reduction in $P_{i,n}^{(k)}$ always has the highest priority, $x_i^{(k)}$ is to determine whether to accept the request, acceptance is 1, and refusal is 0.

The prediction error probability for the general flow category can be derived from Equation (6):

$${\eta }_i^{(k)}=h_i^{(k)}{\Omega }_{\eta }\sqrt{V_{ar}(m_{i,y}^{(k)})}={\widehat{d}}_i^{(k)}$$

(8)

$h_i^{(k)}$ as a historical penalty function, is defined as follows:

$$h_i^{(k)}=m^{{\displaystyle \frac{nm}{W_c+nm}}}$$

(9)

where $nm$ is defined as the number of times the penalty is zero, $P_{i,j}^{(k)}=0,\forall j$. $W_c$ is defined as the length of the season in the forecasting process. The historical penalty function sets the control policy in order to control the system from SLA violations, derives a larger prediction error probability ${\eta }_i^{(k)}$ in the case of prediction failure, obtains a smaller gain, and prevents the waste of resources. The objective of setting three types of slices determines the metric ${\delta }_t$ of each user traffic request in the prediction model.

eMBB: Enhanced mobile broadband is a key application scenario of 5G technology, aiming to provide wide-area coverage and hotspot connectivity. In hotspot scenarios, where user density is high, there is a need for very high traffic capacity, while the requirements for mobility are lower and user data rates are higher. This type of slice does not require strict delay constraints or abundant resources. Therefore, the deployment goal of eMBB slices should be to maximize the remaining resources on physical nodes, which can be represented as follows:

$$\max [{\displaystyle \sum _{n_s\in N_s}{C}_s(n_s)-{\displaystyle \sum _{n_v\in N_v}{C}_v(n_v)\ast {\mu }_{n_v,n_s}}}]$$

(10)

where $n_s$ and $n_v$ represent the sets of physical nodes and virtual nodes, respectively, $C_s\left(n_s\right)$ denotes the resource capacity of physical node, $C_v\left(n_v\right)$ denotes the resource demand of virtual node, and ${\mu }_{n_v{n}_s}$ represents the mapping relationship of virtual node $n_v$ on physical node $n_s$.

mMTC: Massive machine-type communications (mMTC) primarily address the large-scale connectivity needs of Internet of Things (IoT) devices, characterized by a vast number of connected devices, typically transmitting low volumes of data with latency insensitivity. Due to the need to handle a large number of connections, there is a high demand for computational resources and low congestion rates. Consequently, this type of slice aims to minimize bandwidth usage on physical links. In other words, it should maximize the remaining bandwidth on physical links. Therefore, the deployment objective for mMTC slices can be expressed as follows:

$$\max [{\displaystyle \sum _{l_i\in L_i}{B}_i(l_i)}-{\displaystyle \sum _{l_v\in L_v}{B}_v(l_v)}*{\rho }_{l_v,l_i}]$$

(11)

where $l_i$ and $l_v$ represent the sets of physical link and virtual link, respectively, $B_i\left(l_i\right)$ represents the bandwidth of the physical link, $B_v\left(l_v\right)$ denotes the bandwidth of the virtual link, and ${\rho }_{l_v{l}_i}$ represents the number of virtual links corresponding to the physical link.

uRLLC: Ultra-reliable low-latency communication aims to provide extremely high reliability and very low communication latency to meet the needs of applications requiring high real-time performance and stability. uRLLC has strict latency requirements, and the deployment objective should be to minimize the slice’s latency. We translate latency time into hops, so minimizing latency is equivalent to minimizing the length of each physical path. Therefore, the deployment objective for uRLLC slices is as follows:

$$\min [{\displaystyle \sum _{l_v\in L_v}{\rho }_{l_v,l_i}}]$$

(12)

The constraints corresponding to the overall deployment objectives are as follows:

$${\displaystyle \sum _{n_v\in N_v}{\mu }_{n_v,n_i}}=1, \forall n_i\in N_i$$

(13)

$${\displaystyle \sum _{n_i\in N_i}{\mu }_{n_v,n_i}}\le 1, \forall n_v\in N_v$$

(14)

$${\displaystyle \sum _{l_i\in L_i}{B}_i(l_i)\le B_i}$$

(15)

where the constraints of the above equation are expressed as follows: (a) each virtual node is guaranteed to be mapped to only one physical node; (b) each physical node is guaranteed to host only one virtual node; (c) ensuring that the bandwidth occupied by each node does not exceed the total available bandwidth.

In order to meet the demands of the virtual nodes, it is necessary to add two constraints to ensure that the resource capacity of each physical node can meet the total demand of the virtual nodes mapped to it. At the same time, we ensure that the available computing resources of the physical nodes mapping the virtual nodes are not less than the demands of the virtual nodes, which can be expressed as follows:

$${\displaystyle \sum _{n_v\in N_v}{\mu }_{n_{v,}{n}_i}}\ast C_v(n_v)\le C_i(n_i), \forall n_i\in N_i$$

(16)

$${\displaystyle \sum _{n_i\in N_i}{\mu }_{n_{v,}{n}_i}}\ast C_i(n_i)\le C_v(n_v), \forall n_v\in N_v$$

(17)

3. Algorithm Design

We use multiple network slicing agents as multiple agents to update their own policies and undertake MDP training of multiple agents to ensure a stable environment. Markov decisions of N agents can be represented as multivariate groups $(S,A,r_1,r_2,\cdots ,r_n,p,\gamma )$, $S$ is the state space $S=\{s_1,\cdots ,s_n\}$, and $A$ is the action space $A=\{a_1,\cdots ,a_n\}$. In the whole system, $r_n$ is the payoff function of the nth slicing agent (the amount of resources that the slicing agent requests to expect to be allocated), and $p$ is the transfer probability, where all the agents are in the current state $s_n\in S$, take action $a_n\in A$, and transfer to a new state $s_n^{\prime }\in S$. The transfer probability becomes $p(s_n^{\prime }|s_n,a_1,\cdots ,a_n)$. Then, the learning goal of each agent is to maximize cumulative returns:

$$R_N={\displaystyle \sum _{\mathrm{n}=0}^{T_s}{\gamma }^n{r}_n^n}$$

(18)

where ${\gamma }_n$ represents the discount factor $\gamma$ raised to the power of the time step $n$. The discount factor, which typically takes values between 0 and 1, balances immediate and future rewards by assigning a higher weight to rewards closer to the present. $r_n^n$ represents the immediate reward obtained by the agent at time step $n$.

$T_s$ is the time window containing all categories of slice requests and $\gamma \in [0,1)$ is the discount factor. For each slicing agent, which is centrally trained and decentralized for execution, the policy of all the agents, denoted as $\mu =\{{\mu }_1,\cdots ,{\mu }_n\}$, and the action-value function of the agent $n$ is as follows:

$$Q_n^{\mu }(s,a)=E_{s^{\prime },r_n-E}[r_n+\gamma Q_n^{\mu }(s^{\prime },\mu (s^{\prime }))]$$

(19)

The gradient of the expected payoff $J_n=E[R_n]$ of an agent $n$ is as follows:

$${\nabla }_{{\theta }_n^{\mu }}{J}_n\approx E_{s,a-D}[{\nabla }_{{\theta }_n^{\mu }}{Q}_n(s,a|{\theta }_n^Q){|}_{a_n=\mu (s_n|{\theta }_n^{\mu })}]$$

(20)

where ${\theta }_n^Q$ represents the parameters of the policy network of agent $n$ and ${\theta }_n^Q$ represents the parameters of the action-value network of agent $n$. $D$ is the experience pool in which the historical information of each intelligence is stored. $s$ and $a$ represent the state and action, respectively. $\mu \left(s_n|{\theta }_n^{\mu }\right)$ represents the action generated by the policy network $\mu$ in state $s_n$. In the MADDPG-based 5G network slicing deployment algorithm, the actor network and the critic network of all agents are represented as $\mu =\{{\mu }_1,\cdots ,{\mu }_n\}, Q=\{Q_1,\cdots ,Q_n\}$.

3.1. State Space

The real-time resource situation of each physical node in the 5G physical network is the state space. We define two feature vectors ${\phi }_S(G_S,t)$ and ${\phi }_V(G_V,t)$ that represent the physical network and network slice request states at time $t$.

${\phi }_S(G_S,t)$ consists of the following characteristics: (1) Number of physical nodes $N^{\prime }$: the number of physical nodes $N^{\prime }(t)=\{\forall n\in N_S|A_N(n,t)>0\}$ available at time $t$. (2) Number of physical links $L^{\prime }$: The number of physical links $L^{\prime }(t)=\{\forall l\in L_S|A_L(l,t)>0\}$ with available bandwidth capacity at time t. (3) Free/occupied resources: the total amount of available/occupied resources in the physical network at time $t$. (4) Available/unavailable bandwidth: the total available/unavailable bandwidth at time $t$.

${\phi }_V(G_V,t)$ consists of the following characteristics: (1) Number of virtual nodes $N_V^{\prime }$: the number of virtual nodes at time $t$. (2) Number of virtual links $L_V^{\prime }$: the number of virtual links at time $t$. (3) User’s traffic request $r_i^{(k)}$: the request expressed as a specific requested resource metric. (4) Traffic category corresponding to the type of slice request.

3.2. Action Space

The action at every moment t contains two parts. One is represented by $x_i^{(k)}$, accepting a slice request gives a score of as 1 and otherwise 0, and the other is the resource allocation for the three slice types. Therefore, each action of the action set is denoted as $a=\{x,\varphi \}$.

$\varphi \in \{{\varphi }_1, {\varphi }_2, {\varphi }_3\}$; the resource allocation action for the three types of slices first requires the selection of nodes in the physical network to host the virtual nodes for the slicing requests while satisfying the capacity requirements. Considering the degree and centrality of nodes, we first assess the node importance. The degree and centrality of nodes are normalized and the node degree is expressed as follows:

$$d_i^{\prime }={\displaystyle \frac{d_i}{N-1}}$$

(21)

Node centrality is expressed as follows:

$$b_i^{\prime }={\displaystyle \frac{2b_i}{(N-1)(N-2)}}$$

(22)

Based on the above normalization criteria, combined with the resources and structure of the nodes, the node importance is defined as follows:

$$N_R(i)=C(i)\ast {\displaystyle \sum _{l\in L_{\mathrm{i}}}B(l)}$$

(23)

$$N_I(i)=N_R(i)\ast ({\displaystyle \frac{d_i^{\prime }+b_i^{\prime }}{2}})$$

(24)

$N_R(i)$ represents the resources of the node, including the capacity $C(i)$ of the node; the set $L_i$ of links connected to node $i$; and the available bandwidth $B(l)$ of link $l$. Based on the importance of the nodes, Sort the virtual nodes using Algorithm 1, and then perform the node mapping.

Algorithm 1. Virtual node sorting algorithm

Input: $N_v$, the set of virtual nodes requested by the slice.

Output: $N_I(i)$, the sequence of the sorted virtual nodes.

(1) Calculate the importance degree $N_I(i)$ of each virtual node; (2) Sort the virtual nodes in non-increasing order according to their importance values; (3) (Select the virtual node with the highest importance value as the root node, labeled as $P$; (4) Traverse the network slice request graph using the BFS algorithm to obtain the BFS tree $T$; (5) Sort the virtual nodes in each layer of $T$ in non-increasing order according to their importance values; (6) Return $N_I(i)$.

Secondly, for the three types of slicing requests, the node mapping and link mapping are also classified into three specific types. We use Algorithms 2–4 to complete this process, and the details of the algorithms are as follows:

Algorithm 2. eMBB slice deployment algorithm

Input: Network slicing request and physical network.

Output: Success or failure of the node mapping process.

(1) Sort virtual nodes $v\in N_{v_i}$, according to Algorithm 1; (2) For $v\in N_{v_i}$, do; (3) If $v$ is the root node; (4) Map $v$ to the physical node with the highest physical resource capacity; (5) Else; (6) Find the parent virtual node $P$ of virtual node $v$ in tree $T$; (7) Find the physical node $PP$ to which $P$ is mapped; (8) Set the physical node adjacent to $PP$ to $Q$; (9) Map to the physical node in $Q$ with the highest physical resources; (10) End; (11) End. (12) Sort the virtual link $(i,j)\in L_{v_i}$ in a non-increasing order according to the bandwidth requirements of the network slice request; (13) For $(i,j)\in L_{v_i}$, do; (14) Remove the physical links that cannot meet the bandwidth requirements; (15) Find the physical nodes a, b that map virtual nodes $i$, $j$; (16) Map the virtual link $(i,j)$ to the physical shortest path between $a$, $b$; (17) End.

Algorithm 3. mMTC slice deployment algorithm

Input: Network slicing request and physical network.

Output: Success or failure of link mapping process.

(1) Sort the virtual links in non-increasing order according to the bandwidth requirements of the network slicing requests $(i,j)\in L_{v_i}$; (2) For $(i,j)\in L_{v_i}$, do; (3) Find all physical candidate paths AB that can map virtual link $(i,j)$; (4) For $ab\in AB$, do; (5) Find the physical link $l\in ab$ with the minimum bandwidth resource; (6) End; (7) Select the path $mal$ with the maximum value of $l$; (8) Map virtual link $(i,j)$ to path $mal$; (9) Map virtual nodes $i$, $j$ to physical source and physical destination nodes on path $ml$; (10) End.

Algorithm 4. uRLLC slice deployment algorithm

Input: Network slicing request and physical network.

Output: Success or failure of link mapping process.

(1) Sort the virtual links in non-increasing order according to the bandwidth requirements of the network slicing request $(i,j)\in L_{v_i}$; (2) For $(i,j)\in L_{v_i}$, do; (3) Find all physical candidate paths AB that can map virtual link $(i,j)$; (4) Select the path $mil\in AB$ with the minimum number of hops; (5) Map virtual link $(i,j)$ to $mil$; (6) Map virtual nodes $i$, $j$ to physical source and physical destination nodes on path $mil$; (7) End.

3.3. Reward Functions

The learning of the whole system is driven by the reward function. The agent maximizes the reward by interacting with the environment. When the agent decides to access the network slicing request, it chooses the type of slicing request to improve the efficiency of resource utilization and reduce the cost. When the agent rejects the slicing request, it directly ends the action without wasting time. The reward function is as follows:

$$r_x\{\begin{array}{c}{z}_{i,n}^{\left(k\right)}+\xi P_{i,n}^{\left(k\right)}, x=1\\ 0, x=0\end{array}$$

(25)

Based on the state space and action space mentioned above, the training is undertaken with the goal of obtaining the optimal reward. The flow of the MADDPG-based 5G network slicing deployment algorithm comprises two parts of centralized training and decentralized decision-making, as shown in Figure 2.

During training, the intelligence runs at each time slot j and selects the sliced traffic requests $r_i^{(k)}$ in the allowed network slicing requests $x_i^{(k)}=1$ by the prediction module reward function by giving higher priority to network slicing requests for requests closer to the deadline $T_i^{(k)}$. Slicing requests $r_i^{(k)}$ that are not fully satisfied are reserved for the next time slot $j$ + 1. If it is not delivered within the last time $T_i^{(k)}$, a penalty value $P_{\mathrm{i},\mathrm{j}}^{(k)}$ is set. The prediction parameters are dynamically adjusted by the penalty value. The centralized training algorithm for MADDPG-based 5G network slicing deployment is shown in Algorithm 5.

Algorithm 5. MADDPG-based centralized training algorithm for 5G network slicing deployment

Input: Virtual network parameters, physical network parameters, network slice request, $S$, $r_i^{(k)}$, and $\vartheta$.

Output: Target actor network parameters ${\theta }^{\mu }$.

(1) Randomly initialize the actor network and critic network parameters ${\theta }^{\mu }$, ${\theta }^Q$, $z_{i,n}^{(k)}=0, P_{i,n}^{(k)}=0; \forall i\in N$; (2) Initialize the experience replay area; (3) Set the initialization state of all agents $s^0=\{s_1^0,\cdots ,s_n^0\}$; (4) For $r_{i,y}^{(k)},T_i^{(k)}$, do; (5) If $x_i^{(k)}==1$; (6) If $(T_i^{(k)}-n)\ge 0$; (7) $A\sim {\mu }_i={\displaystyle \frac{(r_{i,y}^{(k)}-{\displaystyle \sum _{t=0}^n{z}_{i,t}^{(k)}})}{(T_i^{(k)}-n)\vartheta }}$; (8) Else (9) $P_{i,n}^{(k)}=r_i^{(k)}-{\displaystyle \sum _{t=0}^n{z}_{i,t}^{(k)}}$; (10) $r_i^{(k)}=0$; (11) End if; (12) End if; (13) $S=\vartheta$, performing actions, obtaining rewards, and observing the next state; (14) Place the generated states, actions, and returns in the experience replay area; (15) While $S>0$, do; (16) Execute ${\mu }_i$ in action space $A$, with rewards obtained in non-increasing order; (17) Place the highest award in H; (18) $z_{i,n}^{(k)\prime }=\min \{r_i^{(k)},{\displaystyle \frac{S}{\left|H\right|}}\}$; (19) $S^{\prime }=S-{\displaystyle \sum _{i\in N}{z}_{i,n}^{(k)\prime }}$; (20) $r_i^{(k)\prime }=r_i^{(k)}-z_{i,n}^{(k)\prime }$; (21) End while; (22) Update the critics’ network; (23) Update the network of actors; (24) Update the target network; (25) End for.

The decentralized decision process will take the centrally trained policy input and output the value of the action Q to determine the good or bad choice of the action to make a specific action selection. The specific algorithm steps are shown in Algorithm 6:

Algorithm 6. Decentralized decision-making algorithm for 5G network slicing deployment based on MADDPG

Input: Virtual network parameters, physical network parameters, target network parameters ${\theta }^{\mu }$.

Output: $Q$-value.

(1) Import the parameter $q$${\theta }^{\mu }$; (2) Observe the state $S^{\prime }=\{s_1^{\prime },\cdots ,s_n^{\prime }\}$; (3) Select actions based on the action set $a^{\prime }=\{x^{\prime },{\varphi }^{\prime }\}$ and allocate resources to the three types of slicing requests; (4) Execute action $a^{\prime }$ and calculate the system return; (5) Return the $Q$-value.

4. Simulation Result

Time complexity and space complexity help to evaluate performance and resource requirements, compare the efficiency of different algorithms, optimize performance bottlenecks, manage limited computing resources, determine the feasibility of algorithms under specific hardware and time constraints, and theoretically prove the correctness and optimality of algorithms. To this end, we analyzed the time complexity and space complexity of the six proposed algorithms as shown in

Table 3. Complexity analysis of the algorithm.

Algorithm	Time Complexity	Space Complexity
Sorting	$O\left(n+n\log n\right)$	$O\left(n\right)$
eMBB Slice Deployment	$O\left(mn+n\log n\right)$	$O\left(n+m^2\right)$
mMTC Slice Deployment	$O\left(lpm\right)$	$O\left(lpm\right)$
uRLLC Slice Deployment	$O\left(lp+l\log l\right)$	$O\left(lpm\right)$
Centralized Training	$O\left(Tn\log n\right)$	$O\left(Tn\right)$
Decentralized Decision-Making	$O\left(n\right)$	$O\left(n\right)$

, where $n$ represents virtual nodes, $m$ represents physical nodes, $T$ represents the time steps required for the training process, $l$ represents virtual links, and $p$ is the number of candidate paths.

This section presents the simulation results of the 5G network slicing deployment algorithm based on MADDPG. In the underlying physical network, various physical resources are unified as the physical resources of nodes. We used Python for simulation, with specific parameter settings as follows: the time window is defined as 3600 s and 7200 s, the number of physical nodes is 30, 60, and 90, the capacity of physical nodes is in the range of [20, 50], the link bandwidth is in the range of [20, 50], the virtual node capacity requirement is in the range of [5, 25], and the virtual link bandwidth requirement is in the range of [5, 25]. In our analysis, we mainly focus on the dynamic analysis of the system, the impact of the prediction module on the system, the resource efficiency of the three types of network slice requests, and the overall system operational efficiency. By comparing our proposed MADDPG algorithm with the DDPG algorithm and the ILP algorithm in terms of resource efficiency and slice request acceptance rate, the simulation results demonstrate the advantages of multi-agent deployment in slice requests. The specific simulation results are as follows:

In the test process for system returns, the impact of prediction on the system is compared. As shown in Figure 3 in the training process for the system returns $R=\vartheta -{\displaystyle {\sum }_{i,k}{z}_{i,n}^{(k)}}, n\in T_F$, the prediction process provides traffic request information to the scheduling module, and based on this information the scheduling module selects whether to access the network slicing request to ensure the quality of service and the rational use of resources and reduce costs.

Figure 4 compares the resource efficiency of three types of network slicing requests (eMBB, mMTC, and uRLLC) using three different algorithms. As the number of slicing requests and the mapped physical paths increase, the resource efficiency decreases in all three cases. However, multi-agent collaboration significantly improves the resource allocation efficiency for slicing requests. The other two algorithms are less efficient than the one proposed in this paper, indicating its superiority in achieving on-demand resource allocation. Specifically, our algorithm allocates resources more effectively, ensuring the quality of service for various slicing requests while maximizing the utilization of the available physical resources.

Figure 5 compares the running times of three types of network slicing requests (eMBB, mMTC, and uRLLC). As shown in the figure, eMTC has the longest running time, followed by eMBB, and uRLLC has the shortest running time. This is consistent with our time complexity analysis of the algorithms. In the mMTC algorithm, node mapping and link mapping need to consider more physical nodes and physical links, which increases its time complexity. In the uRLLC algorithm, since the focus is on selecting the path with the fewest hops, the time complexity of path selection and link mapping is usually lower.

Figure 6 compares the performance of the proposed MADDPG algorithm with two other algorithms (DDPG and ILP) in terms of network slicing request acceptance rates. The results show that as the number of slicing requests increases, resource consumption also increases, while the acceptance rate tends to decrease. Due to the predictive capabilities of the prediction module and the presence of the experience replay area, the proposed algorithm significantly outperforms the other two algorithms in accepting slicing requests. This advantage makes the MADDPG algorithm particularly effective in handling complex network environments and high-load conditions, thereby helping to maintain a high level of service quality.

Figure 7 compares the running time of the proposed MADDPG algorithm with the other two algorithms (DDPG and ILP) as the number of network slicing requests increases. The results show that although our proposed algorithm is slightly inferior to the other two algorithms in terms of running time, it has significant advantages in resource allocation efficiency and acceptance rate. This is because MADDPG and DDPG both require multiple iterations for training, whereas ILP solves for the optimal solution in a single run, resulting in a shorter solving time under the same conditions. MADDPG needs to train multiple agents simultaneously, each with its own policy and value network, significantly increasing the computational complexity and time. However, our proposed algorithm significantly outperforms the other two algorithms in other aspects, making it worthwhile to consider the increased running time in exchange for improved network efficiency.

5. Conclusions

This paper proposes a 5G network slicing deployment algorithm based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG), aimed at efficiently implementing network slicing requests while providing high reliability and high-quality service. Through prediction and scheduling modules, the algorithm can deploy and allocate resources for three typical slicing requests (eMBB, mMTC, uRLLC). Utilizing the concept of deep reinforcement learning, slicing agents act as intelligent bodies, taking the real-time resource status of the underlying physical network as the state space and using admission requests and resource allocation as actions. The optimal actions are produced through centralized training, and optimal returns are achieved through decentralized decision-making in interaction with the environment. The final simulation results show that the proposed algorithm performs excellently in terms of resource utilization and the slice deployment success rate, demonstrating its effectiveness and advantages. By improving network efficiency, this algorithm effectively implements network slicing requests on physical networks, showing significant application prospects. However, our algorithm still has room for improvement in terms of running time. In future work, we will optimize the algorithm process to reduce its time complexity.