Knowledge-Assisted Actor Critic Proximal Policy Optimization-Based Service Function Chain Reconfiguration Algorithm for 6G IoT Scenario

Abstract

Future 6G networks will inherit and develop Network Function Virtualization (NFV) architecture. With the NFV-enabled network architecture, it becomes possible to establish different virtual networks within the same infrastructure, create different Virtual Network Functions (VNFs) in different virtual networks, and form Service Function Chains (SFCs) that meet different service requirements through the orderly combination of VNFs. These SFCs can be deployed to physical entities as needed to provide network functions that support different services. To meet the highly dynamic service requirements in the future 6G Internet of Things (IoT) scenario, the highly flexible and efficient SFC reconfiguration algorithm is the key research direction. Deep-learning-based algorithms have shown their advantages in solving this type of dynamic optimization problem. Considering that the efficiency of the traditional Actor Critic (AC) algorithm is limited, the policy does not directly participate in the value function update. In this paper, we use the Proximal Policy Optimization (PPO) clip function to restrict the difference between the new policy and the old policy, to ensure the stability of the updating process. We combine PPO with AC, and further bring the historical decision information as the network knowledge to offer better initial policies, to accelerate the training speed. We also propose the Knowledge = Assisted Actor Critic Proximal Policy Optimization (KA-ACPPO)-based SFC reconfiguration algorithm to ensure the Quality of Service (QoS) of end-to-end services. Simulation results show that the proposed KA-ACPPO algorithm can effectively reduce computing cost and power consumption.

1. Introduction

Looking towards 2030 and beyond, 6G will provide lower latency and more reliable end-to-end connectivity to power Industry 4.0 to Industry 5.0 [1]. The network environment and service requirements are highly dynamic in future 6G IoT scenarios. To cope with highly dynamic service requirements, 6G networks will inherit and develop NFV architecture. With the NFV-enabled network, it becomes possible to establish different virtual networks within the same infrastructure and create different VNFs in different virtual networks. When the service requirements arrive, VNFs are combined in an orderly manner to form SFCs to meet different service requirements. In this way, SFC deployment based on mobile edge computing and network function virtualization can be a viable solution for providing flexible and controlled network services [2]. When the service requirement changes, the corresponding SFC needs to be reconfigured to guarantee the QoS requirements. To meet the low latency requirements of 6G services, highly flexible and efficient SFC orchestration and reconfiguration algorithms have become a crucial problem.

The SFC reconfiguration problem involves the study of how to modify, re-organize, and monitor the deployed SFC to cope with network dynamic changes and meet new service requirements. SFC reconfiguration is always implemented by VNF migration. Reference [3] modeled the SFC reconfiguration problem as an Integer Linear Programming (ILP) formulation and proposed a greedy-based heuristic algorithm to solve the problem Reference [4] proposed a threshold-dependent scalable cluster VNF migration algorithm to minimize the cost of embedding VNF under the condition of satisfying delay constraints. Reference [5] described the VNF migration problem as a new graph theory problem and proposed an efficient heuristic algorithm based on dynamic programming to effectively alleviate dynamic traffic and reduce total traffic costs. Reference [6] studied the reconfiguration problem of a set of SFCs with different priorities. A polynomial time heuristic algorithm was proposed to quickly deploy emergency SFCs while meeting the requirements of emergency SFC input traffic and resource constraints. Another precise algorithm was also proposed to achieve maximum profit for service providers. Reference [7] aimed to minimize the total migration cost and used a temporal conventional network to predict network traffic, proposing a fast and efficient heuristic VNF migration algorithm.

Nowadays, with the rapid development of Artificial Intelligence (AI), deep learning has been widely investigated and has shown advantages in coping with this type of dynamic problem. Reference [8] proposed a DRL-based algorithm to provide fast VNF migration decisions in highly dynamic environments, with the goal of minimizing the weighted total latency and cost of VNF migration. Reference [9] used a mixed-density neural network to accurately model complex user migration patterns in reality, in order to support the prediction of user edge cloud access probabilities and minimize the sum of operational costs and potential losses caused by downtime. Reference [10] proposed a deep Dyna-Q approach to solve the SFC reconfiguration problem under the premise of guaranteeing QoS and resource constraints. Reference [11] proposed a SFC management scheme based on the prediction of service. Reference [12] proposed the Dueling Double Deep Q Network (D3QN)-based SFC orchestration scheme to ensure QoE of users in the highly dynamic and resource-constrained Unmanned Aerial Vehicle (UAV) scenario.

The aforementioned research demonstrates the tremendous potential of deep learning algorithms. The AC algorithm, as a policy-based algorithm, has been used in many dynamic optimization problems. However, the efficiency of classical AC is limited in that the policy does not directly participate in the value function update. So, we use the principle of PPO, that is, proximal ratio pruning, to limit the magnitude of policy updates, and form the ACPPO algorithm. Based on this, we further utilize the historical decision information as the network knowledge to offer better initial policies and accelerate the training speed.

The main contributions of this paper are synthesized as follows:

• We formulate the SFC reconfiguration as a VNF migration problem, aiming to minimize the migration cost, to cope with the highly dynamic service requirements in 6G IoT scenarios.
• We use proximal ratio pruning to limit the magnitude of policy updates, reducing the impact of coupling relationship in the AC algorithm, and combine the advantages of the AC and PPO algorithm to solve the VNF migration problem. And we further introduce the 6G knowledge base, which is dynamic updating by saving and updating the excellent historical policies, to expand the action space and formulate the KA-ACPPO algorithm.
• Simulation results show that our proposed KA-ACPPO can effectively reduce computing cost and power consumption.

This paper is organized as follows. In Section 1, the system model is presented and the problem is formulated. Section 2 proposes the KA-ACPPO algorithm. Section 3 gives the simulation results. Finally, Section 4 concludes the paper and gives future research directions.

2. System Model and Problem Formulation

2.1. Network Model

Consider the 6G IoT scenario. Service requirements are satisfied by SFC orchestration. SFC orchestration is complemented by global control with the help of the 6G knowledge base. When the device produces a service request, it is assigned to the closed-edge node and communicates with the top-level SDN controller and NFV manager. The VNF manager complements the VNF orchestration, forming an ordered set of VNFs, that is, the SFC deployed policy. Finally, the policy is issued to the infrastructure layer to deploy the network functions to meet the service requirements. And when the service requirements change, SFC can be reconfigured through VNF migration, for example, VNF is migrated from one edge node to another edge node.

It is shown in Figure 1 that the edge nodes and the physical links in the infrastructure layer are expressed as the undirected graph $G=(X,L)$, in which X and L, respectively, describe the set of the edge nodes and the physical links. The available computing resource of the edge node $x\in X$ can be expressed as $C_x^{CPU}$. $l_{x,y}$ describes the physical link between the edge node x and the edge node y, in which $x,y\in X$ and $x\ne y$. SFC reconfiguration is implemented through VNF migration, so the physical link needs to reserve bandwidth B for VNF migration. In fact, every SFC can be expressed as an ordered set of VNF, denoted as $V=\{v_1,v_2,\cdots ,v_{\left|V\right|}\}$. We divide IoT services into two categories: delay-sensitive services and delay-tolerant services. The corresponding SFC requests can be denoted as delay-sensitive request $j\in J$ and delay-tolerant request $k\in K$, and the corresponding computing resource provided by the edge nodes can be, respectively, denoted as $b_j$ and $b_k$. This paper ensures the QoS of service requests by controlling the reconstruction costs of VNF migration in three aspects: computing cost, power consumption cost, and delay cost.

For clear presentation, we summarize the notations used in the following formulation in

Table 1. Notations used in this paper.

Notation	Description
X	the set of edge nodes
L	the set of physical links
$l_{xy}$	the link between edge node x and edge node y
$j\in J$	the index of delay-sensitive requests
$k\in K$	the index of delay-tolerant requests
$b_j$	computing resource allocated to request j
$b_k$	computing resource allocated to request k
$z_x$	the computing resource consumption of edge node x
$U_x^V$	the variable of SFC deployment
$W_x^V$	the variable of SFC reconfiguration
$P_x$	the power consumption cost
${\lambda }_j,{\lambda }_k$	the arriving rates of delay-sensitive and delay-tolerant requests
${\lambda }^{\prime }$	total arriving rate of requests
${\mu }^{\prime }$	the serving rate of VNF
${\rho }^{\prime }$	the traffic intensity
$D_q$	the queuing delay of SFC requests
$D_j$	the delay of the delay-sensitive SFC requests
${\beta }_j$	the data size of jth delay-sensitive SFC requests
$D_m$	the total delay of VNF migration caused by the position change of VNFs
$D_o$	the additional delay caused by prolonged downtime
$Q_j$	QoS degradation rate of jth delay-sensitive SFC requests

.

2.2. The Computing Cost of VNF Migration

SFCs are deployed in the edge node and are allocated computing resources from the edge node to meet the service requirements. To represent the computing resource consumption of node x, we denote the decision-making variables of SFC reconfiguration as follows:

$${\theta }_{x,j}=\left\{\begin{array}{cc}1\hfill & \mathrm{i}\mathrm{f} \mathrm{r}\mathrm{e}\mathrm{s}\mathrm{o}\mathrm{u}\mathrm{r}\mathrm{c}\mathrm{e} b_j \mathrm{i}\mathrm{s} \mathrm{a}\mathrm{l}\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{s} \mathrm{t}\mathrm{o} \mathrm{r}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t} j\hfill \\ 0\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}\right.,$$

(1)

$${\theta }_{x,k}=\left\{\begin{array}{cc}1\hfill & \mathrm{i}\mathrm{f} \mathrm{r}\mathrm{e}\mathrm{s}\mathrm{o}\mathrm{u}\mathrm{r}\mathrm{c}\mathrm{e} b_k \mathrm{i}\mathrm{s} \mathrm{a}\mathrm{l}\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{s} \mathrm{t}\mathrm{o} \mathrm{r}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t} k\hfill \\ 0\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}\right..$$

(2)

The computing resource consumption of the edge node x can be denoted as follows:

$$z_k=\sum _{j=1}^J{\theta }_{x,j}{b}_j+\sum _{k=1}^K{\theta }_{x,k}{b}_k.$$

(3)

During the VNF migration process, only the CPU state between the initial node and the target node is considered. Therefore, the computing cost of VNF migration can be denoted as follows

$$Cos{t}_1=\sum _{x=1}^X\sum _{v=1}^V{W}_x^V(1-U_x^V)z_k,$$

(4)

where $U_x^V$ is the variable of SFC deployment, and $W_x^V$ is the variable of SFC reconfiguration. For example, if the first VNF of the current SFC is not deployed at edge node 1, it can be denoted as $U_1^1=0$. If the VNF is migrated to the edge node 1, it can be denoted as $W_1^1=1$, and $W_1^{1,1}(1-U_1^{1,1})=1$, which means the VNF is migrated.

2.3. The Power Consumption Cost of VNF Migration

VNF migration can cause changes in the working status of certain edge nodes, resulting in additional power consumption. Power consumption is related to the usage of computing resources at edge nodes. Based on the current SFC deployment policy, the power consumption of the edge node x can be denoted as [13]:

$$P_x=P(0\%)+(P(100\%)-P(0\%))(2z_x-{\left(z_x\right)}^{1.4}),$$

(5)

where $P(0\%)$ denotes the power consumption of the edge nodes in idle modes, and $P(100\%)$ denotes the power consumption of the edge nodes in full-load modes.

Therefore, the power consumption cost of VNF migration can be denoted as:

$$Cos{t}_2=\sum _{x=1}^X\sum _{v=1}^V{W}_x^V(1-U_x^V)P_x.$$

(6)

2.4. The Delay Cost of VNF Migration

In this paper, the delay-sensitive services and the delay-tolerate services are both modeled as $M/M/1$ queuing, where the arriving rates are, respectively, depicted as ${\lambda }_j$ and ${\lambda }_k$. And the total arriving rate at edge node x can be denoted as follows:

$${\lambda }^{\prime }=\sum _{j=1}^J{\lambda }_j+\sum _{k=1}^K{\lambda }_k.$$

(7)

The CPU working frequency of the edge node x is denoted as F, and the serving rate of VNF can be denoted as:

$${\mu }^{\prime }=z_{x}F.$$

(8)

And the traffic intensity can be denoted as:

$${\rho }^{\prime }={\displaystyle \frac{{\lambda }^{\prime }}{{\mu }^{\prime }}}.$$

(9)

Therefore, the queuing delay of the SFC requests can be denoted as:

$$D_q={\displaystyle \frac{{{\rho }^{\prime }}^2}{\lambda (1-{\rho }^{\prime })}}.$$

(10)

Simply, only the queuing delay of the delay-sensitive SFC requests is considered, which is denoted as:

$$D_j={\displaystyle \frac{{\beta }_j}{b_j}},$$

(11)

where ${\beta }_j$ denotes the data size of the jth delay-sensitive SFC request.

Considering the network overhead of the infrastructure layer and the stability of network services, the impact of VNF migration latency and downtime should be minimized to the greatest extent possible. The VNF migration latency is related to the position change of VNF, and the number of migrated VNF at edge node x can be denoted as:

$${\mu }_x=\sum _{v=1}^V|W_x^V-U_x^V|.$$

(12)

And the number of VNF migration between edge node x and edge node y can be denoted as:

$${\mu }_{x,y}=\sum _{v=1}^V{W}_x^V(1-U_y^V).$$

(13)

The corresponding delay of VNF migration between edge node x and edge node y can be denoted as:

$$D_{x,y}^v={\displaystyle \frac{d_j}{B}}{W}_x^v{U}_y^v\underset{(x,y)\in X}{max}({\mu }_x,{\mu }_{x,y},{\mu }_y).$$

(14)

The total delay of VNF migration caused by the position change of VNFs can be denoted as:

$$D_m=\underset{(v,x,y)}{max}{D}_{x,y}^v.$$

(15)

Downtime refers to the time period during which the migrated VNF is unresponsive, as the VNF state is either migrating from the initial node to the target node or the network has not yet aggregated. Therefore, it is necessary to minimize the possibility of network service interruption caused by prolonged downtime. Using the current resource utilization rate of network edge nodes as a penalty factor to calculate the additional delay caused by excessive downtime, the CPU utilization rate of current network edge node x can be expressed as:

$${\rho }_x={\displaystyle \frac{z_x}{C_x^{CPU}}}.$$

(16)

The additional delay caused by prolonged downtime is denoted as

$$D_o=\sum _{x=1}^X\sum _{v=1}^V(D_j+D_q)W_x^{v,s}(1-U_x^{v,s}){\rho }_x.$$

(17)

When the downtime of VNF migration is too long, it can lead to delay-sensitive SFC requests not being processed in a timely manner during migration, resulting in QoS degradation of delay-sensitive SFC requests. The QoS degradation rate of delay-sensitive SFC requests caused by long VNF downtime can be expressed as:

$$Q_j={\displaystyle \frac{D_o}{D_q+D_j+D_m+D_o}}.$$

(18)

Taking into account the total latency of VNF migration and the additional latency caused by excessive downtime, the latency cost of VNF migration can be expressed as:

$$Cos{t}_3=D_m+D_o.$$

(19)

2.5. Problem Formulation

The total cost of SFC reconfiguration in 6G IoT scenario can be denoted as:

$$Cost={\displaystyle \frac{Cos{t}_1}{{\sigma }_1}}+{\displaystyle \frac{Cos{t}_2}{{\sigma }_2}}+{\displaystyle \frac{Cos{t}_3}{{\sigma }_3}},$$

(20)

where ${\sigma }_1$, ${\sigma }_2$, and ${\sigma }_3$ are normalized constants. In this paper, we minimize the total cost of SFC reconfiguration, and formulate the problem as follows

$$\underset{U_x^V,W_x^V}{min}(Cost={\displaystyle \frac{Cos{t}_1}{{\sigma }_1}}+{\displaystyle \frac{Cos{t}_2}{{\sigma }_2}}+{\displaystyle \frac{Cos{t}_3}{{\sigma }_3}}),$$

(21)

Satisfy the constraint:

$$\begin{array}{c}\left\{\begin{array}{cc}\hfill & C_1:z_x\le C_x^{CPU},\forall j\in X\hfill \\ \hfill & C_2:\sum _{x=1}^X{W}_x^V=v,\forall v\in V\hfill \\ \hfill & C_3:D_q+D_j+D_m+D_o\le D,\forall v\in V,\forall a,y\in X\hfill \\ \hfill & C_4:z_{x}F-(\sum _{j=1}^J{\lambda }_j+\sum _{k=1}^K{\lambda }_k)\ge 0,\forall x\in X,\forall j\in J,\forall k\in K\hfill \end{array}\right.,\hfill \end{array}$$

(22)

where $C_1$ ensures that the edge node x would not be overloaded. $C_2$ ensures that reasonable quantity of VNFs are deployed in every SFC. $C_3$ is the constraint of delay constraint of the delay-sensitive services. $C_4$ ensures the stability of the SFC request queueing.

3. Knowledge-Assisted ACPPO Algorithm

In this section, the Markov Decision Process (MDP) is used to model the above problem. The MDP can be simply defined as $<S,A,R>$, where state S, action A, and reward R are, respectively, defined as follows.

3.1. State

Define the state S as system state at time t, which can be written as follows:

$$s\left(t\right)=\{c_j\left(t\right),b_k\left(t\right),z_x\left(t\right)\},$$

(23)

where $c_j\left(t\right)$ and $b_k\left(t\right)$, respectively, denote the resource requirements of delay-sensitive and delay-tolerate SFC requests, which are detailed as follows:

$$c_j\left(t\right)=\{c_1\left(t\right),c_2\left(t\right),\cdots ,c_J\left(t\right)\},$$

(24)

$$b_k\left(t\right)=\{b_1\left(t\right),b_2\left(t\right),\cdots ,b_K\left(t\right)\},$$

(25)

$$z_x\left(t\right)=\{z_1\left(t\right),z_2\left(t\right),\cdots ,z_X\left(t\right)\}.$$

(26)

3.2. Action

Action A denotes the sets of VNF migration, and the VNF migration at time t can be denoted as

$$a\left(t\right)=\{a_1\left(t\right),a_2\left(t\right),\cdots ,a_i\left(t\right),\cdots ,a_V\left(t\right)\},\forall i\in V,$$

(27)

where $a_i\left(t\right)$ describes that if the ith VNF is migrated. $a_i\left(t\right)=1$ means that ith VNF is migrated, and otherwise, $a_i\left(t\right)=0$. In order to better adapt to the network changes, the knowledge-molded VNF transfer method is introduced into the 6G network knowledge base as prior knowledge to expand the action space, allowing for more choices in VNF transfer decision-making and optimization processes, thereby improving the flexibility and adaptability of the algorithm. At the same time, it is possible to explore more fully, thus having the opportunity to find better solutions and improve algorithm performance and efficiency.

3.3. Reward

Since the section aims to minimize the configuration cost of SFC, the reward function can be denoted as

$$r\left(t\right)=-\alpha Cost,$$

(28)

where $\alpha >0$ is the reward factor, and the larger $r\left(t\right)$, the smaller SFC reconfiguration cost.

Based on the above MDP model, the KA-ACPPO algorithm is proposed to solve the SFC reconfiguration problem. Combining the advantages of the AC algorithm and the PPO algorithm, the ACPPO algorithm is efficient, stable, and adaptable. However, the selection of hyperparameters and the efficiency of sample processing are still issues that need to be considered when using the ACPPO algorithm. Therefore, under the 6G autonomous control framework, the KA-ACPPO algorithm introduces network knowledge related to the algorithm from the 6G network knowledge base, initializes the Actor network parameters and the Critical network parameters, and accelerates the learning process and reduces training time by providing a good initial strategy in the initial stage, avoiding a lot of exploration. Meanwhile, another VNF transfer method is introduced from the 6G network knowledge base as prior knowledge to expand the action space, helping the algorithm converge and learn faster, improving its performance in complex tasks, and enabling it to cope with a wider range of 6G scenarios.

The framework of the KA-ACPPO algorithm is shown in Figure 2. Input the current network topology G and the VNF deployment state, then the Actor network factor and the Critic network factor are initialized according to the module of the 6G knowledge base. The historic policies in the module knowledge base are updated based on the SFC reconfiguration policy. The Actor network is responsible for selecting VNF migration actions, while the Critic network evaluates the value of each action. The gradient descent method is used to update the parameters of the Critic network and the Actor network, and the optimal SFC reconstruction strategy can be denoted as:

$${\pi }^{*}=arg\underset{a}{max}{V}^{\pi }(s,a),\forall s,a.$$

(29)

The Critical network evaluates the Actor network by using the advantage function to select actions based on the current output strategy, thereby assisting the Actor in policy updates. In the current state $s\left(t\right)$, the advantage function of selecting action $a\left(t\right)$ based on the strategy of the Actor network can be denoted as:

$${\hat{A}}_t=Q_{\pi old}(s_t,a_t)-V_{\pi old}\left(s_t\right),$$

(30)

where $Q_{\pi old}(s_t,a_t)$ denotes the value $V_{\pi old}\left(s_t\right)$ is the value at state $s_t$.

According to generalized advantage estimation [14], the advantage function can be denoted as:

$${\hat{A}}_t={\theta }_t+\left(\gamma \lambda \right){\theta }_{(t+1)}+\cdots +{\left(\gamma \lambda \right)}^{T-t+1}{\theta }_{(T-1)},$$

(31)

where ${\theta }_t=r\left(t\right)+\gamma V_{\left(s_{t+}\right)}-V_{\left(s_t\right)}$ is the error of Temporal Difference (TD), and $\gamma$ is the discount factor, Let $\lambda =1$, the advantage function can be denoted as:

$${\hat{A}}_t=-V\left(s_t\right)+r_t+\gamma r_{t+1}+\cdots +{\gamma }^{T-t+1}{r}_{t-1}+{\gamma }^{T-t}V\left(s_T\right).$$

(32)

The larger the value of the advantage function, the better the selected action. The advantage function is designed to increase the stability of policy and avoid unstable optimization processes caused by excessive policy updates. At the same time, by limiting the amplitude of action policy updates, the Actor network’s action update amplitude will not be too large. The update amplitude of the Actor network action strategy represents the ratio of the probability of the current strategy taking action $a\left(t\right)$ in state $s\left(t\right)$ to the probability of the old strategy taking action $a\left(t\right)$ in state $s\left(t\right)$, which is denoted as:

$$r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{old}}\left(a_t\right|s_t)}}.$$

(33)

The clip function limits the size of the policy update step, preventing excessive policy updates and thus avoiding instability issues. The clip function can be denoted as:

$$clip(r_t\left(\theta \right),1-\epsilon ,1+\epsilon )=\left\{\begin{array}{cc}1+\epsilon \hfill & r_t\left(\theta \right)>1+\epsilon \hfill \\ 1-\epsilon \hfill & r_t\left(\theta \right)>1+\epsilon \hfill \\ r_t\left(\theta \right)\hfill & otherwise\hfill \end{array}\right..$$

(34)

The clip function restrict the difference between the new policy and the old policy, to ensure the stability of the updating process. The proximal clip loss function in this section can be denoted as:

$$L^{clip}\left(\theta \right)=E[min{r}_t\left(\theta \right){\widehat{A}}_t,clip(r_t\left(\theta \right),1-\epsilon ,1+\epsilon ){\widehat{A}}_t].$$

(35)

Critic network updates as follows:

$$\varphi =\varphi +\beta {\nabla }_{\varphi }L\left(\varphi \right).$$

(36)

where $\beta$ is the learning rate of the Critic network, and $L\left(\varphi \right)$ is the loss function of the Critic network to minimize TD error, which can be denoted as:

$$L\left(\varphi \right)=r_t+\gamma r_{t+1}+\cdots +{\gamma }^{T-t+1}{r}_{T-1}+{\gamma }^{T-1}V{(s(T+1)-V\left(s\left(T\right)\right))}^2.$$

(37)

The Actor network updates as follows:

$$\theta =\theta +{\beta }^{\prime }{\nabla }_{\theta }{L}^{clip}\left(\theta \right),$$

(38)

where ${\beta }^{\prime }$ is the learning rate of the Actor network. Algorithm 1 detailed the training process of KA-ACPPO.

Algorithm 1 The Training Process of KA-ACPPO.

1: Input: the current network topology G and current VNF deployment location 2: Output: the best SFC reconfiguration policy ${\pi }^{*}$ 3: Initialize the Actor network factor $\theta$ and the Critic network factor $\varphi$ 4: for  $thread-number=1,\cdots ,N$  do 5:     for $interation-number=1,\cdots ,K$ do 6:         Obtain the current network state $s\left(t\right)$, and bring into the knowledge-based VNF migration to expand the action space 7:         Select the action $a\left(t\right)$ according to the policy of Actor network 8:         if the constraints $C_1{C}_4$ are satisfied then 9:            Execute the action $a\left(t\right)$, get the instant reward $r\left(t\right)$, and turn to the next state $s(t+1)$ 10:         else 11:            Reselect the action $a\left(t\right)$ 12:         end if 13:     end for 14:     Calculate ${\widehat{A}}_t$ and $L^{clip}\left(\theta \right)$ 15:     Update the Critic network based on Equation (35) 16:     Update the Actor network based on Equation (37) 17: end for

4. Simulation and Result Analysis

4.1. Parameter Setting

In this section, we set up the simulation environment to evaluate the performance of the proposed KA-ACPPO algorithm. The key parameters are listed in

Table 2. Simulation parameters.

Simulation Parameters	Simulation Values
number of edge nodes	10
computation resource of edge nodes	[100,200] (units)
power consumption of the edge node at idle state	87 (W)
power consumption of the edge node at full-load state	145 (W)
reserve bandwidth for VNF	[5,15] (Mbps)
required computing resource for delay-sensitive SFC	[10,30] (units)
required computing resource for delay-tolerate SFC	[5,15] (units)
length of SFC	[4,5,6,7]
end-to-end delay constraint	60 (ms)
the parameter of clip function	0.2
learning rate of Actor network	0.001
learning rate of Critic network	0.01
discount factor	0.99

[15]. And it is worth noting that the parameter setting of the service requests are generated by random selection from the sum of Gaussian functions with different parameters [15].

4.2. Results Analysis

Figure 3 shows KA-ACPPO algorithm rewards for different number of neurons. It can be seen that, as the number of neurons increases, the reward of the KA-ACPPO algorithm also increases. The reason is that increasing the number of neurons can improve the representation ability of neural networks, enabling them to better learn and fit complex functional relationships, and enable the network to better express the complex relationships between states and actions, thereby improving rewards. However, increasing the number of hidden layer neurons can increase the complexity of the model and may lead to a slower training process. Therefore, when selecting the number of hidden layer neurons, it is necessary to balance training efficiency and increasing rewards. Considering training efficiency and rewards comprehensively, the number of neurons for the KA-ACPPO algorithm is determined to be 128 in the following simulation in this paper.

Figure 4 shows the normalized reconfiguration costs for different algorithms, which is our optimization object in Equation (21). From the figure, we can see that the AC algorithm can quickly converge into a higher convergence value, but it fluctuates greatly throughout the entire training cycle, resulting in a lower convergence value. This is because the strategies and value functions in the AC algorithm are interdependent, and their updates affect each other. This coupling relationship leads to fluctuations in the training process of the AC algorithm. ACPPO algorithm uses proximal ratio pruning to limit the magnitude of policy updates, reducing the impact of this coupling relationship to ensure a stable training process. Moreover, the proposed KA-ACPPO algorithm introduces 6G knowledge on the basis of the ACPPO algorithm, which helps the algorithm learn and converge quickly, ultimately achieving smaller fluctuations and higher convergence.

Figure 5 shows the computing cost for VNF migration using different algorithms, which is related to Equation (4). The length of SFC affects the computational resource cost of VNF migration from two aspects: the current usage of computing resources of edge nodes, and the number of VNF migrations. As the length of SFC increases, the number of VNFs per SFC increases, and the current usage of edge node computing resources becomes more complex. With the goal of minimizing reconstruction costs, the number of VNF migrations should be controlled while migrating VNFs that use fewer computing resources. From the figure, it can be also observed that as the length of SFC increases, the computational resource cost of VNF migration for all three algorithms increases. The computational resource cost of VNF migration using the proposed KA-ACPPO algorithm is always lower than the other two algorithms, ultimately controlling the computational resource cost of VNF migration at $34\%$.

Figure 6 shows the power costs of VNF migration for different algorithms, which is related to Equation (6), and the comparison algorithm is Reinforcement Learning (RL)-based VNF deployment scheme proposed in [16]. The power cost of VNF migration is related to the usage of network node computing resources. Therefore, the length of SFC also affects the power cost of VNF migration in two aspects: the current usage of edge node computing resources and the number of VNF migrations. As the length of SFC increases, the number of VNFs per SFC increases, and the computing resources of the network edge nodes currently used increase, The power consumption increases accordingly. Considering that VNF migration may change the working mode of edge nodes, with the goal of minimizing reconstruction costs, the number of VNF migrations should be controlled to minimize the additional power consumption caused by such changes. From Figure 6, it can be observed that as the length of SFC increases, the power cost of VNF migration for all three algorithms increases. The power cost of VNF migration for the KA-ACPPO algorithm is always lower than the other two algorithms.

Figure 7 shows the QoS degradation rates, which is related to Equation (18), of delay-sensitive SFC requests using different algorithms. According to Formula (18), the QoS degradation rate of delay-sensitive SFC is defined as the proportion of additional delay caused by excessive downtime to the total end-to-end delay. The length of SFC affects the QoS degradation rate of delay-sensitive SFCs from two aspects. Firstly, as the length of the SFC increases, the total end-to-end latency also increases. Secondly, as the length of the SFC increases, the number of VNF migrations per SFC may also increase. The negative impact of migration latency and downtime on the QoS of delay-sensitive SFC will also increase. From Figure 7, it can be observed that as the length of the SFC increases, the QoS degradation rate of the delay-sensitive SFC of all three algorithms increases. The QoS degradation rate of the delay-sensitive SFC of the KA-ACPPO algorithm is always lower than the other two algorithms, and the QoS degradation rate of the delay-sensitive SFC is ultimately controlled at 39%.

5. Conclusions and Discussion

6G will inherit and develop NFV architecture. With this SFC-enabled flexible network architecture, SFC orchestrates different VNFs to improve maintaining the QoS of different services. Considering that the coupling relationship leads to fluctuations in the training process of the traditional AC algorithm, this paper brings proximal ratio pruning to limit the magnitude of policy updates and combine the advantages of the AC algorithm and the PPO algorithm. Moreover, we propose to introduce the 6G knowledge base to provide a better initial policy to accelerate training speed and form the KA-ACPPO-based VNF migration algorithm. Simulation results show that the proposed algorithm can effectively reduce computing cost and power consumption.

However, in this paper, we consider using historic policies as network knowledge to improve performance. In future work, more dimensions of knowledge, such as model-based algorithms and expert experiences, could be researched to further improve the performance of deep learning algorithms.