Priority-Aware Resource Management for Adaptive Service Function Chaining in Real-Time Intelligent IoT Services

Abstract

The growth of the Internet of Things (IoT) in various mission-critical applications generates service heterogeneity with different priority labels. A set of virtual network function (VNF) orders represents service function chaining (SFC) for a particular service to robustly execute in a network function virtualization (NFV)-enabled environment. In IoT networks, the configuration of adaptive SFC has emerged to ensure optimality and elasticity of resource expenditure. In this paper, priority-aware resource management for adaptive SFC is provided by modeling the configuration of real-time IoT service requests. The problem models of the primary features that impact the optimization of configuration times and resource utilization are studied. The proposed approaches query the promising embedded deep reinforcement learning engine in the management layer (e.g., orchestrator) to observe the state features of VNFs, apply the action on instantiating and modifying new/created VNFs, and evaluate the average transmission delays for end-to-end IoT services. In the embedded SFC procedures, the agent formulates the function approximator for scoring the existing chain performance metrics. The testbed simulation was conducted in SDN/NFV topologies and captured the average of rewards, delays, delivery ratio, and throughput as −48.6666, 10.9766 ms, 99.9221%, and 615.8441 Mbps, which outperformed other reference approaches, following parameter configuration in this environment.

1. Introduction

Programmability and elasticity of management layers in Internet of Things (IoT) networking are the primary prerequisites for future autonomous systems (zero-touch network and service management), which can be obtained predominantly based on three enabler technologies: (1) software-defined networking (SDN), (2) network function virtualization (NFV), and (3) multi-access edge computing (MEC) [1,2,3]. Each paradigm consists of significant contributions to activate automation on network path configuration, multiple virtual resource blocks on dedicated hardware, flexible/resource-gathering application programming interfaces, and enhanced computational distribution. Towards the future of self-managing capabilities in real-time intelligent IoT services (e.g., mission-critical deep learning-based applications), each technology requires further enhancement as follows [4,5,6]:

• Routing optimization, priority-aware traffic classification (e.g., application-based/flow-based approaches), quality-of-service (QoS) and quality-of-experience estimation, advanced security, and resource allocation in SDN systems;
• Increasing flexibility in terms of instantiation, modification, and placement of virtual network functions (VNFs) in the NFV layer with model scalability and transferability;
• Reactive/proactive caching placement, improved service slicing, virtualized resource mapping, and task offloading decisions in resource-constrained MEC.

Numerous deep reinforcement learning (DRL) approaches have been studied and recommended by researchers and standardization to enhance these abovementioned perspectives, which consider optimization of joint weighted sum models for communication, computation, caching, and controllability [7,8]. Experience-driven and data-driven mechanisms (e.g., replay batch- and state-aware enhancement) are well known as efficient facilitators for scalable self-organizing IoT networking. With a variety of taxonomies, namely actor–critic or deep deterministic gradient policy, DRL algorithm selection has to be optimized before deployment based on the discrete/continuous action spaces and final policy expectation. In multi-service networks, DRL is a promising decision-making entity, particularly in NFV-based dynamic service function chaining (SFC), that considers the isolation of each service based on a set of VNF forwarding graphs (VNFFGs) [9]. Figure 1 presents an overview of correlations from ingress to egress VNF descriptors, which are specified through the capabilities, resource properties, connection points, and virtual links. Each specific service has to be determined within the selected forwarding graphs. The ingress index is denoted as $a$ followed by sequential indices for representational services $1$ and $2$. The egress index is denoted as $b$ followed by services $3$ and $n$, which refers to the multi-service environment of heterogenous IoT tenants. The function-specific VNF descriptors are executed at least once throughout the selected services to perform the setup mechanism. By setting up the model environment and formulating the evaluation rewards on load balancing, upper-bound delays, and required computation resources, VNF costs and service chain selection can be optimized by the proposed agent.

In massive real-time IoT services, the major domain to tackle in SDN systems and NFV-enabled MEC architectures is the adaptivity of SFC orchestration, which requires dealing with five challenging issues: (1) insufficient virtual resource mapping, (2) deficient delay-sensitive service flow classification, (3) inaccurate priority labels, (4) unstable workload in diversified network congestion states, and (5) incomplete adaptive policy self-modification [4,7]. To handle these issues, several aspects of deployed SFC need to be jointly considered as weighted sum models of allocation costs, computation delays, descriptor modification requirements, and virtual linkages. The upper-bound delays have to be detected/predicted for ensuring the prioritization of both mission-critical and non-mission-critical IoT services. The total execution delays and resource consumption have to be minimized throughout multiple agents to meet the requirements of each aspect.

In this paper, a system model of NFV-based SFC is given by considering the computation resource and orchestration models (VNF placement, virtual link, and modification costs) in multi-service IoT networks. The system architecture of SFC deployment is given by presenting the correlations between instantiated VNF descriptor with defined VNFFG, virtual machine (VM) blocks in NFV infrastructure, and SFC orchestration policy. The multi-agent approaches, including (1) agent 1, virtual resource mapping; (2) agent 2, VNF placement; and (3) agent 3, priority-aware service classification, are primarily based on deep Q-networks (DQNs) which independently observe specific states, actions, and rewards on integrated purposes of priority-aware management. Experimental simulation is conducted in both SDN/NFV-enabled and DRL-based environments.

The rest of the paper is structured as follows: Section 2 presents the preliminary studies on DRL-based SFC. Section 3 presents the system models for the setup environment and orchestration/computation models in the NFV-enabled SFC environment. The working flow and execution processes of proposed multiple agents on priority-aware resource management in real-time intelligent IoT services are given in Section 4. Section 5 shows the performance evaluation, which includes the simulation setup, indication metrics, reference schemes, and result discussions. Finally, the conclusion is presented in Section 6.

Table 1. List of important acronyms.

Abbreviation	Description
IoT	Internet of Things
DQN	Deep Q-Network
DRL	Deep Reinforcement Learning
MEC	Multi-Access Edge Computing
NFV	Network Function Virtualization
NFVO	NFV Orchestrator
QoS	Quality-of-Service
SDN	Software-Defined Networking
SDNC	SDN Controller
SFC	Service Function Chaining
SFC-FRFP	SFC Flow Rule Forwarding Path
SFC-OP	SFC Orchestration Policy
TOSCA	Topology and Orchestration Specification for Cloud Applications
VIM	Virtual Infrastructure Manager
VM	Virtual Machine
VNFs	Virtual Network Functions
VNFFG	VNF Forwarding Graph

presents a list of important acronyms used in this paper.

2. Preliminary Studies on DRL-Based SFC Deployment

In this section, the promising selected studies on applying DRL for optimizing the procedures of SFC deployment are outlined, including the autonomous VNF placement via DRL-based performance prediction, optimal placement via double DQN, optimal resource usages, and adaptive online SFC with DRL.

Reinforcement learning problems have been modeled and dealt with using a Markov decision process framework throughout the interactions between four tuple elements including a set of state spaces, action spaces, transition probability function, and reward, which are denoted as $S$, $A$, $P$, and $R$ [10,11,12]. The setup environment, namely the NFV-based SFC scenario, samples a set of state features, denoted as $s_t$, throughout numerous timeslot $t$-iteration. The agent executes the action $a_t$ by exploring or exploiting selection to the environment for altering the conditions and evaluating the update metrics. The transition function as a probability distribution presents the likelihood of updating the state features in the NFV-based SFC environment towards $s_{t+1}$. $r_t$ can be adjusted in SFC environment as the expected objectives to illustrate the satisfactions of applied action in that particular state characteristic within timeslot $t$. $r_t$ can be referred to as the end-to-end execution delay or computational network function latency [13,14,15,16]. As an experience-driven mechanism, each element is stored within the replay batch for synchronously optimizing the parameters of online/target function approximators. Throughout this processing flow, the agent can maximize the reward for particular states by selecting the optimal action, which is collectively known as an optimal policy. This overview procedure has been adopted and modified to various promising approaches in SFC orchestration.

In [17], the authors use a DRL agent to interact with an SDN/NFV-enabled environment to optimize the VNF placement through a weighted sum model of cost minimization on allocation, VNF instance execution, and SFC request rejection. The placement model is formulated as binary integer programming. A good reward function, which jointly considers each aspect of the environment, efficiently generates reliable and long-term agent flexibility. In [18], the authors propose an adopted reinforcement learning scheme for autonomous VNF placement via performance prediction. To predict the transmission efficiency as an approximation of expected VNF performance, the elements for reinforcement learning agent are described as follows:

• State consists of all the possible ranging predictions classified into three categories, namely underestimation (predicted value is less than the actual value), equivalent (predicted value is equal to the actual value), and overestimation (predicted value is more than the actual value).
• Action alters states by setting decrement/increment values of the prediction by 0.01.
• Reward considers an outcome range between 0 and 1 as a deficiency and efficiency of the state–action pairs for improving immediate and next state. The acceptable error margin is defined.

The communication and computation resource utilization in SFC architecture is optimized by adjusting the flow specifications and service function instances. In [19,20,21], the optimization problem is formulated in SFC-aware flow systems for minimizing resource usage in each chain. The optimized scheme needs to consider the constraints of SFC-aware flow conservation, capacity, and service function processing order. In DRL-based SFC deployment, the is a need to consider the system description on orchestration entity and Markov decision process models, which includes the elements of service-required VNF, service-related SFC, and characteristics of service requests. The components that interact with this setup environment consist of primary features as follows [19]:

• State defines the features of remaining resource capacities, output bandwidth, and characteristics of processed VNFs.
• Action indicates the placement index of deploying particular VNFs in the selected server node. The action alters the placement specifications which output different performances, whereas the null action indicates the undeployed status of VNFs.
• Reward formulates the optimization model of the weighted total between provider and client profits, which can be obtained by the approved requests and expenditures for deployment.

Throughout these selected promising studies, SFC processing flows have been enhanced with DRL-based approaches. However, the challenge in multi-service prioritization of real-time intelligent IoT networks remains a critical aspect that needs to converge between autonomous placement, resource mapping, and classification of service criticality. Particularly, in deep learning-based applications, the procedures of data-driven models need an adaptive function-specific VNF selection with minimization of total execution time and resource consumption. To acquire an efficient priority-aware resource orchestration in SFC deployment, a setup system model and problem formulation, jointly tackling in-depth aspects of an NFV-based system and the Markov decision process, are required for ensuring the reliability and scalability of the proposed multi-agent scheme via confluence models in total reward function.

3. System Models

In this section, the collaboration models between NFV-enabled SFC architecture and the Markov decision process framework are given for specifying the features that matter in the agent policy formulation. Figure 2 presents the exemplary entity relationship diagram of the system architecture [22,23,24]. In this proposed SDN/NFV-assisted SFC, the connectivity between multi-service IoT users to specific network services is through the orchestration of SFC policy, which is associated with the VNFFG/VNF specifications and VM resource building block determination. In an example execution timeslot, the end user sends a single flow, which belongs to a unique service chain forwarding path instance. The forwarding decisions and rules follow the optimal SFC orchestration policy. Each path instance is based on the occupied VNFFG descriptor using the topology and orchestration specification for cloud applications (TOSCA) template in NFV, which carries many VNF descriptors as a set of ordered functions. The VM resource blocks and other resource properties are allocated within the configuration of the descriptor. The proposed weighted sum model has to reflect the performance of policy with the network service. The expectations on QoS class identifiers are key primary features for priority-aware management, which are varied between each service criticality.

The SDN and NFV entities comply with each other to flexibly modify the instances, adjust the virtual resource blocks, orchestrate the descriptors, and install the forwarding rules. The data and control plane are decoupled by SDN concepts which list the main entities of (1) the data plane, namely a set of small base stations, denoted as $BS=\left\{1,2,\dots ,b\right\}$, equipped with NFV-enabled nodes with allocated virtual resources, denoted as $vS=\left\{1,2,\dots ,s\right\}$, and (2) control plane, namely the configuration from NFV management and orchestration proposed for assigning weights to apply increment or decrement actions on virtual resource capacities. Application programming interfaces are used to extract the state information of maximum/available resources, congestion states, etc. Within this architecture, the SDN/NFV-assisted SFC system models require tackling the orchestration and computational resources.

Table 2. Key notations used in the system models and proposed mechanism formulation.

Notation	Description
$BS=\left\{1,2,\dots ,b\right\}$	A set of small base stations
$vS=\left\{1,2,\dots ,s\right\}$	A set of NFV-enabled nodes with allocated virtual resources
$vFG=\left\{1,2,\dots ,i\right\}$	A set of VNFFGs
$vNF=\left\{1,2,\dots ,j\right\}$	A set of VNFs
$vL=\left\{1,2,\dots ,l\right\}$	A set of virtual links
$s{t}_{j\left(i\right)}^s$	The status variable of deploying VNF$-j$ in service path $i$
$s{t}_{l\left(j\to j^{\prime }\right)}^s$.	The status variable of placing virtual link $l$ connecting between VNFs
$a_{j\left(i\right)}^s$	The placeme cost of VNF$-j$
${\theta }_i$	Critical weight of priority-aware service $i$
$b{w}_{l\left(j\right)}^{s\to s^{\prime }}$	The bandwidth allocation between NFV-enabled nodes
$c_{l\left(j\right)}^s$	Link costs of VNF$-j$ in NFV-enabled node
$f\left(r_{j\left(i\right)}^s{\theta }_i\right)$	Function for returning predetermined resource allocation in node $s$ for executing VNF$-j$(primarily based on the ${\theta }_i$)
$r{r}_{j\left(i\right)}$	Required resources for computing VNF$-j$
$C_{alloc}$	Allocation cost in VNFFG
$C_{util}$	Cost of total utilization in VNFFG
$C_{comp}$	Computational resource model in VNFFG
$T_{total}^i$	The total delay in a particular chain $i$
$T_{mod}^i$	Modification delay in a particular chain $i$
$T_{orc}^j$	Orchestration delay of VNF$-j$
$T_{exe}^j$	Execution delay of VNF$-j$

presents important notations used in this paper.

To fully formulate the model for SFC orchestration, a weighted sum of VNF placement in a complete VNFFG and virtual link usages is given for outputting the costs. The computation resource is expected to be minimized for further enhancement. Afterward, the total execution delays for processing from ingress to egress VNFs in a particular service path instance require consideration in the proposed system as well.

3.1. Costs of Allocation and Virtual Link Usages

A set of VNFFGs is denoted as $vFG=\left\{1,2,\dots ,i\right\}$, and a set of VNFs is denoted as $vNF=\left\{1,2,\dots ,j\right\}$. The status of deploying VNF-$j$ into forwarding graph VNFFG-$i$ (particular service instance) with a specified resource block in node $s$ is presented as $s{t}_{j\left(i\right)}^s$. The placement cost is considered as $a_{j\left(i\right)}^s$; however, with a defined weight ${\theta }_i$ of priority-aware service criticality, the total cost can be significantly decreased or increased, which guides the orchestration emphasis after the weights are predetermined via the proposed algorithm. For allocation cost in a VNFFG, denoted as $C_{alloc}$, the summation of each placement is formulated to associate with the VM resource blocks that operate the defined function, which is given in (1). Moreover, in a VNFFG, the utilization of virtual links for connection points consists of a configuration cost. With $x$ number of VNFs in a forwarding path, the number of virtual links is required to be one less than the total number of VNFs ($x-1)$, which connects within each ordered function. A set of links is denoted as $vL=\left\{1,2,\dots ,l\right\}$. The costs of total utilization, denoted as $C_{util}$, can be formulated as Equation (2) by considering four essential variables as follows: (1) the bandwidth allocation between NFV-enabled nodes, denoted as $b{w}_{l\left(j\right)}^{s\to s^{\prime }}$; (2) link costs of each VNF in NFV-enabled node $s$, denoted as $c_{l\left(j\right)}^s$; (3) the critical weight ${\theta }_i$ of the forward path; and (4) the status of placing virtual link $l$ connecting between ingress VNF$-j$ and egress VNF$-j^{\prime }$, denoted as $s{t}_{l\left(j\to j^{\prime }\right)}^s$.

$$C_{alloc}={\displaystyle \sum }_{i \in vFG}{\displaystyle \sum }_{j \in vNF}{\displaystyle \sum }_{s \in vS}s{t}_{j\left(i\right)}^s{a}_{j\left(i\right)}^s{\theta }_i$$

(1)

$$C_{util}={\displaystyle \sum }_{i \in vFG}{\displaystyle \sum }_{j \in vNF}{\displaystyle \sum }_{l \in vL}{\displaystyle \sum }_{s \in vS}s{t}_{l\left(j\to j^{\prime }\right)}^{s}b{w}_{l\left(j\right)}^{s\to s^{\prime }}{c}_{l\left(j\right)}^s{\theta }_i$$

(2)

3.2. Computation Model

In a particular defined service, the allocated VM resource blocks are essential for adequately managing computing a complete service path in SFC systems within a required QoS class expectation. To ensure reliability, the pre-allocated resource properties need to optimally match with the required computation resources per end-to-end service transmission. For real-time mission-critical services, the resources can be extended for securing the future congestion states, which are assigned as a primary variable in dynamic configuration. Equation (3) expresses the computation model, denoted as $C_{comp}$, that is based on the parameters of (1) allocation status $s{t}_{j\left(i\right)}^s$, (2) predetermined resource allocation in node $s$ for executing VNF$-j$ primarily based on the ${\theta }_i$, denoted as $f\left(r_{j\left(i\right)}^s{\theta }_i\right)$, and (3) required resources for computing VNF$-j$, denoted as $r{r}_{j\left(i\right)}$. The model outputs 0 if VNF$-j$ does not belong in that particular service; otherwise, the model returns a ratio between 0 to 1 for optimizing the placement, where 1 indicates the optimal point. If the predetermined allocation leads to output higher than 1, it indicates a service drop due to exceeding the upper-bound limitation, which severely affects the real-time model building in intelligent IoT services.

$$C_{comp}={\displaystyle \sum }_{i \in vFG}{\displaystyle \sum }_{j \in vNF}{\displaystyle \sum }_{s \in vS}s{t}_{j\left(i\right)}^s\frac{ r{r}_{j\left(i\right)}}{f\left(r_{j\left(i\right)}^s{\theta }_i\right)}$$

(3)

3.3. Execution Delays

Within a particular service, the total execution delay is the most critical metric to be minimized by optimally adjusting adequate computation resources, bandwidth for NFV-enabled node $s$, and accurate predictive VNF performance on the ordered path. Equation (4) expresses the total delay in a particular chain $i$, denoted as $T_{total}^i$, where each element is gathered from the computation and placement models. $T_{mod}^i$ determines the modification delay to reconfigure the chain descriptor following the proposed agent policy in any required intervals. In chain $i$, the total orchestration and execution delays of each placed VNF$-j$, which are denoted as $T_{orc}^j$ and $T_{exe}^j$, are considered in terms of function instantiation, configuration, and processing.

$$T_{total}^i=T_{mod}^i+{\displaystyle \sum }_{j \in vNF}s{t}_{j\left(i\right)}^s(T_{orc}^j+T_{exe}^j)$$

(4)

4. Priority-Aware Agent for Resource Management in SFC

To interact between SDN/NFV-assisted SFC system models with the proposed agent, the Markov decision process model is applied for determining the state observations, action spaces, and rewards as primary objectives in policy orchestration. The process of each agent follows four primary phases, namely SFC environment initialization, state gathering, reward-maximized action selection, experience-driven exploitation, and improved parameters for the function approximator. In the early stage, the proposed model of each agent learns the possible actions for fitting within state observations in different congestion intervals of massive IoT. The management layer of the setup environment uses the application programming interfaces and protocols to sample the workload of VNF instances within each path for the proposed agent in the core control. The agent selects actions via an optimization process and experience-driven mechanism. Each applied action is stored for iterative improvement and synchronously transitioning to future states. Figure 3 presents an overview architecture of the interaction between SDN/NFV-assisted SFC systems and proposed agents. There are three predominant agents which aim for (1) virtual resource mapping, (2) VNF placement/creation policy, and (3) priority-aware service labeling. The agents observe the states and apply the action from/to the SFC orchestration policy (SFC-OP) and SFC flow rule forwarding path (SFC-FRFP) in the NFV orchestrator (NFVO) and SDN controller (SDNC), respectively.

SFC-OP responses control the forwarding path, initialize/modify/delete service chains, and adjust the priority. SFC-FRFP manages the flow rule installation and instance selection. The virtual service nodes at the virtual infrastructure manager (VIM) consist of virtualized resources and maps with workload-based functions. To execute this processing flow and acquire an optimal reward, the primary objectives, model formulation of each agent, and SDN/NFV-based SFC orchestration requires in-depth descriptions, which will be presented in this section as follows.

The confluence of reinforcement learning and deep neural networks (function approximators) enables an optimal output on action selection and advanced Q-value function. The proposed agent uses the components of the Markov decision process in the SFC environment for handling the optimization learning. There are four primary components for multiple agents to feed and iteratively establish for the final learning policy, which includes a collective batch, in this system architecture:

• State spaces indicate the significant features of the SDN/NFV-assisted SFC environment, where the placement costs, bandwidth allocation, link costs, and required resources per VNF are observed for formulating the central policy in both SFC-OP and SFC-FRFP. The proposed multi-agent DRL utilizes the Markov decision process framework to tackle the problem by synchronously storing the sequential experienced IoT networking. SFC-OP ensures the provision of the placement $a_{j\left(i\right)}^s$ and link $c_{l\left(j\right)}^s$ costs for agents. SFC-FRFP responds with $b{w}_{l\left(j\right)}^{s\to s^{\prime }}$ and $r{r}_{j\left(i\right)}.$ The reference points are assigned within NFV management and orchestration entities. Representational state transfer is used mainly in SDNC for interactions between OF-enabled networks (cluster heads), SFC-FRFP, and central multiple agents. These interfaces allow communications within central software mechanism systems for high and reliable performance.

  $$s_t=\left\{a_{j\left(i\right)}^s, b{w}_{l\left(j\right)}^{s\to s^{\prime }}, c_{l\left(j\right)}^s, r{r}_{j\left(i\right)}\right\}$$

  (5)
• Actor spaces consist of the configuration parameters in priority-aware SFC systems that highly impact the system execution. The status of deploying VNF in any particular path $s{t}_{j\left(i\right)}^s$ can be adjusted between 0 and 1, which acts as a decision variable of the placement. The proposed agent considers the workload of services, application criticality, and congestion as major factors. Similarly,$s{t}_{l\left(j\right)}^s$ is also a decision variable for placing virtual links between VNFs, which outputs between 0 and 1. This configuration aims to emphasize the difference between connectivity points in non-real-time and real-time IoT services. The service criticality is incorporated with the application plane in the SDN-based management and experience-driven entity for enhancing the priority awareness, which integrates weight ${\theta }_i$ for any service path $i$. The critical ratio is a ranging float value from 0.0 to 1.0, where 1.0 allows the allocation to be completely placed in critical real-time labels. The predetermined resource allocation from $f\left(r_{j\left(i\right)}^s{\theta }_i\right)$ maximizes or stabilizes its value based on the service-critical weights. For example, the critical weight of 0.5 in service $i$ from agent 3 decides the final values of placement and resource capacities from both agents 1 and 2 as described in Equations (1) and (2).

  $$a_t=\left\{s{t}_{j\left(i\right)}^s, s{t}_{l\left(j\right)}^s,{\theta }_i, f\left(r_{j\left(i\right)}^s{\theta }_i\right)\right\}$$

  (6)
• Reward determines the evaluation of applied $a_t$, in terms of placement decision variables and weights, on current costs, required computation, and available bandwidth resources as $s_t$. With the performance of the state–action pair $\left(s_t, a_t\right)$, the collective rewards on each agent, as shown in (7), are reflected to reward agents 1, 2, and 3 as expressed in (8), (9), and (10), respectively. The reward for agent 1, denoted as $r_1$, presents the resource efficiency by aiming to compare the optimal variables that minimize the overload resource placement. The minimization follows the constraints of decreasing the output when it exceeds 1, and the optimal output is 1 since the model meets the best matching orchestration when $r{r}_{j\left(i\right)}$ is equal to $f\left(r_{j\left(i\right)}^s{\theta }_i\right)$. The reward for agent 2, denoted as $r_2$, aims to range the scoring reward based on a container with minimization of the placing costs. A tradeoff between high cost and latency occurs when the ${\theta }_i$ is equal to 1 (highly mission-critical weight). The latency performance of executing the service path is emphasized as a reward $r_3$ in agent 3. The latency in orchestration, execution, and modification in the processing SFC systems reflects the performances compared with the lower-bound and upper-bound threshold of each service QoS class identifier.

  $$R=r_1+r_2+r_3$$

  (7)

  $$r_1=\min \left({\displaystyle \sum }_{i \in vFG}{\displaystyle \sum }_{j \in vNF}{\displaystyle \sum }_{s \in vS}s{t}_{j\left(i\right)}^s\frac{ r{r}_{j\left(i\right)}}{f\left(r_{j\left(i\right)}^s{\theta }_i\right)}\right)$$

  (8)

  $$r_2=\min \left({\displaystyle \sum }_{i \in vFG}{\displaystyle \sum }_{j \in vNF}\left({\displaystyle \sum }_{s \in vS}s{t}_{j\left(i\right)}^s{a}_{j\left(i\right)}^s{\theta }_i+{\displaystyle \sum }_{l \in vL}{\displaystyle \sum }_{s \in vS}s{t}_{l\left(j\right)}^{s}b{w}_{l\left(j\right)}^{s\to s^{\prime }}{c}_{l\left(j\right)}^s{\theta }_i\right)\right)$$

  (9)

  $$r_3=\min \left(T_{total}^i\right)$$

  (10)
• Q-value and loss optimization aims to optimize long-term targets and parameters for the deep networks of function approximators. The target network uses the replay batch parameters for training. However, in every training phase, loss values appear as either high or low compared to the satisfactory point. The loss follows the mean square error concept, which is a squared difference between the target $Q^{*}\left(s,a\right)$ and predicted q-value. With optimal q-value approximation, the agent can take actions that maximize the performance of each real-time service with long-term efficiency. Equation (11) presents the primary target q-value, which follows the rewards in (8)–(10) with the discount factor $\gamma$ (uncertainty rate) of the expected maximum in the next state. The networks (online and target) iteratively modify the parameters and synchronously exchange the weights. The policy iteration of multi-agent DQNs consists of a flow transition between each component in the Markov decision process framework, which is depicted in Figure 4.

  $$Q^{*}\left(s,a\right)={\displaystyle \sum }_{k\in R}{r}_k+\gamma \underset{a^{\prime }}{\max }{Q}^{*}\left(s^{\prime },a^{\prime }\right)$$

  (11)

When the service task is offloaded in the SFC environment, the requests will be checked in the SFC-OP and SFC-FRFP entities to delete, occupy, or instantiate the new/existing service type with the same or different required VNFs. If the service is instantiated, the states of costs, computing properties, and bandwidth for transmission will be observed for the agents. The descriptors are created in SFC-OP. The existing set of VNFs will be iteratively checked in the VNF manager for matching, and the virtual links will be placed. Agent 1 slices the virtual resource mapping based on the optimal policy in the current timeslot. After finalizing the creation of the required VNFs, VNFFG, and SFC flows, the SDNC appends the flow rules and executes the service path.

5. Performance Evaluation

This section shows the performance evaluation of the proposed priority-aware policy with multi-agent DQNs for optimal resource management, termed PA-DQNs-RM. The SDN/NFV testbed is integrated for considering the complete processes of instantiating/declining the service chain (using ordered descriptor representations with the TOSCA NFV template).

Table 3. Software-defined control system configuration.

Purpose/Platform	Specifications
SDN Controller (c0, c1, c2, c3)	RYU 4.32 (controller and NFV-enabled nodes)
Mininet	2.3.0d6
Open vSwitch	2.13.0
Iperf3 (QoS metrics)	Fair-queuing-based socket-level packing, UDP
RESTful	4.1.1
Device Configuration	mininet.node.CPULimitedHost
Control and Interfaces	mininet.link.TCintf and mininet.link.OVSIntf
N4, N9	OpenFlow-enabled protocol
N9	1 Gb/s to 2 Gb/s
N6	9 Gb/s
Payload Sizes	1024 bytes

presents the software-defined control system configuration [25,26,27,28]. With a defined SFC topology on SDN/NFV-enabled architecture, the data features are extracted for computing the agent model on OpenAI Gym and TensorFlow 2.0 [29,30,31,32]. Each agent consists of five primary entities, namely environment initialization, deploying action, next-state transition, reset/complete statuses, and immediate rewards/Q-values/epsilon-greedy on the main agent function. The hyperparameters of the proposed agent are 0.01, 0.95, and 300 as learning rate, discount factor, and episodes, respectively. The overall simulations were executed on a computing server with an Intel(R) Xeon(R) Silver 4280 CPU @ 2.10 GHz, 128 GB memory, and an NVIDIA Quadro RTX 4000 GPU.

5.1. Proposed and Reference Schemes

The proposed scheme, termed PA-DQNs-RM, follows the multi-agent DQN mechanism as described in Section 3 and Section 4, primarily using (1) the state features of the SFC environment including costs, bandwidth availability, and virtual links; (2) actions of making decisions on adjusting weights depending on the service priority; and (3) a multi-perspective reward valuation. To handle the uncertainty of congestion and performance instability, the proposed scheme gathers the state features and experienced path for dynamically adjusting the critical weight. In the aspect of experiments, the assigned application programming interfaces abstract devices/resources for state observations and adjust the simulation parameters to heavily congested, normally congested, and non-congested communications. By fully merging the outputs between each agent, the policy orchestration and forwarding rules on SDN/NFV management entities are capable of managing the resources efficiently. To present the performance efficiency of the proposed scheme, three reference schemes are selected for experimentation and comparison as follows:

• DQN-RM indicates the single-agent mechanism which tackles resource management without considering the critical weights ${\theta }_i$ of service priority awareness. A variety of congestion intervals leads DQN-RM to consist of deficient action selection, particularly when parallel service requests occur. By solely tackling the resource predetermination $f\left(r_{j\left(i\right)}^s\right)$, the efficiency of resources is high; however, it remains an optimizable mechanism for real-time intelligent IoT services [33,34,35]. This reference scheme observes the same states as the proposed priority-aware scheme.
• Random-RM is used as a conventional approach in various complementary studies, where the orchestration policy allocates the system resources by random method [15,34]. The rewards are still formulated in this scheme, by executing the service requests with adjusted random resources and service-critical weights.
• MR-RM follows the greedy method by managing the resources based on either cost or real-time latency optimization. In our experience, we prioritize real-time latency over cost, which leads to a configuration maximal rate of ${\theta }_i$.

5.2. Performance Metrics

In this study, the main objective is to illustrate the performance optimization within the execution of SFC environments from IoT tenants to end services. The optimization mechanism is located in the control plane/orchestrator as a core system of virtual resource mapping, virtual function placement, and priority awareness. Other than the core system, the flows are executed via default simulation topology. The interfaces for N4, N6, and N9 are adopted for SDN-based systems. The payload is assigned for IoT services. The detailed configuration is described in Table 3. The selected metrics for evaluation are captured and calculated in the aspects of executing per service chain in heterogeneous IoT networks. The performance metrics include the following:

• The average immediate total reward per episode identifies the output of evaluating the state–action $\left(s_t,a_t\right)$ execution as expressed in Equations (7), (8), (9), and (10). In the experiment, the agent operates for 300 episodes. Epsilon-greedy balances the exploring and exploiting phase. The immediate rewards are most likely to be stable after 150 episodes. The average metric sums up every immediate reward of that episode and then divides it by the total numbers. The visualization is considered per 50-episode average to present the accumulative outputs. The proposed and reference schemes operate using the same hyperparameters. A negative reward (−10, −1) identifies deficient action selection. The maximum value is 0.
• End-to-end executing delay of requested services is measured in milliseconds (ms) within 300 s simulation time. This metric is considered the total time from IoT tenants to end service, which differentiates between the proposed and reference schemes only by $T_{total}^i$. The propagation, queueing, transmission, and proposed agent orchestration are calculated for this metric.
• Delivery ratio: tasks of requested services represent the efficiency of the decision-making agent when the service requests are being operated, whether there is a reliable path or not. The instantiation, deletion, and modification of service requests are the key management in this environment and variables for ensuring the reliability of the offloading and placed service tasks on any particular chain. With low delivery ratios, the policy is not suitable for real-time IoT scenarios. The priority-aware scheme takes the upper-bound tolerable delays more critically, and they are assigned a critical weight for preventing the high drop possibility.
• Throughput indicates the success data over the allocated bandwidth and VM computation resources for going through each VNF on SFC systems. This metric is measured in megabits per second (Mbps).

5.3. Result and Discussions

In this sub-section, the outputs of the proposed and reference schemes are shown in terms of four main performance metrics. Each scheme follows the simulation setup for overall topologies. SFC-OP and SFC-FRPR differentiate the performance metrics as depicted in Figure 5, which include (a) the average immediate total reward per 50 episodes over a 300-episode experiment; (b) end-to-end execution delay of requested services in ms; (c) delivery ratio: tasks of requested service in percentile ratio; and (d) throughput in Mbps. Over a 300 s simulation, the fluctuation is greatly shown in 150 and 250, where the configuration of requesting a new service is assigned to the experiment. The uncertainty of network states is randomized following congestion levels in both peak and off-peak hours. The states of congestion and non-congestion are observed for the agent.

Figure 5a illustrates how the agent explored and exploited throughout each episode. In the early first 50 iterations, the proposed PA-DQNs-RM obtains −91 negative rewards, which are 8, 10, and 57 positive scores higher than DQN-RM, Random-RM, and MR-RM, respectively. By 150, the proposed policy is capable of alleviating 39.5301% of errors and obtains a great reward. By the end of 300, the agent reached optimal exploitation satisfaction with −9 scores. This score indicates only nine non-optimal orchestrations over numerous possible states. With a multi-aspect weighted sum reward, the agent is more reliable for estimating a long-term SFC instantiation, particularly for multi-taxonomy IoT applications. The priority-aware algorithm with nine non-optimal statuses ensures the reliability and transferability of scalable SFC-IoT management systems. Since the total reward $R$ considers the optimization points of costs, resources, and latencies, the output presents a whole tradeoff in different congestion states, where multiple service requests from real-time IoT services are labeled into different priority classes.

The end-to-end execution delay of the requested service primarily tackles the IoT tenant task of determining whether (1) to request new chain instantiation or (2) to check for an existing chain until the flow rule is installed into Open vSwitch and execute the service with its required set of VNFs. Figure 5b shows the results of the proposed and reference delays. In 150 s, the reference schemes increase the delays by 25.6014%, 28.1801%, and 32.7842%, which gives an unsatisfactory jitter for real-time IoT experiences. Within the configuration of new request instantiation on 150 and 250, the proposed scheme has a deviation of only 1.9911 ms and 5.9932 ms compared to the first 50 s of the experiment.

The execution delay of the proposed scheme shows the sustainability of SFC-OP and SFC-FRFP for long-term self-managing SFC when it comes to instantiating new chain requests or modifying the VNF/VNFFG instances. In terms of reliability, Figure 5c shows the delivery ratio of the requested service tasks. Since the real-time IoT QoS class identifier sets the required upper-bound tolerable delays lower than non-real-time applications, the priority-aware approach is significant for deployment within adaptive agent observation. PA-DQNs-RM reached an average successful chain execution ratio of 99.9221% over a 300 s simulation, which is 0.0423%, 0.0751%, and 0.1085% higher than DQN-RM, Random-RM, and MR-RM, respectively. Each agent has different responsibility for this performance metric, and it is equally significant to map adequate resource capacities, place appropriate VNFs, and proactively label the service-critical weight. By predetermining the communication and computation resources, the delivery ratio on future bottlenecks can be greatly reduced.

Figure 5d presents the total throughput per second in the conducted experiments. In 150, IoT tenants request new chains for the central SFC systems, which heavily degrades the throughput of reference schemes. Compared to a 100 s experiment point, MR-RM degrades the performance by 12 time differences, which can be explained by that when the new chain request is required, the maximal rate configuration cannot configure the optimal weight and lacks self-organizing on unseen requests. This drawback severely affects real-time IoT services. For Random-RM and DQN-RM, a great fluctuation is not shown in 150; however, the throughputs are noticeably lessening until the ending point. The proposed scheme has a stable performance, where the difference between the beginning and the ending point of the experiment is only 0.005 Mbps. The balance of serving both existing/new requests contributes to being a sufficient system. In real-time deep learning-based modeling in IoT services, each VNF can be modified over time, as can the VM allocation; therefore, the priority-aware resource management with dynamic multi-agent DQNs is an efficient approach for maintaining an adaptive SFC system.

6. Conclusions and Future Work

This paper presented a priority-aware resource management scheme for SDN/NFV-assisted SFC systems. The proposed scheme, termed PA-DQNs-RM, primarily relies on multi-agent DQNs, which consist of three aspects, namely virtual resource mapping, VNF placement/creation, and awareness of different real-time IoT service priorities. The system models of NFV-based SFC and SDNC for SFC forwarding path installation were described in this paper. Each model consists of action-configurable parameters such as the critical weight of prioritized service, the status variables of deploying VNF/virtual links, and the predetermination function of resource allocation in each VNF/VNFFG descriptor. A multi-aspect total reward, including resource valuation, placing costs, and latency performance, was assigned for scoring the efficiencies of applied actions over the proposed SFC environment. The simulation setup was given, and the experiment was conducted by including both SDN/NFV-enabled testbed and agent models. PA-DQNs-RM achieved a variety of satisfying performance metrics, including (1) average immediate total reward per episode, (2) end-to-end executing delay of requested services, (3) delivery ratio: tasks of requested services, and (4) throughput, for supporting multi-priority services in real-time IoT. Within this simulation environment, the proposed agent obtained averages of rewards per episode, end-to-end delays, delivery ratio of requested new/created services, and throughput of −48.6666, 10.9766 ms, 99.9221%, and 615.8441 Mbps, respectively. Compared to the reference approaches, the proposed scheme offers an optimized awareness policy for multi-service priorities in real-time IoT networks.

In a future study, experience-driven virtual resource scheduling for IoT network slicing will be further extended over this proposed approach. The SDN/NFV-enabled system architecture for on-demand network slicing will be modeled for a deep deterministic policy gradient-based agent to configure continuous actions and optimize the complete end-to-end real-time IoT network slicing.