Utility-Driven End-to-End Network Slicing for Diverse IoT Users in MEC: A Multi-Agent Deep Reinforcement Learning Approach

Abstract

Mobile Edge Computing (MEC) is crucial for reducing latency by bringing computational resources closer to the network edge, thereby enhancing the quality of services (QoS). However, the broad deployment of cloudlets poses challenges in efficient network slicing, particularly when traffic distribution is uneven. Therefore, these challenges include managing diverse resource requirements across widely distributed cloudlets, minimizing resource conflicts and delays, and maintaining service quality amid fluctuating request rates. Addressing this requires intelligent strategies to predict request types (common or urgent), assess resource needs, and allocate resources efficiently. Emerging technologies like edge computing and 5G with network slicing can handle delay-sensitive IoT requests rapidly, but a robust mechanism for real-time resource and utility optimization remains necessary. To address these challenges, we designed an end-to-end network slicing approach that predicts common and urgent user requests through T distribution. We formulated our problem as a multi-agent Markov decision process (MDP) and introduced a multi-agent soft actor–critic (MAgSAC) algorithm. This algorithm prevents the wastage of scarce resources by intelligently activating and deactivating virtual network function (VNF) instances, thereby balancing the allocation process. Our approach aims to optimize overall utility, balancing trade-offs between revenue, energy consumption costs, and latency. We evaluated our method, MAgSAC, through simulations, comparing it with the following six benchmark schemes: MAA3C, SACT, DDPG, S2Vec, Random, and Greedy. The results demonstrate that our approach, MAgSAC, optimizes utility by 30%, minimizes energy consumption costs by 12.4%, and reduces execution time by 21.7% compared to the closest related multi-agent approach named MAA3C.

1. Introduction

The rapid growth of the Internet of Things (IoT) and mobile Internet has significantly increased user data traffic on edge networks. Cloud computing provides on-demand services such as data processing and storage to users. However, due to limited resource availability in cloud servers, this can lead to excessive bandwidth utilization, as well as data transfer and latency issues [1,2]. For example, in an emergency situation where users require rapid real-time responses, additional latency and obtaining resources from the cloud server could result in a major disaster. The current scenario encourages the distribution of computational resources from the cloud server to the edge of the network [3]. Advancements in machine learning (ML) techniques and artificial intelligence (AI) have enabled the rapid processing and training of immense amounts of data. This progress not only allows for adaptation to dynamic environments but also leads to a considerable reduction in latency and improvement in computational performance [4,5]. Mobile edge computing (MEC) intelligence, the combination of MEC with ML and AI, will undoubtedly be a significant booster for the potential advancement of networks. However, these developments may pose major challenges when resources are scarce in MEC cloudlets and users demand an urgent response with quality of services (QoS) [6,7,8,9].

In the MEC paradigm, network service providers (SPs) lease resources in cloudlets deployed at various locations near IoT users in a large area. Users utilize these resources, and SPs charge for their usage to maximize profit [10]. Therefore, in dense areas such as shopping malls and city centers—and in contrast to rural areas—where a high concentration of IoT users surrounds a single cloudlet, efficiently providing services becomes challenging due to the limited availability of cloudlet resources. Effectively utilizing MEC resources and balancing utilization across cloudlets is a complex task, particularly when user requirements demand minimal latency and urgent responses [11]. When IoT users send network slicing requests to the nearest cloudlet, these requests can have varying requirements. Some tasks might be flexible with processing times, while others may demand the fastest possible computing power to prevent significant user losses.

The limited computing capacity of cloudlets, while enabling the provision of diverse services to IoT users, presents a significant challenge in optimizing network slicing resources within our approach. This capacity is represented by the availability of Virtual Network Function (VNF) instances in activated or deactivated states within cloudlets. Chaining these VNFs allows for the creation of network slices that meet the demands of IoT users [12]. However, effectively fulfilling both common and urgent request requirements from a high volume of IoT users becomes difficult due to these resource constraints. In scenarios with a large number of IoT users, available cloudlet resources may be constrained, leading to challenges in handling numerous simultaneous requests. Specifically, when approximately 1000 IoT requests arrive at the same time, resource limitations can result in delays of up to 30% for urgent requests and a reduction of 20% in overall efficiency due to inadequate distribution [13].

ML techniques such as deep reinforcement learning (DRL) and reinforcement learning (RL) have been employed by scholars to address the issues mentioned earlier. These techniques have demonstrated their significant impact in resolving real-world optimization challenges, including resource allocation, computational outsourcing, adapting to environmental changes, and maintaining QoS. This efficiency is particularly notable in the environment of network slicing, including MEC and IoT [14].

Developing an effective resource optimization technique is critical for MEC. This technique should optimize scarce cloudlet resources, minimizing latency and cost while maximizing SP utility. Various prior works have investigated resource allocation in MEC, focusing on historical traces for joint request placement and resource allocation [15], adaptive VNF deployment and migration of network slices [16,17], maximization of utility by accepting the maximum number of network slicing requests [18], and maximization of SP revenue by providing on-demand network slicing [19]. However, these approaches often lack the necessary adaptability to rapidly changing and diverse environments, particularly when IoT users require urgent computing in the MEC network. By implementing real-time resource allocation mechanisms, MEC can ensure that urgent IoT user requests are fulfilled promptly, potentially mitigating disaster risks.

Our primary objective is to develop a resource optimization technique for MEC that can dynamically allocate resources to meet the demands of both urgent and common IoT user requests. This approach aims to achieve a balance of cloudlet resource utilization, ensuring timely completion of common tasks while prioritizing urgent requests with minimal latency. All resource allocation decisions are made while adhering to established QoS requirements. However, despite considerable improvements in achieving time-sensitive operations, potential challenges remain. These include limited resources in cloudlets, congestion in links due to restricted available bandwidth, and the lack of precise prediction of request requirements. Centralizing all requests, both common and urgent, on the primary cloudlet can provide low-latency responses for users. However, this approach introduces a single point of failure and a resource bottleneck. Even if additional VNF instances are activated, the limited capacity of the primary cloudlet may prove insufficient for sustained high volumes of diverse requests [20]. A key challenge lies in dynamically deciding to process requests locally or not, considering system constraints. DRL, in conjunction with network slicing within MEC, offers a promising solution. In a multi-time-slot scenario where IoT user requests arrive randomly, we investigate the use of DRL to address the challenge of end-to-end network slicing in MEC for network resource optimization [21,22]. Traditional DRL often utilizes centralized or decentralized learning with a single agent. However, this approach presents limitations for network slicing in MEC due to the following two key factors: a vast number of geographically dispersed cloudlets and a large, continuous action space [23].

In contrast to these studies, in our end-to-end network slicing utility optimization problem, we propose a multi-agent DRL approach that intelligently handles a large, continuous action space and accommodates the diverse needs of each user request. By avoiding premature convergence, we train the model to optimize system utility, maximize rewards, reduce energy consumption costs, mitigate execution times for both common and urgent user requests, balance the utilization of cloudlet resources, and achieve higher accuracy compared to existing methods. These aspects have not been well investigated in the current network slicing problem.

To be more precise, our approach evaluates the following crucial questions:

• How to prevent assigning all common and urgent requests to the home cloudlet to avoid overloading a single cloudlet location;
• How to decide which request to assign to the adjacent cloudlet;
• When it is necessary to activate a VNF instance and when to deactivate it;
• How to ensure that common requests are assigned within their deadlines, even when urgent requests are also arriving;
• How to manage available resources in the cloudlet if a user’s demand changes in real time.

Therefore, our work presents significant contributions to the field of network slicing in MEC, particularly regarding the efficient handling of both common and urgent IoT user requests with minimal latency. To address the previously identified challenges, this research proposes a multi-agent-based approach with the following key findings:

• We design a network slicing-based MEC system using multi-agent soft actor–critic (MAgSAC), where cloudlets are placed at various locations near IoT users to provide end-to-end resource allocation services. These cloudlets have computing capabilities in the form of VNF instances, which can be activated or deactivated as needed. This setup accommodates both common and urgent IoT user requests while balancing resource allocation across cloudlets, ultimately ensuring QoS.
• We propose an extensive optimization problem model that aims to optimize the overall utility of the MEC network. This is achieved through intelligent network slice utilization, which involves a trade-off between revenue, energy consumption cost, and overall execution time. By transforming this complex optimization problem into a DRL problem, we describe it as a Markov Decision Process (MDP) and approach it as a multi-agent DRL problem.
• We devise a multi-agent DRL-based MAgSAC algorithm, which intelligently provides resources for both common and urgent requests through prediction by activating and deactivating VNF instances in home cloudlets, as well as adjacent cloudlets. It minimizes energy consumption costs by reconsidering idle or remaining capacity before deactivating VNF instances, thereby maximizing overall utility and minimizing latency. This scheme efficiently facilitates user needs and prevents cloudlets from creating imbalanced network slicing during resource allocation. Our approach aims to intelligently handle the optimization challenges mentioned earlier.
• We conduct extensive simulations to compare our MAgSAC approach with benchmark methods, including MAA3C, SACT, DDPG, S2Vec, Random, and Greedy. The results indicate that our MAgSAC scheme achieves the highest utility with the lowest execution time, as well as minimum delay and energy consumption cost compared to the other approaches.

2. Related Work

Network slicing has recently attracted significant attention from researchers due to the challenges of efficient resource allocation and latency-aware optimization in MEC networks [24,25,26,27,28]. In network communication, ML approaches are rapidly developing as an emerging trend to improve QoS and solve optimization problems such as resource allocation in MEC [29] and real-time service provisioning for both common IoT users and those requiring urgent services [30]. State-of-the-art research focuses on ML approaches for resource optimization in MEC, highlighting this area as a key focus of existing studies.

For example, to balance edge network resources and user demand in bursty traffic, a multi-agent-based two-fold algorithm for resource allocation and request redirection (RA-RR) was introduced in [31]. This approach matches and coordinates slice resource demand using Lyapunov optimization theory to predict changes in demand and efficiently make coupled decisions. The authors of [32] optimized the problem of resource management, considering how previous actions impact the future long-term reward of MEC servers regarding computational latency and energy utilization. To solve this optimization problem, they introduced a multi-agent Deep Deterministic Policy Gradient (DDPG)-based strategy for resource allocation, which also reduces the action space. The authors of [33] proposed a cooperative multi-agent DRL (Coo-MADRL) method to tackle challenges in multi-cloud and multi-edge environments, such as network topology and bandwidth constraints. This method maximizes server and link resources while minimizing task latency, using centralized training and decentralized implementation to optimize rewards. The authors of [34] introduced a single- and multi-agent DRL-based one-shot agent scheme that autonomously manages network slice requests. The one-shot agent decides where to place these requests on the physical infrastructure to maximize the reward and the total number of accepted requests, thereby improving QoS.

A joint optimization model integrating resource constraints was proposed to address the joint optimization problem of service migration and resource allocation (SMRA). The authors of [35] introduced a DRL-based algorithm for SMRA, where Long Short-Term Memory (LSTM) predicts mobile user behavior and identifies an optimal resource allocation scheme, while a Parameterized Deep Q Network (P-DQN) resolves the migration policy to maintain service continuity. The authors of [36] addressed the issue of completing tasks before their deadlines while minimizing energy consumption and cost. They proposed an end-to-end DRL scheme to handle the large action space by maximizing rewards and offloading tasks to the best server, thereby mitigating computational costs. The authors of [37] used an intelligent natural actor–critic DRL to minimize energy costs and latency while maximizing available processing capacity (APC), thereby improving system performance and rewards. The authors of [38] proposed a DRL-based dueling DQN scheme to handle the resource allocation and energy cost minimization problem. This scheme addresses the issue in multi-tenant networks where lower-priority tenants dynamically change their behavior to that of high-priority tenants, potentially leading to increased energy consumption costs.

The authors of [39] addressed the issues of task delay and extra energy consumption regarding the joint offloading and resource allocation (JORA) problem in MEC networks, aiming to maintain the Quality of Experience (QoE) for end users. They introduced the Lyapunov optimization approach to maximize long-term QoE and an energy deficit queue to guide real-time, online solutions for the energy consumption problem. By addressing the issues of the Age of Information (AoI), energy consumption, and convergence, the authors of [40] focused on minimizing the AoI in a wirelessly powered IoT environment. The challenge of optimizing transmission selection, channel selection, transmission duration, and transmit power was demonstrated to be NP-hard. The proposed strategy, a distributed multi-node resource allocation method called distributed multi-node resource allocation (DDMRA), combines the DDPG approach with the discrete action selection technique from the DQN.The research reported in [41] focused on maximizing QoE by correlating it with QoS through joint task and resource allocation. A multi-agent DRL-based model for Distributed Joint Task and Computing Resource Allocation (DJTCRA) dynamically allocates resources to user tasks, partially observing real-time states in edge computing to enhance QoE.

To solve the network slicing optimization problem of computing, communication, and cache (3C), along with QoS in MEC, the authors of [42] maximized utility using a DRL-based twin-actor DDPG algorithm. This algorithm intelligently adapts to changes in the environment and effectively allocates resources. In [43], the authors aimed to maximize the expected utility based on the reservation of slices by introducing a two-time-scale scheme. In the long time scale, tenants decide whether to activate the slice or not, and in the short time scale, they reconfigure the active slice by adapting to user demand. The Frank–Wolfe algorithm was proposed to solve the long-time-scale convex approximation problem, while Least Absolute Shrinkage and Selection Operator (LASSO) regularization is used for short-time-scale slice reconfiguration.

To address resource utilization for VNF sharing and new-instance creation, the authors pf [44] introduced an integer linear programming (ILP)-based core network slicing (OCNS) model. They also proposed a heuristic backtracking algorithm for network slicing (HBA-NS) that allocates resources sequentially while considering service-level agreements (SLA). Additionally, a deep learning (DL)-based Convolutional Neural Network (CNN)+LSTM model predicts requests using historical data to adjust resources for future time slots. The study reported in [45] highlights network slicing as a key technology for next-generation networks, efficiently utilizing resources to increase the profit of SPs. The SLA-NS framework optimizes SP profit through network slice pricing, demand forecasting, and resource allocation. By employing a two-layer game model and an LSTM predictor, SLA-NS effectively decouples pricing and allocation to enhance efficiency. The authors of [46] proposed an online DRL-based model-free policy gradient strategy to address the problem of slice tenants (STs) requesting and selecting resources from infrastructure providers based on predefined configurations. A two-stage reward function (TRF) was designed to optimize the objective and minimize the cost for slice brokers (SBs).

To improve network slice mobility (NSM) through future predictions, the authors of [47] proposed a federated DRL (FDRL) mechanism. This approach improves system scalability and flexibility and boosts long-term profit using FDRL predictions and double deep Q learning (DDQN) for decision making. The authors of [48], introduced a probabilistic forecasting technique for the prediction of network slicing demand, building upon prior deterministic techniques. A DeepAR-based slice admission control mechanism was proposed for making sequential decisions in Software-Defined Networks (SDNs), supported by the SDN controller. Additionally, a closed-loop parameter-updating method was developed to enhance the admission control process. The efficacy of these methodologies was demonstrated using actual traffic data from real-world scenarios. The authors of [49] addressed power allocation and request acceptance by considering resource limitations, user priorities, system stability, and long-term performance. They introduced the following two approaches: prediction-aided weight DRL (PW-DRL) for online power allocation and request acceptance and trust region policy optimization (TRPO) for adaptation to environmental changes.

Compared to the aforementioned resource optimization studies, our proposed DRL-based MAgSAC end-to-end network slicing approach considers both common and urgent requests of IoT users on MEC networks. It aims to optimize utility, mitigate energy consumption costs, and maximize revenue when resources on the cloudlets are limited. Additionally, our approach balances resource consumption with minimal time in the cloudlets. In terms of previous extended DRL approaches, MAgSAC represents a way of integrating a multi-agent framework with an extensive reward function and a modern MDP model. This integration effectively manages the fluctuating nature of IoT traffic and resource limitations, resulting in higher efficiency in handling VNF instances and fair resource allocation among cloudlets.

3. Motivation

Our main goal is to deliver dynamic and flexible services to IoT users, optimizing utility by balancing revenue, energy costs, latency, and QoS. Users select services for specific periods, while service providers (SPs) allocate resources based on available capacity and QoS efficiency. We use various examples to substantiate this approach. IoT users have diverse needs; some require uninterrupted connectivity, while others need a flexible QoS. For instance, soil moisture sensors in agriculture monitor humidity at intervals, while autonomous vehicles need constant connectivity for real-time traffic updates and safety. In emergencies, such as fire alarms, immediate responses are essential.

These examples demonstrate that IoT requirements vary by situation. Unknown user demands and resource availability can severely impact QoS. Efficient resource use requires the identification of user needs and allocation resources accordingly. The reservation of resources can lead to either under- or over-utilization depending on whether IoT users make regular or periodic requests. A lack of requests can waste resources and increase costs, while simultaneous requests from multiple users can cause network congestion.

Congestion varies by time and location, with some areas experiencing higher demand at different times. SPs offer services based on their capacity and coverage, but some may struggle to meet specific service levels. For example, urgent requests with strict delay requirements may conflict with common requests needing flexible delays, leading to increased latency, higher costs, reduced profit, and degraded QoS without a proper strategy. Assigning all urgent requests to the primary cloudlet can reduce latency but may create an imbalance in resource allocation, overburdening one site. To the best of the authors’ knowledge, existing research does not adequately address the challenge of balancing urgent and normal requests across cloudlets while optimizing overall utility and QoS.

4. System Model

Consider a mobile edge computing network represented by $\mathcal{G}=\left(\mathcal{Z}\cup \mathcal{V},\mathcal{E}\right)$ in which $\mathcal{V}$ denotes a set of access points (APs), i.e., $\mathcal{V}=\left\{v_1,v_2,v_3,\dots ,v_n\right\}$ and $\mathcal{E}$ describes the set of edges in the network, i.e., $\mathcal{E}=\left\{e_1,e_2,e_3,\dots ,e_n\right\}$. In this edge computing network, resource-limited cloudlets, denoted by $\mathcal{Z}=\left\{z_1,z_2,z_3,\dots ,z_n\right\}$, are placed at various locations close to users and are interconnected to perform end-to-end network slicing. Each cloudlet ($z_j\in \mathcal{Z}$) has virtualized computing resources available as VNF instances to implement the different requested services in $\mathcal{G}$. Let ${\mathcal{F}}_l\in \mathcal{VNF}$ and $l=\left\{1,2,3,\dots .,\left|\mathcal{F}\right|\right\}$ represent the set of various VNF instances at each $z_j$, indicating the capacity ($z_j^{cap}$) of $\mathcal{Z}$ in the form of activated ($f_l^{\oplus }$) and deactivated ($f_l^{\oslash }$) VNF instances. We assume that in each cloudlet ($z_j$), a VNF instance ($f_l$) can be assigned to one IoT user request at a time, but the nature and demand of the user request need to be estimated in advance. Let $p\left(f_l\right)$ denote the computing resource requirement of the VNF instance ($f_l$). Additionally, we assume that the monitoring time period (T) is divided into equal time slots denoted by t and indexed by $1\le t\le T$. A limited amount of bandwidth (B) is available for each cloudlet to transmit and receive data. Let $D_e$ and $B_e$ be the delay and bandwidth resource capacity of each edge ($\mathcal{E}$) of the cloudlet ($z_j$), respectively. The cloudlets ($z_j$) are interconnected with a set of links ($\mathcal{E}(i,j)$). In $\mathcal{G}$, via $v\in \mathcal{V}$, the cloudlets $\mathcal{Z}$ receive the network slice requests ($r_i^s\in R^s$), with delay requirements sent by users from various IoT devices to the nearest $z_j$ in that area. Figure 1 illustrates the MEC network, where APs with interconnected cloudlets have the capacity to provide end-to-end network slicing services to IoT users in the form of activated and deactivated VNF instances. The terminology employed in this paper is defined in

Table 1. Terminology.

Term	Definition
$\mathcal{G}=\left(\mathcal{Z}\cup \mathcal{V},\mathcal{E}\right)$	MEC network, where $\mathcal{Z}$ is a set of cloudlets, $\mathcal{V}$ is a set of APs, and $\mathcal{E}$ is a set of edge links
${\mathcal{F}}_l$	A set of VNF instances
$z_j^{cap}$	Capacity of a cloudlet $z_j$
$f_l^{\oplus }$	An activated VNF instance
$f_l^{\oslash }$	A deactivated VNF instance
$p\left(f_l\right)$	Computing resource requirement of VNF instance $f_l$
t	Time-slot index $1\le t\le T$
$B_e$	Bandwidth at the edge (e)
$D_e$	Delay at the edge (e)
$\mathcal{E}(i,j)$	A set of links
$r_i^s\in R^s$	Network slice requests
$\lambda$	Arrival rate of requests
${\varrho }_{j,l}^t$	Total number of unfinished requests
${\rho }_{i,j}^t$	Request currently being processed
$r_i^s$	Common requests
$r_{urg}^s$	Urgent request
$S{D}_i$	T distribution
$\Delta$	Standard deviation
${\mathfrak{U}}^{up}$	Upper bound
${\mathfrak{U}}^{lo}$	Lower bound
$\eta$	Average T distribution
$d_n^{u,h}$	Overall computational delay in the home cloudlet for urgent requests
$d_{q{w}^{\prime }}^{r^s}$	Additional queue delay in adjacent cloudlet
$d_{\overline{qw}}^{r^s}$	Queue waiting delay
$d_n^{u,ad}$	Overall computational delay ub adjacent cloudlet for urgent requests
${\chi }_n^{r^{s,t}}$	A binary decision variable $\{0,1\}$
$d_n^{c,h}$	Overall computational delay in home cloudlet for common requests
$d_n^{c,ad}$	Overall computational in adjacent cloudlet for common requests
$b_k^t$	Required bandwidth to transfer a request via the $k^{th}$ link
${\mathcal{D}}_j^t$	Overall delay faced by common and urgent requests in the home and adjacent cloudlet
$c\left(f_l\right)$	Computing cost of accommodating one unit of traffic
$c_n^{u,h}$	Overall energy consumption cost for urgent requests in the home cloudlet
$c_n^{u,ad}$	Overall energy consumption cost for urgent requests in the adjacent cloudlet
$c_n^{c,h}$	Overall energy consumption cost for common requests in the home cloudlet
$c_n^{c,ad}$	Overall energy consumption cost for common requests in the adjacent cloudlet
${\mathcal{C}}_j^t$	Overall energy consumption cost for common and urgent requests in the home and adjacent cloudlet
$P{R}_j^t$	Total profit earned by the cloudlet
${\left(f_l^{\oplus }\right)}^{idl}$	An idle VNF instance
$z_j^{\ast cap}$	Remaining computing capacity of a cloudlet $z_j$
${\varphi }_{j,l}^t$	The decision on the assignment of resources for action $a_i^t$
${\theta }_i^{\iota }$	Critic network with parameters ${\theta }_i^{\iota }$ $(\iota =1,2)$
${\tilde{\theta }}_i^{\iota }$	Target network
${\alpha }_i$	Temperature parameter
${\mathcal{B}}_r$	Experience reply buffer
${\mathcal{B}}_{\varpi }$	Mini-batch
£	Loss function
${\pi }_{{\zeta }_i}$	Policy parameter
$a^t$	Action space
$s^t$	State space

.

4.1. IoT Users Requests

In MEC networks, cloudlets ($\mathcal{Z}$) are located at different locations, and the number of incoming IoT user requests ($r_i^s\in R^s$) varies from time to time and area to area. For example, during peak hours in a shopping mall, the number of incoming requests from IoT users can be significantly higher than in a residential area during the same time. Each user request ($r_i^s$) is received by the nearest cloudlets through v. Cloudlet $z_k$, which is near a shopping mall, airport, or train station, is much busier than cloudlets in rural areas. Due to the limited availability of computing resources in each cloudlet, it is hard to handle all the requests ($r_i^s$). Cloudlet $z_j$ sends overloaded requests to the adjacent cloudlet ($z_{k,i}\subseteq z_{k,j}$) in the case of the unavailability of the required resources to avoid imbalance in data overhead and latency issues.

By considering the continuous agility of IoT users moving between different cloudlets, we assume that incoming requests for required resources arrive randomly from various IoT users with an arrival rate of $\lambda$. Therefore, the arrival rate of user requests is denoted by

$$\lambda ={\displaystyle \frac{\mathcal{Z}[r_i^s.p\left(f_l\right)]}{t}},$$

(1)

where $r_i^s$ represents the demand of the user for computing resource ($p\left(f_l\right)$) from the set of cloudlets ($\mathcal{Z}$) at time slot t.

We denote the flow of requests from one cloudlet to another as $F(i,j)$. To ensure that all incoming requests are accommodated, we ensure that the flow (F) of incoming requests does not exceed the edge-link capacity, i.e., $F(i,j)\le {\mathcal{E}}^{cap}$. We assume that if the sum of incoming requests equals the sum of outgoing requests, denoted as $\sum T_{in}=\sum T_{out}$, there is no rejection of requests and no delay in request processing.

It is important to remember that ${\varrho }_{j,l}^t$ represents the total number of unfinished requests for service ($f_l$) during the resource allocation phase of time slot t. Therefore, we denote the unfinished requests as follows:

$${\varrho }_{j,l}^t=\sum _{r_i^s\in f_l\cap R^{s,t}}{\rho }_{i,j}^t,\forall i,j,l,t,$$

(2)

where ${\rho }_{i,j}^t$ indicates that the request ($r_i^s$) is currently being processed by cloudlet $z_j$ in time slot t.

4.2. Common and Urgent Requests

IoT users send heterogeneous kinds of requests to fulfill their specific requirements. Based on the demand, we divide these requests into two categories.

4.2.1. Common Requests

IoT users’ requests that have specific resource demands but not an urgent deadline are considered common requests ($r_i^s$). Moreover, it is accurate to characterize these requests as less critical. A few examples of common requests include real-time video conferences, room temperature monitoring, and virtual reality applications.

4.2.2. Urgent Requests

IoT users’ requests that arrive earlier than the regular period and demand strict and rapid assignment are classified as urgent requests ($r_{urg}^s$). Missing the deadline for these requests may lead to significant disasters and system failures. Urgent requests typically involve delay-sensitive requirements and an immediate response from the SP to mitigate further substantial losses. For instance, in natural disaster situations, an ambulance requires a prompt response to identify a shorter and less congested traffic route or a fire alarm sensor continuously analyzes data to prevent major damage. These types of IoT requests fall under urgent requests. MEC resources are deployed in such circumstances to process the sensed information and make swift decisions based on the collected data. In other words, the faster the response from the MEC network, the better the mitigation of major losses. To further distinguish between common and urgent requests, we introduce the following variable:

$$r_{urg}^s=\left\{\begin{array}{cc}1,\hfill & \mathrm{urgent}\mathrm{requests}\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(3)

In the MEC network, the system differentiates between common and urgent requests based on several factors, including IoT users’ application type, request requirements/characteristics, and priority level. Through the analysis of information collected from sensors, input sources, and historical records, relevant characteristics are identified and utilized to ascertain the nature of the requests, their sensitivity, the location of request origination, and environmental conditions. To distinguish between regular requests and urgent requests in human body monitoring and autonomous vehicle systems, T-distribution modeling [50] and anomaly detection approaches [51] are utilized to ensure effective and timely responses. In emergencies, these approaches respond effectively, allocate resources promptly according to the requirements, and ensure safety.

By integrating the previously outlined approaches of autonomous vehicles, we formulate an estimation range encompassing both upper and lower bounds based on the historical record of characteristics, required reaction time, location, and environmental factors [52]. The T distribution, denoted by $S{D}_i$, represents the sample characteristic associated with the response time for a normal sample size of n, where $n>0$. The sample mean value ($\overline{SD}$) is calculated as follows:

$$\overline{SD}={\displaystyle \frac{{\sum }_{i=1}^{n}S{D}_i}{n-1}}.$$

(4)

Therefore, the sample characteristics of the standard deviation can be defined by $\Delta$ as follows:

$$\Delta =\sqrt{{\displaystyle \frac{1}{n-1}}\sum _{i=1}^{max}S{D}_i-\overline{SD}}.$$

(5)

Therefore, the upper and lower bound can be described as follows:

$${\mathfrak{U}}^{up}=\overline{SD}-\sqrt{{\displaystyle \frac{1}{n-1}}}.\Delta .\eta ,$$

(6)

$${\mathfrak{U}}^{lo}=\overline{SD}+\sqrt{{\displaystyle \frac{1}{n-1}}}.\Delta .\eta ,$$

(7)

where $\eta$ denotes the average T-distribution coefficient for a sample size of n. The appropriate range for each monitored parameter of autonomous vehicles should fall within the ranges of Equations (6) and (7), which indicate the appropriate response time for the vehicle. At a given time slot (t), ${\mathfrak{U}}^t$ depicts the parameter of the recorded value of autonomous vehicles. Therefore, urgent requests at time slot t can be calculated as follows:

$$r_{urg}^s=\left[{\displaystyle \frac{{({\mathfrak{U}}^{up}-{\mathfrak{U}}^t)}^2-{({\mathfrak{U}}^{lo}-{\mathfrak{U}}^t)}^2}{{({\mathfrak{U}}^{up}-{\mathfrak{U}}^{lo})}^2}}\right].$$

(8)

Higher values of $r_{urg}^s$ indicate that the corresponding requests demand an urgent response, and lower values indicate that the request can be dealt with normally if $r_{urg}^s=0$. In some situations, the value of the $r_{urg}^s$ may be theoretically greater ($r_{urg}^s>1$). To ensure that the value remains uniform throughout, we constrain the maximum value by adjusting the urgent-request demand level to $r_{urg}^s=1$, which is the highest possible level of urgent requests. To ensure uniformity in value retention, we constrain the maximum value by adjusting the urgent-request demand level to $r_{urg}^s=1$, representing the highest possible level of urgent requests for demanding resources. For clarity regarding response time for urgent and common requests, we establish a reference range with an upper bound of ${\mathfrak{U}}^{up}=5.5$ ms and a lower bound of ${\mathfrak{U}}^{up}=3.5$ ms [53]. A required response time of 3 ms indicates an urgent request, which must be rapidly assigned to the relevant cloudlet. Computing urgent requests according to user requirements requires a specific mechanism, clear end-to-end communication, and an efficient resource allocation approach to QoS. Therefore, it is essential to introduce a method capable of handling both common and urgent user requests in a timely manner, which is described in Section 5.

4.3. End-to-End Delay

From the IoT users’ end to the destination, each request usually has delay requirements with specific QoS requirements. Considering the end-to-end delay requirements, we divide the delay of incoming requests into the following two parts: urgent-request delay and common-request delay.

Urgent-Request Delay: Each request is assumed to disintegrate and can be subdivided into sub-requests, which have to be processed in sequence. For each sub-request, knowledge about available computing resources in the home cloudlet is needed for fair allocation. The decision to allocate resources is based on the availability of computing resources in the related cloudlet; either the available capacity is enough to accommodate the urgent requests, idle instances, or remaining capacity or the request has to be transferred to the adjacent cloudlet if the delay requirements are flexible. However, we prioritize the primary cloudlet. The decision with respect to the allocation of the request at each time slot is denoted by $T_n^{u,h}=\{1,2,3,\dots ,T_n\}$. Therefore, the overall computation delay for users’ urgent requests can be written as follows:

$$d_n^{u,h}=\sum _{t\in T_n^{u,h}}(d_{qw}^{r^s}+d_{f_l^{\oplus }}^{r^s}+d_{exc}^{r^s}).\sigma t,\forall n,t,$$

(9)

where $d_{qw}^{r^s}$ is the queue waiting delay if $r_i^s$ is already in processing (${\rho }_{i,j}^t$), $d_{f_l^{\oplus }}^{r^s}$ is the VNF instance’s activation delay, $d_{exc}^{r^s}$ is the request-execution delay, and $\sigma t$ shows the time-slot period.

On the other hand, if the urgent request is assigned to the adjacent cloudlet in time slot t due to the unavailability of the required resources and only when the delay requirements are flexible, then the delay includes the transfer delay as well. Therefore, the additional delay for requests waiting in a queue for the adjacent cloudlet can be written as follows:

$$d_{q{w}^{\prime }}^{r^s}=\sum _{t\in T_n^{u,ad}}(d_{tr}^{r^s}+d_{exc}^{r^s}+d_{f_l^{\oplus }}^{r^s}+d_{\overline{qw}}^{r^s}),$$

(10)

where $d_{tr}^{r^s}$ is transmission delay and $d_{\overline{qw}}^{r^s}$ is the extra queue waiting delay. Therefore, the overall request delay to the adjacent cloudlet can be denoted as follows:

$$d_n^{u,ad}=\sum _{t\in T_n^{u,ad}}\sum _{k\in K}{b}_k^t.{\chi }_n^{r^{s,t}}\left(\sigma t+{\displaystyle \frac{c\left(f_l\right).(d_{q{w}^{\prime }}^{r^s}+d_{tr}^{r^s}+d_{exc}^{r^s})}{f_l}}\right)\forall k,n,t,$$

(11)

where ${\chi }_n^{r^{s,t}}\in \{0,1\}$ is a binary decision variable, where ${\chi }_n^{r^{s,t}}=\left\{0\right\}$ when the urgent request is assigned to the home cloudlet and ${\chi }_n^{r^{s,t}}=\left\{1\right\}$ otherwise. $f_l$ denotes the available computing resources in the adjacent cloudlet that can be allocated to the transferred request (n) if the bandwidth $b_k^t$ at the kth link is enough to transfer the request for processing, and $c\left(f_l\right)$ is the computing resource demand of the user at time slot $\sigma t$.

Common-Request Delay: Common requests sent by users are flexible and can be assigned to the home cloudlet or transferred to the adjacent cloudlet. Common requests play an essential role in maintaining a balance of cloudlet resources for allocation. Therefore, the decision with respect to resource allocation at each time slot is depicted by $T_n^{c,h}=\{1,2,3,\dots ,T_n\}$, and the overall computational delay of common requests is illustrated as follows:

$$d_n^{c,h}=\sum _{t\in T_n^{c,h}}1-{\chi }_n^{r^{s,t}}(d_{qw}^{r^s}+d_{f_l^{\oplus }}^{r^s}+d_{exc}^{r^s}).\sigma t,\forall n,t,$$

(12)

where ${\chi }_n^{r^{s,t}}$ is a binary decision variable $\{0,1\}$. If ${\chi }_n^{r^{s,t}}=\left\{0\right\}$, the request is processed in the home cloudlet, and ${\chi }_n^{r^{s,t}}=\left\{1\right\}$ otherwise. The terms $d_{qw}^{r^s}$, $d_{f_l^{\oplus }}^{r^s}$, and $d_{exc}^{r^s}$ represent queue waiting, VNF activation, and request execution delays, respectively. If, due to the limited availability of resources in the home cloudlet or to balance the resource allocation process among cloudlets, a request is transferred to the adjacent cloudlet at time slot t [54]; then, such a request can face an overall delay, which includes a processing delay in the home cloudlet, delayed transmission delay to the adjacent cloudlet, and computational delay in the adjacent cloudlet, which can be written as

$$d_n^{c,ad}=\sum _{t\in T_n^{c,h}}\sum _{k\in K}{\varrho }_{j,l}^t.b_k^t.{\chi }_n^{r^{s,t}}\left(\sigma t+{\displaystyle \frac{c\left(f_l\right).(d_{q{w}^{\prime }}^{r^s}+d_{tr}^{r^s}+d_{exc}^{r^s})}{f_l}}\right),\forall j,k,l,n,t,$$

(13)

where ${\varrho }_{j,l}^t$ represents the total number of unfinished requests already in processing, which may cause an extra delay and can be released in the next time slot ($t+1$). $b_k^t$ denotes the available bandwidth required to transfer the request in the $k^{th}$ link to the adjacent cloudlet. ${\chi }_n^{r^{s,t}}$ is a binary decision variable $\{0,1\}$. $p\left(f_l\right)$ shows the computing resource demand of the request at time slot t, and $f_l$ represents the available computing resources in the cloudlet. $d_{qw}^{r^s}$ is the queue waiting delay, $d_{f_l^{\oplus }}^{r^s}$ is the VNF activation delay, and $d_{exc}^{r^s}$ is the execution delay on all kth links in time slot t.

Therefore, the overall delay faced by urgent and common users requests in the home cloudlet ($z_j$) or the adjacent cloudlet ($z_k$), along with the delay in links, which should not exceed the overall delay requirements ($D_e$), can be written as

$${\mathcal{D}}_j^t=\sum _{t\in T}^{r_i^s\in R^s}\sum _{k\in K}(d_n^{u,h}+d_n^{u,ad}+d_n^{c,h}+d_n^{c,ad})\le D_e,\forall i,j,k,n,t.$$

(14)

4.4. Energy Consumption Cost and Profit

The network SP accommodates users and charges costs in return by implementing requests on demand. The cost of request implementation is based on the utilization of network resources. We assume that the computing resources are already leased from infrastructure providers and are available to the cloudlets in the form of deactivated VNF instances. To assign a request to a network slice, the VNF instance must be in an activated condition. Activating an instance incurs energy usage, which is converted into the cost of activating and deactivating the VNF instances. This cost is divided into the following two parts: urgent-request cost and common-request cost.

Urgent-Request Cost: We consider that some requests sent by IoT users require a rapid response. Let $c\left(f_l\right)$ denote the computing cost of accommodating one unit of traffic ($r_i^s$) in cloudlet $z_j$ during time slot t. The cost is determined based on where the request is implemented. In cases where an urgent request ($r_i^s$) is processed in the home cloudlet ($z_j$) and $r_i^s\le z_j$ with at least one VNF instance ($f_l^{\oplus }$) activated, the overall cost of energy consumption for urgent requests can be written as

$$c_n^{u,h}=\sum _{l=1}^{\left|\mathcal{F}\right|}\sum _{t\in T_n^{u,h}}\left(c\left(f_l\right)+{\displaystyle \frac{c\left(f_l^{\oplus }\right)+c{\left(f_l^{\oplus }\right)}^{idl}+c{\left(f_l^{\oplus }\right)}^{pro}+c\left(f_l^{\oslash }\right)}{f_l}}\right)\sigma t,$$

(15)

where $c\left(f_l^{\oplus }\right)$ denotes the cost of activating VNF instances, $c{\left(f_l^{\oplus }\right)}^{idl}$ is the cost generated by idle VNF instances, $c{\left(f_l^{\oplus }\right)}^{pro}$ is the processing cost to host requests, $c\left(f_l^{\oslash }\right)$ is the cost of deactivating the VNF instance, and $\sigma t$ represents the time period.

When the available resource capacity at $z_j$ is not sufficient to fulfill the demand of $r_i^s$, requests may be transferred to the adjacent cloudlet for processing in the case of flexible requests. Then, the cost ($c\left(f_l\right)$) is charged for the consumption of the bandwidth ($b_k^t$) at the kth link at time slot t, and the energy consumption cost to process the user request in the neighboring cloudlets can be denoted as

$$c_n^{u,ad}=\sum _{l=1}^{\left|\mathcal{F}\right|}\sum _{t\in T_n^{u,ad}}\sum _{k\in K}1-{\chi }_n^{r^{s,t}}\left(\sigma t+{\displaystyle \frac{c\left(b_k^t\right)+c\left(f_l^{\oplus }\right)+c{\left(f_l^{\oplus }\right)}^{idl}+c{\left(f_l^{\oplus }\right)}^{pro}}{f_l}}\right),$$

(16)

where $c\left(b_k^t\right)$, $c\left(f_l^{\oplus }\right)$, $c{\left(f_l^{\oplus }\right)}^{idl}$, and $c{\left(f_l^{\oplus }\right)}^{pro}$ represent the bandwidth cost when a request is transferred to the adjacent cloudlet at each time slot (t), the cost of activating the VNF instance, the cost of idle VNF instances, and the processing cost, respectively.

Common-Request Cost: The cost generated by common requests ($r_i^s$) allocated to the home cloudlet ($z_j$) with acceptable delay requirements can be calculated as follows:

$$c_n^{c,h}=\sum _{l=1}^{\left|\mathcal{F}\right|}\sum _{t\in T_n^{c,h}}1-{\chi }_n^{r^{s,t}}\left({\displaystyle \frac{c\left(f_l^{\oplus }\right)+c{\left(f_l^{\oplus }\right)}^{idl}+c{\left(f_l^{\oplus }\right)}^{pro}+c\left(f_l^{\oslash }\right)}{f_l}}\right)\sigma t,$$

(17)

where ${\chi }_n^{r^{s,t}}$ is the decision variable, $c\left(f_l^{\oplus }\right)$ represents the cost of activating VNF instances, $c{\left(f_l^{\oplus }\right)}^{idl}$ is the cost generated by idle VNF instances, $c{\left(f_l^{\oplus }\right)}^{pro}$ is the processing cost, $c\left(f_l^{\oslash }\right)$ is the cost of deactivating the VNF instance, and $\sigma t$ denotes the time period.

We assume that some common requests arrive without real-time requirements for execution but can also be transferred to the adjacent cloudlet ($z_{j\to k}$) due to their flexible demand, maintaining balance among the cloudlets. Thus, we can express the energy consumption cost of common requests as

$$c_n^{c,ad}=\sum _{l=1}^{\left|\mathcal{F}\right|}\sum _{t\in T_n^{c,ad}}\sum _{k\in K}{\chi }_n^{r^{s,t}}\left(c\left(f_l\right)+{\displaystyle \frac{c\left(b_k^t\right)+c\left(f_l^{\oplus }\right)+c{\left(f_l^{\oplus }\right)}^{idl}+c{\left(f_l^{\oplus }\right)}^{pro}+c\left(f_l^{\oslash }\right)}{f_l}}\right)\sigma t,$$

(18)

where ${\chi }_n^{r^{s,t}}$ is the decision variable to transfer the requests to the adjacent cloudlet at time slot t, $c\left(b_k^t\right)$ is the bandwidth consumption cost on the links during transfer of requests, $c\left(f_l^{\oplus }\right)$ is the VNF activation cost, $c{\left(f_l^{\oplus }\right)}^{idl}$ is the cost of idle VNF instances, $c{\left(f_l^{\oplus }\right)}^{pro}$ is the request processing cost, and $c\left(f_l^{\oslash }\right)$ is the VNF instance deactivation cost in cases in which it cannot meet the user requirements.

Therefore, the overall cost of energy consumption for the SP to provide the required resources to urgent and common requests in the home cloudlet ($z_j$) or an adjacent cloudlet ($z_k$) can be written as

$${\mathcal{C}}_j^t=\sum _{l=1}^{\left|\mathcal{F}\right|}\sum _{t\in T}^{r_i^s\in R^s}\sum _{k\in K}(c_n^{u,h}+c_n^{u,ad}+c_n^{c,h}+c_n^{c,ad}).$$

(19)

The SP earns profit by charging the cost to provide the required resources for IoT user requests. We assume that $P{R}^{f_l}$ represents the profit obtained by the SP for providing a service instance ($f_l$) for a user request at each time slot. The total profit earned by cloudlet $z_j$ at each time slot (t) is expressed as

$$P{R}_j^t=\sum _{r_i^s\in R^s}{\tau }_i.P{R}_i^{f_l}.$$

(20)

5. Optimization Problem for End-to-End Network Slicing

In this section, we illustrate the multi-agent Markov decision process (MAMDP)-based problem formulation and the DRL-based MAgSAC procedure to solve the utility optimization problem in end-to-end network slicing.

5.1. MAMDP-Based Problem Formulation

In the proposed multi-agent approach [55], the primary action space is divided into various sub-action spaces, with each agent assigned responsibility for a specific sub-action space. Each agent can observe and analyze the actions of its peer agents. The reward is based on the aggregation of actions from all agents rather than from an individual agent. Therefore, it is crucial for all agents to collaborate to maximize the overall reward, as failure to do so would result in suboptimal outcomes. These actions impact the environment, which subsequently changes.

We present MAMDP as a tuple $(\mathcal{N},\mathcal{S},\mathcal{A},\mathcal{U})$, where $\mathcal{N}$ denotes the set of agents, $\mathcal{S}$ is the set of possible environment states, $\mathcal{A}$ represents the set of actions associated with agents, and $\mathcal{U}$ illustrates the utility function. The main objective of the agents is to learn an optimal policy to maximize the accumulated reward at each time slot (t). This procedure continues until the agent discovers an optimal policy. A DRL-based approach can quickly map different operating states to actions, in contrast to traditional optimization methods that must solve the problem for every operating state [56]. The end-to-end network slicing problem can be formulated as an MAMDP as follows:

Agent $\mathcal{N}$: In the MEC system, each IoT user requests network resources. An agent, represented by $\mathcal{N}$, makes decisions based on available resources. It assigns the required resources to requests in the targeted cloudlet ($z_j$) and manages the entire process. These agents (${\mathcal{N}}_i\in \mathcal{N}$) are trained to utilize the MAgSAC approach, using their observations of the environment and action space to collectively achieve the optimal policy at each time slot (t).

State Space $\mathcal{S}$: In each time slot (t), the agents observe the status of the MEC environment and aggregate the following information:

• Requests being processed in cloudlet $z_j$ at the $t^{th}$ time slot: ${\varrho }_{j,l}^t$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;
• Available computing capacity of cloudlet $z_j$ at the $t^{th}$ time slot: $z_j^{cap,t}$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;
• Idle VNF instances in cloudlet $z_j$ at the $t^{th}$ time slot: ${\left(f_l^{\oplus }\right)}^{idl,t}$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;
• Active VNF instances in cloudlet $z_j$ at the $t^{th}$ time slot: $f_l^{\oplus ,t}$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;
• Deactivated VNF instances in cloudlet $z_j$ at the $t^{th}$ time slot: $f_l^{\oslash ,t}$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;
• Available bandwidth resources at each edge at the $t^{th}$ time slot: $B_{e,i,j}^t$, $\forall {\mathcal{N}}_i\in \mathcal{N}$.

Therefore, the overall status of the MEC system can be described by state $s_i^t\in \mathcal{S}$, where $s_i^t=(s_i^t,s_2^t,\dots ,s_n^t)$. This represents the system state of cloudlet i at time slot t.

$$s_i^t=\left[r{\varrho }_{j,l}^t,\dots ,z_j^{cap,t},\dots ,{\left(f_l^{\oplus }\right)}^{idl,t},\dots ,f_l^{\oplus ,t},\dots ,f_l^{\oslash ,t},\dots ,B_{e,i,j}^t\right].$$

(21)

Action Space $\mathcal{A}$: After observing the system state, the agents collaborate to manage the resource allocation process. This includes activating or deactivating necessary VNF instances in the cloudlets to process IoT user requests at each time slot (t). The action space ($\mathcal{A}$) is defined as follows: $\mathcal{A}=(a_1,a_2,\dots ,a_n)$, which encompasses the following actions in the observed environment:

• Amount of computing resources in cloudlet $z_j$ that can be assigned to a VNF instance at the $t^{th}$ time slot: $f_l^t$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;
• Activation of a VNF instance in cloudlet $z_j$ to fulfill user requirements at the $t^{th}$ time slot: $f_l^{\oplus ,t}$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;
• Amount of remaining computing resources in cloudlet $z_j$ that can still be assigned at the $t^{th}$ time slot: $z_j^{\ast cap,t}$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;
• Deactivation of a VNF instance in cloudlet $z_j$ at the $t^{th}$ time slot: $f_l^{\oslash ,t}$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;
• Transfer of the request to the adjacent cloudlet ($z_i$) at the $t^{th}$ time slot: $z_{i\to j}^t$, $\forall {\mathcal{N}}_i\in \mathcal{N}$;

Therefore, the action space ($a_i^t\in \mathcal{A}$) of the agents in the $i^{th}$ cloudlet at time slot t can be written as

$$a_i^t=\left[a_{f_l^t},\dots ,a_{f_l^{\oplus ,t}},\dots ,a_{z_j^{\ast cap,t}},\dots ,a_{f_l^{\oslash ,t}},\dots ,a_{z_{i\to j}^t}\right]$$

(22)

While taking actions to satisfy user requirements, the following constraints must also be considered:

• One VNF instance must be activated in the cloudlet to assign the user request;
• If an idle instance is available, it must meet the needs of the user request to promote reuse;
• The remaining capacity in the cloudlet for the instance should be greater than the incoming request;
• To transfer the request to an adjacent cloudlet, the bandwidth available between the two cloudlet edges ($e_i\leftarrow e_j$) should be sufficient for this transfer.

When deciding whether to activate or deactivate the VNF instance for assignment, it is necessary to keep track of unfinished requests. Thus, the remaining capacity in cloudlet $z_j$ that can still be allocated at time slot t is expressed as $z_j^{\ast cap,t}=z_j^{cap}-{\sum }_{l=1}^{\left|\mathcal{F}\right|}{\varrho }_{j,l}^t·p\left(f_l\right)$. Then, the decision with respect to the assignment of resources for action $a_i^t$ is further specified by ${\varphi }_{j,l}^t=⌊{\displaystyle \frac{z_j^{\ast cap,t}·{\stackrel{‘}{a}}_j}{p\left(f_l\right)}}⌋$.

Utility Function $\mathcal{U}$:

With 5G network slicing, the resource demands of IoT users can vary greatly. Because different services, from security cameras to online gaming, have distinct utility functions, they prioritize different network aspects. For example, a weather sensor might prioritize low latency (fast response) for timely data transmission, while a video call might prioritize high bandwidth for smooth video streaming. Well-defined utility functions enable agents to make well-informed judgments on the most effective way to utilize resources, resulting in the optimal distribution of vital resources. In utility optimization functions, our goal is to optimize cloudlet resources intelligently while allocating them to IoT user requests with minimum latency. Intelligent allocation involves effectively optimizing overall utility by intelligently activating and deactivating VNF instances, reutilizing idle instances and remaining capacity, efficiently utilizing bandwidth during request transfer, and maintaining QoS according to user demand and MEC network states.

For each IoT user request ($r_i^s$), the agent (${\mathcal{N}}_i$) takes an action ($a_i^t$) based on the given environmental state ($s_i^t$), and the environment responds with the immediate utility. The utility of the MEC network is based on revenue and cost. Revenue comprises efficient utilization of MEC resources, considering QoS, while the cost includes the energy consumption required to accommodate IoT users. According to Equations (19) and (20), after excluding the cost from the revenue ($P{R}_j^t-{\mathcal{C}}_j^t$), the immediate utility function of cloudlet $z_j$ of the ${\mathcal{N}}_i$ at each time (t) can be presented as

$${\mathcal{U}}_i^t=\sum _{t=1}^T\sum _{j=1}^{\left|\mathcal{Z}\right|}(P{R}_j^t-{\mathcal{C}}_j^t).$$

(23)

Based on the utility model ($u_i^t\in {\mathcal{U}}_i^t$) defined above, we can express the optimization problem in the following way:

$$\underset{{(f_l,f_l^{\oplus },z_j^{\ast cap},f_l^{\oslash },z_{i\to j})}^t}{\mathrm{Maximize}}\lim \underset{T\to \infty }{\sup }\mathbb{E}\left[\sum _{t=0}^{T-1}{\mathcal{U}}_i^t,\right]$$

(24)

subject to

$$\sum _{l=1}^{\left|\mathcal{F}\right|}{f}_l^t.p\left(f_l\right)\le z_j^{cap},\forall j,l,t$$

(24a)

$$\sum _{l=1}^{\left|\mathcal{F}\right|}{f}_l^t.p\left(f_l\right)\le z_j^{\ast cap},\forall j,l,t$$

(24b)

$$\sum _{l=1}^{\left|\mathcal{F}\right|}{f}_l^t.p\left(f_l\right)+\sum _{r_i^s\in R^{s,t}}{f}_l^{\oplus }.p\left(f_l\right)\le z_j^{cap},\forall i,j,l,t$$

(24c)

$$\sum _{f_l\in \mathcal{F}}{\left(f_l^{\oplus }\right)}^{idl}\le f_l^t-{\varrho }_{j,l}^t,\forall j,l,t$$

(24d)

$${\varrho }_{j,l}^t\le f_l^t,\forall j,l,t$$

(24e)

$$\sum _{t=1}^T\sum _{j=1}^{\left|\mathcal{Z}\right|}{\chi }_n^{r^{s,t}},\forall j,n,t$$

(24f)

$$f_l^t\ge 0,\forall l,t$$

(24g)

$${\varrho }_{j,l}^t\in \mathbb{Z},\forall j,l,t$$

(24h)

$${\chi }_n^{r^{s,t}}\in \{0,1\}\forall n,t$$

(24i)

$${\chi }_n^{r^{s,t}}{\left(f_l^{\oplus }\right)}^{idl}\in \{0,1\}\forall l,n,t$$

(24j)

$${\chi }_n^{r^{s,t}}\left(f_l^{\oplus }\right)\in \{0,1\}\forall l,n,t$$

(24k)

$${\chi }_n^{r^{s,t}}\left(f_{l)}^{\oslash }\right)\in \{0,1\}\forall l,n,t$$

(24l)

where $\mathbb{E}[.]$ represents an estimate of the agent’s long-term utility. Constraint (24a) ensures that the computing demand of the user for resources at each time slot does not exceed the total amount of available capacity in the related cloudlet. Constraint (24b) indicates that the computing demand of the user does not exceed the remaining available capacity in the cloudlet. Constraint (24c) serves as a preventive measure to ensure that the closest cloudlets do not exceed their computational capabilities by enabling an excessive number of VNF instances. The initial part of constraint (24d) denotes the total number of requests that demand a VNF instance ($f_l$) and that are assigned to an idle instance in cloudlet $z_j$. This total must not exceed the threshold stated in the second part of the constraint. The second portion of the constraint refers to the number of idle VNF instances that are currently available. Constraint (24e) denotes the number of VNF instances, which is equal to or greater than the number of instances being processed. This is essential for efficient resource allocation, since the scaling of idle instances should be minimized. Constraint (24f) guarantees that each user request is solely assigned to a single cloudlet—either the home cloudlet or an adjacent cloudlet.

5.2. Multi-Agent Soft Actor–Critic-Based Learning

We adopt a model-free, SAC-based multi-agent DRL strategy to execute IoT requests within the MEC network by modifying the standard reinforcement learning (RL) approach. Figure 2 illustrates the multi-agent SAC-based architecture. The multi-agent SAC framework comprises three primary components aimed at enhancing the performance of the approach [57].

• Actor–Critic Structure: SAC operates according to the actor–critic structure, which includes an actor part and a critic part. The actor contributes to determination of the optimal strategy that maximizes expected utility, whereas the critic provides an estimation of the state and state–action value over that period. By leveraging the actor–critic structure, SAC effectively combines policy-based and value-based RL, which is a positive aspect of this approach.
• Entropy Maximization: By incorporating entropy assessments of policies into the utility function, the stochasticity of SAC’s policy substantially improves, thereby enabling the exploration of a wider range of potentially optimal decisions. Compared to previous policy-based DRL methods, the SAC approach demonstrates greater adaptability and scalability, allowing it to adapt effectively in stochastic environments. In short, maximizing entropy in the SAC algorithm promotes exploration and enhances the ability of the policy to adapt to complex and extremely large environments.
• Off-Policy Learning: To train network parameters based on the experience replay strategy, SAC utilizes an off-policy formulation. This approach enables the efficient utilization of sampled experiences to achieve smooth convergence. SAC leverages the following three key features: off-policy learning, the actor–critic framework, and entropy maximization. These features collectively contribute to SAC’s effectiveness in continuous control actions.

Following this, we offer a detailed description of how the SAC strategy is applied during the learning process. Initially, we delve into the concept of soft value functions within the SAC framework. Then, we explain the specific learning processes of both the critic (policy assessment) and the actor (policy improvement).

5.2.1. Soft Value Function

Our approach aims to find a policy (${\pi }_i\left({\mathit{a}}_i\right|s)$) that maximizes the expected utility in the long run. In the SAC-based DRL technique [58], an entropy term ($\mathcal{H}\left({\pi }_i\left({\mathit{a}}_i\right|s)\right)=-log\left({\pi }_i\left({\mathit{a}}_i\right|s)\right)$) is included in the utility function to ensure continuous exploration of the environment. The objective related to maximizing entropy is known as the entropy objective function and can be defined as follows:

$${\pi }_i^{\ast }=arg\underset{{\pi }_i}{max}\underset{{\mathit{s}}^t,{\mathit{a}}_i^t\sim {\pi }_i}{\mathbb{E}}\left[\sum _{t=0}^{\infty }{\gamma }^t\left(u_i^t+{\alpha }_i\mathcal{H}\left({\pi }_i\left(·|{\mathit{s}}^t\right)\right)\right)\right],$$

(25)

where the function $\mathcal{H}(·)$ is used to compute the entropy of policy ${\pi }_i$, represented as $\mathcal{H}\left({\pi }_i(·|s)\right)={\mathbb{E}}_{{\mathit{a}}_i\sim {\pi }_i}\left[-log\left({\pi }_i\left({\mathit{a}}_i\right|s)\right)\right]$. The symbol $\gamma$ denotes the discount factor used in long-term utility calculation. Additionally, ${\alpha }_i$ functions as the temperature parameter that controls the degree of entropy regularization. It serves as a learning parameter crucial for achieving an optimal balance between exploration and exploitation actions.

The soft state-value function enables the determination of how profitable state s is, considering the expected return. The soft Q-value function describes the long-term benefit of action ${\mathit{a}}_i^t$ in state s, facilitating the analysis and improvement of policy ${\pi }_i$ during training. The soft state-value function integrates the expected return augmented by entropy, formulated as follows:

$$V_i^{\pi }\left({\mathit{s}}^t\right)=\underset{{\mathit{s}}^t,{\mathit{a}}_i^t\sim {\pi }_i}{\mathbb{E}}\left[\sum _{t=0}^{\infty }{\gamma }^t\left(u_i^t+{\alpha }_{i}H\left({\pi }_i\left(·\mid {\mathit{s}}^t\right)\right)\right)\mid {\mathit{s}}_0={\mathit{s}}^t\right].$$

(26)

Therefore, the soft Q function can be expressed similarly as

$$\begin{array}{c}\hfill Q_i^{\pi }({\mathit{s}}^t,{\mathit{a}}_i^t)=\underset{{\mathit{s}}^t,{\mathit{a}}_i^t\sim {\pi }_i}{\mathbb{E}}\left[\sum _{t=0}^{\infty }{\gamma }^t{u}_i^t+{\alpha }_i\sum _{t=1}^{\infty }{\gamma }^{t}H\left({\pi }_i\left(·\mid {\mathit{s}}^t\right)\right)\mid {\mathit{s}}_0={\mathit{s}}^t,{\mathit{a}}_{i,0}^t={\mathit{a}}_i^t\right].\end{array}$$

(27)

SAC utilizes the provided policy (${\pi }_i$) to iteratively compute the soft value functions. The relationship between the soft state-value function ($V_i^{\pi }\left({\mathbf{s}}^t\right)$) and the soft Q function ($Q_i^{\pi }({\mathit{s}}^t,{\mathit{a}}_i^t)$) is presented in accordance with the Bellman equation as follows:

$$\begin{array}{cc}\hfill Q_i^{\pi }({\mathit{s}}^t,{\mathit{a}}_i^t)& =u_i^t+\underset{{\mathit{s}}^{t+1}\sim {\pi }_i}{\mathbb{E}}\left[V_i^{\pi }\left({\mathit{s}}^{t+1}\right)\right],\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill V_i^{\pi }\left({\mathit{s}}^t\right)& =\underset{{\mathit{s}}^t,{\mathit{a}}_i^t\sim {\pi }_i}{\mathbb{E}}\left[Q_i^{\pi }({\mathit{s}}^t,{\mathit{a}}_i^t)+\alpha \mathcal{H}\left({\pi }_i\left(·|{\mathit{s}}^t\right)\right)\right].\hfill \end{array}$$

(29)

Employing DNNs in resource optimization challenges allows the algorithm to precisely represent the soft action-value function, considering the complex state and action space. This capability is effective in handling a large part of the problem space and improving decision making. Such a feature enhances the algorithm’s ability to navigate the complexities of resource allocation, ultimately optimizing resource consumption and QoS efficiency [59].

Afterwords, fully a connected DNN is employed with the parameters of ${\theta }_i$ and ${\zeta }_i$, the soft action-value function of $Q_i^{\pi }({\mathit{s}}^t,{\mathit{a}}_i^t)$, and the policy of ${\pi }_i\left({\mathit{a}}_i^t\right|s^t)$, which can also be parameterized as $Q_{{\theta }_i}({\mathit{s}}^t,{\mathit{a}}_i^t)$ and ${\pi }_{{\zeta }_i}\left({\mathit{a}}_i^{\mathit{t}}\right|s^t)$, respectively.

5.2.2. Policy Evaluation

During training, the experience replay strategy is used to disrupt the temporal correlations of samples. The performed actions, the transition of network environment states in each time slot (t), and the achieved immediate utility can be represented by a tuple $\left\{s^t,{\mathit{a}}_i^t,u_i^t,s^{t+1}\right\}$. These tuples are stored in a fixed-size experience replay buffer (${\mathcal{B}}_r$). Subsequently, the actor and critic are updated by randomly sampling a mini batch (${\mathcal{B}}_{\varpi }$) of tuples (${\left\{s^{t,b},{\mathit{a}}_i^{t,b},u_i^{t,b},s^{t+1,b}\right\}}_{{\mathcal{N}}_i\in \mathcal{N}}$) from this buffer.

To mitigate the overestimation of state values, two separate critic neural networks with parameters of ${\theta }_i^1$ and ${\theta }_i^2$ are employed. The smaller value between $Q_{{\theta }_i^1}$ and $Q_{{\theta }_i^2}$ is selected as the actual Q value. Each evaluation critic network parameter (${\theta }_i^{\iota },\forall \iota \in \left\{1,2\right\}$) is updated individually by minimizing the loss function ($\mathcal{L}\left({\theta }_i^{\iota }\right)$). To reduce positive bias in policy improvement, a target soft action-value function with parameters pf ${\tilde{\theta }}_i^{\iota }$ is introduced. The parameters of the target networks, denoted as ${\tilde{\theta }}_i^{\iota }$, are regularly updated by blending the current parameters with a fraction of the previous target parameters. This approach enhances training stability by providing more consistent target values to the critic networks. The target soft action-value function, represented by ${\tilde{\theta }}_i^{\iota }$, is then expressed as follows:

$$\begin{array}{cc}\hfill {\ell }_i^{t,b}& =u_i^{t,b}+\gamma (\underset{\iota =1,2}{min}{Q}_{{\widetilde{{\theta }^{\iota }}}_i}\left({\mathit{s}}^{t+1,b},{\mathit{a}}_i^{t+1,b}\right)\hfill \\ & -{\alpha }_{i}log\left({\pi }_{{\zeta }_i}\left({\mathit{a}}_i^{t+1,b}|{\mathit{s}}^{t+1,b}\right)\right)).\hfill \end{array}$$

(30)

After that, the critic loss function can be stated as

$$\pounds \left({\mathit{\theta }}_i^{\iota }\right)={\displaystyle \frac{1}{2{\mathcal{B}}_{\varpi }}}\sum _{b=1}^{{\mathcal{B}}_{\varpi }}{\left(\underset{\iota =1,2}{min}{Q}_{{\mathit{\theta }}_i^{\iota }}\left({\mathit{s}}^{t,b},{\mathit{a}}_i^{t,b}\right)-{\ell }_i^{t,b}\right)}^2.$$

(31)

Therefore, after updating the critic networks, we can find the stochastic gradient as follows:

$$\begin{array}{cc}\hfill & {\nabla }_{{\mathit{\theta }}_i^{\iota }}\pounds \left({\mathit{\theta }}_i^{\iota }\right)={\displaystyle \frac{1}{{\mathcal{B}}_{\varpi }}}\sum _{\iota =1}^{{\mathcal{B}}_{\varpi }}{\nabla }_{{\mathit{\theta }}_i^{\iota }}{Q}_{{\mathit{\theta }}_i^{\iota }}\left({\mathit{s}}^{t,b},{\mathit{a}}_i^{t,b}\right)\hfill \\ & ·(Q_{{\mathit{\theta }}_i^{\iota }}\left({\mathit{s}}^{t,b},{\mathit{a}}_i^{t,b}\right)-\left(u_i^{t,b}+\mathit{\gamma }\left(Q_{{\tilde{\mathit{\theta }}}_i^{\iota }}\right({\mathit{s}}^{t+1,b},{\mathit{a}}_i^{t+1,b}\right)\hfill \\ & -{\alpha }_{i}log\left({\pi }_{{\zeta }_i}\left({\mathit{a}}_i^{t+1,b}|{\mathit{s}}^{t+1,b}\right)\right)\left)\right)).\hfill \end{array}$$

(32)

Additionally, to ensure the stability of the learning procedure, the parameters of the target critic network (${\tilde{\theta }}_i^{\iota }$) are updated based on the parameters of the evaluation critic network (${\theta }_i^{\iota }$) using a soft-updating approach, which is expressed as follows:

$$\widetilde{{\mathit{\theta }}_i^{\iota }}=\vartheta {\mathit{\theta }}_i^{\iota }+(1-\vartheta )\widetilde{{\mathit{\theta }}_i^{\iota }},\forall \iota =1,2,$$

(33)

where $\vartheta \in (0,1)$ indicates the update factor.

5.2.3. Policy Improvement

The goal is to operate the model at a lower cost while maximizing total utility within a given time frame through policy optimization. Conversely, an inferior current policy may lead to lower training accuracy or generate significant costs, ultimately reducing utility. Therefore, enhancing the policy is crucial. Iteratively refining the policy parameters ensures continual progress towards improved effectiveness and efficiency. Ongoing policy improvement also enhances the model’s adaptability to evolving environments and data variations, facilitating effective decision making and ensuring sustained long-term improvements. The policy parameters are refined by minimizing the expected Kullback–Leibler (KL) divergence in the algorithm as follows:

$$J\left({\mathit{\zeta }}_i\right)=\underset{{\mathit{s}}^t\sim {\mathcal{B}}_{\varpi }}{\mathbb{E}}\left[{\mathrm{D}}_{\mathrm{KL}}\left({\pi }_{{\zeta }_i}\left(·|{\mathit{s}}^t\right)\left|\right|{\displaystyle \frac{exp\left({\displaystyle \underset{\iota =1,2}{min}}{Q}_{{\mathit{\theta }}_i^{\iota }}\left({\mathit{s}}^t,·\right)\right)}{{{\rm Y}}_{{\mathit{\theta }}_i^{\iota }}\left({\mathit{s}}^t\right)}}\right)\right],$$

(34)

where the difference between the distributions (${\pi }_{{\mathit{\zeta }}_i}\left(·|{\mathit{s}}^t\right)$ and ${\displaystyle \frac{exp\left({\displaystyle \underset{\iota =1,2}{min}}{Q}_{{\mathit{\theta }}_i^{\iota }}\left({\mathit{s}}^t,·\right)\right)}{{{\rm Y}}_{{\mathit{\theta }}_i^{\iota }}\left({\mathit{s}}^t\right)}}$) is measured by the KL divergence. ${{\rm Y}}_{{\mathit{\theta }}_i^{\iota }}\left({\mathit{s}}^t\right)$ is a constant partition function that does not affect the updated policy gradient. Therefore, by scaling with the temperature parameter (${\alpha }_i$) and removing the term ${\mathbb{E}}_{a^t\sim {\pi }_{\zeta }}[{\alpha }_{i}log{{\rm Y}}_{{\theta }_i^{\iota }}\left(s^t\right)]$, the KL divergence can be reformulated as follows:

$$J\left({\mathit{\zeta }}_i\right)=-\underset{{\mathit{s}}^{t+1}\sim {\mathcal{B}}_{\varpi }}{\mathbb{E}}[\underset{{\mathit{a}}^{t+1}\sim {\pi }_{{\zeta }_i}}{\mathbb{E}}\left[\underset{\iota =1,2}{min}{Q}_{\widetilde{{\theta }_i^{\iota }}}\left({\mathbf{s}}^{t+1,b},{\mathbf{a}}_i^{t+1,b}\right)\right.$$

$$-{\alpha }_{i}log({\pi }_{{\zeta }_i}({\mathit{a}}_i^{t+1,b}|{\mathit{s}}^{t+1,b}))]].$$

(35)

In this context, ${\delta }_{{\zeta }_i}({\epsilon }^t;s^t)$ represents the neural network transformation used to reparameterize the policy, where ${\epsilon }^t$ denotes an input noise sample from a Gaussian distribution. Thus, we can rewrite Equation (30) as

$$J\left({\mathit{\zeta }}_i\right)=-\underset{{\mathit{s}}^{t+1}\sim {\mathcal{B}}_{\varpi },{\epsilon }^t\sim {\pi }_{{\zeta }_i}}{\mathbb{E}}[\underset{\iota =1,2}{min}{Q}_{{\tilde{\theta }}_i^{\iota }}\left(s^{t+1,b},{\delta }_{{\zeta }_i}\left({\epsilon }^t;s^{t,b}\right)\right)$$

$$-{\alpha }_{i}log({\pi }_{{\zeta }_i}({\delta }_{{\zeta }_i}({\epsilon }^t;s^{t,b})\mid s^{t+1,b}))].$$

(36)

Therefore, we can obtain an approximate representation of the policy gradient as follows:

$$\begin{array}{cc}\hfill {\nabla }_{{\zeta }_i}J\left({\mathit{\zeta }}_i\right)=& {\nabla }_{{\mathit{\zeta }}_i}{\alpha }_{i}log\left({\pi }_{{\mathit{\zeta }}_i}\left({\mathit{a}}_i^{t,b}|{\mathit{s}}^{t,b}\right)\right)\hfill \\ & +\left({\nabla }_{{\mathit{\zeta }}_i}{\alpha }_{i}log\left({\pi }_{{\mathit{\zeta }}_i}\left({\mathit{a}}_i^{t,b}|{\mathit{s}}^{t,b}\right)\right)\right.\hfill \\ & -\left.{\nabla }_{{\mathit{\zeta }}_i}Q\left({\mathit{s}}^{t,b},{\mathit{a}}_i^{t,b}\right)\right){\nabla }_{{\mathit{\zeta }}_i}{\delta }_{{\mathit{\zeta }}_i}\left({\epsilon }^t;{\mathit{s}}^{t,b}\right).\hfill \end{array}$$

(37)

The learning process alternates between soft policy evaluation and soft policy improvement until convergence towards the optimal policy with maximum entropy is achieved. This iterative cycle maintains policy flexibility and exploratory capability, enabling adaptation to various scenarios and the unpredictability of the environment. By increasing entropy, the policy strikes a balance between exploration and exploitation, facilitating the exploration of the state–action space while avoiding premature convergence to sub-optimal solutions.

After minimizing the loss function, according to Equations (31) and (36), the gradient can be calculated to update the temperature parameters automatically as follows:

$$J\left({\alpha }_i\right)=\underset{{\mathit{a}}^{t,b}\sim {\pi }_{{\mathit{\theta }}_i}}{\mathbb{E}}\left[-{\alpha }_{i}log\left({\pi }_{{\mathit{\theta }}_i}\left({\mathit{a}}^{t,b}|{\mathit{s}}^{t,b}\right)\right)-{\alpha }_i\overline{H}\right],$$

(38)

where $a^{t,b}=\left\{a_i^{t,b}\right\}$ describes the set of actions and $\overline{H}$ denotes the target entropy value.

5.3. Detailed Examination of Algorithms

The primary goal of Algorithm 1 is to identify resources in the cloudlet where IoT user requests arrive, assess the nature of each request, and allocate resources accordingly. Upon the arrival of a request in the cloudlet during each time slot, it searches within the set of cloudlets in the MEC for activated VNF instances that closely match the request requirements, including latency constraints. If the request is not common and requires urgent execution, it locates an activated VNF instance in the cloudlet with sufficient capacity to meet the request’s requirements. Alternatively, it searches for an idle VNF instance in the home cloudlet that has completed its processing and is ready for deactivation or has remaining capacity. If the latency requirements are flexible, it explores activated instances in adjacent cloudlets. Otherwise, it activates a new VNF instance that precisely matches the requirements to minimize resource waste. Urgent requests are prioritized to the primary cloudlets, and to balance resource allocation among cloudlets, common requests may be transferred to other cloudlets.

Algorithm 1 Request Assignment

Input: A set of cloudlets $\mathcal{Z}$ at the MEC network $\mathcal{G}=(\mathcal{Z}\cup \mathcal{V},\mathcal{E})$ and at each time slot t a set of IoT users requests $R^s$. Output: Find the related cloudlet $z_j$ to assign the request $r_i^s$ at each time slot t. 1: for $i=1,2,\cdots ,|R^{s,t}|$ do 2:    for $j=1,2,\cdots ,\left|\mathcal{Z}\right|$ do 3:      Find the set of cloudlets $z_j$ which have activated VNF instances $f_l^{\oplus }$ and that instance capacity matches the requirements of $r_i^s$ along with delay $D_e$ requirements. 4:      if $r_{urg}^s>1$ then 5:         if $f_l^{\oslash }=1$, find the $f_l^{\oplus }$ in $z_j$ with $z_j^{cap}-{\sum }_{l=1}^{\left|\mathcal{F}\right|}{\varrho }_{j,l}^t.p\left(f_l\right)$ still meet the demand of the $r_{urg}^s$, or any idle instance ${\left(f_l\right)}^{idl}$ already finished computing reutilize it, and if delay requirements assign to $z_{i\to j}$, otherwise initialize a new $f_l^{\oplus }$ with exact requirement at $z_j\in \mathcal{Z}$ and assign $r_{urg}^s$ to it. 6:         if $r_{urg}^s\le 1$ then 7:           At any $z_j\in \mathcal{Z}$ find a $f_l^{\oplus }$ with residual capacity or ${\left(f_l^{\oplus }\right)}^{idl}$. 8:         else 9:           Transfer $r_i^s$ to the $z_{i\to j}$. 10: $\mathbf{return}$ The decision of requests $r_i^s$ assignment at cloudlet $z_j$.

Algorithm 2 presents an online soft actor–critic (SAC) mechanism for resource allocation to requests in cloudlets. Each agent in this approach takes system states as inputs and employs an actor network to generate policies for decision making. A critic network assesses the quality of these actions. The parameters of the actor network are updated using policy gradients. This process facilitates decisions on resource assignments based on the available capacity of VNF instances in the cloudlets, ensuring efficient resource allocation.

Algorithm 2 Online Soft Actor–Critic-based Process to Assign Resources

Input: The system states $s_i^t=(r{\varrho }_{j,l}^t,z_j^{cap,t},{\left(f_l^{\oplus }\right)}^{idl,t},f_l^{\oplus ,t},f_l^{\oslash ,t},B_{e,i,j}^t)$, the actor network with the parameter ${\theta }_i$ and the critic network with the parameter ${\zeta }_i$ for each agent ${\mathcal{N}}_i$. Output: The decision of resource assignment at the cloudlet $z_j$. 1: The given system states $s_i^t$ as input, the output of the actor network is ${\pi }_i\left(a_i^t\right|s^t;{\theta }_i)$ for each agent ${\mathcal{N}}_i$; 2: Select sample action $s_i^t$ based on the distribution ${\pi }_i\left(a_i^t\right|s^t;{\theta }_i)$ for each agent ${\mathcal{N}}_i$; 3: By using the critic network evaluate the action-value function $Q_i(s^t,a_i^t;{\zeta }_i)$ for each agent ${\mathcal{N}}_i$; 4: Update the actor network parameters ${\theta }_i$ by calculating the policy gradient for each agent ${\mathcal{N}}_i$; 5: ${\varphi }_{j,l}^t=\lfloor {\displaystyle \frac{z_j^{\ast cap,t}.{\stackrel{‘}{a}}_j}{p\left(f_l\right)}}\rfloor$, $\forall l\in \{1,2,3,\dots ,|\mathcal{F}\left|\right\}$; 6: $\mathbf{return}$ The decision ${\varphi }_{j,l}^t$ of resource assignment to $R^s$ at cloudlet $z_j$.

The main steps of our proposed DRL-based MAgSAC algorithm are illustrated here. In the first stage, the parameters of the soft Q-value functions ($Q_{{\theta }_i}^{\iota }(s^t,a_i^t),\forall \iota \in \left\{1,2\right\}$) are initialized with weights of ${\theta }_i^{\iota }$. Subsequently, the target parameters (${\tilde{\theta }}_i^{\iota }$) of the soft Q-value functions are initialized with the same weights (${\theta }_i^{\iota }$). Following this, the policy parameters (${\pi }_{{\zeta }_i}\left(a_i\right|s)$) are initialized with random weights, and experiences are sampled from the experience replay memory (${\mathcal{B}}_r$).

In the second stage, within each cloudlet and episode, agents observe the initial state ($s^0$) of the mobile edge computing environment. In the third stage, at each time slot, each agent observes the network state( $s_i^t$); then, the actor takes action $a_i^t$ based on the observed state ($s_i^t$) and the defined policy (${\pi }_{{\zeta }_i}\left(a_i^t\right|s^t)$). The decision to allocate resources according to the requirements is made by invoking Algorithm 2, while Algorithm 1 predicts the nature of the requests during the assignment stage. After taking actions in the environment, each agent receives feedback in the form of utility $u_i^t$, and the network environment transitions from $s^t$ to $s^{t+1}$.

In the fourth stage, the experiences of agent actions in the environment are stored in the experience replay memory (${\mathcal{B}}_r$) as tuples $\{s^t,a_i^t,u_i^t,s^{t+1}\}$. In the fifth stage, a mini batch (${\mathcal{B}}_{\varpi }$) of experiences is randomly selected from the experience replay memory (${\mathcal{B}}_r$), and the soft Q-value parameters (${\theta }_i^{\iota }$) and policy parameters (${\zeta }_i$) are updated by minimizing loss functions $\pounds \left({\mathit{\theta }}_i^{\iota }\right)$ and $J\left({\mathit{\zeta }}_i\right)$ according to Equations (31) and (36), respectively. The gradients can be calculated automatically using Equation (38); then, the temperature parameters are updated.

In the last stage, the parameters of the target critic network (${\tilde{\theta }}_i^{\iota }$) are updated based on the assessment of the critic network parameters (${\theta }_i^{\iota }$) according to Equation (33).

The algorithm, referenced as Algorithm 3, employs an online SAC approach to assign requests to suitable cloudlets and allocate resources accordingly. Actions related to request assignment and resource allocation in the cloudlets are facilitated by invoking Algorithms 1–4.

Algorithm 3 Online Soft Actor–Critic-Based Algorithm to Assign Requests and Resource Allocation

Input: A set of cloudlets $\mathcal{Z}$ with VNF instances $\mathcal{F}$ at the MEC network $\mathcal{G}=(\mathcal{Z}\cup \mathcal{V},\mathcal{E})$ and at each time slot t a set of IoT users requests $R^s$. Output: The decision of request $r_i^s$ assignment and resource allocation at each time slot t. 1: Train the actor and critic networks by apply MAgSAC Algorithm 4; 2: for $t=1,2,\dots ,T$ do 3:    for $i=1,2,\dots ,\mathcal{Z}$ do 4:      Invoke the Algorithm 2 to allocate the resources by deciding ${\varphi }_{j,k}^t$ to accommodate the $r_i^s$ the VNF instances $f_l\in \mathcal{F}$ will be activate $f_l^{\oplus }$ or deactivate $f_l^{\oslash }$ at the cloudlet $z_j$; 5:    To assign the requests $r_i^s$ and $r_{urg}^s$ invoke the Algorithm 1 and calculate the $f_l^{\oplus }$, ${\left(f_l^{\oplus }\right)}^{idl}$ at each time slot t and update the system according; 6: $\mathbf{returrn}$ The decision of the request assignment and resource allocation after action $a_i^t$ with $z_j^{\ast cap},{\left(f_l^{\oplus }\right)}^{idl},f_l^{\oplus }.$

Algorithm 4 Training of MAgSAC

Input: A set of cloudlets $\mathcal{Z}$ with VNF instances $\mathcal{F}$ at the MEC network $\mathcal{G}=(\mathcal{Z}\cup \mathcal{V},\mathcal{E})$ and at each time slot t a set of IoT users requests $R^s$. Output: The trained networks of updated policy $a=a_1,a_2,\dots a_n$ 1: Initialize the critic network with parameters ${\theta }_i^{\iota }$$(\iota =1,2)$, initialize the target critic network with parameters ${\tilde{\theta }}_i^{\iota }$ = ${\theta }_i^{\iota }$, Initialize the actor network with parameters ${\zeta }_i$, Initialize the temperature parameter ${\alpha }_i$, and experience reply memory ${\mathcal{B}}_r$ for each agent ${\mathcal{N}}_i$. 2: for $i=1,2,\dots ,\left|\mathcal{Z}\right|$  do 3:    for each episode = 1 do 4:      Initialize the state $s^0=\left\{s_i^0\right\}$ for all agents $\mathcal{N}$; 5:      for $t=1,2,\dots ,T$ do 6:         for each agent $\mathcal{N}$ do 7:           Observer the system states $s_i^t=(r{\varrho }_{j,l}^t,z_j^{cap,t},{\left(f_l^{\oplus }\right)}^{idl,t},f_l^{\oplus ,t},f_l^{\oslash ,t},B_{e,i,j}^t)$; 8:           Take the decision with action $a_i^t=(a_{f_l^t},a_{f_l^{\oplus ,t}},a_{z_j^{\ast cap,t}},a_{f_l^{\oslash ,t}},a_{z_{i\to j}^t})$ for all $\mathcal{N}$ by following the policy ${\pi }_{{\zeta }_i}\left(a_i^t\right|s^t)$; 9:           By invoking the Algorithm 2 find the ${\varphi }_{j,l}^t=\lfloor {\displaystyle \frac{z_j^{\ast cap,t}.{\stackrel{‘}{a}}_j}{p\left(f_l\right)}}\rfloor$, $\forall k\in \{1,2,3,\dots ,|\mathcal{F}\left|\right\}$; 10:          At the request assignment stage call the Algorithm 1 for common $r^s$ and urgent requests $r_{urg}^s$; 11:          Aggregate the positive reward of the $z_j$ at each time slot t by following the Equation (20); 12:          Observe the utility $u_i^t$ and get the next state $s^{t+1}$; 13:          Store these tuples $\{s^t,a_i^t,u_i^t,s^{t+1}\}$ in ${\mathcal{B}}_r$ replace the oldest experience; 14:          A mini-batch of ${\mathcal{B}}_{\varpi }$ experience $\{s^t,a_i^t,u_i^t,s^{t+1}\}$ randomly selected from ${\mathcal{B}}_r$; 15:          Calculate the target Q-value ${\ell }_i^{t,b}$ from Equation (30); 16:          Minimize the loss function from Equation (31) by updating the parameters ${\theta }_i^{\iota }$ of soft Q-value function; 17:          Update the policy parameters ${\zeta }_i$ from the gradient in Equation (37); 18:          Temperature parameter ${\alpha }_i$ update by computing the gradient from Equation (38); 19:          Update the target networks ${\theta }_i^{\iota }$ with Equation (33);

6. Complexity Analysis

In the initialization phase (Line 1), the critic and actor networks, temperature parameter, and experience replay memory are initialized for each agent, which runs in constant time $O\left(1\right)$. In the main loop (Lines 2–6), there are three nested loops; the first loop iterates over the number of cloudlets ($T\left(\mathcal{Z}\right)$), the second and fourth loops iterate over the number of agents ($T\left(n\right)$), and the third loop iterates over the number of time slots ($T\left(t\right)$). This gives a combined complexity of $O(T\left(\mathcal{Z}\right)·T{\left(n\right)}^2·T\left(t\right))$. The subsequent steps (Lines 7–19) are constant time operations ($O\left(1\right)$), as they involve updating parameters using mini-batch experience, which is performed in constant time. Therefore, the overall time complexity of the algorithm is $O\left(1\right)+O(T\left(\mathcal{Z}\right)·T{\left(n\right)}^2·T\left(t\right))+O\left(1\right)=O(T\left(\mathcal{Z}\right)·T{\left(n\right)}^2·T\left(t\right))$.

7. Results and Discussion

In this section, we conduct comprehensive simulations to assess the efficiency of our proposed approach in mobile edge computing scenarios. Additionally, we evaluate the influence of significant parameters on the effectiveness of the proposed technique.

7.1. Parameter Setup

The simulations are performed on a Dell Core i7-9850H CPU @ 3.00 GHz (12 CPUs) with an Intel UHD 630 graphics card, NVIDIA Quadro T2000, and 64GB RAM running on a Windows 11 Pro 64-bit operating system. The experimental parameters are set in accordance with the research work reported in [60,61,62]. In an MEC network, we consider a network size ranging from 10 to 200 nodes; each topology is generated using GT-ITM [63], and the number of cloudlets is set to 20% of the network size. The computing capacity of each cloudlet ranges from 40,000 to 120,000 MHs [64], and the bandwidth of each link varies from 30 Mbps to 100 Mbps [65]. The computing resource requirements for five types of network functions—firewall, NAT, collision detector, and IDS—are adopted from [66,67]. Each network slicing request is randomly generated from 10 to 100 Megabytes, and the delay requirements for common and urgent requests range from 1 to 20 ms [68]. We utilize randomly generated networks ranging from 10 to 200 nodes to train our model. Subsequently, we employ the trained instances for examination on networks with varied sizes and parameter variations to demonstrate the model’s adaptability. The resulting values of the proposed algorithm in each figure represent the mean values, and the algorithm’s running time depends on the machine. These parameter settings are adopted in our experimental analysis unless otherwise specified.

We present a comparative analysis of the performance of our proposed MAgSAC algorithm with that of the following six benchmark approaches:

• MAA3C-based approach: The first benchmark [69] jointly considers the selection of edge nodes and resource allocation to optimize energy consumption, delay, and computing capabilities. We employ this approach with the same parameter settings for fair comparison.
• SAC-based approach: The second benchmark is a traditional approach referred to as SACT, which sets resource allocation to cloudlets based on available computing capacity.
• DDPG-based approach: The third benchmark, DDPG, makes resource allocation decisions based on environmental feedback.
• Structure2Vec approach: The forth benchmark is Structure2Vec (referred to as S2Vec), which facilitates learning through feature-embedding strategies.
• Random approach: The fifth benchmark randomly chooses cloudlets for resource allocation.
• Greedy approach: The sixth benchmark selects cloudlets greedily based on resource availability, considering the available bandwidth in links by assessing the closest paths.

7.2. Performance Analysis

We first examine the performance of the $\mathtt{MAgSAC}$ algorithm against the other six benchmark algorithms ($\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, $\mathtt{DDPG}$, $\mathtt{S}\mathtt{2}\mathtt{Vec}$, $\mathtt{Random}$, and $\mathtt{Greedy}$) in terms of accumulated utility, end-to-end average delay, running time, overall execution time, and average energy consumption, with network sizes varying from 10 to 200 nodes, while setting the number of requests to 100. The proposed algorithm’s results are shown in Figure 3.

As we can see from Figure 3a, the $\mathtt{MAgSAC}$ algorithm has a higher accumulated utility than the benchmark algorithms, outperforming $\mathtt{MAA}\mathtt{3}\mathtt{C}$ by 19.9%, $\mathtt{SACT}$ by 35.2%, $\mathtt{DDPG}$ by 49.8%, $\mathtt{S}\mathtt{2}\mathtt{Vec}$ by 104.2%, $\mathtt{Random}$ by 147.2%, and $\mathtt{Greedy}$ by 167.7%. The reason is that when the network size is 10, $\mathtt{MAgSAC}$ has fewer activated VNF instances available, and with a network size of 200, not only are more VNF instances available to facilitate IoT user requests, but the algorithm also utilizes the remaining capacity and idle VNF instances efficiently. However, the $\mathtt{Random}$ and $\mathtt{Greedy}$ approaches gain the lowest utility compared to $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, $\mathtt{DDPG}$, and $\mathtt{S}\mathtt{2}\mathtt{Vec}$ because they choose to activate new VNF instances rather than utilize existing ones. When the network size is 100, the $\mathtt{Random}$ and $\mathtt{Greedy}$ algorithms perform almost identically due their choice of the closest path to accommodate user demand.

In Figure 3b, it can be observed that the end-to-end average delay experienced by $\mathtt{MAgSAC}$ is much lower than that of its counterpart algorithms, with an average delay 13.1% less than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 32.7% less than that of $\mathtt{SACT}$, 35.9% less than that of $\mathtt{DDPG}$, 43.9% less than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 46.2% less than that of $\mathtt{Random}$, and 45.9% less than that of $\mathtt{Greedy}$. The reason is that $\mathtt{MAgSAC}$ intelligently finds the activated VNF instance in the primary cloudlet with minimum delay, maximizing utility.

Figure 3c represents the running time of all the comparison algorithms, where we can see that the running time of all the algorithms gradually increases from a network size of 10 to 200. However, the running time of $\mathtt{MAgSAC}$ is 27.8% higher than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 59.5% higher than that of $\mathtt{SACT}$, 78.9% higher than that of $\mathtt{DDPG}$, 250% higher than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 312.4% higher than that of $\mathtt{Random}$, and 435.6% higher than that of $\mathtt{Greedy}$. Compared to the six benchmark algorithms, $\mathtt{MAgSAC}$ obtains the highest utility with minimum delay and more feasible solutions than the other algorithms.

From Figure 3d, it can be seen that the overall execution time to compute the IoT user requests of all the algorithms varies with network size from 10 to 200. Algorithm $\mathtt{MAgSAC}$ performs much better than the other six comparison counterparts. When the network size is 10, the execution time is 59.3, meaning most requests are implemented in the home cloudlet by assigning the activated idle or remaining capacity of VNF instances, and fewer requests are referred to the adjacent cloudlet. As the network size increases, more VNF instances are available to accommodate the requests, and the execution time decreases up to a network size of 200. On the other hand, when the network size is 150 and 200, the execution time of the $\mathtt{S}\mathtt{2}\mathtt{Vec}$ and $\mathtt{Random}$ algorithms is identical because the feature-embedding strategy of $\mathtt{S}\mathtt{2}\mathtt{Vec}$ and the random resource selection employed in the cloudlet of $\mathtt{Random}$ deliver similar results. Thus, the overall execution time to compute the requests of $\mathtt{MAgSAC}$ is 17.4% lower than, and 435.6% higher than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 33% lower than, and 435.6% higher than that of $\mathtt{SACT}$, 35.6% lower than, and 435.6% higher than that of $\mathtt{DDPG}$, 45.8% lower than, and 435.6% higher than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 47% lower than, and 435.6% higher than that of $\mathtt{Random}$, and 48.6% lower than, and 435.6% higher than that of $\mathtt{Greedy}$.

Figure 3e depicts the average energy consumption costs to implement IoT user requests to activate and deactivate the VNF instances of all the algorithms. The $\mathtt{MAgSAC}$ algorithm has the lowest cost to implement user requests compared to the other six algorithms, even when active resources are limited at a network size of 10. As the network size increases, the cost decreases. The reason is that with increasing network size, more VNF instances finish processing and become available for reuse, and our algorithm chooses the lowest-cost implementation resources with minimum delay, maximizing total utility. Thus, the average energy consumption cost to compute the requests using $\mathtt{MAgSAC}$ is 27% less than that using $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 35.2% less than that using $\mathtt{SACT}$, 39.2% less than that using $\mathtt{DDPG}$, 53.7% less than that using $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 56.3% lower than that using $\mathtt{Random}$, and 59.5% lower than that using $\mathtt{Greedy}$.

Next, we examine how the number of cloudlets impacts performance in a real network topology, namely AS1755, ranging from 10 to 100 cloudlets, using the $\mathtt{MAgSAC}$, $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, $\mathtt{DDPG}$, $\mathtt{S}\mathtt{2}\mathtt{Vec}$, $\mathtt{Random}$, and $\mathtt{Greedy}$ algorithms.

In Figure 4a, we can notice that the $\mathtt{MAgSAC}$ algorithm delivers the highest utility compared to the other six algorithms, corresponding to 18.1% more than $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 40.9% more than $\mathtt{SACT}$, 49.4% more than $\mathtt{DDPG}$, 85.2% more than $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 127.6% more than $\mathtt{Random}$, and 150.3% more than $\mathtt{Greedy}$, with $\mathtt{Greedy}$ having the lowest utility among all algorithms. The reason is that with the growth in the number of cloudlets, more user requests are implemented by considering the requirements and available resources. This also minimizes the end-to-end average delay, providing resources not only for common requests but also for urgent IoT user requests. The delay of the $\mathtt{MAgSAC}$ algorithm dramatically decreases by 31.3% compared to $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 48.9% compared to $\mathtt{SACT}$, 53% compared to $\mathtt{DDPG}$, 58.3% compared to $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 59.4% compared to $\mathtt{Random}$, and 61% compared to the $\mathtt{Greedy}$ benchmark when the number of cloudlets ranges from 60 to 100, as we can observe in Figure 4b.

Despite this, a high number of cloudlets may result in a longer search for an optimum solution, thereby unavoidably prolonging the running time of all seven algorithms. The running time of $\mathtt{MAgSAC}$ is 29.3% more than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 57.7% more than that of $\mathtt{SACT}$, 102.7% more than that of $\mathtt{DDPG}$, 166.4% more than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 182% more than that of $\mathtt{Random}$, and 252.8% more than that of the $\mathtt{Greedy}$ method, as shown in Figure 4c.

Figure 4d presents the overall execution times of the algorithms. The overall execution time of $\mathtt{MAgSAC}$ is 14% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 29.7% lower than that of $\mathtt{SACT}$, 36.4% lower than that of $\mathtt{DDPG}$, 42.4% lower than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 46.6% lower than that of $\mathtt{Random}$, and 47% lower than that of $\mathtt{Greedy}$. Since $\mathtt{MAgSAC}$ enables the agents to learn and improve the policy, it efficiently mitigates the execution time with the increasing number of cloudlets, maximizing the accumulated utility and reducing the overall energy consumption cost to activate and deactivate the VNF instances according to the requirements of IoT users. This results in the lowest energy consumption cost of $\mathtt{MAgSAC}$, which is 26.4% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 35.3% lower than that of $\mathtt{SACT}$, 41.6% lower than that of $\mathtt{DDPG}$, 49.5% lower than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 53.5% lower than that of $\mathtt{Random}$, and 56.4% lower than that of $\mathtt{Greedy}$, as can be observed in Figure 4e.

Next, we investigate the impact of the number of service providers on the performance of the $\mathtt{MAgSAC}$, $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, $\mathtt{DDPG}$, $\mathtt{S}\mathtt{2}\mathtt{Vec}$, $\mathtt{Random}$, and $\mathtt{Greedy}$ algorithms by varying the number of service providers in a real network topology (AS1755). Results of the accumulated utility, average delay of IoT user requests, running time, overall execution time of the requests, and average energy consumption cost are defined in Figure 5. In Figure 5a, we can see that the accumulated utility gained by $\mathtt{MAgSAC}$ and $\mathtt{MAA}\mathtt{3}\mathtt{C}$ increases with the growth in the number of service providers compared to the $\mathtt{SACT}$, $\mathtt{DDPG}$, $\mathtt{S}\mathtt{2}\mathtt{Vec}$, $\mathtt{Random}$, and $\mathtt{Greedy}$ algorithms. The reason is that more resources are allocated by the service providers to the users, which gradually increases the accumulated utility. The utility achieved by $\mathtt{MAgSAC}$ is 30% higher than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 44% higher than that of $\mathtt{SACT}$, 57.9% higher than that of $\mathtt{DDPG}$, 57.4% higher than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 85% higher than that of $\mathtt{Random}$, and 118% higher than that of the $\mathtt{Greedy}$ strategy. However, the end-to-end average delay of all seven compared counterparts slightly decreases. The average delay of $\mathtt{MAgSAC}$ is 22.4% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 43.7% lower than that of $\mathtt{SACT}$, 45.4% lower than that of $\mathtt{DDPG}$, 47.6% lower than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 47.7% lower than that of $\mathtt{Random}$, and 48.2% lower than that of $\mathtt{Greedy}$, as seen in Figure 5b.

Meanwhile, as shown in Figure 5c, the running time of all the algorithms increases with the growth of service providers, but $\mathtt{MAgSAC}$ has the highest running time of almost 68.7 ms. The strategy of $\mathtt{MAgSAC}$ for allocating resources requires more time, including training time, to reach the optimal level, which is acceptable. The increasing running time percentage of $\mathtt{MAgSAC}$ is 37.4% more than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 71% more than that of $\mathtt{SACT}$, 89.3% more than that of $\mathtt{DDPG}$, 161.3% more than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 176.4% more than that of $\mathtt{Random}$, and 205.7% more than that of $\mathtt{Greedy}$.

Figure 5d illustrates the overall execution time of all seven algorithms. We can see that the execution time required by $\mathtt{MAgSAC}$ is 21.7% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 34.5% lower than that of $\mathtt{SACT}$, 37.4% lower than that of $\mathtt{DDPG}$, 44.7% lower than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 46.6% lower than that of $\mathtt{Random}$, and 47.6% lower than that of $\mathtt{Greedy}$ from 10 to 600 service providers. The reason is that an optimal number of shared resources from service providers is needed to execute IoT user requests. More resource availability not only minimizes the delay but also the execution time. We can also observe that the execution time of the $\mathtt{Random}$ and $\mathtt{Greedy}$ algorithms is almost identical with 200 to 400 service providers. This is because the $\mathtt{Random}$ algorithm chooses resources to fulfill the demand randomly, which is excessive with 200 service providers, and the $\mathtt{Greedy}$ algorithm allocates resources based on availability. Therefore, the two strategies perform identically in this range.

On the other hand, the average energy consumption cost obtained the $\mathtt{MAgSAC}$ algorithm is 12.4% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 26% lower than that of $\mathtt{SACT}$, 31.4% lower than that of $\mathtt{DDPG}$, 37.7% lower than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 41.8% lower than that of $\mathtt{Random}$, and 44% lower than that of $\mathtt{Greedy}$ with an increasing number of service providers, as evidenced in Figure 5e. The reason behind this decrease is that more service providers are involved in the process of resource allocation, which not only increases the utility of the service providers but also increases the processing cost of IoT user requests, including the bandwidth utilization cost in the links.

We now investigate the impact of the computing capacity of the cloudlets in a real network topology (AS1755) from 5 MHz to 90 MHz on the performance of the seven benchmark algorithms, namely $\mathtt{MAgSAC}$, $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, $\mathtt{DDPG}$, $\mathtt{S}\mathtt{2}\mathtt{Vec}$, $\mathtt{Random}$, and $\mathtt{Greedy}$, in terms of accumulated utility, average end-to-end delay, running time, overall execution time, and average energy consumption cost, as depicted in Figure 6. We can see that the $\mathtt{MAgSAC}$ algorithm gains around 8.7%, 37.5%, 54.1%, 93.7%, 114%, and 135.8% accumulated system utility relative to the $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, $\mathtt{DDPG}$, $\mathtt{S}\mathtt{2}\mathtt{Vec}$, $\mathtt{Random}$, and $\mathtt{Greedy}$ algorithms, then remains stable afterwards when the cloudlet computing capacity is 90 MHz. The reason is that the computing capacity of the cloudlets is stable, achieving more utility with the growth of the capacity, as shown in Figure 6a.

The end-to-end average delay of all the algorithms can be seen in Figure 6b. The delay from 5 MHz to 10 MHz is slightly higher for all the algorithms, although that of $\mathtt{MAgSAC}$ is 8% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 19.6% lower than that of $\mathtt{SACT}$, 21.9% lower than that of $\mathtt{DDPG}$, 27% lower than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 30% lower than that of $\mathtt{Random}$, and 30.8% lower than that of $\mathtt{Greedy}$. As the computing capacity of the algorithms increases, the delay decreases because more capacity is available, and with minimal delay, more requests are implemented.

Figure 6c shows the running time of all the comparison algorithms. We can observe that the running time of $\mathtt{MAgSAC}$ and $\mathtt{MAA}\mathtt{3}\mathtt{C}$ is higher than that of the other algorithms due to the improved policy in the resource allocation process. However, the running time of the $\mathtt{SACT}$, $\mathtt{DDPG}$, $\mathtt{S}\mathtt{2}\mathtt{Vec}$, $\mathtt{Random}$, and $\mathtt{Greedy}$ algorithms shows a gradually reducing but steady trend with the growth of the computing capacity of the cloudlets. The running time of $\mathtt{MAgSAC}$ is 20.3% more than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 53% more than that of $\mathtt{SACT}$, 56.1% more than that of $\mathtt{DDPG}$, 77.5% more than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 80.4% more than that of $\mathtt{Random}$, and 100.3% more than that of $\mathtt{Greedy}$.

We further investigate the impact of increasing the computing capacity of the cloudlets on all the algorithms, as seen in Figure 6d. The findings demonstrate that when the computing capacity of the cloudlets improves, the overall time taken to implement the requests decreases. Each scheme exhibits a consistent pattern as a consequence of the enhanced computational capabilities of MEC networks. As the computing capacity of the cloudlets grows, our $\mathtt{MAgSAC}$ approach demonstrates a significant reduction in overall execution time compared to the other six algorithms, with an overall execution time 0.27% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 1.2% lower than that of $\mathtt{SACT}$, 2.2% lower than that of $\mathtt{DDPG}$, 6.5% lower than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 11.3% lower than that of $\mathtt{Random}$, and 18.8% lower than that of $\mathtt{Greedy}$.

The average energy consumption costs of the $\mathtt{MAgSAC}$, $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, $\mathtt{DDPG}$, $\mathtt{S}\mathtt{2}\mathtt{Vec}$, $\mathtt{Random}$, and $\mathtt{Greedy}$ algorithms are presented in Figure 6e. The average energy consumption cost computing IoT user requests decreases with the growth of the computing capacity of all seven algorithms. This is because with the increase in computing capacity of the cloudlets, more users are accommodated by the service providers with the available resources without activating new VNF instances, which gradually decreases the cost with increasing cloudlet capacity. Therefore, in terms of average energy consumption cost regarding computing capacity of the cloudlets, $\mathtt{MAgSAC}$ achieves the lowest cost that is 17.6% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 29.3% lower than that of $\mathtt{SACT}$, 30.6% lower than that of $\mathtt{DDPG}$, 41.7% lower than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 43.2% lower than that of $\mathtt{Random}$, and 44.5% lower than that of $\mathtt{Greedy}$.

Figure 7 shows the impact of the number of requests on the performance of the $\mathtt{MAgSAC}$, $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, $\mathtt{DDPG}$, $\mathtt{S}\mathtt{2}\mathtt{Vec}$, $\mathtt{Random}$, and $\mathtt{Greedy}$ algorithms. The primary purpose of our approach is to facilitate both common and urgent IoT user requests as much as possible to minimize the failure percentage within the given time frame. Minimizing request failure is very important in emergency cases where an autonomous vehicle or real-time application needs a quick response. The common-request failure percentages of the seven comparison algorithms are slightly higher than those of the urgent-request algorithms, as seen in Figure 7a. However, our algorithm ($\mathtt{MAgSAC}$) has a request failure percentage that is 44.3% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 65.7% lower than than that of $\mathtt{SACT}$, 69.2% lower than than that of $\mathtt{DDPG}$, 72.8% lower than than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 78% lower than than that of $\mathtt{Random}$, and 80% lower than than that of $\mathtt{Greedy}$ for common requests. For urgent requests, it is 52% lower than than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 71% lower than than that of $\mathtt{SACT}$, 75.5% lower than than that of $\mathtt{DDPG}$, 79.3% lower than than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 81.9% lower than than that of $\mathtt{Random}$, and 84.5% lower than than that of $\mathtt{Greedy}$.

The higher percentage of common-request failures is due to our prioritization of urgent requests. In primary cloudlets, all urgent requests are handled with minimal execution time, which results in common requests being transferred to adjacent cloudlets when resources are unavailable in the home cloudlet. This prioritization is reflected in the higher execution time for common requests, as evidenced in Figure 7b. The overall request execution time for $\mathtt{MAgSAC}$ is 18.1% less than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 31.7% less than that of $\mathtt{SACT}$, 39.8% less than that of $\mathtt{DDPG}$, 56.1% less than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 63.7% less than that of $\mathtt{Random}$, and 68.1% less than that of $\mathtt{Greedy}$ for common requests. For urgent requests, the execution time of $\mathtt{MAgSAC}$ is 21.6% less than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 41.7% less than that of $\mathtt{SACT}$, 48% less than that of $\mathtt{DDPG}$, 61% less than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 67% less than that of $\mathtt{Random}$, and 71.9% less than that of $\mathtt{Greedy}$.

Figure 7c shows that the overall execution time of the requests increases with the growth of the total number of requests. The rationale behind this is that as the number of requests increases, not only does prediction require differentiation common and urgent requests, but appropriate resources must also be allocated according to the requirements. In some cases, requests need to be transferred to adjacent cloudlets for implementation, and congestion may occur, which increases the overall execution time. The execution time for the $\mathtt{MAgSAC}$ algorithm is 23.2% lower than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 36.8% lower than that of $\mathtt{SACT}$, 45.9% lower than that of $\mathtt{DDPG}$, 53.6% lower than that of $\mathtt{S}\mathtt{2}\mathtt{Vec}$, 59.9% lower than that of $\mathtt{Random}$, and 66.3% lower than that of $\mathtt{Greedy}$. Thus, the $\mathtt{MAgSAC}$ algorithm has the lowest execution time among all the comparison counterparts.

Finally, we evaluate the convergence performance of our proposed $\mathtt{MAgSAC}$ approach in comparison with three DRL-based benchmarks, namely $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, and $\mathtt{DDPG}$, based on MINIST [70], as shown in Figure 8. In Figure 8a, we can notice that the accumulated utility over the first 10 episodes is at the lowest level but gradually increases and starts converging to an approximate optimal point from episode 50 onwards. The $\mathtt{MAgSAC}$, $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, and $\mathtt{DDPG}$ algorithms all converge to a stable solution from episode 50 onwards, which means that as the number of episodes grows, the speed of convergence tends to increase for all the algorithms. While the $\mathtt{SACT}$ and $\mathtt{DDPG}$ algorithms aim to maximize the estimated long-term return, their overall utility is slightly lower than that obtained by the other two algorithms, namely $\mathtt{MAgSAC}$ and $\mathtt{MAA}\mathtt{3}\mathtt{C}$. The notable enhancement of the proposed $\mathtt{MAgSAC}$ approach and the $\mathtt{MAA}\mathtt{3}\mathtt{C}$ scheme may be due to the use of a maximum-entropy regularized stochastic strategy instead of a deterministic policy. Furthermore, our proposed $\mathtt{MAgSAC}$ algorithm clearly achieves more rapid convergence and superior utility, outperforming $\mathtt{MAA}\mathtt{3}\mathtt{C}$ by 21.8%, $\mathtt{SACT}$ by 38.7%, and $\mathtt{DDPG}$ by 51.8%. The proposed $\mathtt{MAgSAC}$ algorithm achieves convergence after around 50 episodes. The reason is that the proposed $\mathtt{MAgSAC}$ method allows for the efficient allocation of resources not only in the home cloudlet but also in adjacent cloudlets, thereby preventing the system from reaching a sub-optimal solution.

In Figure 8b, we examine the accuracy achieved by our proposed $\mathtt{MAgSAC}$ algorithm against the other DRL-based algorithms, namely $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, and $\mathtt{DDPG}$. It is worth mentioning that all algorithms indicate an immediate rise in accuracy during the first 10 episodes, then gradually reach a stable state after about 50 rounds. Efficiency reduction may occur only when there is a lack of resources in the cloudlets of concern. The results indicate that $\mathtt{MAgSAC}$, $\mathtt{MAA}\mathtt{3}\mathtt{C}$, $\mathtt{SACT}$, and $\mathtt{DDPG}$ exhibit superior accuracy. The accuracy gain by $\mathtt{MAgSAC}$ is 38% at 10 episodes, which is much higher than that of the comparison counterparts. Afterwards, with the increase in episodes, the accuracy percentage increases to 4.2% higher than that of $\mathtt{MAA}\mathtt{3}\mathtt{C}$, 6.4% higher than that of $\mathtt{SACT}$, and 8.7% higher than that of $\mathtt{DDPG}$. However, $\mathtt{MAgSAC}$ outperforms all of them, obtaining an accuracy rate of over 97% in up to 300 episodes. It is evident that as the number of episodes increases, our technique improves significantly. Therefore, the $\mathtt{MAgSAC}$ algorithm achieves exceptional performance, clearly showing its superiority compared to previously reported techniques.

8. Research Findings

Our work presents a MAgSAC algorithm designed for efficient network slicing in MEC environments. This research addresses the challenge of uneven traffic distribution and resource constraints by predicting and classifying IoT requests as common or urgent, thereby optimizing resource allocation and minimizing latency. The proposed MAgSAC algorithm enhances the management of VNF instances within cloudlets, preventing overloading and ensuring balanced resource distribution. By transforming the optimization problem into an MDP, the algorithm tackles the complexity of resource allocation through intelligent decision making. Extensive simulations demonstrate that the MAgSAC approach outperforms six benchmark methods, namely MAA3C, SACT, DDPG, S2Vec, Random, and Greedy, in terms of accumulated utility, energy consumption, and execution time. It successfully balances trade-offs between revenue, energy costs, and latency while ensuring the timely completion of both common and urgent requests. Additionally, the algorithm addresses the challenge of adapting to real-time changes in user demand and resource availability. These findings highlight the algorithm’s effectiveness in optimizing MEC systems, ensuring high QoS for diverse IoT applications, and mitigating potential risks associated with resource limitations.

9. Conclusions

This paper addresses the challenge of resource allocation in MEC environments, particularly the uneven distribution of IoT requests. We propose a network slicing-based MEC system that utilizes a MAgSAC algorithm. Our approach leverages strategically placed cloudlets with on-demand VNF activation/deactivation capabilities to efficiently cater to both common and urgent IoT requests. To optimize the resource allocation process, we formulate a comprehensive optimization model that considers utility, energy consumption cost, and latency. By transforming this model into a multi-agent deep reinforcement learning problem, we achieve intelligent resource allocation through the use of the MAgSAC algorithm while optimizing overall utility. Extensive simulations demonstrate the effectiveness of our approach, achieving the highest utility, minimizing execution time, and reducing energy consumption costs compared to existing methods. Our work paves the way for efficient resource allocation in MEC systems, ensuring optimal QoS for diverse IoT users.

Future research can build on this foundation by investigating new optimization parameters, extending the system’s scalability, and incorporating more advanced prediction models to further optimize resource allocation techniques in MEC scenarios. Furthermore, investigating the integration of serverless computing paradigms could lead to novel solutions to address complex resource allocation challenges by disassociating resource management from individual instances. Additionally, evaluating the impact of practical constraints, such as network latency and device heterogeneity, on the proposed method’s efficiency would be advantageous. Future research could explore the enhancement of the MAgSAC algorithm’s adaptability to dynamic and unpredictable IoT traffic patterns. Comparative analysis with other cutting-edge DRL techniques could offer a deeper comprehension of the algorithm’s performance and potential areas for enhancement.