Differentiated Security Requirements: An Exploration of Microservice Placement Algorithms in Internet of Vehicles

Abstract

In recent years, microservices, as an emerging technology in software development, have been favored by developers due to their lightweight and low-coupling features, and have been rapidly applied to the Internet of Things (IoT) and Internet of Vehicles (IoV), etc. Microservices deployed in each unit of the IoV use wireless links to transmit data, which exposes a larger attack surface, and it is precisely because of these features that the secure and efficient placement of microservices in the environment poses a serious challenge. Improving the security of all nodes in an IoV can significantly increase the service provider’s operational costs and can create security resource redundancy issues. As the application of reinforcement learning matures, it is enabling faster convergence of algorithms by designing agents, and it performs well in large-scale data environments. Inspired by this, this paper firstly models the placement network and placement behavior abstractly and sets security constraints. The environment information is fully extracted, and an asynchronous reinforcement-learning-based algorithm is designed to improve the effect of microservice placement and reduce the security redundancy based on ensuring the security requirements of microservices. The experimental results show that the algorithm proposed in this paper has good results in terms of the fit of the security index with user requirements and request acceptance rate.

1. Introduction

The Internet of Vehicles (IoV) is a very promising technology [1], and in recent years, with the rapid development of the Internet and the IoV, the number of vehicle users has been growing, and the generation and processing of large amounts of data have become common phenomena [2]. Meanwhile, as the cost of a network channel’s bandwidth and computing resources decreases, internet applications are becoming larger and larger, with too much coupling existing between functions and features for system changes and later maintenance [3,4]. Unlike traditional networks, bandwidth and computing resources are generally scarce in IoV environments, and the use of individual applications may face quality of service problems such as high latency and network congestion. Microservices are one solution to this problem because they have the feature of splitting a large application into multiple small and independent services [5,6], each running in an independent process. Microservices communicate with other services through open service interfaces to jointly fulfill the functions of the application [7]. Several studies have shown that microservice technology can be ported to the IoV by combining containerized microservice technology with the Kubernetes orchestration tool [8,9,10], as well as by designing orchestration algorithms to manage the whole process. By applying microservice technology within the IoV, the performance in terms of network flexibility and scalability can be improved.

Although microservice management can be implemented in the IoV, the large scale of microservices creates a complex chain of microservice calls as the number of microservices grows exponentially. The vehicle may not be able to handle the task when it reaches the IoV due to its limited computational resources [11]. In addition, network communication between microservices exposes a larger attack surface because communication between vehicles and between vehicles and fixed base stations on the road, servers, and other facilities uses wireless communication [12]. In the communication process, the number of data transfers increases, and the increased number of data transfers not only increases communication overheads but also poses challenges to the security of user data. Protecting personal privacy and sensitive information in this process has become a topic of great concern [13]. Existing security placement solutions are not as effective as traditional security defense techniques such as intrusion detection in the face of large, dynamically deployed microservices, and are more difficult to implement [14,15]. In addition, users have different needs concerning security levels; for example, the transmission of citizens’ personal information and governmental data requires a high level of communication security, whereas behaviors such as the prediction of traffic flow in the city do not require a high level of security.

Figure 1 shows the network environment where microservices are placed in the Internet of Vehicles (IoV) as an example on the left side, and the result of the microservices’ deployment on the right side. Figure 1 shows that information between vehicles and with fixed actors such as roadside base stations is transmitted via wireless signals, which makes it much easier for an attacker to intercept these data using specialized devices or software than wired communications that use physical links for data transmission, thereby stealing sensitive information or interfering with the normal operations of microservices [16]. In addition, when it comes to information-sensitive applications such as navigation, vehicles collect a large amount of important information such as roadside and geographic coordinates while driving, and if this service is placed in a node that does not meet the security standards, it will increase the likelihood of sensitive information leakage and attacks. Designing a deployment algorithm that meets the security expectations of microservices can greatly improve the security performance of the entire network. This allows the operator’s entire service to provide services to users at a high level of security, and the user’s security requirements are guaranteed at the same time, resulting in greater user satisfaction and promoting the development of a network society [17,18]. However, higher security necessitates greater costs, and with a large number of network nodes, these costs need to be borne by network operators and users, so it is not practical to upgrade every node where microservices can be placed to a highly reliable and secure node. Each microservice has different security requirements. For example, assume there are two services involving two high-security to-be-placed nodes and one low-security to-be-placed node, and the two microservices are the police linkage service and the road information reporting service. Compared to the police linkage service, the road information reporting service does not have high security requirements. Thus, the most suitable placement scenario is to place the police linkage service in the high-security node and the road condition information service in the low-security node. This requires reducing the likelihood of attacks on microservices and improving the security of the entire network, taking into account the constraints of the CPU and other resources of the nodes as well as the QoS. There are several deployment options. This study has chosen B, C, G, and F as the nodes where each of the four microservices are placed, and the red link is the actual deployment link. In Figure 1, the network topology used contains eight nodes and nine links, which is a relatively simplified network. However, in real-world network environments, the number of network nodes, the complexity of the links, and the volume of requests for microservices are usually far beyond these numbers. Currently, numerous scholars, such as He et al. [19,20,21,22,23,24], use traditional mathematical methods or heuristic algorithms to deal with the placement of microservices. Although these methods are effective in some cases, they tend to consume a lot of computational resources when dealing with large-scale, complex networks and may lead to obtaining only locally optimal solutions. In addition, other scholars, such as Li et al. [25,26,27,28], have begun to explore the use of deep learning techniques, but these studies tend to ignore post-deployment security issues. Considering the current practical challenges of microservice deployment, the trend of increasing network size in the future [29], and the research value of microservice-related problems, it is particularly important to develop an algorithm that considers both security and efficiency when handling microservice deployment [30]. This algorithm should not only provide solutions to existing problems but should also be able to adapt to future developments in the network environment to ensure long-term viability and effectiveness. This will ensure that operators can provide services to their subscribers at a highly secure level of performance, while at the same time fulfilling the subscribers’ requirements for security, thereby enhancing subscriber satisfaction. This enhancement not only strengthens subscriber trust but also contributes to the overall development of a network society.

The main contributions of this paper are as follows:

• This study proposes a microservice placement strategy based on differential security requirements, targeting environments like the IoV that heavily rely on wireless communication and are susceptible to network attacks. This study models the microservice placement process as a graph-embedding process. This study combines the node placement process with an asynchronous reinforcement learning algorithm by setting up an agent to make the algorithm converge quickly and to make it perform well in a network environment with a large number of microservices to be placed.
• Unlike traditional placement methods, the proposed algorithm dynamically handles the security parameters of each microservice placement node. The security index is set for the placement node to cope with the users’ differentiated security requirements, ensuring that each microservice is placed in a node that meets its security requirements as much as possible to reduce the likelihood of cyber-attacks.
• In addition, this study also conducts simulation experiments based on the real network environment, and the algorithm proposed in this paper shows good results in the request acceptance rate and security demand satisfaction index in comparison with other algorithms.

The overall structure of this paper is as follows. The work related to the secure placement of microservices is presented from two perspectives in Section 2. Immediately after that, the proposed asynchrony-based differential microservice placement algorithm is described in detail in Section 3. Section 4 describes the environment setup for simulation experiments and analyzes the results in detail. Finally, in Section 5, this study is concluded and possible future research directions for microservice security placement are analyzed.

2. Related Work

This section introduces and analyzes research advances in recent years in the areas of microservice security and microservice placement, respectively.

2.1. Microservices Security

In the study by Gopal [31] and Hossain et al. [32], it is proposed that although microservices, as a lightweight architecture that can fully utilize the capabilities of edge nodes, can effectively address challenges such as poor scalability of monolithic programs, the complex invocation relationships that exist between microservices expose a larger attack surface. The placement, scheduling, and auto-scaling of microservices may have security threats, so it is crucial to develop lightweight cryptographic mechanisms or secure placement algorithms. Jin et al. [33] proposed a framework that can sense changes in the cloud environment and dynamically optimize for changes in mobile target defense. Jin et al. [33] formalized various attack scenarios to build a multidimensional attack model. The effectiveness of container technology and mobile target defense in a microservices architecture is accurately evaluated. Rishi et al. [34] attempt to detect attacks launched towards microservices and propose a Docker-based lightweight architecture that can reduce traffic overhead by deploying the solution at the edge for earlier detection and mitigation of attacks. Zdun et al. [35] proposed a model-based approach to determine compliance with safety standards by predicting whether the environment conforms to predefined metrics. Experimental results show the accuracy of this method. In contrast to setting security policies, Xu et al. [36] propose the idea of setting security components separately. They deployed an additional security gateway that supports authentication and authorization in the original gateway. The experimental results also show that the communication latency of microservices provided based on the secure gateway is lower. Ibrahim et al. [37] devise a set of security rules to constrain network function placement behavior using a mixed-integer linear programming optimization model, but the paper focuses more on latency minimization. Overall, there is relatively little existing research on the security aspects of microservice placement and a lack of algorithms for microservice placement based on users’ differentiated security requirements.

2.2. Microservice Placement

For this type of node placement problem, the classical approach proposed by Cheng et al. [38] uses a Markov model to rank the network nodes according to their resources and topological properties. Experimental results also show that the method proposed by Cheng et al. yields some improvement in the evaluation metrics of long-term yield and acceptance rate. Similarly, Selimi et al. [39] design a low-complexity heuristic algorithm to solve the microservice placement problem. The information required for the microservice placement decision is derived from the node availability and network bandwidth dimensions of the network information. Experimental results show that the microservice placement algorithm proposed by Selimi et al. provides a significant improvement over the random placement method in terms of both customer response time and packet loss rate. However, the present placement method does not consider security in specific usage scenarios and other factors that affect placement. In contrast, Gu et al. [40] first proved that the microservice placement problem is an NP-hard problem, modeled the microservice placement problem in the form of integer linear programming, and then solved the microservice deployment problem by combining intra-service and inter-server sharing. Experiments show that the approach proposed by Gu et al. provides a large improvement in edge throughput rate. He et al. [19] define the service placement problem in microservice systems with complex dependencies and multiple instances as a fractional polynomial problem and transform it into a quadratic ratio and fractional problem to be solved by a greedy algorithm. The experimental results also show the computational speed advantage of the algorithm proposed by He et al. [19] Moreover, Li et al. [41] also use heuristic algorithms to establish a microservice layout optimization model to minimize the average transmission latency and load balancing and use genetic algorithms to effectively achieve low latency and load balancing. Liu et al. propose a multifactor evolutionary algorithm for the container placement problem in the microservices framework, setting multiple optimization objectives to cope with the container placement problem for a wide range of applications of different sizes in the heterogeneous clustering environment, solving the container placement problem in heterogeneous clustered environments. Su et al. [42] address the problem of microservice placement in satellite networks in a space–air–ground integrated network and propose an approach that uses an attention mechanism to efficiently capture attributes, such as computational resources, that affect microservice placement. Simulation experiments show that the proposed method of Su et al. performs well in terms of gain–overhead ratio and placement request acceptance rate.

In general, existing solutions to the node placement problem are mainly classified into several categories such as using traditional algorithms, heuristic algorithms, and deep learning methods. However, there is not much research on the microservice placement problem considering security. This paper is inspired by the above research and proposes a microservice placement strategy with differentiated security requirements.

3. Algorithm Description

In this section, the modeling scheme of the problem is first presented, and reasonable constraints and performance evaluation metrics are designed according to the network. The proposed algorithm is presented in detail immediately after based on the problem modeling.

3.1. Problem Definition

In this study, the set $MSR$ is used to represent the set of service requests, and its expression is shown in Equation (1).

$$MSR=\{G_1^{ms},G_2^{ms},\cdots ,G_x^{ms}\},$$

(1)

where this study abstracts all microservices that complete a full service into a directed weighted graph G, as shown in Equation (2).

$$G^{ms}=\{N^{ms},L^{ms}\},$$

(2)

where $N^{ms}$ represents the set of all microservices that complete this service and $L^{ms}$ represents the set of open links invoked between microservices.

For each microservice node ($n^{ms}\in N^{ms}$), there is a $Cos{t}_{CPU}\left(n^{ms}\right)$ representing the CPU resources that need to be consumed to run this microservice. Similarly, for each link $l^{ms}$, there is a $Cos{t}_{BW}\left(l^{ms}\right)$ representing the bandwidth resources that need to be consumed by the two microservices to call through the open interface. Furthermore, this study defines the abstraction of the entity nodes where the microservices are placed as an undirected weighted graph $G^{env}$, as shown in Equation (3).

$$G^{env}=\{N^{env},L^{env}\},$$

(3)

where $N^{env}$ represents the collection of physical nodes where the microservice is deployed and $L^{env}$ represents the collection of communication link between the physical nodes. For any $n^{env}\in N^{env}$, there exists $Remain\left(n^{env}\right)$, representing the remaining to compute and storage resources of that node; $Typ{e}_n^{env}$, representing the set of types of microservices that this node supports for deployment, and $Sec\left(n^{env}\right)$, representing the security level of the node. Similarly, for a link linking physical nodes, there is $Band\left(l^{env}\right)$, representing the remaining bandwidth of the link. Security requirements for the IoV include, but are not limited to, security requirements for remote wireless update packet authentication, geo-referenced security requirements when providing location information services, and data security requirements in data analytics services. Due to the number of services and the variety of security requirements in the IoV, it is not easy to quantify. Therefore, this study quantifies the security requirements of different microservices in the IoV in terms of security levels.

In a real network environment, every node is not exactly the same in terms of security due to hardware and technology differences. In other words, they have different levels of security. For example, high-security nodes usually integrate stronger firewall features or can support multiple encryption protocols such as WPA3, which can effectively block malicious traffic and potential threats. Deploying microservices in higher-security nodes can effectively reduce the likelihood of malicious attacks on microservices, thus reducing the probability of the entire application being affected. This study establishes a positive correlation between security index and security performance, where the higher the security index, the better the security. Similarly, microservices have different security-level needs. Microservices that have less impact on national and personal interests (e.g., commodity information management services) require lower placement node security levels than sensitive services (e.g., payment services).

Given that microservice requests eventually need to be deployed on physical nodes in the environment, the relationship between $G_i^{ms}$ and $G^{env}$ can be formalized as Equation (4).

$$G_i^{ms}\mapsto G^{env},\forall G_i^{ms}\in MSR,$$

(4)

The deployment process is considered to be completed when each of the microservices contained in the service request and their corresponding links are successfully deployed in the physical network. In the early stages of microservice deployment, due to the relative abundance of available resources in the physical network, each service request arriving in the network has a variety of potential deployment locations to choose from when deploying the microservice and its links. Each deployment option will have an impact on the subsequent deployment process. This study uses two binary variables, Equations (5) and (6), to express whether the microservices and links in the request $G_i^{ms}$ are successfully deployed to the corresponding physical nodes or links, respectively.

$${\Phi }_j^i=\left\{\begin{array}{cc}1,\hfill & n_i^{ms}\mathrm{successfully}\mathrm{deployed}\mathrm{to}{n}_j^{env}\hfill \\ 0,\hfill & n_i^{ms}\mathrm{unsuccessfully}\mathrm{deployed}\mathrm{to}{n}_j^{env}\hfill \end{array}\right.$$

(5)

where $n_i^{ms}$ represents the i-th microservice in this service request and $n_j^{env}$ represents the j-th physical node in the physical network topology. A binary variable of 1 means that $n_i^{ms}$ has been successfully deployed to node $n_j^{env}$, and conversely, a variable of 0 means the microservice $n_i^{ms}$ has not been deployed to $n_j^{env}$. Similarly, the binary variable $\Psi$ shown in Equation (6) follows this rule.

$${\Psi }_{rs}^{pq}=\left\{\begin{array}{cc}1,\hfill & l_{pq}^{ms}\mathrm{successfully}\mathrm{deployed}\mathrm{to}{l}_{rs}^{env}\hfill \\ 0,\hfill & l_{pq}^{ms}\mathrm{unsuccessfully}\mathrm{deployed}\mathrm{to}{l}_{rs}^{env}\hfill \end{array}\right.$$

(6)

where $l_{pq}^{ms}$ represents the link between microservice $n_p^{ms}$ and $n_q^{ms}$ and $l_{rs}^{env}$ represents the link between physical node $n_r^{env}$ and $n_s^{env}$. The formulae for calculating the remaining computing and storage resources are shown in Equations (7) and (8).

$$Band\left(l^{env}\right)=O_{bw}\left(l^{env}\right)-\sum _{i=1,j=1,l^{ms}\Uparrow l^{env}}^{\left|MSR\right|}Cost\left(l_{ij}^{ms}\right),$$

(7)

where $O_{bw}\left(l^{env}\right)$ represents the bandwidth at the idle time when no microservices are placed on the physical link $l^{env}$ and $\left|MSR\right|$ represents the number of microservice requests.

$$Remain\left(n^{env}\right)=O_{cpu}\left(n^{env}\right)-\sum _{i=1,n^{ms}\Uparrow n^{env}}^{\left|MSR\right|}Cost\left(n_i^{ms}\right),$$

(8)

where $O_{cpu}\left(n^{env}\right)$ represents the total amount of computational and storage resources in the physical node when no microservices are deployed.

The constraints shown in Equations (9)–(13) also need to be satisfied during microservice placement. Firstly, to ensure the proper operation of the microservices, the remaining computational resources of the server must be greater than the computational resources required by the microservices, which is expressed as shown in Equation (9).

$$\begin{array}{c}Remain\left(n_p^{env}\right)-{Cost}_{CPU}\left(n_q^{ms}\right)\ge 0,\\ n_p^{env}\in N^{env},n_q^{ms}\in N^{ms},if{\Phi }_q^p=1\end{array}$$

(9)

Similarly, the remaining bandwidth of the communication links between servers must also meet the bandwidth consumed by the calls between microservices. Otherwise, network delays and packet loss will occur, which will affect the normal operation of the whole service. The expression is shown in Equation (10).

$$\begin{array}{c}Band\left(l_{rs}^{env}\right)-{Cost}_{BW}\left(l_{ij}^{ms}\right)\ge 0,\\ l_{rs}^{env}\in L^{env},l_{ij}^{ms}\in L^{ms},if{\Psi }_{rs}^{ij}=1\end{array}$$

(10)

In addition, the security index of each microservice placement request should be at least greater than or equal to the actual user requirement for the security index. The expression is shown in Equation (11).

$$\sum _{n^{ms}\Uparrow n^{env}}Sec\left(n_i^{env}\right)\ge {Sec}_{user}$$

(11)

Security-related information is self-contained attribute information in the network nodes, similar to other resource attributes such as CPU and bandwidth, which can be obtained directly from the network. Finally, each microservice can only complete its lifecycle in one physical node, and this expression is shown in Equation (12).

$$\sum _{j=1}^{|N^{env}|}{\Phi }_j^i=1$$

(12)

In contrast to nodes, the transmission of data emanating from a microservice over a physical link can take multiple hops to reach the destination node, and this expression is shown in Equation (13).

$$\sum _{i=1}^{|L^{env}|}{\Psi }_{l_i^{env}}^{pq}\ge 1$$

(13)

Finally, consider the rapid development of container technologies such as Docker and container orchestration technologies such as Kubernetes. In the future, microservice architectures will not be affected by inherent placement requirements and will achieve smoother deployment and usage, so the inherent placement requirements for certain microservices are not constrained.

3.2. Evaluation Indicators

The placement acceptance rate of microservice requests is the most basic metric for evaluating the quality of placement services. The placement acceptance rate is directly proportional to the performance of the placement algorithm. Under a certain number of elements in the MSR of the microservice placement request set, if the placement acceptance rate is high, it means that the more applications are successfully deployed in the physical network, the stronger the stability of the whole microservice architecture, and the better the user experience. The expression for the placement acceptance rate is shown in Equation (14).

$$R_{\mathrm{accept}}=\frac{|MS{R}_{\mathrm{accepted}}|}{|MS{R}_{\mathrm{arrived}}|}\times 100\%,$$

(14)

where $|MS{R}_{\mathrm{accepted}}|$ is the total number of microservice placement requests arriving at the network and $|MS{R}_{\mathrm{accepted}}|$ represents the number of microservices that were successfully placed. Under the premise of ensuring the acceptance rate, the security index of the whole link is made to meet the user’s needs as far as possible. The security index after the final placement is completed will neither fall short of the user’s security needs nor cause the risk of privacy leakage. At the same time, it will not be substantially higher than the user’s security needs, causing unnecessary waste of security resources. Equation (15) is the calculation method for the evaluation index of user fit.

$$R_{\mathrm{srfr}}=\frac{{Sec}_{user}}{{\sum }_{n^{ms}\Uparrow n^{env}}Sec\left(n_i^{env}\right)}\times 100\%$$

(15)

where ${\mathrm{Sec}}_{user}$ is the security requirement and ${\sum }_{n^{ms}\Uparrow n^{env}}Sec\left(n_i^{env}\right)$ is the sum of the security levels of the nodes occupied by the actual microservices prevented. As mentioned earlier, this paper tries to place services in nodes that meet their security requirements as much as possible, reducing the redundancy of security resources while safeguarding their basic security needs. To measure this effect, this paper proposes the evaluation metric of security fit. A higher security fit R means less waste of security resources and better results of the algorithm.

3.3. Algorithm Design

To achieve the optimization goals mentioned above, and to adapt to complex and dynamic microservice placement environments and constraints, as well as to reduce human intervention in placement, this study proposes an algorithm based on asynchronous reinforcement learning (RL) for secure microservice placement. This study uses Actor–Critic as the basic framework, but unlike the traditional Actor–Critic model, the asynchronous security algorithm proposed in this paper improves convergence by introducing multiple parallel intelligences. Each agent possesses its own parameters and policies, allowing them to interact and learn in diverse network environments. After a certain number of learning iterations, the gradient of the loss function in each agent is calculated and updated in the central neural network. The whole architecture is similar to a tree of public ancestors. Because each intelligence learns individually in its own environment, the empirical data collected are relatively independent and do not affect each other, thus increasing the diversity and richness of the samples. After updating to the established rounds, the model of the central network is the final model this study uses for making the selection of nodes for microservice placement.

The detailed architecture of the method is shown in Figure 2. The Actor and Critic are the two key parts of the algorithm. In the algorithm, the Actor is mainly used for the selection of microservice placement actions, and the Critic is used to evaluate the value generated by the microservice placement and provide feedback to the Actor to help it improve the placement strategy. In carrying out the above process, three core concepts are involved: state space, action space, and reward. Here, the precise definition of the state space is the key to implementing an effective reinforcement learning algorithm. This is because the Actor must select the appropriate action based on the current state and learn the optimal behavioral strategy in the process. In addition, the network environment changes (e.g., node computational resource availability) after the microservice deployment task is completed, which requires the algorithm to be able to perform state updates in real time. At the same time, the reward mechanism needs to give feedback to the Actor based on the change in state. The complexity of the network environment further increases the difficulty of obtaining the state of the network environment. In particular, the size of the state space directly determines the complexity of the problem, as the intelligence cannot understand the network environment directly. If the information extracted from the environment contains too much irrelevant data, it will reduce the efficiency of the algorithm. To solve this problem, this study extracts key information from the environment that affects the effectiveness of microservice deployment to form a feature matrix. Specifically, this study selects the following network features from the network environment to construct the feature matrix.

• Available resources of nodes: If the amount of available resources of a physical node does not reach the resources consumed by a microservice’s deployment, it cannot be deployed, so it is evident that the amount of available resources of a node is directly related to the success rate of the microservice deployment. Given this, this study includes available node resources as a key feature extracted from the network environment. Its calculation is shown in Equation (8).
• Bandwidth resources of links: The greater the available bandwidth resources of a link, the more data will be allowed to be transmitted simultaneously. If the available bandwidth of a link to which a physical node is connected is greater, the likelihood of a successful deployment of a microservice is greater. This study uses $A{v}_{BW}\left(n^{env}\right)$ to denote this feature, which is computed as shown in Equation (16).

  $$A{v}_{BW}\left(n^{env}\right)=\sum _{l^{env}\in L_{n^{env}}}Band\left(l^{env}\right)$$

  (16)
• Connectivity of nodes: The more connected a physical node is to other nodes, the more optional the communication links between deployed microservices are, and the higher the probability is of implementing microservice placement using a multi-hop format. This study uses the degree $De\left(n^{env}\right)$ of a physical node to denote the connectivity of a node.
• Security level of nodes: Deploying microservices in physical nodes that meet the security level is the only way to reduce the possibility of attacks and thefts of the sensitive information they carry, so the security level of the physical node is an important factor that affects the secure placement of microservices. This study denotes security level by $Sec\left(n^{env}\right)$.

Based on this, the feature information of each physical node is aggregated into a feature matrix, as shown in Equation (17). During training, the intelligences simply extract this feature matrix directly from the network environment.

$$\mathbf{M}=\left[\begin{array}{c}{F}_{n_1^{env}}\\ F_{n_2^{env}}\\ ⋮\\ F_{n_{|N^{env}|}^{env}}\end{array}\right]=\left[\begin{array}{cccc}Remain\left(n_1^{env}\right)& A{v}_{BW}\left(n_1^{env}\right)& De\left(n_1^{env}\right)& Sec\left(n_1^{env}\right)\\ Remain\left(n_2^{env}\right)& A{v}_{BW}\left(n_2^{env}\right)& De\left(n_2^{env}\right)& Sec\left(n_2^{env}\right)\\ ⋮& ⋮& ⋮& ⋮\\ Remain\left(n_{|N^{env}|}^{env}\right)& A{v}_{BW}\left(n_{|N^{env}|}^{env}\right)& De\left(n_{|N^{env}|}^{env}\right)& Sec\left(n_{|N^{env}|}^{env}\right)\end{array}\right]$$

(17)

This study uses a dominance function to measure the superiority of each placement action relative to the average to help estimate the error of the value function and to aid in updating the parameters of the value function. The Q function is used to represent the expected value of the cumulative future rewards that intelligence will receive for executing its strategy, and it is expressed in Equation (18).

$$Q\left(s_t,a_t\right)=\sum _{i=0}^{\sigma -1}{\gamma }^i{r}_{t+i}+{\gamma }^{\sigma }V\left(s_{t+\sigma }\right)=r_t+\gamma V\left(s_{t+1}\right),$$

(18)

where $s_t$ represents the state at step t and $a_t$ represents the placement action chosen at state $s_t$. $\sigma$ represents the offset of the time step used to calculate the cumulative reward, starting at the current time step t. The cumulative reward calculation takes into account the rewards at the next $\sigma$ time steps. To achieve low latency updates and to improve the stability of the algorithm, $\sigma$ takes the value of 1. $\gamma$ is a discount factor between $[0,1]$ that weighs the importance of current and future rewards. The placement behavior of microservices is a holistic one and needs to be considered in the long run, so the discount factor in this paper takes the value of $0.9$. $r_{t+i}$ denotes the reward obtained at $t+i$ time steps. $V\left(s_{t+\sigma }\right)$ denotes the expected value of the cumulative reward obtained in the future under the implementation of the microservice placement policy. This formula is subtracted from the value function $V\left(s_t\right)$ in the current state, which is determined in Equation (19), to obtain a dominance function that can measure the degree of superiority or inferiority of the action.

$$A\left(s_t,a_t\right)=Q\left(s_t,a_t\right)-V\left(s_t\right)$$

(19)

Finally, the dominance function and the entropy $H\left(\pi \left(s_t\right)\right)$ used to encourage exploration and increase the randomness of the strategy together form Equation (20) to enable strategy updating.

$$\theta ={\nabla }_{{\theta }^{\prime }}log\pi \left(a_t\mid s_t\right)A(s_t,a_t)+\beta {\nabla }_{{\theta }^{\prime }}H\left(\pi \left(s_t\right)\right)$$

(20)

where ${\nabla }_{{\theta }^{\prime }}log\pi \left(a_t\mid s_t\right)$ is the strategy gradient, which is used to indicate how the parameters should be adjusted to increase the probability of taking a favorable action. ${\nabla }_{{\theta }^{\prime }}$ is a gradient operator. The policy network uses a multilayer perceptron. An MLP is a feed-forward neural network that contains a series of linear transformations and nonlinear activation functions, components that are admittedly differentiable. $A\left(s_t,a_t\right)$ is the dominance function mentioned in Equation (19), which instructs which nodes will increase the probability of which microservices will be placed. $\beta {\nabla }_{{\theta }^{\prime }}H\left(\pi \left(s_t\right)\right)$ is the entropy regularization term, and $H\left(\pi \left(s_t\right)\right)$ is the entropy of the policy in state $s_t$, which is used to encourage the policy to remain exploratory and prevent premature convergence to a locally optimal strategy.

The training pseudo-code for the algorithm is shown in Algorithm 1. The inputs to the algorithm include the microservice placement request, the physical network on which the microservice is placed, and the training parameters. Among them, the training parameters include a maximum number of training rounds of 100, a discount factor of $0.9$, an update frequency of 10, a policy network learning rate of $0.005$, and an evaluation network learning rate of $0.001$. The two evaluation indicators introduced in Section 3.2 are used as a reward function to assist in training. At the end of the training, the output model contains weighted parameters that allow the network state to be mapped to a probability distribution of microservice placement actions. This study uses these model parameters to determine the probability of microservices’ placements in each node and uses the Dijkstra algorithm to complete the placement of call links between microservices.

Algorithm 1: Training.

1: Input: training parameters, physical network, MSR; 2: Output: network model; 3: Initialization of optimizers, global networks, worker networks, training parameters; 4: while ($epoc{h}_{current}<epoc{h}_{max}$) do 5:     reset $epoch\_reward$; 6:     while $True$ do 7:         Select the placement action; 8:         Calculating the $stat{e}_{next}$, $rewar{d}_{now}$; 9:         $epoch\_reward$ += $rewar{d}_{now}$; 10:         Update work node parameters; 11:         if Arriving to update global agent rounds then 12:            Synchronizing global network parameters; 13:         end if 14:         if Placement complete then 15:            Save the best model under the current round; 16:         end if 17:         $epoc{h}_{current}++$; 18:         $stat{e}_{now}$ = $stat{e}_{next}$; 19:     end while 20: end while 21: return model;

4. Experiments

Experiments were run on a computer with an Intel i7-11800H processor and 16GB of RAM on an NVIDIA RTX3060 graphics card with one 6G video memory. The algorithms were run in a Miniconda environment, and the code was written in Python 3.8. In order to reflect the effectiveness of the algorithm proposed in this paper, the simulation experiments are chosen to be carried out in the data from a real environment. This study uses the network topology data provided by SNDlib [43] for the experiments. SNDlib is a database for telecommunication network design created by the Warsaw University of Technology in 2005, which aims to provide researchers with data about real telecommunication networks to facilitate the optimization and operation of telecommunication networks and their operations. The Germany50 [43] dataset this study used is one of the real classical network topologies provided by SNDlib. The instance contains 50 nodes and 88 edges. Since it does not contain data on security per se, this study only uses its topology. The range of experimental parameter settings was generated based on the real parameters as well as the joint parameter settings of the studies [38,44,45]. This study sets the computational resources and link bandwidths of each physical node to conform to a normal distribution in the interval $[50,100]$ with 1000 service microservice placement requests. Moreover, this study sets the microservice placement request arrival frequency to 5 per 100 time units. Each placement request contains microservices that obey a uniform distribution within $[2,6]$. In addition, regarding the setting of security parameters and security requirements, this study sets the security index of each physical node to be uniformly distributed between $[10,20]$. Specific experimental and training parameters are demonstrated in

Table 1. Parameter settings.

Types	Description	Value
Training parameters	training rounds	100
	discount factor	$0.9$
	update frequency	10
	policy network learning rate	$0.005$
	evaluation network learning rate	$0.001$
	offset of the time step	0
Experimental parameters	number of nodes	50
	number of edges	88
	compute resource	$[50,100]$
	link bandwidth	$[50,100]$
	number of microservices included in a single microservice request	$[2,6]$
	security level	$[10,20]$
	security requirements of microservices	$[5,20]$
	request arrival frequency	5
	service microservice placement requests	1000

.

This study documents in Figure 3 the changes in the request acceptance rate and the fit of the security requirements of the algorithm during the training process. In the pre-training phase, the algorithm is in an exploratory phase and underperforms on the two evaluation indicators because it has not learned a good microservice placement strategy. As the training proceeds, the algorithm improves its performance on the evaluation indicators based on the feedback of the reward changes. As the training proceeds, the changes in the evaluation indicators gradually slow down, and the algorithm gradually reaches a converged state. Thus, Figure 3 demonstrates that the performance of the algorithm proposed in this paper can be significantly improved with more training iterations and also demonstrates the convergence of the algorithm.

Since the microservice placement process studied in this paper can be abstracted as a network embedding problem, three state-of-the-art algorithms in the field of network embedding, namely NodeRank, PSM, and CS-RL-VNE, are chosen as the comparison methods in order to prove the effectiveness of the proposed algorithms in this paper. The basic principle of the comparison algorithm is described as follows:

• PSM [44]: This is a classical embedding algorithm and there is a large body of literature using it as a baseline algorithm. The basic idea of this algorithm is to construct a subset of deployable nodes for each of the microservices to be deployed using the greedy conceptm and then, for the available subset, select the node with the greatest available resources as the deployment node.
• NodeRank [38]: The NodeRank method calculates the physical node rank based on Markov random wandering. This method uses the amount of resources of a node as a measure of its deployment priority, ranks the nodes based on the resource and topological attributes of the network nodes, and deploys the microservices in the nodes with the largest amount of remaining resources.
• CSS-RL-VNE [45]: CSS-RL-VNE uses traditional RL methods as a means to improve the algorithm’s performance. The important attribute features of the underlying network are extracted for training by involving the policy network; the deployment possibilities are ranked based on the training results, and the physical node with the highest probability of successful deployment is selected as the deployment node.

In addition, to ensure the fairness of the experiments, all the algorithms are run in network environments with the same parameters, and for link deployment between microservices, this study uniformly uses the breadth-first search algorithm. Figure 4 and Figure 5 show the performances of the proposed algorithm on two key metrics: placement request acceptance rate and user security requirement fit, respectively.

As can be seen in Figure 4, algorithm generally outperforms the other two classical placement algorithms in terms of request acceptance rate. This is because this work includes consideration of microservice placement request acceptance rates in the reward during reinforcement learning training, and asynchronous-based reinforcement learning can interact extensively with the environment and learn towards a higher reward. In addition, to accommodate the other three comparison algorithms, this study only calculates the fit of successful placement requests.

As can be seen in the right-hand side of Figure 5, the proposed algorithm also performs better than the other three algorithms in terms of the fit to the user’s security needs, thanks to the security considerations this study has included in the algorithm. Among the comparison algorithms, PSM and NodeRank show very low levels of fit because they do not take security into account. The reason why the CSS-RL-VNE algorithm performs better than PSM and NodeRank is because it takes security into account. However, compared to the ASMP algorithm proposed in this paper, it does not consider the fit to the user’s security requirements, which leads to excess performance even if the minimum security requirement is met. For example, if a microservice only needs to be deployed into a node with security level 3 to satisfy its minimum security requirement, the CSS-RL-VNE algorithm may place it into a physical node with security level 8 to satisfy its security requirement. In this process, there is an excess of resources. Unlike the CSS-RL-VNE algorithm, the proposed algorithm is designed with fit to user security requirement in mind. The algorithm proposed in this paper achieves a security requirement fit of 0.85 in the Gemmery50 network topology. This means that on the basis of satisfying the basic security requirements of the microservices, the chosen security level does not exceed the microservices’ needs too much and thus reduces the redundancy of security resources. As shown in the experimental results, there are two main reasons why the algorithm does not fully achieve a 100% fit to user security requirements. First, since the primary goal is to ensure the normal deployment and operation of the application, the algorithm will first meet the basic security requirements and resource requirements to ensure placement with the greatest success rate and then will consider the degree of fit to security requirements, which inevitably will in the microservices being deployed in a node that exceeds the security requirements. Secondly, the distribution of security levels in the Gemmery50 nodes is not regular, and it is difficult to make the sum of the number of nodes in each security level exactly equal to the security requirements of the applications. Moreover, normal development may involve business expansion and security level upgrading, so leaving an appropriate security margin can reduce the costs of future maintenance.

Comprehensive experimental results show that the proposed algorithm has 5.9% and 8.97% better performance over the latest algorithms in terms of request acceptance rate and fit to user’s security requirements, respectively, and that it ensures security without wasting security resources.

5. Conclusions

Compared to traditional monolithic applications, microservice architecture still faces many challenges, although it can be used to split complex applications into small and autonomous sub-modules, providing greater flexibility and scalability for business-level expansion. On the one hand, the splitting of services and the increase in network size causes an increase in placement complexity. On the other hand, after splitting the services, the number of data transfers and the exposure time increases, bringing about great security risks. Ensuring high security for all nodes requires greater cost overheads. This paper reduces the operational costs and reduces the redundancy of security resources for service providers using an ingenious approach. This study designs an asynchronous reinforcement learning algorithm for differentiated security requirements, which uses the parallel training of multiple intelligences that interact with the microservice placement environment in parallel, and introduces a policy network for the selection of placement nodes, which ensures the acceptance rate of the placement requests and reduces the redundancy of the security resources based on the basic security requirements of the microservices. The experimental results show the effectiveness of this proposed algorithm.

In future studies, researchers should try to consider different scenarios and larger-scale network environments. In addition, future research could also consider starting from the intruder’s point of view, and constructing a multi-layered and all-round security protection system for aspects such as the transport layer encryption, authentication, and security mechanism of the API Gateway covered by the microservice security protocols. Finally, subsequent research can also consider the security of the link into the algorithm as well.