Agent-Based Virtual Machine Migration for Load Balancing and Co-Resident Attack in Cloud Computing

Featured Application

Applicable to application scenarios where virtual machine live migration is used to simultaneously solve load-balancing and co-resident attack issues.

Abstract

The majority of cloud computing consists of servers with different configurations which host several virtual machines (VMs) with changing resource demands. Additionally, co-located VMs are vulnerable to co-resident attacks (CRA) in a networked environment. These two issues may cause uneven resource usage within the server and attacks on the service, leading to performance and security degradation. This paper presents an Agent-based VM migration solution that can balance the burden on commercially diverse servers and avoid potential co-resident attacks by utilizing VM live migrations. The Agent’s policies include the following: (i) a heuristic migration optimization policy to select the VMs to be migrated and the matching hosts; (ii) a migration trigger policy to determine whether the host needs to relocate the VMs; (iii) an acceptance policy to decide if the migration request should be accepted; and (iv) a balancer heuristic policy to make the initial VM allocation. The experiments and analyses demonstrate that the Agents can mitigate CRA in a distributed way to mitigate the associated risks while achieving acceptable load balancing performance.

1. Introduction

Cloud computing is based on a group of networked servers, typically comprised of various commodity hosts with diverse computing resources [1,2]. Virtualization technology is commonly utilized in cloud computing to mask server heterogeneity, permit server consolidation, and enhance server utilization [3,4,5]. Virtualized hosts can host multiple virtual machines (VMs) with different resource requirements and varied workload types [6]. When servers host heterogeneous VMs with differing resource requirements, which may fluctuate unpredictably, it can cause an imbalance in resource usage within hosts, leading to service level agreement breaches and performance degradation. Unequal computing demands and workload imbalances can result in an accumulation of resource usage imbalances within hosts. For example, a VM that needs higher CPU resources but fewer memory resources [7] may result in imbalanced resource usage on the host. As a result, VM allocation to hosts should consider both computing resource parameters, such as the number of cores and memory size, as well as the dynamic workload. The dynamic balancing of workloads in cloud computing is increasingly achieved through live VM migration [8], which transfers VMs from an overloaded host to another with idle computing resources [9].

At the same time, it should be noted that the co-location of VMs produced for service requests from different customers on the same physical server in cloud computing introduces distinct security and reliability risks that have been overlooked by current load balancing algorithms. Co-Resident Attacks (CRAs) are malicious attacks on cloud computing services, which means that malware-infected VMs are on the same server as a target VM. Then, the malicious VMs may establish a side channel to enable data theft or corruption [10,11]. Various CRA mitigation techniques have been proposed, including side-channel elimination, VM assignment or migration, and game-theory-based strategies [12,13,14,15]. Ref. [16] provides an overview of several existing CRA defense mechanisms. In recent years, CRA mitigation strategies based on probabilistic combinatorial models have been proposed and analyzed, including data partitioning, replication and Non-Volatile Memory (NVM). These strategies have been developed to address both data reliability and security issues.

Although existing load balancing mechanisms in cloud computing generally do not consider co-resident attack factors, VM allocation and migration can mitigate these attacks while also being the primary means of achieving cloud computing load balancing. Therefore, by integrating load balancing and co-resident attack factors into a unified VM migration mechanism, both issues can be mitigated simultaneously. Motivated by this notion, our research endeavors to establish a comprehensive approach that integrates VM migration, load balancing, and CRA while addressing these concerns within a unified framework.

Agent-based methods have emerged as an effective approach for load balancing and co-resident attack mitigation in cloud computing environments. Agent-based cloud computing is a productive and efficient approach to managing cloud resources, as demonstrated by various examples [17,18]. This paradigm uses Agent-based strategies to achieve complex goals, such as load balancing and CRA mitigation, in a constantly evolving cloud computing environment. Agent-based load balancing is a method of distributing incoming requests across multiple servers in a distributed system, with the goal of optimizing resource utilization and minimizing response time. This approach employs a software agent that is deployed on each server, which communicates with other agents to coordinate the distribution of incoming requests.

Our research objective is to develop policies for VM live migration that address both load-balancing and security concerns in a Cloud Computing environment. In this manuscript, we present an Agent-based cloud computing load balancing approach that integrates load balancing and co-resident attack mitigation to address the challenges arising from (i) dynamic workloads in cloud computing, (ii) volatile VM resource utilization, and (iii) potential co-resident attacks on servers. Our Agent-based VM migration technique combines (i) real-time VM migration and (ii) the interaction and collaboration capabilities of Agents to achieve the distributed design objectives, such as load balancing and CRA mitigation. The load-balanced multi-agent protocol can balance workloads and steer clear of servers that are susceptible to co-resident attacks on a collection of heterogeneous hosts. This protocol defines roles for three distinct types of agents, including Balancer Agent, Server Agent, and Virtual Machine Agent, which dynamically cooperate using balancer heuristics, migration policies, acceptance policies, and migration heuristics to balance workloads. The balancer heuristics are responsible for the initial allocation of VMs. The distributed migration heuristics enables each Server Agent to decide which VM should be migrated when, while the Server Agent that needs to migrate a VM and the Server Agent that is willing to host a VM collaborate to determine the target host.

Our contributions are as follows:

(1) We propose an Agent-based cloud computing VM migration architecture that combines load balancing with co-resident attack mitigation and allows for the trade-off between load balancing and co-resident attack mitigation within the same framework.

(2) We develop a cooperative dynamic VM migration Agent-based system that can withstand potential co-resident attacks on servers while considering two factors: (i) fluctuating workloads and (ii) irregular resource use of VMs.

(3) We propose Agent-based VM migration mechanisms that enable the Agents to decide the initial allocation of VMs, the timing of VM migration, the migration destination, and whether to accept the migration requests.

The rest of the paper is structured as follows: Section 2 provides a review of related work, Section 3 defines the Agent-based VM migration problem, Section 4 and Section 5 provide a comprehensive description of the Agent-based VM migration mechanism, and Section 6 presents experiments and analyses. In Section 7, we provide concluding remarks and suggest some future research directions.

2. Related Works

2.1. Load Balancing in Cloud Computing

Load balancing is a critical issue in cloud computing [19,20], and various approaches have been proposed to address it, including network traffic load balancing [21], energy-aware load balancing [22], and stream processing load balancing [23]. Current solutions focus on balancing CPU and memory resources by leveraging live VM migration and considering the heterogeneity of servers and the variability of VM resource utilization. However, these solutions often overlook the fact that a single host may be configured to host multiple VMs, making the co-located VMs vulnerable to co-resident attacks in the event of a cyber assault.

Traditionally, cloud computing load balancing mechanisms are centralized. However, this approach may become a bottleneck of the system, highlighting the need for a more decentralized approach to load balancing. Deploying VMs on a heterogeneous collection of hosts can be treated as a multidimensional boxing problem [24,25], and mathematical approximations exist for this problem. However, these solutions are not directly applicable to the load balancing problem in cloud computing since they assume empty hosts and do not consider the number of moves required to balance the load through VM migration [26].

To address these issues, Hyser et al. [26] presented a VM allocation approach that monitors and forecasts host resource utilization using a simulated annealing algorithm and predefined heuristics to determine the source and target hosts for VM migration. However, their proposed centralized architecture has scalability limitations, and maintaining the resource utilization of hosts close to the entire system average may result in computational resource waste of computing-intensive hosts (because the same computational load is allocated as for ordinary hosts), which may not be optimal in heterogeneous commercial cloud computing environments.

Load balancing methods for cloud computing have been approached from various perspectives, with a trade-off between load balancing and the number of VMs to migrate being a common concern. For example, Red Hat Enterprise Virtualization Manager [27] uses live migration of VMs for load balancing, employing a uniformly distributed heuristic load balancing strategy that migrates the VMs to the host with the lowest CPU usage. In contrast, VMware’s resource scheduler [28] utilizes affinity criteria based on past information about the resource utilization of VMs and then statically matches the VM and the host.

Load balancing methods that rely on resource use prediction [29,30,31,32,33,34] are influenced by the dynamic performance changes of heterogeneous commercial hardware in cloud settings. The performance of a cloud computing is impacted by various factors, such as network circumstances and workload type, making it challenging to estimate cloud performance. The load-balancing technique based on resource prediction requires a valid sample of the factors. Thus, this method utilizes behavioral patterns [29], historical workload data [31], and VM resource utilization traces [30] to achieve load balancing.

Anderson et al. [35] proposed an Agent-based load balancing strategy, where an Agent placed on a VM monitors the location of consumer requests, and the VM is transferred to the external network once the number of consumer requests from the external network exceeds the number of (threshold-based) local requests. However, relocating VMs based only on consumers’ locations may not improve the cloud’s overall load balancing. In contrast, the Agent-based load balancing system [35] uses the real-time movement of VMs. The Agent managing the host is categorized as idle, overloaded, or underloaded, where the idle Agent is deemed balanced and does not participate in load balancing, and the underloaded and overloaded Agents collaborate to move VMs from the overloaded Agent to the underloaded Agent. Nevertheless, since cloud computing typically consist of heterogeneous commercial servers, dividing hosts into three categories may result in the underloaded computing-extensive Agent receiving VMs with high computing resource usage after crossing the threshold directly from underload to overload, leading to the migration of VMs away from it. This is due to the lack of previous samples of the exact resource utilization of the hosts to select the optimal target host based on the criteria.

2.2. Co-Resident Attack

A co-resident attack is a two-phase attack that aims to co-locate the attacker’s VMs on the same physical server as their intended victim. Once the attacker gains co-residency, they proceed to create side channels to collect sensitive information about the victim [36,37,38]. To achieve co-residency, the attacker may utilize brute force tactics, such as launching numerous VMs, subject to cost limitations, or taking advantage of sequential and parallel positions in VM placement. Notably, in the Amazon EC2 cloud [39,40], launching one VM soon after the termination of another VM or launching both VMs almost simultaneously increases the likelihood of these VMs being allocated to the same server.

In theory, co-resident VMs should not interfere with one another, but this is not always the case in existing cloud systems. This interference enables attackers to construct side channels between VMs and gain access to sensitive data. For instance, experiments [39] have shown that an attacker can use the Prime+Probe [41,42] approach to deduce cache utilization and workload information of the victim, as cache usage significantly affects cache read operation execution time. Additionally, an attacker can assess the victim’s network traffic. Furthermore [43], attackers can leverage common hardware resources, such as instruction caches, to recover cryptographic keys.

Previous research has identified several potential co-resident attack defenses, which can be broadly categorized into five types. The first category involves side-channel removal, which refers to techniques described in [44,45]. These strategies operate at the hardware layer and may include deleting high-resolution clocks [46], introducing delays for potentially malicious activities [47], and removing all internal reference clocks [48]. Another approach involves imposing isolation by prohibiting the sharing of critical resources. For example, page coloring [49] can be utilized to limit cache-based side channels, and hypervisors [50] can be replaced with hardware techniques to separate access to shared resources. However, the implementation of these techniques often requires significant modifications to current cloud platforms, making their adoption by cloud service providers doubtful.

The second category aims to increase the challenge of confirming coexistence by utilizing network measurements to identify whether two VMs are on the same server [39]. To conceal Dom0’s IP address, which is used to administer all VMs on the host, cloud providers can block the customer’s access to it. However, the effectiveness of simply hiding Dom0’s IP address as a means of detecting co-residence is increasingly challenged by the emergence of new detection methods. Various approaches have been proposed to identify co-residence, including those described in [51,52,53]. Therefore, additional measures beyond IP address hiding are needed to mitigate the security risks associated with co-resident attacks.

The third category involves detecting abnormalities in system calls, CPU and RAM consumption, and cache miss behavior that may indicate an attacker is using the Prime+Probe [54,55] approach to harvest information from a victim. Researchers have proposed several methods for detecting these characteristics and constructing defense systems appropriately.

The fourth category involves utilizing VM allocation rules that make colocation difficult. One such strategy is the co-resident co-build (CLR) system [56], which classifies servers as open or closed based on their ability to accept more VMs. The system maintains a constant number of open servers and assigns a new VM at random. If the selected server cannot accept further VMs, it will be designated as closed, and a new server will be started. The greater the number of open servers, the greater the algorithm’s co-resident resistance.

The fifth and final category involves using the Vickrey–Clarke–Groves (VCG) [57,58] technique to periodically move VMs to different hosts to reduce the likelihood of co-residence. The number of VMs to be moved and the destination hosts are determined, and a VM allocation strategy is generated to minimize the total security risk. However, frequent VM movement may increase power consumption and result in performance degradation, which could breach the Service Level Agreement (SLA) with the cloud provider [59,60].

3. Research Aim and Problem Statement

In this part, we outline the purpose of our research. Then, we define the problem.

3.1. Research Aim and Metrics

The objective of this study is to develop an Agent-based VM migration methodology that integrates co-resident attack (CRA) mitigation strategies into live migration policies, with the objective of ensuring equitable distribution of workloads and the secure operation of cloud computing. The effectiveness of our approach is evaluated using a set of three key performance indicators. To clarify the calculation method for the key performance indicators utilized in this method, and to facilitate the development of subsequent virtual machine migration strategies, we have included the metrics used in this paper in

Table 1. Definitions regarding the metrics.

Name	Definition
$i$	index of VMs in the Cloud $\left(i=1,2,\dots ,n\right)$, $n$ indicates the number of VMs in the Cloud.
$j$	index of hosts in the Cloud $\left(j=1, 2, \dots ,m\right)$, $m$ indicates the number of hosts in the Cloud.
$t$	index of load-balancing decision time points $\left(t=1, 2, \dots , \mathrm{T}\right)$, $\mathrm{T}$ indicates the total operation time.
${\alpha }_i$	Number of cores required by $V{M}_i$
${\alpha }_{it}^{\%}$	Percentage of CPU usage of $V{M}_i$ at time $t$
${\beta }_i$	Number of gigabytes of memory required by $V{M}_i$
${\beta }_{it}^{\%}$	Percentage of memory usage of $V{M}_i$ at time $t$
${\gamma }_{it}$	Security level of $V{M}_i$ at time $t$
$C_{jt}$	Idle number of cores of host $j$ at time $t$
$M_{jt}$	Idle memory (expressed in gigabytes) of host $j$ at time $t$
$x_{jt}^{\%}$	Percentage of CPU usage of host $j$ at time $t$
$y_{jt}^{\%}$	Percentage of memory usage of host $j$ at time $t$
$S_{jt}$	Security level of host $j$ at time $t$
$D$	Number of hosts covered by malicious VMs
$d$	Number of VMs threated by malicious VMs

.

To support live migration of VMs, it is necessary to determine whether $V{M}_i$ needs to be migrated to host $j$ at each decision time point, and this decision variable is also used in the calculation of the load-balancing metric. The decision variable is as follows.

$$m_{ijt}=\{\begin{array}{c}1 \mathrm{i}\mathrm{f} V{M}_i \mathrm{m}\mathrm{i}\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{s} \mathrm{t}\mathrm{o} \mathrm{h}\mathrm{o}\mathrm{s}\mathrm{t} j \mathrm{a}\mathrm{t} \mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e} t\\ 0 \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}$$

(1)

Balance—For server administrators, it is desirable to balance cloud computing, i.e., the proportions of computing or memory resource usages of hosts that are close to each other. A scale statistic can describe the balance of resource utilization, and in this paper, we choose one of the commonly used scale statistics, the sample standard deviation. The sample is the CPU/memory usage of each host $j$ at time $t$. Hence, we define Balance as the arithmetic mean of resource utilization sample standard deviations represented as a percentage.

$${\overline{\sigma }}_{CPU}=\frac{ {{\displaystyle \sum }}_{t=1}^T{\sigma }_{CP{U}_t}}{T}, {\sigma }_{CP{U}_t}=\sqrt{\frac{1}{m-1}{{\displaystyle \sum }}_{j=1}^m{\left(x_{jt}^{\%}-\overline{x_t^{\%}}\right)}^2}$$

(2)

$${\overline{\sigma }}_{memory}=\frac{ {{\displaystyle \sum }}_{t=1}^T{\sigma }_{memor{y}_t}}{T},{ \sigma }_{memor{y}_t}=\sqrt{\frac{1}{m-1}{{\displaystyle \sum }}_{j=1}^m{\left(y_{jt}^{\%}-\overline{y_t^{\%}}\right)}^2}$$

(3)

$$Balance={\overline{\sigma }}_{CPU}+{\overline{\sigma }}_{memory}$$

(4)

Coverage—One criterion to measure the security performance is Coverage, the rate of hosts attacked, which equals the number of targeted hosts co-located with malicious VMs, divided by the number of total hosts $m$.

$$Coverage=D/m$$

(5)

CVR—Another criterion, CVR (Co-located VMs Rate), is the percentage of VMs next to malicious VMs. A VM belonging to a malicious user undermines the server’s security and the security of all VMs operating on the same server.

$$CVR=d/n$$

(6)

We use the following example to illustrate the definition of Coverage and CVR. As seen in Figure 1, a legal user L launches four VMs (VM_L1, VM_L2, VM_L3 and VM_L4), which run on four separate servers (Server 1, Server 2, Server 3 and Server 4). Attacker A then launches eight VMs (VM_A1, VM_A2, …, VM_A8), four of which co-locate with three VMs of L. The Coverage is 7/8, and the CVR is 3/12.

3.2. Problem Statement

Our research objective is to develop policies for VM live migration that address both load-balancing and security concerns in a cloud computing environment. To achieve this, we consider workload balance and security risk as the key factors. Workload balance is determined by the CPU and memory utilization equilibrium across hosts. Equation (4) is used to calculate the metric. However, assessing the security performance of the system is not straightforward, as it is difficult to determine whether a VM is legitimate or malicious. To address this challenge, we assign security levels to each user based on behavior-based algorithms that detect co-resident attacks. VMs belonging to users with similar security levels are placed on hosts of matching security levels, which prevents low-security-level VMs from launching co-resident attacks on high-security-level VMs.

Cloud computing is evaluated at discrete periods $t=1, 2, \dots , \mathrm{T}$, which are uniformly distributed over the policy assessment horizon T. The main challenge is to develop policies that balance workload across the hosts while minimizing VM migrations.

4. Agent-Based VM Migration Architecture

The present study describes an agent-based architecture designed for VM application, allocation and migration, as illustrated in Figure 2.

The architecture comprises three kinds of agents: Balancer Agents (BA), Server Agents (SAs), and Virtual Machine Agents (VMAs).

The BA is responsible for forwarding VM allocation requests from users to the SAs according to a predefined set of VM allocation policies, as detailed in Section 5.5. Whenever the BA receives a VM application request, it broadcasts a request to the SAs, with the goal of identifying the optimal server for hosting the VM, based on a VM migration heuristic. If no host has sufficient computational power and matching security level, the BA recommends retaining the VM, as described in Section 5.1. The BA is intended for deployment on the balancer servers of cloud computing facilities.

The SAs use server-centric migration policies, including workload thresholds and security level mismatch, to initiate VM migrations, with the objective of optimizing the allocation of resources and security risk mitigation. The SAs use migration heuristics to identify the optimal destination host and the most appropriate VM to migrate, as outlined in Section 5.1 and Section 5.4. Additionally, SAs are equipped with VM acceptance policies to determine whether to accept VMs that are being migrated by other SAs, as elaborated in Section 5.3. When an SA is unable to migrate a VM due to conflicts with the VM acceptance policies of other SAs, the SA waits for a predefined interval before reinitiating the proposal. SAs are also responsive to VM allocation requests from the BA. The SAs are deployed on each host.

The VMAs are responsible for monitoring the resource utilization of VMs and transmitting monitoring reports to the SAs, which manage the physical hosts. The SAs coordinate and collaborate to distribute workloads evenly across a heterogeneous set of hosts using a multi-agent protocol for VM migration, as detailed in Section 5.1.

5. Agent-Based VM Migration

5.1. Agent-Based VM Migration for Load-Balancing and Co-Resident Attacks (ALBA)

ALBA is a mechanism with two distinct roles: initiator and participant. The initiator role is triggered by an SA whose resource usage surpasses the threshold or by an SA with a VM that does not match the security level, which then initiates a VM migration. The SA broadcasts a call for proposals to participant SAs within the same cloud computing unit (Figure 3), containing the VM’s description that the initiator is attempting to migrate. The participant SAs which have the same security level as the VM’s description respond with a proposal message, including their current resource utilization statistics, indicating that they can host the VM without breaching their migration thresholds or security level.

The initiator SA analyzes the received proposed messages from the participating SAs with specified algorithm. The resource utilization data are reviewed. If the initiator SA employs only the CPU-based migration threshold, the participant SA with the highest idle CPU resource is chosen. Similarly, if the initiator SA relies solely on the memory-based migration threshold, the participant SA with the highest memory resource is chosen. Furthermore, if the initiator SA employs both the CPU-based and memory-based migration criteria simultaneously, the participant SA with the highest CPU and memory resource is selected.

Once the most suitable participant SA is selected, the initiator SA migrates the VM to the chosen participant SA (Figure 3). In the event that the initiator SA receives no accept proposals, i.e., no participant SA has sufficient CPU and memory resources, or a matching security level, the initiator SA waits for a defined time interval and then resends the proposal to the participants (Figure 3).

5.2. VM Migration Policies

The VM migration policy is used to decide when to initiate a VM migration from one Server Agent (SA) to another. This policy comprises two essential components: a migration threshold and an activation threshold.

The migration threshold indicates that the resource utilization level is about to degrade the performance of the host or that a VM does not match the security level of the current host. SAs employ VM migration thresholds to initiate VM migrations. SAs are provided with both resource-based and security-based migration thresholds and can activate either one or both simultaneously. For instance, a host may face a considerable performance decline when the CPU resource is relatively exhausted while there is still idle memory. Consequently, its SA uses the migration threshold of CPUs to trigger VM migrations.

The activation threshold prevents unnecessary VM migrations. That is, peaks in resource usage that last only for a short period of time. It indicates the maximum time that the resource usage surpasses the migration thresholds (based on the host monitoring rate) before attempting to initiate VM migrations, as in [27].

Both thresholds are decided by the administrator based on the resources of each host. Hosts with more computing resources will of course have higher resource-based thresholds, but the security-based threshold is only related to the matching of the security level of the host and the VM.

5.3. VM Acceptance Policies

A VM acceptance policy defines the criteria for accepting VMs through live migration from other overloaded SAs based on their resource usage levels or from other security-level-violated SAs based on the security level mismatching with a certain VM.

The policy comprises three elements: (i) idle computing resources including CPU and memory; (ii) after accepting the migrated VM, whether the resource utilization level reaches the specified threshold; and (iii) a security level. Both resource-based and security-based acceptance thresholds are provided to enable SAs to accept or reject the migration proposal.

The idle computing resources and resource utilization level of a VM acceptance policy outlines the remaining resources of the host and the resource utilization of the host after the VM occupies its required resources. For example, if a VM with a high CPU resource requirement is to be accepted, the host must not only have sufficient idle CPU resources, but after accepting the VM, its resource utilization level should not exceed the source host of the VM. The resource utilization levels must be evaluated by the cloud computing administrator as part of this project.

To enhance the security of cloud computing, the VM acceptance policy incorporates a security level. Specifically, when a host receives a VM, it will only accept VMs with a matching security level. The security level of the VM must be the same as the security level of the host, irrespective of whether the VM’s security level is higher or lower than that of the host. This policy is designed to prevent unauthorized access or data breaches resulting from a mismatch in security levels between the host and the VM. By adopting this security matching level, cloud computing can better protect its resources and data from security risks.

5.4. VM Migration Heuristics

The VM migration heuristic also serves to reduce the load of overloaded SAs and transfer VMs whose security levels do not match. For the former, it assists overloaded SAs in selecting VMs to migrate to underloaded SAs by ranking VMs according to their resource usage. When VM migration thresholds, as well as the activation threshold, are exceeded, an SA employs migration heuristics to select VMs for migration. For the latter, the security matching level serves as a “one-vote veto.” This means that if the security level of a VM is not equivalent to that of the host, it must be mitigated to the host.

SAs have access to resource-based migration algorithms that prioritize VMs based on CPU and memory utilization data. The migration heuristic considers the ranking of CPU and memory usage, and the security level.

An SA employing a resource-based migration heuristic selects the VM to be migrated using the following process (Algorithm 1).

Algorithm 1: Resource-based migration heuristic

Input: N (Vector of VMs), VM CPU usage, VM memory usage Output: VM for migration (Nk) 1. Sorting vector N based on ascending VM CPU usage 2. $N_{CP{U}_{RankOrder}}^k$ = $k$, i.e., VM $k$ has a CPU weight of $k$, VMs with higher CPU usage have higher CPU weight 3. Sorting vector M based on ascending VM memory usage 4. $N_{ME{M}_{RankOrder}}^k$ = $k$, i.e., VM $k$ has a memory weight of $k$, VMs with higher memory usage have higher memory weight 5. $\mathrm{Let} N_{OverallRankOrder}^k=N_{CP{U}_{RankOrder}}^k+N_{ME{M}_{RankOrder}}^k$ 6. Selecting the VM $N^k$ with the highest value of $N_{OverallRankOrder}^k$ (ties are broken randomly)

To select a VM to be migrated, an SA ranks VMs by CPU consumption in ascending order (Algorithm 1, step 1) and assigns a weight to each VM based on its CPU rank order (Algorithm 1, step 2). The SA then ranks the VMs by memory consumption in ascending order (Algorithm 1, step 3) and assigns a weight to each VM based on its memory rank order (Algorithm 1, step 4). The SA calculates the total rank order by adding the CPU and memory rank orders (Algorithm 1, step 5) and selects the VM with the highest total rank order (Algorithm 1, step 6).

An SA that employs a security-based migration heuristic selects the VM to be migrated using a much simpler process. Once the SA detects a mismatch of VM security levels, it starts the migration heuristic, and selects all the VMs with the mismatch security levels.

5.5. Allocating Heuristics of BA

The allocating heuristic also serves to balance the load of SAs and match the security level of the VMs to that of the host.

For load balancing, the hosts in cloud computing are regarded as a host resource pool, and four commonly used VM migration algorithms [61,62,63,64] were implemented into BA to facilitate the selection of initial destination hosts for newly arriving VMs. These algorithms include:

Round-robin—this algorithm evenly distributes newly acquired VMs by assigning each VM to each host in turn, and this continues in cycle. For an illustration, refer to [64].

Greedy—this algorithm assigns a newly arrived VM to the host with the lowest sum of CPU and memory utilization percentages. For an illustration, refer to [61].

First-fit—this algorithm assigns an incoming VM to the first host in the pool with sufficient idle resources to match the VM’s specifications. For an illustration, refer to [62].

Best-fit—this algorithm assigns an incoming VM to the host with the smallest number of idle cores. Given that virtualized hosts can allocate significantly more (virtual) memory than their available physical memory to VMs [35], the number of accessible cores was chosen over the quantity of free memory. For an example, see [63].

For security, after the host is selected according to the migration algorithm, it is also checked whether the security levels of the host and the VM match. If there is a match, migrate to the selected host; otherwise skip the host and continue the selection. If there is no suitable host, BA reserves the VM request for a period of time and re-initiates the application.

6. Evaluation and Empirical Results

To assess the performance of ALBA, we conducted three sets of experiments that were implemented using the NetLogo platform [65], where each SA controls a turtle (representing a host) and VM migration is simulated through linking and delinking VM turtles (representing a VM) with SA turtles. All experiments were run on a computer with an 11th Gen Intel(R) Core (TM) i7-1165G7 processor operating at 2.80 GHz, with 16 GB of RAM and a 64-bit Windows 10 operating system.

6.1. Agent-Based VM Migration (ALBA) versus Centralized VM Migration

6.1.1. Objective

The objective of this experiment was to evaluate and compare the VM migration and security performance of ALBA with a centralized VM migration approach, as well as to analyze the advantages and disadvantages of distributed VM migration compared to the centralized approach in different aspects.

6.1.2. Experimental Settings

To validate the empirical results, we created a centralized VM migration strategy, which is commonly used in the literature [61,62,63,64]. The central manager was equipped with the same four load-balancing strategies as frequently employed in Bas (see Section 5.5). Notably, the commonly used benchmark heuristics do not utilize live migration of VMs.

The experiment included three sets of parameters: (i) cloud computing parameters, (ii) load-related parameters (

Table 2. Load-related parameter for experiments.

Parameter	Alternative Values
Number of VMs	300
Specifications of VMs
Number of cores	{1, 2, 4, 6, 8}
Memory sizes	{4, 8, 12, 16, 20} GBs
User number	{1, 2, 3, …, 28, 29, 30}
Initial low-security users	{1, 2}
Low-security user detected in operation	{3}
User security level	{Low, High}
Mean inter-arrival time of VMs	5 ticks
Mean inter-departure time of VMs	100 ticks

), and (iii) the agent-based testbed settings (

Table 3. Agent-based testbed parameters for experiments.

Parameter	Experiment 6.1	Experiment 6.2
BA’s VM migration heuristics	{Round-robin, greedy, first-fit, best-fit} heuristics	{Round-robin, greedy, first-fit, best-fit} heuristics
SAs’ proposal selection mechanism	Greedy selection	Greedy selection
SAs’ CPU-based acceptance policy	Above threshold	Above threshold
SAs’ memory-based acceptance policy	Above threshold	Above threshold
SAs’ security-based acceptance policy	Level Matching
SAs’ CPU acceptance threshold	0%	0%
SAs’ memory acceptance threshold	0%	0%
SAs’ CPU-based migration threshold	50%	50%
SAs’ memory-based migration threshold	50%	50%
SMAs’ activation threshold	10 ticks	10 ticks
SMAs’ monitoring frequency	1 tick	1 tick

).

1.

Cloud Computing Parameter

The simulated cloud computing environment consisted of 20 hosts, with 6 hosts having 20 cores and 48 GB memory, 8 hosts having 40 cores and 96 GB memory, and 6 hosts having 60 cores and 144 GB memory. This resulted in a heterogeneous computing environment with an arbitrary distribution of relatively weak hosts, regular hosts, and relatively powerful hosts. The host requirements were derived from [66]. Furthermore, 3 hosts (1 for each type of host) were assigned a low security level to host VMs requested by users with a low security level. To accommodate the load specifications provided in Table 2, the number of hosts simulated by Sas was set to 20.

2.

Load-Related Parameters

The load-related parameters in Table 2 include various parameters, such as the number of VMs, VM specifications, user security levels, and the mean inter-arrival and mean inter-departure times of VMs. To investigate the efficiency of ALBA with a larger number of VMs than what can be instantiated simultaneously using existing open-source frameworks for cloud computing, such as OpenNebula (which can instantiate up to 148 VMs simultaneously, as reported in [67]), we set the number of VMs to 300. To generate a heterogeneous load, we arbitrarily selected the number of logical cores and memory capacity from the possible values presented in Table 1.

To mitigate co-resident attacks via VM migration, we added a security-level attribute to each user requesting a VM. If an SA identifies a VM performing anomalous tasks during host operation, the user who applied for the VM may be identified as the attacker, and all VMs requested by the same user may be considered attack tools. Consequently, the security level of all VMs requested by this user should be reduced.

We set the mean inter-arrival times of VMs to 5 ticks and mean inter-departure times of VMs to 100 ticks, as determined experimentally by overloading cloud computing to its maximum capacity for hosting VMs. This approach forces Sas to migrate VMs and results in VMs with a relatively long lifetime that may migrate multiple times.

3.

Agent-Based Testbed Settings

The parameters for the experiment are presented in Table 2, which includes the VM migration heuristics of the BA and the parameters of the Sas. These parameters consist of VM migration heuristics, VM acceptance policies, VM migration thresholds, an activation threshold, and a monitoring frequency.

The resource-based acceptance policies were set to 0%, which means that Sas would react to all proposals from other Sas seeking to move VMs provided that sufficient computing resources were idle. Some benchmark VM migration strategies, such as the greedy and first-fit heuristics, consider both CPU and memory utilization. To facilitate a comparison, Sas were equipped with a combined migration heuristic, which considers both CPU and memory usage. This heuristic was chosen due to its ability to achieve relatively good load balancing while achieving fewer VM migrations.

To trigger VM migrations based on CPU or memory use, the resource-based migration criteria were set to 50%. Additionally, the activation threshold of Sas was set to 10 ticks, which refers to the length of time that Sas are permitted to exceed their migration criteria for computational resource usage.

To ensure robustness and reliability of the simulation results, 10 different experiments were performed for each configuration of the testbed, resulting in a total of 90 simulations with the instantiation of 27,000 VMAs.

6.1.3. Performance Indicator

In order to evaluate the effectiveness of agents utilizing decentralized VM migration in comparison to centralized VM migration, the performance metric Balance was utilized and calculated using Equation (4). Since Sas and the central manager are required to balance VM load on the hosts, a smaller value for Balance would indicate more evenly distributed cloud computing.

6.1.4. Results and Analyses

The results of the experiments are presented in Figure 4 and Figure 5. In Figure 4, the horizontal axis shows a combination of the distributed approach (ALBA) and central approach using different VM migration algorithms, while the vertical axis represents the Balance metric. Based on the results, three main conclusions can be drawn.

4.

Analysis 1

The use of SAs equipped with ALBA results in more evenly distributed load among heterogeneous hosts compared to central managers with frequently used VM migration algorithms. As shown in Figure 4, SAs with ALBA demonstrated lower Balance indicators compared to central managers. The central managers allocate newly arrived VMs to hosts based on the hosts’ current resource usage and the VMs’ initial resource requirements, whereas SAs with ALBA distribute loads through dynamic migration of VMs. Moreover, overloaded SAs can relocate VMs autonomously to other SAs that respond to the call for migration proposals. By working together, SAs without global knowledge of cloud computing can dynamically and autonomously balance loads across hosts, outperforming central managers using frequently used VM migration algorithms and complete knowledge of cloud computing.

5.

Analysis 2

As shown in Figure 4, the best (lowest) two Balance indicators were found for ALBA with the round-robin heuristic (8th column) and the greedy heuristic (6th column). BAs assign incoming VMs evenly (using a round-robin approach) among hosts, leading to a well-balanced load. Hence, SAs can benefit from BAs’ initial allocation of VMs. As described in Analysis 1, SAs can balance loads more evenly than central balancers by considering the dynamic fluctuations of VMs’ resource usage. ALBA combines primary allocation of VMs (through BAs) and secondary migration (through the cooperation of SAs), allowing for optimal VM migration in cloud computing.

6.

Analysis 3

As shown in Figure 5, the central manager achieved a higher Coverage and CVR than ALBA, indicating that it is inferior to ALBA equipped with live migration in terms of security performance. It is likely that the user’s reputation, i.e., security level, is not always known when cloud computing receives the user’s application for a VM. The experimental design specifies that the security level of some users is known at the beginning of the experiment, while the security level of other users needs to be updated only after dangerous behaviors are detected in the VMs they own. Both ALBA and Central benefit from the initial allocation of VMs. However, the central manager does not support live migration, which results in VMs with dangerous behavior found in operation, and other VMs of the users they belong to, not being migrated to hosts with matching security levels, increasing the risk of co-resident attacks.

6.2. ALBA versus Agent-Based VM Migration without Considering CRA (MapLoad)

6.2.1. Objectives

The objective of this experiment is to compare the effectiveness of the ALBA load balancing method with MapLoad, an Agent-based VM migration method that disregards co-resident attacks (CRA). The first aim of the experiment is to determine whether ALBA’s consideration of CRA successfully mitigates the security risk of cloud computing. The second aim is to assess whether the additional costs of implementing CRA mitigation in ALBA are justifiable.

6.2.2. Experimental Settings

To compare and validate the empirical results, we used MapLoad [66] as the reference method. MapLoad is an Agent-based VM migration method that considers both CPU and memory usage, and employs the same four frequently used load-balancing strategies of BAs (as discussed in Section 5.5).

The testbed was designed with three different sets of parameters: (i) cloud computing parameters, (ii) load-related parameters (Table 1), and (iii) agent-based testbed settings (Table 2).

6.2.3. Performance Indicators

The effectiveness of ALBA and MapLoad were evaluated using the performance metric Balance, calculated as shown in Equation (4). A lower Balance value indicates a more evenly distributed computing resource usage across (possibly heterogeneous) hosts.

To compare the efficiency of ALBA’s and MapLoad’s CRA mitigation mechanisms, we used the security measures of Coverage and CVR. These measures were calculated using Equations (5) and (6) respectively.

6.2.4. Results and Analyses

The experimental results are presented in Figure 6. Two conclusions can be drawn from the findings.

7.

Analysis 1

As illustrated in Figure 6, MapLoad achieved a lower balance (1st, 3rd, 5th, and 7th columns) than ALBA using the four commonly used VM migration heuristics (2nd, 4th, 6th, and 8th columns). MapLoad distributes the load more evenly across heterogeneous hosts because it does not consider CRA, allowing low-load hosts to be fully utilized. Conversely, ALBA restricts the migration of load from high-load hosts to other hosts, even if the latter have low CPU and memory resource usage, because it cannot accept VMs that do not match the security level of the host.

8.

Analysis 2

As shown in Figure 7, MapLoad achieved significantly higher coverage and CVR than ALBA with CRA mitigation mechanisms, indicating that ALBA is inferior to MapLoad in terms of security performance. ALBA distributes VMs of the same security level on matching hosts, confining high-risk users to specific hosts and significantly reducing the risk of CRA across cloud computing. Furthermore, by equipping the SAs with ALBA, the user’s security level can be reduced after identifying risky VM activity, and this information can be broadcast to other SAs to adjust their migration and acceptance policies for VMs from the respective users. All SAs can make migration decisions based on the security level of the user to which the VM belongs and migrate VMs that do not match that host, avoiding the loss of high-risk VMs to ordinary users’ VMs. This further reduces CRA risk compared to considering the security level of the VM only at the time of VM assignment.

7. Conclusions and Future Work

This work presents a novel agent-based load-balancing approach (ALBA) that dynamically balances loads while accounting for host heterogeneity, VM resource utilization heterogeneity, and co-resident attacks. ALBA enables agents to balance loads in a distributed form while mitigating security risks from co-resident attacks by utilizing VM live migration. The empirical evaluation in Section 6 shows that agents without global knowledge of cloud computing can balance loads, outperforming central managers with complete knowledge. Furthermore, agents adopting ALBA exhibited higher security performance (in terms of Coverage and CVR) than simplified ALBA while achieving slightly lower load-balancing performance, which is an acceptable trade-off to improve the security of the cloud computing.

This research contributes an Agent-based VM migration architecture that combines load balancing with co-resident attack mitigation and allows for the trade-off between load balancing and co-resident attack mitigation. It also presents a novel agent-based load-balancing mechanism (ALBA) that can balance workloads across hosts cooperatively, considering heterogeneity in both CPU and memory, as well as security levels. Additionally, this work enables the Agents to decide the initial allocation of VMs, the timing of VM migration, the migration destination and whether to accept the migration requests.

Future research will focus on other issues that can be solved with VM migration, such as zero-day exploit and physical security (such as theft or unauthorized access to a physical host; live VM migration can be used to move VMs to other secure hosts). While this work demonstrates the desirable properties of an Agent-based VM migration approach (e.g., load balancing, security), more security and reliability issues that can be solved by VM migration can be studied in the same framework, and these issues can be solved eventually through comprehensive trade-offs.