Energy-Efficient Resource Provisioning Using Adaptive Harmony Search Algorithm for Compute-Intensive Workloads with Load Balancing in Datacenters

Abstract

Drastic variations in high-performance computing workloads lead to the commencement of large number of datacenters. To revolutionize themselves as green datacenters, these data centers are assured to reduce their energy consumption without compromising the performance. The energy consumption of the processor is considered as an important metric for power reduction in servers as it accounts to 60% of the total power consumption. In this research work, a power-aware algorithm (PA) and an adaptive harmony search algorithm (AHSA) are proposed for the placement of reserved virtual machines in the datacenters to reduce the power consumption of servers. Modification of the standard harmony search algorithm is inevitable to suit this specific problem with varying global search space in each allocation interval. A task distribution algorithm is also proposed to distribute and balance the workload among the servers to evade over-utilization of servers which is unique of its kind against traditional virtual machine consolidation approaches that intend to restrain the number of powered on servers to the minimum as possible. Different policies for overload host selection and virtual machine selection are discussed for load balancing. The observations endorse that the AHSA outperforms, and yields better results towards the objective than, the PA algorithm and the existing counterparts.

1. Introduction

The evolution of cloud computing paves the way for realizing the long-held dream of the deliverance of computing resources as a utility to users. The applications, development environments and infrastructure services are delivered to the users through large and small datacenters [1]. However, the datacenters persistently suffer from their ever-increasing energy consumption and CO₂ emissions. Besides this, the money spent on energy consumption by a datacenter plays a crucial role in deciding the cloud provider’s service cost [2]. According to the natural resources defense council (NRDC) report [3], U.S. datacenters’ energy consumption is expected to reach 140 billion kilowatt-hours of electricity in the current year of 2020, which is equivalent to 50 large coal-based power plants emitting nearly 150 million tons of carbon dioxide (CO₂). Further, the report emphasizes that the best energy efficiency practices and policies could reduce electricity use by 40%. Due to this, energy consumption reduction has become a crucial design parameter in modern datacenters [4]. Interestingly, the primary source of energy consumption in a datacenter is its enterprise servers, which consume more than 60% of the overall energy [5]. At the granular level, the power consumption of enterprise servers can be categorized into static and dynamic: static power consumption is due to powered-on servers with no circuit activity, whereas dynamic power consumption is due to the switching of transistors during workload execution. On average, 30% of the existing servers are comatose, meaning that they are idle and contributing to static power consumption [6]. Further, more than 80% of the information technology (IT) budget is spent to keep them running as an essential measure to ensure availability, which is an overestimated resource requirement. To meet the service level agreement, it is essential to keep all servers running. An idle server consumes approximately two-thirds energy of its 100% utilization at full load [7,8,9]. In addition, it is noteworthy that idle power and dynamic power consumption on a utilization level varies based on the different power models of physical servers. The energy reduction caused by shrinking the number of existing resources by virtual machine (VM)-consolidation may spin-off lower-level resource availability resulting in jeopardizing the goodwill and credibility of the provider. Utilization of the server at a high voltage results in a high temperatures and shorter lifetime. Optimal server utilization provides an optimized power consumption of central processing unit (CPU) resources resulting in the increased lifetime of the resource with a power consumption that is limited to its specification. The computation capacities of the servers have to be realized and optimum utilization of resources should be achieved [10] to reduce idle and active servers’ power consumption. By considering the above-mentioned advantage, in this research work an adaptive harmony search algorithm (AHSA) and power award (PA) algorithm is proposed to achieve minimum power consumption of the servers.

The rest of the paper is organized as follows: Firstly, general facts about the datacenter power consumption are outlined in Section 1. Secondly, several closely associated research works are given in Section 2. Following that, the research problem formulation is given in Section 3. Subsequently, Section 4 elaborates on the algorithms that are considered in this research work. Afterwards, the virtual machine distribution algorithm is explained in Section 5. Then, the experimental set-up for evaluation, results and discussions about the significance of load balancing are discussed in Section 6. Finally, Section 7 concludes the findings of this research work.

2. Related Works

A multi-objective genetic algorithm is proposed for dynamic demand prediction to minimize energy consumption and increase resource utilization [11]. Many heuristic algorithms have been used in the VM placement problem: ant-colony-based VM placement is formulated as a multidimensional bin-packing problem to control resource wastage and energy consumption simultaneously [12]. Group technology-based cuckoo optimization technology is used to control the datacenter’s operating cost, considering task migration, VM creation and energy consumption [13]. An improved practical swarm optimization is used to increase the quality of service with reduced power consumption [14]. A nature-inspired genetic algorithm and bacterial-foraging algorithm are used for efficient task scheduling to reduce energy consumption and to minimize the overall makespan of the task [15]. A modified genetic algorithm is proposed by generating an initial population with the Max–Min approach to get a better makespan [16]. A new heuristic embedded with the ant colony is proposed to reduce energy consumption [17]. The nature of the harmony search algorithm adopts itself to suit both discrete and continuous variable problems [18]. Virtual machine consolidation is performed using harmony memory search by reducing the number of active machines along with the quality of service requirement [19]. Ant-colony-based consolidation of VMs is utilized to minimize energy consumption with acceptable performance levels [20]. The modified practical swarm optimization approach is used to perform virtual machine consolidation with an aim to reduce power reduction using CloudSim [21]. Meta-heuristic approaches are a considerable alternate to obtaining a solution that offers a trade-off between computational cost and the best fitness solution for NP (non-deterministic polynomial-time)-hard problems. There is no single global optimal algorithm that derives the best optimal solution in all optimization schemes [22]. A few optimization algorithms, like ant colony and genetic algorithms, follow the search path of recently generated partial solutions, few others provide a single-point search solution, in which the search path is decided based on the solution generated at each interval. Many meta-heuristic algorithms are population-based, storing a set of feasible solutions in the search path, which paves the way for efficient exploration in global search space.

Metaheuristic algorithms have been identified as efficient tools to solve engineering design problems [23]. Harmony Search Algorithm (HSA) has been used in the majority of applications, like power and energy management, distributed generation and capacitor placement, transmission network planning, job scheduling, neural network training, water pump switching, water distribution network, heat and power economic dispatch, charge scheduling of energy storage systems with renewable power generators, estimation of life cycle cost and carbon dioxide equivalent emission of the buildings [24,25,26,27,28,29,30] and so on. A hybrid approach, coral reefs optimization has been used to tackle the feature selection problem for an extreme learning machine prediction model for short-term wind prediction in [31]. Optimization techniques are widely used in many sustainable energy sources such as building, environment and energy. The finding of much recent scientific research works highlights the growth and popularity of optimization approach towards the reduction in the CO₂, energy consumption and power cost. The potential and strength of meta-heuristic algorithms in exploration and exploitation have made its applicability in sustainable energy optimization problems possible, which is the current need [32]. The increasing demand for the power and environmental challenges of global warming has made renewable energy (solar, wind and hydro) an inevitable substitute of existing power sources to reduce greenhouse gas emissions. The attention and expansion of solar energy have increased due to a decrease in its installation cost. Despite its cost, there is a need to overcome the regulatory, technical barriers to achieving its growth [33].

The most efficient server first greedy task scheduling algorithm considers the energy profile of the servers and allocates the task to the servers with a minimum increment in overall power consumption. The energy reduction is achieved by two factors: (i) by reducing the number of active servers, (ii) by choosing an appropriate server for task placement [34]. The most urgent, CPU-intensive, Bag-of-Task scheduling is performed by introducing a dynamic voltage frequency scaling intelligence in the scheduling algorithm to achieve power reduction [35]. Energy reduction is achieved by executing the processors in minimum frequency level and the quality of service is maintained by executing the tasks within the deadline. The unbalanced resource usage, high energy usage and the number of migrations are handled by task categorization and a resource utilization square model [36]. The weight function is applied to each task based on its resource requirement for task categorization. The individual resource is ranked based on its utilization via the resource utilization square model and the resource famine is reduced by efficient VM consolidation by selecting energy-efficient target servers. The sufficient utilization of green energy, maintaining the required resource demand within the datacenter is achieved by combining the heuristics and statistical approaches. The optimization problem for cost reduction is formulated considering the server and cooling device’s power consumption, aiming at overall net revenue.VM migration is based on the joint optimal planning on two parameters: maximum profit and maximum green energy utilization [37]. Scheduling of workflow considering data file assignment reduces the execution time of the task with reduced overall makespan. The distribution of the tasks and data are considered as a single dependent problem, rather than two independent problems. A hybrid evolutionary algorithm-based task scheduling and data assignment approach outperforms the standard approaches, Min-Min and heterogeneous earliest finish time [38]. VM placement and data placement are considered as a single problem to reduce cross-network traffic and bandwidth usage. Ant colony optimization was used for the selection of physical machine’s (PMs) for the placement of data adjacent to VMs. Data, placed in close proximity to the VM, with the decrease in VM communication distance, resulted in a reduction in job completion time [39]. A queuing structure to handle a large set of VMs and crow-search-based multi-objective optimization was used to reduce resource wastage and power consumption in datacenters [40]. The results were compared against the genetic algorithm and the first fit decreasing approach. The live migration was handled in serial, parallel and improved serial methods. A hybrid krill herd optimization algorithm with an eagle strategy was used for VM expansion during heavy load to maintain the quality of service agreement. The problem of eccentrics and congestion was handled by a new proposed change and response and agreement protocol by monitoring the packet delay, latency and throughput. The experimental results of the hybrid krill herd optimization approach were compared against particle swarm optimization, ant colony optimization, genetic algorithm and simulated annealing [41].

In this article, AHSA, a population-based method suitable for solving continuous and discrete optimization problems, is proposed to reduce the power consumption of the servers. AHSA holds its strength in obtaining a new solution by considering both the probabilistic parameters of all the individual values of the initial harmony and independent variable tuning (stochastic operators). The additional credit is that the rate of convergence and quality of the solution does not depend on initial values, such as a gradient-search. The objective function used in AHSA could satisfy the analytical and differentiable metric.

3. Problem Formulation

Let V = {V₁, V₂, V₃, …, V_N} represent the N number of reserved VMs with operating frequency (F₁, F₂, F₃, …, F_N), cores (C₁, C₂, C₃, …, C_N) and execution time (τ₁, τ₂, …, τ_N). Each V_i where i ε [1, N] can be characterized as a triplet (F_i, C_i, τ_i), thus F_i represents reserved frequency, and C_i represents the number of cores, τ_i represents the execution time reserved for V_i. Let S = {S₁, S₂, S₃, …, S_M} represents M number of heterogeneous servers, each with k number of discrete frequencies (f₀, f₁, f₂, f₃, …, f_k+1) with utilization(U₀, U₁, U₂, …, U_k+1), U₀ = 0%,U_k+1 = 100% and dynamic power (P₀, P₁, P₂ P₃, …, P_k+1). U₀ is idle state with power consumption P₀. Each S_j, where j ∈ [1,M] with utilization (U_j,0, U_j,1, U_j,2, …, U_j,k+1) and power consumption (P_j,0, P_j,1, P_j,2, P_j,3, …, P_j,k+1) can be characterized as a triplet (Utl_j, P_j, TC_j), where Utl_j is the current utilization of server S_j, P_j, is the power consumption of server S_j, where as the TC_j is the total processing capacity of S_j. The relation R between jth physical machine (PM) and ith VM indicates whether VM_i is placed in PM_j, i.e.,

$$\mathrm{R}\left(\mathrm{j},\mathrm{i}\right)=\left\{\begin{array}{cc}1\hfill & \hfill {\mathrm{VM}}_{\mathrm{i}}^{\mathrm{CPU}}\le {\mathrm{PM}}_{\mathrm{j}}^{\mathrm{CPU}} and {\mathrm{VM}}_{\mathrm{i}}^{\mathrm{mem}}\le {\mathrm{PM}}_{\mathrm{j}}^{\mathrm{mem}},{\mathrm{VM}}_{\mathrm{i}}{\mathrm{is} \mathrm{allocated} \mathrm{to} \mathrm{PM}}_{\mathrm{j}}\\ & \hfill \mathrm{i}\in \left[1,\mathrm{N}\right] \mathrm{and} \mathrm{j}\in \left[1,\mathrm{M}\right]\\ 0\hfill & \hfill \mathrm{otherwise}\end{array}\right\}$$

(1)

The Service Level Agreement (SLA) is measured using the Ratio of VM Acceptance (RVA) calculated as

RVA(V) = T(R)/N

(2)

where, N is the total number of VM requests submitted and T(R) is the total number of VM requests accepted and mapped to available PMs. It is derived as

$$T\left(R\right)={\displaystyle \sum }_{j=1}^M{\displaystyle \sum }_{i=1}^{N}R\left(j,i\right)$$

(3)

3.1. Objective Function

The VM requests (ReqQ) are accepted at the beginning of each reservation cycle. The VM request in ReqQ with (F_i, C_i, τ_i) remains constant. τ_i is the total number of reservation cycles reserved by the VM for the resource (F_i × C_i). Utl_j(t) is the current utilization of server j. The power consumption of jth physical machine with utilization l at time t is represented as $P_{j,l}\left(t\right)$ and derived as [42]

$$P_{j,l}\left(t\right)=\frac{(Ut{l}_j\left(t\right))-(U_{j,l})}{(U_{j,l+1})-(U_{j,l})}\times \left(\left(P_{j,l+1}\right)-\left(P_{j,l}\right)\right)+\left(P_{j,l}\right)$$

(4)

where, U_j,l ˂ Utl_j(t) ˂ U_j,l+1, 0 ≤ l < k + 1.

The energy consumption of jth PM within a reservation cycle (rc) can be calculated as

$${\int }_0^{rc}{\displaystyle \sum }_{l=0}^{k+1}{P}_{j,l}\left(t\right)dt$$

(5)

The total energy consumption of all the PMs within a reservation interval [0, rc] can be calculated as

$${\displaystyle \sum }_{j=1}^M{\int }_0^{rc}{\displaystyle \sum }_{l=0}^{k+1}{P}_{j,l}\left(t\right)dt$$

(6)

The energy consumption (E) of all the physical machines in the data center within an interval [0, T] can be partitioned as number of reservation cycles (nrc) segments and it is obtained by

$$E\left(D\right)={\displaystyle \sum }_{j=1}^M{\displaystyle \sum }_{rc=1}^{nrc}{\displaystyle \sum }_{l=0}^{k+1}{\displaystyle \sum }_{i=1}^N{R}_{j,i}\left(t_{rc}\right)\times P_{j,l}\left(t_{rc}\right)\times \left(t_{rc}-t_{rc-1}\right)$$

(7)

The resource allocation problem in datacenters is formulated as

$$G\left(x\right)=\left\{\begin{array}{c}{G}_{energy}\left(x\right)=\min E\left(D\right)\\ s.t\left(9\right)-\left(13\right)\end{array}\right\}$$

(8)

The objective function G(x) is subject to the following constraints:

The total number of VM’s allocated to a machine should not exceed the servers computing (CPU) and memory capacity (mem).

$${\displaystyle \sum }_{i=1}^N{R}_{j,i}^{cpu}\le P{M}_i^{cpu.max}$$

(9)

$${\displaystyle \sum }_{i=1}^N{R}_{j,i}^{mem}\le P{M}_j^{mem.max}$$

(10)

The relation R between VM and PM is many-to one, i.e., R ⊆ N × M if

$$\forall iϵN\&\forall j,kϵM:\left(i,j\right)\in R \wedge \left(i,k\right)\in R\Rightarrow j=k$$

(11)

The total energy (eng) consumed by all the VM’s should not exceed the available brown energy (B) at the datacenter

$${\displaystyle \sum }_{i=1}^N{R}_{j,i}^{eng}\le Total Available B$$

(12)

The total brown energy consumed should not exceed the cloud providers’ agreed upon grid electricity consumption (G)

$$Total available B\le Total assigned G.$$

(13)

3.2. System Model Overview

Figure 1 presents the system model of the proposed work. Each component of the system model is explained below:

• Request Queue (ReqQ): The ReqQ stores the entire reserved VM provisioning request.
• Physical Machine Repository (HostQ): The resource details related to available memory, CPU capacity, current operating frequency, power consumption, percentage of CPU utilization, number of active VMs, state (on/off) about all the PMs are stored in the HostQ.
• Target VM Queue (TargetVMQ): TargetVMQ contains VM details about the server assigned, the percentage of CPU time utilized by the VM, submission time, placement time, remaining execution time, power consumption.
• Management Node: The allocation and reallocation management (ARM) algorithm is a daemon executed in the management node. It activates the scheduling algorithm for VM to PM mapping, resource deallocation algorithm for resource recovery and task distribution algorithm for VM reallocation at specific intervals. It updates TargetVMQ, HostQ and ReqQ.
• Physical Machine Manager (PMM): PMM is a daemon that is executed on each physical machine. It monitors and records the machines functional parameters. It is responsible for updating the HostQ repository with server details.
• Virtual Machine Manager (VMM): VMM is a daemon that is executed on each VM. It is responsible for updating the TargetVMQ with VM functional parameter details.

4. Evaluation of Algorithms

The primary objective of this research work is to propose a PA algorithm and AHSA for VM placement and to obtain an overall power reduction in the servers with the required quality of service.

4.1. ARM-Algorithm

The high-level structural design of the ARM algorithm which is executed in the management node is given in Algorithm 1. This algorithm is partitioned into three sections. Section 1: Lines 2–4 perform VM to PM mapping using the scheduling algorithm; Section 2: Lines 5–6 Perform resource deallocation for every interval, starting from minimum execution time (Min-exe-time). Section 3: Lines 7–9 perform task distribution at regular migration intervals.

Algorithm 1: High-level overview of ARM algorithm approach
Input: Hostlist, VMinstancelist
Output: TargetVMQ
1	For each interval do
2	ReqQ ← Get VMs from VMinstancelist;
3	HostQ ← Get hosts from HostList;
4	TargetVMQ ← Call scheduling algorithm (in Section 4.2, Section 4.3, Section 4.4 and Section 4.5);
5	If interval > min-exe-time then
6	Completedlist ← Get VMs with active time completion from TargetVMQ;
7	For VM in Completedlist do
8	Deallocate resources associated with the VM;
9	If interval == Mig-interval, then
10	TargetVMQ ← Call task distribution algorithm (TDA);
11	Return TargetVMQ.

4.2. Proposed Power-Aware Algorithm (PA)

Algorithm 2 presents the pseudo code for the PA algorithm. In the proposed PA algorithm, the VM is allocated to the server, which showcases the minimum increase in the overall power consumption. The main highlight of the proposed PA algorithm is the inclusion of the outgoing tasks’ minimum remaining execution time to predict the dynamic power in host selection (Line 13–23). Algorithm 2 receives host capacity and the VM’s resource request through HostQ and ReqQ. The feasible hosts (with maximum utilization) for VM placements are identified as SelectedHost and the difference in dynamic power before and after VM placement is calculated in Line 8 of Algorithm 2. The power consumption P2 is not constant throughout the execution of the VM, as it depends on the next incoming and outgoing task of the machine to which it is allocated. The decision is performed based on the new VM execution time and the next outgoing task’s remaining execution time, as explained in Lines 13–23 of Algorithm 2.

Complexity Analysis

The time complexity of the algorithm can be analyzed by considering ‘n’ VM requests in ReqQ and ‘m’ number of hosts in HostQ. Lines 3–4 need to execute O(n) times. Lines 1–24 need to execute O(m) times. The sort functions in Line13 and 20 need to execute O(m log(m)). The complexity of the algorithm is conquered by the number of VM requests n, sort operation and the number of host m. The complexity of the algorithm is estimated as O(nmlog(m)).

Algorithm 2: Power-Aware algorithm (PA)

4.3. FFD Algorithm

In first fit decreasing (FFD), the VM selection is based on the first-fit strategy of the VMs ordered in non-increasing order of CPU utilization and first-fit heuristics are used for host selection. FFD has O(nm) complexity.

4.4. BFD Algorithm

In best fit decreasing (BFD), the host selection is based on the first-fit strategy of the PMs in non-decreasing order of CPU capacity and first-fit heuristics is used for VM selection.

4.5. Adaptive Harmony Search Algorithm

The HSA algorithm is derived from the imitation of musical performance in a search for better harmony. The harmony of any musical instruments is a combination of different sound waves depicting the relationship between them. The HSA performs a stochastic search based on the harmony memory consideration rate and pitch adjustment rate [43]. The HSA was used on continuous variables to obtain an optimum design plane in [44]. HSA optimization parameters have been modified and improved from different viewpoints to suit its applicability in the decision-making process for different problems [45]. Many studies on discrete and continuous problems found a slow convergence on the application of basic HSA for more decision variables due to constant HMCR (harmony memory consideration rate) and PAR (pitch adjustment rate). The AHSA avoids constant values for HMCR and PAR, and dynamically modifies it in the optimization process. Taking advantage of AHSA in this work, we have dynamically changed the HMCR and PAR based on the problem search space. The typical HMCR and PAR values lie in the range 0–1 [18].

4.5.1. Initialization

In this work, the number of discrete decision variables is the number (N) of requests in the ReqQ {V₁, V₂, V₃, …, V_N}. The set of available discrete values (D) for each V_i is initialized, D ∈ {id₁, id₂, … id_M}, where M is the number of machines and id represents the machine identifier. The initial structure of the harmony memory vector is randomly generated and sorted based on objective function values as

$$HMV=\left[\begin{array}{c}{v}_1^1,v_2^1,v_3^1\dots v_N^{1}ob{j}^1\\ v_1^2,v_2^2,v_3^2\dots v_N^{2}ob{j}^2\\ \dots \dots \dots \dots \\ v_1^s,v_2^s,v_3^s\dots v_N^{s}ob{j}^s\end{array}\right].$$

where, v₁ⁱ … v_Nⁱ represents the ith vector of N decision variables. objⁱ represents the obtained objective function based on Equation (8). The number of rows in harmony memory vector (HMV) is represented as s, which is the HMV-size and N represents the number of requests in ReqQ. The HSA parameters are to be initialized for HMV-size, and maximum iterations and bandwidth.

4.5.2. Improvisation of HMV

In this work, the size of the decision variable, i.e., the number of requests in ReqQ, varies in each iteration. The HMCR and PAR values are dynamically identified prior to the improvisation process in each interval based on the available PMs, as in

Table 1. Adaptive harmony search algorithm dynamic parameter values.

Parameter	Value
Max-Iteration	5000
Dynamic-HMV-size	10 × N(Number of requests in ReqQ)
Dynamic-HMCR	Max (0.8, (1 − (Total PMs with full utilization/Total number of PMs)))
Dynamic-PAR	Min (0.35, (1/Total PMs with full utilization))
Bandwidth (BW)	1

.

The HMCR value controls the rate of choosing a value from HMV. The rate of choosing a random value from the HMV is 1-HMCR, presented as:

If (rand₁() < HMCR), then

(14)

New-assignVMi = Oldi

Else

New-assignVMi = New-assign-VMi’

Endif

where Oldⁱ represents a set of $\left\{v_1^i,v_2^i,v_3^i\dots v_N^{i}ob{j}^i\right\}$ in HMV. New-assign-VM^i’ represents a new set of assignments of $\left\{v_1^{i\prime },v_2^{i\prime },v_3^{i\prime }\dots v_N^{i\prime }\right\}$ to obtain a new objective $new\_ob{j}^{i\prime }$. rand₁() is the uniformly generated random number between 0 to 1. With an HMCR value of 0.75, the probability of choosing from HMV is 75% and the probability for the new assignment is 25%. The new assignment generated with 25% probability is updated in HMV if the new objective function value is better than the worst objective function value in HMV. With 75% probability, all the decision variables of randomly chosen ith HMV are pitch-adjusted based on BW. With PAR assumed to be 0.10, and if the uniform random number generated is less than PAR, it means the probability of choosing the value from the neighboring values to the right and left of the chosen value is 10% × HMCR probability. The rate of doing nothing is (1-PAR).

The pitch adjustment on the discrete variable is performed as

If (rand₁() < PAR), then

(15)

New-assignVMi’ = New-assignVMi ± BW

Endif

4.5.3. The Measure of Fitness and HMV update

The new harmony generated in step 2 is checked for feasibility using the constraints given in Equations (9)–(13). If the new harmony found is better than the worst harmony in the HMV, the worst harmony is updated. The HMV is sorted based on objective value. In this work, the non-deterministic nature of the problem may lead to non-convergence of the results, aiming for a high SLA and global optima. The optimal solution may not happen to take into account all the decision variables to satisfy the constraints in higher intervals. In that case, the decision variables are reduced one by one after considering the number of attempts made towards an impossible solution.

Figure 2 displays the AHSA design procedure used to solve the power optimization problem described in Section 4.5.1, Section 4.5.2 and Section 4.5.3. In Figure 2, NHMV(V_i)represents the new harmony memory value of the ith decision variable, r-ind represents the random row index chosen for improving the ith decision variable, and PM-id is the set of physical machine identifiers in which the VM is to be placed.

4.5.4. High-Level Overview of AHSA

Algorithm 3 presents the top-level view of the AHSA approach. Considering there are NVM requests to be placed in M PMs, then at the beginning of each interval, the AHSA is activated.

Algorithm 3: High-level overview of AHSA

Input: HostQ, ReqQ

Output: TargetVMQ

For each interval do

(TargetplanVMQ), ← Call Algorithm 4 with (HostQ, ReqQ);

TargetVMQ ← Actual allocation of VM to HostQ as per TargetplanVMQ;

Return TargetVMQ

Algorithm 4 presents the AHSA for optimal host identification. In Algorithm 4, Line 2 generates an initial HMV population vector. The feasible host of the VM is computed using Equation (1). After the initial population generation, exploration is done from lines 15–21, and the exploitation of existing search space is done from lines 6–14 by using Equation (15). The new vector is generated by identifying the next feasible host. If the new vector yields a better objective, it is updated in HMV in Line 11 and 17. The procedure is detailed in Section 4.5.

Algorithm 4: AHSA proposed for power optimization
Input: HostQ, ReqQ
Output: TargetplanVMQ
1	HMV-size = N × M as in Table 1.
2	HMV ← Call Algorithm 5;
3	For each iteration to Max-iteration do
4	Choose HMCR based on Table 1
5	If rand( ) < HMCR then
6	Choose PAR based on Table 1
7	If rand( ) < PAR then
8	i ← randomly chosen row index of HMV;
9	New-assignVM i’ ← New-assignVM i ± BW using Equations (8) and (15);
10	New-obj1’← Objective function value obtained for New-assignVMi’ using Equation(8);
11	if New obji < HMV [1].obj1’ then
12	H MV[1] · obj1 ← new-obji’;
13	H MV[1] 1 ← New-assignVMi’;
14	else
15	set HMS Size to 1 × M
16	New-assignVM i’ ← Call Algorithm 5;
17	If New_obj1’ < HMV [1].obj1 then
18	H MV [1] · obj1 ← New_obj1’;
19	H MV [1]1 ← New-assignVMI’;
20	Sort the HMV in non-increasing order based on HMV · obj;
21	TargetplanVMQ ← HMV[Last];
22	Return TargetplanVMQ

Algorithm 5: AHSA—Initial population generation
Input: HostQ, ReqQ, HMS-size
Output: HMV
1	For each hm in HMS-Size do
2	[host-id] ← Random selections of discrete values from a set of possible values for ReqQ;
3	If feasible [host-id] based on Equations (1), (9)–(11)
4	HMV [hm].add[host-id];
5	HMV [hm] · obj ← Objective function value corresponding to the HMV using Equation (8);
6	else
7	Go to line 2;
8	Sort the HMV in non-increasing order based on HMV · obj;
9	Return HMV

5. Task Distribution Strategy

In the distribution process, a VM of the same type is created in the destination host while the same type is under execution in the source host. After the complete migration of the tasks from the source to the destination, the source VM resources are released. The number of migration and migration time of the VM chosen for migration has to be reduced, as it involves energy overhead, as the VM to be migrated kept executing at both the machines, and there is a possibility of performance degradation due to migration time. Half of the total available bandwidth is used for migration. The migration time involves the communication overhead drawn in by the memory reserved by the VM.

5.1. Overload Server Detection Methods

Two adaptive thresh hold methods of overload detection are used. A robust statistic is a needed alternative to standard classical methods [46].

5.1.1. Median Absolute Deviation

Median Absolute Deviation (MAD) is a statistical measure that is more flexible than the sample variance and standard deviation in handling outliers in the sample data. The utilization threshold is determined as

U_TH = 1 − x.MAD

(16)

where x ∈ R⁺ that decides the persuasive nature of consolidation.

5.1.2. Inter Quartile Range

Inter Quartile Range (IQR) measures the dataset’s statistic dispersion based on third and first quartiles. The quartiles’ breakdown point encloses 25%. The utilization threshold is determined as

U_TH = 1 − x.IQR

(17)

where x ∈ R⁺ that decides the persuasive nature of consolidation.

5.1.3. Static Threshold

In this work, we assume that, to balance the load in all the machines with 65% utilization, the static threshold is assumed as 65 %.

5.2. Virtual Machine Selection Methods

The virtual machine selection for migration is performed using the following two methods.

5.2.1. Minimum Migration Method

VM with higher CPU utilization is considered for migration. It selects the minimum number of VMs to be migrated. Let V_i be the set of VMs running in ith PM, and U_i is the overall utilization of the set greater than the threshold (U_TH), and let P(V_i) be the power set of V_i, then minimum migration method (MMM) finds a subset ϵ P(V_i)using

$$SϵP\left(V_i\right),U_i-{\displaystyle \sum }_{v_iϵS}U\left(v_i\right)<threshold$$

(18)

where, $U\left(v_i\right)$ is the utilization of a VM $v_i$. The $v_i$ ϵ S chosen for migration is subject to the following constraint. $v_i$ should satisfy both Condition 1 and Condition 2 to maintain the utilization of PM close below the threshold (U_TH).

Condition 1: The utilization of $U(v_i)$ should be higher than |Ui −UTH|;

Condition 2: If condition 1(Con1) is true, then the new utilization $|(U_i-U(v_i))-U_{TH}|$ should be minimal among the other VMs running in ith PM.

5.2.2. Maximum Migration with Least Resource Request

The VM with the minimum migration time and a remaining execution time greater than the migration time is considered for migration.

The half of the total available bandwidth (bw) is used for migration. The migration time involves the communication overhead drawn in by the memory reserved by the VM. The communication over head is calculated as

Mbw = bw/2, MT = (Mem-size/Mbw)

(19)

where migration bandwidth (Mbw) and migration time (MT) depends on the memory size (Mem-size) of the VM to be migrated.

5.3. Task Distribution Algorithm

The task distribution presented in Algorithm 6 is activated at regular intervals. The SourceHost is the host from HostQ with a power consumption higher than the threshold and DestHost are the hosts from HostQ with a power consumption less than the threshold. The VMs are selected from the Sourcehost based on any one of the VM selection methods and the destination host is identified by considering the power consumption constraint in Line 13. The algorithm complexity for m number of hosts and t number of tasks considered for migration is (O(mtm) + O(mlog(m)). The final complexity of the algorithm is O(m²t).

6. Experimental Evaluation

Considering the expense and time incurred in the evaluation of large-scale experiments in real-time, Matlab is used to simulate the environment. Each reservation cycle is assumed to have a duration of 300 s (rc = 300 s). An input request is accepted at the beginning of each reservation cycle.

The VM’s resource requirements are assumed constant throughout the interval. A single datacenter with heterogeneous systems capable of provisioning multiple VMs is considered. As the size of the requested virtual resource is not known, with no limitation for VM request, only the active state execution time of the VM is considered. As the CPU is considered to consume a significant fraction of energy, the tasks are considered CPU-intensive. In this work, CPU utilization and power consumption are considered linearly proportional. All the machines are assumed idle when not in use consuming static power. The datacenter assumed to be powered only by grid energy source. The peak IT load (only server) power estimation of a datacenter is estimated ≈ as 44 kW for the physical machine specification given in

Table 2. Physical machine characteristics.

Machines	Frequency(MHz)	No of cores	Power model	Memory(GB)
M1	2500	56	1	192
M2	2200	16	2	24
M3	2200	16	3	24
M4	2500	112	4	384
M5	2200	64	5	128
M6	2300	36	6	64

[47]. A datacenter is assumed to have a floor space of ≈ 500 square feet. Total electricity power requirement is calculated as ≈136.8 kW (including cooling load, lighting and others). The CPU power consumption of all the servers should not exceed 23kW [48].

Algorithm 6: Proposed task distribution algorithm (TDA)

6.1. Physical Machine and VM Reservation Modeling

Table 2 and

Table 3. Power (in Watts) setting of physical machines (PMs).

Power Model	Idle	10%	20%	30%	40%	50%	60%	70%	80%	90%	100%
1	48.2	106	129	154	179	205	235	269	306	347	385
2	69.4	103	117	137	159	182	201	218	240	290	315
3	59.5	87.6	98.5	109	120	136	153	177	204	233	261
4	89.8	237	286	333	383	442	511	592	710	839	939
5	84.9	138	154	168	182	197	214	234	253	271	287
6	47.3	92.3	108	123	138	153	172	199	230	260	289

represent the model of physical machines with real power consumption and the configurations of heterogeneous systems. This is taken from the Standard Performance Evaluation Corporation (SPEC) power benchmark [47], which is considered for simulation. A small-scale datacenter with 50 systems is considered for algorithm evaluation. The VM characteristics given in

Table 4. Virtual machine request types.

Name	ECU	Speed (GHz)	Memory (MB)
M11.small	1	1	1740
M1.large	2	4	7680
M1.xlarge	4	8	15,360
M2.xlarge	2	6.5	17,510
M22.xlarge	4	13	35,020
C1.median	2	5	1740

are used to model the virtual machine reservations. It is based on Amazon EC2 compute unit (ECU) inspiration but customized according to physical machine characteristics to avoid the fragmentation of resources. Normally, the storage and network requested by the VM are not confined to the physical machine in which it is placed. Therefore, the storage and network bandwidth do not influence the placement of VM on a particular PM. The actual location of data storage is decided by the provider in a timely manner. In this real scenario, a feasibility check involves placing VM on a particular PM, mainly based on the availability of memory and the CPU clock cycle which has a strong bond with each other in computing node. The arrival time of VM is modeled using Poisson distribution. Lambda is determined by the average number of requests submitted in a reservation cycle. Uniform distribution in the interval (τmin, τmax) is used to model the execution time (τ) of the VM, where τmin and τmax are the minimum and maximum execution times (VM’s reservation cycles) of 10 and 60, respectively.

6.2. Evaluation Metrics and Baseline Approaches

The adaptive harmony search algorithm and power-aware algorithm are compared with first-fit decreasing and best fit decreasing. To show the effectiveness of the approaches, the following metrics are considered.

6.2.1. Overall Power Consumption

Considering overall power consumption as an essential energy efficiency metric to reduce grid energy consumption, the total power consumed by the servers for the accepted VM request is formulated in Equation (7).

6.2.2. The Ratio of VM Acceptance

This is considered as a service level agreement metric. The ratio equals the total number of VM requests accepted pertaining to overall request submitted.

6.3. Evaluation of the Proposed Approach without VM Distribution

6.3.1. The Scenario I: Mapping of Heterogeneous VM Types to Heterogeneous Machines without Distribution Policy

The mapping of VMs to PMs is modeled as a variable size bin-packing problem. For Scenario I, Figure 3 depicts the variation in number of VM requests and CPU demand in the workload. The VM types given in Table 4 are mapped to the machines given in Table 2. Figure 4 displays the RVA% of AHSA, PA, FFD and BFD approaches for 24-h duration. Figure 4 reveals 1–2% increase in RVA% by AHSA than PA, and 3–13% for FFD and BFD. Figure 5 presents the RVA% for the first 12 reservation cycles. The CPU utilization for the first five reservation cycles with 100% RVA for all approaches is 19.75%, 38.74%, 52.12%, 67.35% and 78.25%. The AHSA achieved an overall power reduction (24-h duration) of 2.18% more than FFD, 1.481% more than PA, and 2.24% more than BFD, as shown in Figure 6. Figure 7 presents the power consumption for the acceptance rate indicated in Figure 5. As in Figure 7, the power reduction for 100% acceptance of AHSA with respect to the first four intervals is 1.05% less than PA, 1.78% less than FFD and 1.86% less than BFD.

6.3.2. Scenario II: Mapping of Heterogeneous VM types to Homogeneous Machines without Distribution Policy

In Scenario II, the total number of 50 M4 machines, as in Table 2, is considered for mapping the VM types given in Table 4, which portrays a fixed size bin-packing problem. The RVA% for the 24-h duration is shown in Figure 8. It can be inferred from Figure 8 that the AHSA has obtained 1–4% more RVA% than PA, and approximately 5–22% more than FFD and BFD.

Figure 9 presents the overall power consumption of all the methods for a day. The overall power reduction achieved by AHSA is approximately 1.75% more than PA, BFD and FFD. In Figure 10, the first 14 reservation cycles depict 100% acceptance of all the algorithms, which can be used as a fair measure for comparison of power for the same utilization. AHSA provides approximately 7% more overall power reduction in the first 14 cycles than BFD and FFD, and approximately 1% more than PA for the same utilization.

6.4. Evaluation of the Proposed Approach with VM Distribution

Simulation reservation cycles are grouped into three categories, namely RAC, RBC and RMC, to evaluate the effectiveness of migration. For every eight RAC, three RBC are allowed to create a demand for migration, followed by one RMC. Figure 11 presents the CPU demand considered for VM distribution with all reservation cycle categories.

Scenario III: Mapping of heterogeneous VM types to heterogeneous machines with different distribution policies

In Scenario III, the overload host selection is based on MAD, IQR and static threshold (ST). Similarly, VM selection is based on MMM and maximum migration with least resource request (MML) policy for task distribution. Figure 12a–c presents the number of migrations, RVA% and power consumption of different combinations of host and VM selection policies. Figure 12a–c concludes that IQR-MML has an average of 1% increase in acceptance rate and considerable increase in power reduction than other approaches. IQR-MML based host and VM selection policy has resulted in a power reduction of 0.1–0.4% more than ST-MMM, 0.2–0.5% more than IQR-MMM, 0.3–0.4% more than MAD-MMM, 0.2–0.4% more than MAD-MML, and 0.1–0.4 %more than ST-MML. The placement of VMs without task distribution policy has resulted in increased an power consumption of 0.13% in AHSA, 0.49% in PA, 0.44% in FFD, and 0.38% in BFD than IQR-MML.

In this work, non-parametric Mann–Whitney U/Wilcoxon rank-sum test is used to check whether there is a significant difference in the results obtained. Figure 13 demonstrates the power consumed by different types of VM-placement approaches. After running the test on two samples, AHSA with FFD, AHSA with BFD, PA with FFD, PA with BFD, the p value is less than 0.0001, and therefore it can be concluded that AHSA- and PA-based allocation has a significant difference with other approaches towards energy consumption. Figure 14 demonstrates the power consumed by different types of load balancing approaches. IQR-based host selection and MML-based VM selection have a minimum power consumption compared to methods. The FFD approach has more power consumption in all distribution policy and VM selection approaches. MMM-based VM selection has more power consumption irrespective of host selection, due to migration of more VMs. The MML-based VM selection works better with IQR than in ST- and MAD-based host selection.

7. Conclusions

This research work proposed two approaches, AHSA and PA, to decrease the power consumption of servers in the datacenters. Further, the proposed task distribution approach for load-balancing among the server facilitates maintaining the maximum number of active servers with the lowest attainable server frequency. The overload host detection was based on MAD, IQR and ST. The virtual machine selection for migration was based on MMM and MML. The mapping of VMs to both homogeneous and heterogeneous machines with and without task distribution is analyzed in terms of power reduction and RVA metric. The simultaneous placement of VMs using the AHSA optimization approach yields a better RVA metric with improved power reduction compared to the proposed sequential-based PA allocation and its counterparts FFD and BFD. The simulation results conclude that AHSA-based allocation with IQR-MML migration policy for load balancing provides a better loss reduction for heterogeneous systems with different power models.