Thermal-Aware Virtual Machine Allocation for Heterogeneous Cloud Data Centers

Abstract

In recent years, a large and growing body of literature has addressed the energy-efficient resource management problem in data centers. Due to the fact that cooling costs still remain the major portion of the total data center energy cost, thermal-aware resource management techniques have been employed to make additional energy savings. In this paper, we formulate the problem of minimizing the total energy consumption of a heterogeneous data center (MITEC) as a non-linear integer optimization problem. We consider both computing and cooling energy consumption and provide a thermal-aware Virtual Machine (VM) allocation heuristic based on the genetic algorithm. Experimental results show that, using the proposed formulation, up to $30\%$ energy saving is achieved compared to thermal-aware greedy algorithms and power-aware VM allocation heuristics.

1. Introduction

Cloud computing has emerged as a new and popular computing paradigm for providing on-demand hosted services over the Internet [1,2]. The computing and storage capacity of data centers have increased rapidly as the demand for Internet-based services has increased [3,4,5,6]. The continuous increase of energy consumption and power density is one of the serious side effects of such rapid growth in the data center facilities and imposes a significant cost on Cloud data centers that reduces the marginal profit of Cloud providers and leads to a growth in the rate of carbon dioxide emissions. Thus, energy efficiency has become a major economic and environmental issue in data centers, which has attracted the attention of researchers and industries [7,8,9].

A considerable amount of literature has been published on energy-efficient computing in data centers [10,11]. One of the most significant parts of the energy consumption in data centers appears to be the energy consumed in the cooling system [12,13]. Recent cooling technologies, such as in-rack, in-row, and container-based cooling, have been proposed to improve data centers’ energy efficiency. However, the majority of data centers are still restricted to older cooling technologies, such as chilled-water air cooling [14]. Thus, it seems that there is still room for further improvement in energy efficiency by applying resource management techniques [15,16].

To the best of our knowledge, most works into minimizing cooling energy consumption have proposed optimization algorithms to maximize the temperature of the cooled air supplied by data centers’ Computer Room Air Conditioning (CRAC) unit [17]. The maximum allowable temperature is determined by the lowest red-line temperature of the installed servers in a data center. More efficient cooling processes allow higher temperatures for supplied cooled air without exceeding the red-line temperature in each server for a given workload. These temperature-based optimization methods reduce the recirculating heat in the data center [18]. The recirculation of exiting hot air from devices’ air outlets back into their air inlets increases inlet temperatures and consequently decreases the maximum allowable temperature of the supplied cooled air.

Several studies have parametrized heat recirculation and heat flow fields based on the temperature distribution in the data center [19,20]. The most detailed thermal model of a flow field to predict temperature distribution is a Computational Fluid Dynamics (CFD) model [21]. Similarly, thermal maps of data centers can predict the temperature distribution for a given CPU utilization pattern [22]. However, these models are not suitable for on-line resource management due to their high computational costs. Thermal models with lower complexity, such as Supply Heat Index (SHI), Recirculation Heat Index (RHI) [23], and Heat Recirculation Factor (HRF) [23], have been proposed in the literature.

In addition to the above-mentioned traditional techniques, recent advances in virtualization methodology have made it possible for designers to reduce the idle power costs of modern data centers by improving system manageability [24,25,26]. A Virtual Machine (VM) is a software abstraction of an actual physical computing system which is machine-independent and easily portable [27]. Virtual Machines have many benefits; for example, they keep the users separate from the underling infrastructure and isolate the workloads that share common facilities. A VM benefits its user by improving performance and scalability and reducing hardware and software requirements, as well as manageability and fault tolerance. Moreover, it improves resource utilization, reduces the energy consumption through the consolidation of multiple workloads on fewer servers, and enables live migration [28,29,30]. It can also provide load balancing between servers by enabling Virtual Machine migration to eliminate thermal hotspots in data centers.

In this paper, we formulate the problem of thermal-aware Virtual Machine allocation in a heterogeneous data center as a non-linear integer optimization problem. To the best of our knowledge, our proposed formulation is the first to mathematically formulate the problem of how to allocate Virtual Machines in a thermal-aware manner among the computing nodes in order to minimize the total energy consumption of the data center (the formulated problem is named MITEC). We provide a novel heuristic approach to solve the MITEC problem based on a genetic algorithm. Furthermore, we compare the performance of the proposed heuristic with thermal-aware greedy algorithms and power-aware VM allocation heuristics.

The main contributions of this paper are the following:

• Creating a formal definition of optimal thermal-aware VM allocation by considering both computing and cooling energy consumption and providing a novel heuristic based on a genetic algorithm to obtain a near-optimal solution in less computing time;
• Designing a trade-off between the power-aware consolidation techniques and thermal-aware load balancing approaches to obtain higher energy savings in Cloud data centers;
• An extensive simulation-based evaluation and performance analysis of the proposed algorithm.

The remainder of this paper is organized as follows: we address the related work in Section 2. The general physical description of the data center and the power and the total energy consumption models are described in Section 3. Section 4 presents the thermal-aware VM allocation problem formulation, while a genetic heuristic approach is provided in Section 5. Simulation results are provided in Section 6. Finally, Section 7 concludes the paper.

2. Related Work

Recently, there has been a large volume of published studies carried out on energy-efficient resource management in data centers [7,8,9,11]. One of the most significant discussions aimed at minimizing the total power consumption in data centers is the Dynamic Voltage/Frequency Scaling (DVFS) of servers [31,32,33]. Pakbaznia et al. [34] formulate the total data center power consumption and propose an efficient algorithm which reduces the server power by appropriately allocating tasks to servers and choosing the optimal voltage-frequency level for servers. The algorithm also considers the cooling power and chooses the proper supplied cold air temperature value in order to minimize the cooling cost. Dynamically “right-sizing” the data center [35] by turning off inactive servers is another technique which can be applied to achieve more energy savings. In [36], the authors consider the problem of server provisioning and DVFS control at hosting centers. The paper formulates this problem and presents three main techniques based on steady-state queuing analysis, feedback control theory, and a hybrid between the two.

Due to the fact that a considerable reduction in the cooling energy requirements of data centers can be achieved by thermal-aware resource management, several studies have been conducted to explore different techniques in this area. As explained by [5,37], common designs for cooling systems for data centers result in an inefficient and limiting phenomenon called heat recirculation. Heat recirculation is caused by the mixing of the two hot and cold air flows and prevents ideal cooling efficiency. In a study by HP Labs and Duke University, Sharma et al. [23] introduced two scalar and dimensionless parameters, the Supply Heat Index (SHI) and Recirculation Heat Index (RHI), as the metrics that can be used to evaluate the thermal efficiency of a data center by assessing of heat recirculation.

The study presented in [38] includes a detailed thermo-fluid analysis of a typical data center to resolve temperature distribution for a given steady-state power density. This paper uses steady-state and transient CFD techniques and proposes a method describing the provisioning of data center cooling resources based on CFD simulation. Studies prior to [39] explored efficient methods of provisioning cooling resources for data centers [40,41,42]. However, Moore et al. propose a thermal-aware workload placement algorithm based on the minimization of the cooling energy needed at the facilities level [39]. They first introduce a parameter to assess the heat recirculation phenomenon, called the heat recirculation factor (HRF), which resulted from thermodynamical considerations. Then, they propose Zone-Based Discretization (ZBD) and Minimize-Heat-Recirculation (MinHR) algorithms for workload placement based on the HRF metric to minimize the effects of heat recirculation on cooling inefficiency.

Tang et al. [43] introduce an abstract heat flow model to characterize the heat recirculation phenomenon in data centers. This model can also predict the inlet air temperature of each chassis based on the supplied cooled air temperature and workload distribution faster than CFD simulations and is fairly accurate. The main idea in this work is profiling a matrix, called the cross interference matrix, using the measurement or prediction of the inlet and outlet temperature of each chassis by on-board and ambient sensors or via CFD simulations. In [44], Tang et al. developed a new, low-complexity, linear heat recirculation model based on the cross interference matrix introduced in [43] to minimize the peak inlet temperature within a data center. They presented two heuristic methods, called XInt-GA (genetic algorithm) and XInt-SQP (Sequential Quadratic Programming) algorithms, to solve the optimization problem in a reasonable time.

Further optimization strategies have been developed based on abstract heat recirculation models in recent years. In [45], the authors proposed a Power and Thermal Management (PTM) engine which determines the number and placement of ON servers and adjusts the supplied cold air temperature simultaneously. Using this method, the total power consumed by servers and CRACs has been minimized. Abbasi et al. [46] developed a method for a thermal-aware dynamic server set provisioning and introduced a thermal-aware workload distribution algorithm among the active servers.

3. System Model

In this section, we provide the general physical description of the data center and present the power and the total energy consumption models.

3.1. Layout of Data Center

The data center room generally contains several rows of racks; each rack consists of few chassis in which a number of blade servers are assembled. Current data centers are designed with hot/cold aisles, as shown in Figure 1. Each row is placed between a hot and a cold aisle. Chilled air for room cooling is supplied by the CRAC unit through the floor tiles. Each server sucks the chilled air into the rack to cool the servers, and the hot air is blown out by the fans into hot aisles formed by the data center flow pattern. Then, the accumulated hot air is extracted by the CRAC vents on the ceiling. The important duty of the CRAC unit is to keep the inlet temperature of each chassis under the red-line temperature ($T_{red}$), which otherwise would degrade performance or harm the data center facilities.

3.2. Heat Recirculation Phenomena

The air circulation system in a data center provides coolant flow for chassis i at its inlet with temperature $T_{in}^i$ and transfers the exhausted hot air from its outlet with temperature $T_{out}^i$. After extracting server-generated heat, the CRAC supplies chilled air of temperature $T_{sup}$ via floor tiles. The key optimization idea is to isolate the cold aisles as much as possible, reducing heat leakage into coolant cold aisles. Obviously, such leakage increases the differences between $T_{in}^i$ and $T_{sup}$ and consequently decreases the maximum $T_{sup}$ that can be supplied given a certain workload pattern.

In order to isolate the cold aisles, the thermal-aware workload placement formulation can be used to minimize the heat recirculation in a certain flow pattern. In this formulation, the data center can be considered an open system with a cold air feed (temperature $T_{sup}$) that is affected by the chassis, which are heat sources. $T_{in}^i$ and $T_{out}^i$ denote inlet and outlet temperatures of ith chassis. The enthalpy change in each chassis is stated by $K_i(T_{out}^i-T_{in}^i)$ in which $K_i={\dot{m}}_i{c}_P$ where ${\dot{m}}_i$ is mass flow rate of chassis i and $c_P$ is the specific heat capacity of air. A workload distribution is described by vector $\overrightarrow{P}$, which quantifies the power dissipated by servers in each chassis. From $\overrightarrow{P}$, we determine the corresponding inlet and outlet temperatures ${\overrightarrow{T}}_{in}$ and ${\overrightarrow{T}}_{out}$. As explained by [43], the contribution of the flow pattern on heat recirculation can be characterized by a cross interference coefficient (CIC) matrix denoted by $\mathbf{A}$. In fact, $\mathbf{A}$ is a matrix representation of the heating contribution of all chassis outlet flows on each others’ inlet temperatures in a certain flow pattern. Using the CIC matrix, the inlet temperatures of each chassis can be calculated as follows:

$$\begin{array}{c}\hfill {\overrightarrow{T}}_{in}={\overrightarrow{T}}_{sup}+\mathbf{D}\overrightarrow{P}\end{array}$$

(1)

in which the heat distribution matrix, $\mathbf{D}$ is defined by

$$\begin{array}{c}\hfill \mathbf{D}\equiv [{(\mathbf{K}-{\mathbf{A}}^T\mathbf{K})}^{-1}-{\mathbf{K}}^{-1}].\end{array}$$

(2)

where ${\overrightarrow{T}}_{in}$ and ${\overrightarrow{T}}_{sup}$ are the corresponding inlet temperature and the cold air supply vectors. The former contains $T_{in}^i$ for each chassis. The latter is a vector with identical components equal to the cool air temperature supplied by the CRAC unit.

3.3. Preliminaries

3.3.1. Cooling Efficiency of the CRAC Unit

Traditionally, the energy efficiency of a cooling system is characterized by the coefficient of performance (CoP). The CoP is defined as the ratio of the amount of heat that is removed by the cooling system $\left(Q\right)$ to the total amount of energy $\left(E\right)$ that is consumed in the cooling system to remove that amount of heat from the system:

$$\begin{array}{c}\hfill \mathrm{CoP}=\frac{Q}{E}.\end{array}$$

(3)

In the current study, the proposed coefficient is based on a typical water-chilled CRAC unit utilized in an HP Utility data center [39]. In this model, the CoP is a function of $T_{sup}$:

$$\begin{array}{c}\hfill \mathrm{CoP}\left(T_{sup}\right)=(0.0068T_{sup}^2+0.0008T_{sup}+0.458)\end{array}$$

(4)

3.3.2. VM Allocation and Power Model

We consider a large computing facility, consisting of N chassis. Each chassis $i\in \{1,\cdots ,N\}$ contains a set of servers which provide the total computing capacity of $M_{cap}^i$ million instructions per second (MIPS). We denote the VM set as $V=\{V{M}_j(M_{req}^j,L_j)|j=1,\cdots ,M\}$, where jth VM requests $M_{req}^j$ million instructions per second and a task of $L_j$ million instructions (MIs) is executed on jth VM. We consider that each VM executes a single task with a length of $L_j$. Thus, the task execution time is equal to the task instruction length ($L_j$) divided by the VM processing capacity ($M_{req}^j$). The total power consumption of the ith chassis with a CPU utilization of $u_i$ can be calculated as [47]

$$P_{comp}^i={\alpha }_i+{\beta }_i·u_i,$$

(5)

where ${\alpha }_i$ and ${\beta }_i$ represent the idle power consumption of the ith chassis and the extra power consumption from CPU utilization, respectively. The computing energy consumption of ith chassis can be defined as

$$E_{comp}^i={\int }_{t_0}^{t_1}{P}_{comp}^i\left(u_i\left(t\right)\right)dt={\alpha }_i·t_{max}^i+{\beta }_i·\sum _{j\in V_i}(u_{ij}·t_j),$$

(6)

where $t_{max}^i$, $V_i$, $u_{ij}$ and $t_j$ represent the maximum time that ith chassis is active, the set of VMs that are allocated to ith chassis, the utilization of jth VM on ith chassis (i.e., $M_{req}^j/M_{cap}^i$) and the fraction of time that jth VM is active (i.e., $L_j/M_{req}^j$), respectively.

3.3.3. Total Energy Consumption

The total energy consumption of a data center is defined as the sum of chassis energy, $E_{comp}$, and the cooling energy of the CRAC unit, $E_{CRAC}$ [47]. The cooling energy consumption of the CRAC unit can be computed as $E_{CRAC}=E_{comp}/CoP\left(T_{sup}\right)$. Thus, the total energy consumption of a data center can be written as

$$E_{total}=E_{comp}+E_{CRAC}=\left(1+{\displaystyle \frac{1}{CoP\left(T_{sup}\right)}}\right)\sum _{i=1}^N{E}_{comp}^i.$$

(7)

In order to prevent performance degradation or damage to the data center facilities, the CRAC unit must keep the inlet temperature of each chassis under the red-line temperature $T_{red}$. Thus, considering Equation (1) we have

$$\begin{array}{cc}\hfill & {\max }_{1\le i\le N}\left\{\overrightarrow{T_{in}}\right\}\le T_{red},\end{array}$$

(8)

$$\begin{array}{cc}\hfill \Rightarrow & T_{sup}+{\max }_{1\le i\le N}\left\{\mathbf{D}\overrightarrow{P}\right\}\le T_{red},\end{array}$$

(9)

$$\begin{array}{cc}\hfill \Rightarrow & T_{sup}\le T_{red}-{\max }_{1\le i\le N}\left\{\mathbf{D}\overrightarrow{P}\right\}.\end{array}$$

(10)

Thus, the maximum supply temperature from the CRAC unit is $T_{sup}=T_{red}-{\max }_{1\le i\le N}\left\{\mathbf{D}\overrightarrow{P}\right\}$. Now, we can rewrite Equation (7) as

$$\begin{array}{cc}\hfill & E_{total}=E_{comp}+E_{CRAC}\hfill \\ \hfill & =\left(1+{\displaystyle \frac{1}{CoP(T_{red}-{\max }_{1\le i\le N}\left\{\mathbf{D}\overrightarrow{P}\right\})}}\right)\sum _{i=1}^N{E}_{comp}^i.\hfill \end{array}$$

(11)

4. Problem Formulation

In this subsection, we provide the non-linear integer formulation for the problem of thermal-aware VM allocation. We define an integer variable $x_{ij}$, which is set to 1 if the jth VM is allocated to ith chassis. Then, we define another integer variable $y_i$, which indicates whether the ith chassis is ON or OFF as follows:

$$y_i=\left\{\begin{array}{cc}0\hfill & \mathrm{if}{x}_{ij}=0,\forall j\in \{1,\cdots M\}\hfill \\ 1\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(12)

The computing power consumption of the ith chassis can be modeled as

$$P_{comp}^i={\alpha }_i·y_i+{\beta }_i·\sum _{j=1}^M{x}_{ij}.u_{ij}.$$

(13)

We define ${\overrightarrow{P}}_{comp}={[P_{comp}^1,\cdots ,P_{comp}^N]}^T$ as a column vector, representing the computing power consumption of each chassis and $t_{max}^i={\max }_{1\le j\le M}\{x_{ij}·t_j\}$ as the maximum time that ith chassis is active. The mathematical optimization problem is defined as

$$\begin{array}{cc}\hfill min.& \left(1+{\displaystyle \frac{1}{CoP(T_{red}-{\max }_{1\le i\le N}\left\{\mathbf{D}{\overrightarrow{P}}_{comp}\right\})}}\right)\hfill \\ \hfill & \times \sum _{i=1}^N({\alpha }_i·t_{max}^i+{\beta }_i·\sum _{j=1}^M(x_{ij}·u_{ij}·t_j)),\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \mathbf{subject}\mathbf{to}\forall j:& {\sum }_{i=1}^N{x}_{ij}=1,\end{array}$$

(15)

$$\begin{array}{cc}\hfill \forall i:& {\sum }_{j=1}^M{x}_{ij}·M_{req}^j\le M_{cap}^i,\end{array}$$

(16)

$$\begin{array}{cc}\hfill \forall i:& y_i\le {\sum }_{j=1}^M{x}_{ij}\le C·y_i,\end{array}$$

(17)

$$\begin{array}{cc}\hfill \forall i:& y_i\in \{0,1\},\end{array}$$

(18)

$$\begin{array}{cc}\hfill \forall i,j:& x_{ij}\in \{0,1\}.\end{array}$$

(19)

The objective function (14) tries to minimize the total energy consumption of a data center including the computing and cooling energy consumption. In practice, there is a trade-off between consolidating VMs on fewer servers to save the idle power consumption and the load balancing of VMs on a higher number of servers to avoid thermal hotspot creation. Our objective function accurately has taken these conflicting objectives into account.

The first constraint (15) ensures that each VM must be allocated to exactly one chassis. The second constraint (16) states that the requested MIPS of all VMs allocated to ith chassis must be less than or equal to the capacity of the chassis. In the third constraint (17), C is a significantly large number (that is, at least as large as the number of VMs). If for the ith chassis, ${\sum }_{j=1}^M{x}_{ij}$ equals 0 (i.e., no VM is allocated to ith chassis), then (16) implies that $y_i$ must be 0. If ${\sum }_{j=1}^M{x}_{ij}$ is bigger than 0 for the ith chassis, then (16) states that $y_i$ must be 1. The proposed non-linear integer formulation can obtain optimal VM placement solutions; however, it is an NP-hard problem and impractical for large-sized data centers. We therefore use a heuristic approach offering a comparably fast running time and still yielding near-optimal solutions.

5. The MITEC-GA Algorithm

In this section, we provide a genetic algorithm (MITEC-GA) to solve the VM allocation problem formulated in the previous section. Our proposed heuristic uses a vector of integer numbers, c, in order to implement the chromosome encoding. Each chromosome c represents a solution for the VM allocation problem. The allocation of jth VM to the ith chassis can be expressed by $c_j=i$. Thus, we have the following:

$$1\le c_j\le N,\forall j=1,...,M.$$

(20)

Let us define the available MIPS of ith chassis as

$$r_i=M_{cap}^i-\sum _{j:c_j=1}{M}_{req}^j.$$

(21)

A chromosome is a feasible solution for the problem if $r_i$ is greater than or equal to 0 for all chassis; that is, no allocation exceeds chassis capacity.

Algorithm 1 details the proposed heuristic. The MITEC-GA begins by generating a pool of random population and calculating the fitness function for each candidate solution (lines 2–5). The algorithm uses $E_{total}$ in Equation (7) as the fitness function. Our genetic algorithm runs in an iterative manner and repeats the following steps in each iteration:

• Selection: The roulette-wheel selection method is used to randomly select two parents from current population;
• Crossover: The new parents are generated by mixing selected chromosomes together and obtaining new allocations for some randomly selected VMs (lines 15–24);
• Mutation: The new solutions are formed by changing the new parents in a random way (lines 25–39).

We define a weighting factor $w_j$ for jth VM in a candidate solution c which is directly proportional to $M_{req}^j$, ${\beta }_{c_j}$ while inversely proportional to $r_{c_j}$. The higher the value of $w_j$, the more likely the allocation of jth VM changes in the solution c. We define $w_j$ as

$$w_j={\displaystyle \frac{M_{req}^j\times {\beta }_{c_j}}{r_{c_j}+ϵ}}.$$

(22)

This weighting factor represents the importance of dynamic energy consumption. When the remaining capacity of a chassis decreases, the utilization of the chassis increases; as a consequence, the dynamic energy consumption of the chassis increases. Similarly, when the requested MIPS of a VM is large, it means that this VM affects the dynamic energy consumption of the chassis more than other VMs. Thus, a large VM has a higher chance to be placed on another chassis resulting in higher load balancing and lower consolidation, which can avoid hot-spot thermal issues and consequently reduce cooling energy costs. In order to mutate chromosomes, the MITEC-GA first sorts VMs according to their weighting factor in descending order. The algorithm then substitutes the allocation of randomly selected VMs with lower weighting factors for the allocation of VMs with higher weighting factors.

Algorithm 1 MITEC-GA

1: procedure Genetic  2:     for $i=1$ to $popSize$ do   3:         $population[$i$]\leftarrow$ select a random feasible solution   4:         $fitness[$i$]\leftarrow$ calculate the fitness function   5:     $CurPop\leftarrow population$   6:     for $i=1$ to $num\_of\_iter$ do   7:         $selParents\leftarrow$ select two solutions from $CurPop$   8:         using roulette-wheel selection   9:         $crossParents\leftarrow$ call CrossOver($selParents$)   10:         $mutateSol\leftarrow$ call Mutate($crossParents$)   11:         Apply the fitness function on $mutateSol$.   12:         $CurPop\leftarrow$ select n solutions from $CurPop$   13:         and $mutateSol$ with low energy consumption   14:     $FinalSolution\leftarrow$ the solution within $CurPop$ with best fitness   15: functionCrossOver($selParents$$S_1$ and $S_2$)   16:     Select two random numbers, ($rand1<rand2$)   17:     for $i=rand1$ to $rand2$ do   18:         $c_1,c_2\leftarrow S_1\left[i\right],S_2\left[i\right]$   19:         $m_{c_1}^a,m_{c_2}^a\leftarrow$ available MIPS in chassis’s $c_1$   and $c_2$ 20:         if $m_{c_1}^a\ge M_{req}^i$ then    21:            Substitute $c_1$ for $c_2$ in $S_2$    22:         if $m_{c_2}^a\ge M_{req}^i$ then    23:            Substitute $c_2$ for $c_1$ in $S_1$    24:       return $S_1$ and $S_2$   25: functionMutate($crossParents$$S_1$ and $S_2$)   26:     for all ${\mathit{VM}}_j$ do             ▹ Calculate weighting factors   27:         $w_1\left[j\right]\leftarrow (M_{req}^j\times {\beta }_{S_1\left[j\right]})/(r_{S_1\left[j\right]}+ϵ)$   28:     Sort $w_1$ in descending order   29:     for all ${\mathit{VM}}_j$ in the first half of $w_1$ do   30:         $selPart\leftarrow$ set of all VMs in the last tenth of $w_1$   31:         repeat  32:            $sel\_VM\leftarrow$ select a VM randomly from $selPart$    33:            if $M_{req}^{{\mathit{VM}}_j}\ge M_{cap}^{S_1[sel\_VM]}$ then    34:                Substitute $S_1[sel\_VM]$ for $S_1\left[{\mathit{VM}}_j\right]$ in $S_1$   35:                break   36:            else  37:                Increment the $selPar{t}^{\prime }s$ space by one   38:         until $selPar{t}^{\prime }s$ space reach to ${\mathit{VM}}_j$   39:     Do the same things for $w_2$ and $S_2$

6. Simulation Results

In this section, the details of our simulation study and the performance results are presented.

6.1. Simulation Setup

In order to evaluate the the proposed scheme, CloudSim is modified and used to simulate thermal-aware VM allocation policies. We consider that a data center consists of two rows of five racks, each containing 50 chassis with 10 servers. The servers are HP D1080 and D2050. There are 35 chassis of D1080 and 15 chassis of D2050 servers. The D1080 server has a power consumption of 2020 W when idle and 2520 W at 100% CPU utilization. Similarly, the D2050 server has a power consumption of 1590 W at the idle state and 2490 W at 100% CPU utilization. The CRAC supplies chilled air at 20 ${}^{\circ }$C into the data center through the raised floor vents (Figure 1). Thermal intervals of 15 min (that is, sufficient time for data center thermal stability) are added to the simulator. In each interval, we calculate the maximum supply temperature from the CRAC unit, $T_{sup}$, based on Equation (10) and measure the energy consumption based on Equation (7).

In order to characterize the workload model, we use Amazon Elastic Compute Cloud (EC2) Standard Instances: Small Instance with one EC2 compute unit (1000 MIPS); Medium Instance with two EC2 compute units (2000 MIPS); Large Instance with four EC2 compute units (4000 MIPS); and Extra Large Instance with eight EC2 compute units (8000 MIPS). We consider different numbers of VMs in our simulations to achieve different data center utilizations (i.e., from 300 VMs to 620 VMs, which means from $48\%$ data center utilization to $98\%$ data center utilization). In the simulations, we assign one job with a random MIs to each VM. The MIs of each job follows a normal distribution from $750,000$ MIs to $4,500,000$ MIs [48,49]. We vary the mean of the normal distribution for each type of instances. For example, for Small Instance VMs, the mean is $1,500,000$ MIs, and for Extra Large Instance VMs, it is $3,750,000$ MIs. All the results are averaged over 50 runs for randomly-generated VMs.

We compare our heuristic with four thermal-aware greedy algorithms and one power-aware heuristic. Minimum Inlet Temperature (MinTinlet) is a greedy algorithm which allocates each VM to a chassis with the minimum inlet temperature. Minimum of Maximum Inlet Temperature (MinMaxTinlet) is a greedy algorithm which allocates each VM to a chassis which causes the minimum increase in the maximum inlet temperature. The third greedy algorithm is Minimum Summation Differential Inlet Temperature (MinSumDTinlet), which allocates each VM to a chassis which causes a minimum increase in the sum of inlet temperatures of all chassis (before and after allocation) within the data center. The main objective is to cause a minimum change in the inlet temperature of all chassis. LRH [50] is another greedy algorithm that assigns VMs to idle servers based on the chassis’ contribution to heat recirculation. Finally, we use one of the implemented VM allocation policies in the CloudSim, “PowerVmAllocationPolicySimple” [51], which allocates each VM to the first chassis with sufficient MIPS capacity.

6.2. Simulation Results

The performance of our proposed heuristic is depicted in Figure 2. On average, more than $30\%$ energy savings are achieved by our heuristics in higher utilizations in comparison with other greedy VM allocation heuristics. This is due to the fact that using the consolidation technique may have an effect on heat recirculation phenomena and increase the total energy consumption. The numbers of powered-off chassis for greedy algorithms are considerably greater than the numbers of powered-off chassis for MITEC-GA; thus, powered-on chassis are over-utilized in these greedy algorithms. MITEC-GA turns on more chassis; however, powered-on chassis are not fully utilized, resulting in less of an effect on the heat recirculation phenomena and consequently less dynamic energy consumption.

As shown in Figure 3 and Figure 4, MITEC-GA can efficiently reduce both computing and cooling energy. It can reduce more cooling energy especially when the utilization of servers is high (i.e., VM number = 620). This is because MITEC-GA turns on more chassis to obtain load balancing within the system, avoiding the creation of hotspots. This load balancing allows higher temperatures for supplied cooled air without exceeding the red-line temperature in each server for a given workload.

The supply temperatures of different algorithms are also shown in Figure 5 and Figure 6. Thermal-aware greedy algorithms provide higher supply temperatures; however, they are not as efficient as MITEC-GA, which considers both thermal and computing power simultaneously. Simulation results also suggest that using a consolidation technique may have an effect on heat recirculation phenomena and increase the total energy consumption. As depicted in

Table 1. Numbers of off-chassis in different VM allocation algorithms for different numbers of VMs.

Number of VMs	MinSumDTinlet	MinMaxTinlet	SimplePower	MinTinlet	LRH	MITEC-GA
300	25	25	25	25	25	23
340	22	22	22	22	22	20
380	19	19	19	18	19	17
420	15	15	15	15	15	13
460	12	12	12	12	12	10
500	9	9	9	9	9	7
540	5	5	5	5	5	4
580	2	2	2	2	2	0
620	1	1	1	1	1	0

, the numbers of off-chassis for greedy algorithms are greater than the numbers of off-chassis of MITEC-GA; thus, on-chassis are over-utilized in these greedy algorithms. The effect of over-utilizing the chassis can be seen in Figure 2, where their energy consumption is significantly higher than MITEC-GA. MITEC-GA turns on more chassis; however, on-chassis are not fully utilized, they consume less dynamic energy consumption, and they have less effect on heat recirculation phenomena.

In terms of the cost–benefit analysis, reducing energy consumption can reduce the power capacity-related costs as well as the energy costs. Assuming $\$0.12$ energy cost per kWh [52], by the energy saving of up to $30\%$ which is achieved compared to thermal-aware greedy algorithms and power-aware VM allocation heuristics, MITEC-GA can save around 131,400–254,040 kWh of energy yearly, meaning $\$15,768$ to $\$30,484$ savings on energy bills per year.

7. Conclusions

In this paper, we formulated the problem of minimizing the total energy consumption of a heterogeneous data center (MITEC) as a non-linear integer optimization problem. We provided a thermal-aware VM allocation heuristic, MITEC-GA, based on the genetic algorithm. The MITEC-GA is a meta-heuristic optimization technique, which randomly searches the feasible solution space and finds a near-optimal solution for the allocation of thermal-aware VMs. The algorithm operates in an iterative manner. In each iteration, the MITEC-GA tries to generate the solutions that are a better fit than the current solution based on the total energy consumption of a data center. By the repeated modification of the existing solution, the MITEC-GA evolves the current solution toward the optimal solution. Experimental results reveal the effectiveness of our proposed heuristic and demonstrate that, on average, more than $30\%$ energy savings are achieved by our heuristic in comparison with other greedy VM allocation heuristics. In future works, we will consider an on-line thermal-aware VM allocation problem and use VM migration techniques to achieve more energy savings. We will also consider the dynamic effect of fan speeds on heat recirculation phenomena and try to provide an on-line fan speed control mechanism.