Energy Aware Virtual Machine Scheduling in Data Centers
Abstract
:1. Motivation
2. Energy Proportionality and the PEEP Metric
2.1. SPECpower Benchmark
2.2. Energy Proportionality Calculation And Its Implications
- (1)
- For all 334 servers that achieve peak energy efficiency at 100% utilization, their EP curves are always above the EP curve of the ideal server.
- (2)
- For the remaining 175 servers that achieve peak energy efficiency at non-100% utilization, six servers are always above the ideal server, 113 servers intersect with the ideal curve once, and 56 servers intersect with the ideal curve twice. Their energy efficiency and EP are listed in Table 3.
2.3. The PEEP Metric
3. Energy Efficiency and Proportionality Aware Virtual Machine Scheduling
3.1. Server Energy Efficiency Identification
- (1)
- Representative of Data Center Workloads: We use SPECpower_ssj, PrimeSearch [17], STREAM [18], and their mixture. The workload is also proportional to the server’s hardware configuration. It varies with multiple workload intensities to simulate a real multitenant virtualized cloud computing environment.
- (2)
- Easy Implementation: Recent commercial servers are equipped with IPMI support through which we get the real time power consumption from embedded sensors in a server mainboard. This includes system power, processor power, and memory bank power. With simulation workload- and IPMI-based power consumption, we can get the server’s energy efficiency and proportionality under different workload types and workload intensity levels. In the authors’ implementation, IPMI-based power data acquisition has negligible overhead on the physical server’s system utilization and extra power consumption (< 0.25 watt).
- (1)
- Computing Intensive Workload: The authors use a prime number computation program, namely, PrimeSearch, as the computing intensive workload. One execution of PrimeSearch will calculate and search prime numbers in 10 intervals (1,200000), (1,400000), (1,600000), (1,800000), (1,1000000), (1,1100000), (1,1200000), (1,1300000), (1,1400000), (1,1500000), respectively. These 10 intervals are 10 subsearching tasks. The completion time of the interval search is calculated and the sum of the completion time of 10 subtasks is calculated as the task completion time of one Primesearch execution.
- (2)
- Memory Intensive Workload: The authors use STREAM as the memory intensive workload. It is a single synthetic benchmark program to measure the amount of memory and corresponding computational rate of the simple vector kernel. The CPU of the computer system runs much faster than memory. As this progresses, more program performance will be limited to memory bandwidth rather than CPU computing performance.
- (3)
- Hybrid Workload: The authors use SPECpower_ssj2008 as the hybrid workload to stress the server’s components, including CPU and memory synthetically.
3.2. VM Workload Characterization
3.3. Virtual Machine Scheduling and Migration
- Server Monitoring and Power Data Collection: CPU and memory contribute to most of the dynamic power consumption in a server. Therefore, the monitoring module of EASE is to collect real-time power consumption, CPU utilization, memory utilization, and other system access and activities statistics. The physical server provides the physical machine CPU, memory, input/output (I/O), and other hardware resource utilization using the sysstat utility.
- Benchmark Execution: EASE runs benchmark tests on dedicated servers under multiple software and hardware configurations. It also collects performance data.
- Energy Efficiency and Proportionality Calculation: Based on system monitoring data and benchmark execution results, the energy efficiency and proportionality are calculated in real time.
- VM Scheduling: When EASE gets each server’s current running status and optimal working range, it determines if the server is overloaded or underloaded. For new VMs, EASE looks for the appropriate target physical server to keep them working in an optimal working range. For existing VMs, EASE first characterizes the machine’s workload type (i.e., computing intensive, memory intensive, or hybrid) and attempts to migrate the VM to an appropriate target physical server. Servers with different hardware and software configurations have different energy efficiency and proportionality. Therefore, if there are multiple servers with different resources (i.e., resource type, capacity, and configuration), EASE first sorts the servers with available resources by resource type in descending order of energy efficiency, EP, and PEEP values. Next, it selects a target host according to the priority that the target host can reach the peak energy efficiency or energy efficiency will be higher in its optimal working range.
4. Experiment Results and Analysis
4.1. Experimental Platform
4.2. Experimental Results
5. Related Work
6. Conclusions and Future Work
Author Contributions
Funding
Conflicts of Interest
References
- Worldwide Server unit Shipments from 1st Quarter 2009 to 1st Quarter 2016. Available online: http://www.statista.com/statistics/267390/global-server-shipments-by-vendor/ (accessed on 26 December 2018).
- Fernández-Cerero, D.; Fernández-Montes, A.; Velasco, F. Productive Efficiency of Energy-Aware Data Centers. Energies 2018, 11, 2053. [Google Scholar] [CrossRef]
- Report to Congress on Server and Data Center Energy Efficiency Public Law 109-431. Available online: https://escholarship.org/uc/item/74g2r0vg (accessed on 26 December 2018).
- Natural Resources Defense Council, Data Center Efficiency Assessment. Available online: https://www.nrdc.org/sites/default/files/data-center-efficiency-assessment-IP.pdf (accessed on 26 December 2018).
- GeSI SMARTer2020: The Role of ICT in Driving a Sustainable Futur. Available online: http://gesi.org/smarter2020 (accessed on 26 December 2018).
- Barroso, L.A.; Clidaras, J.; Hölzle, U. The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, 2nd ed.; Morgan & Claypool Publishers: San Rafael, CA, USA, 2013. [Google Scholar]
- Wu, Q.; Deng, Q.; Ganesh, L.; Hsu, C.; Jin, Y.; Kumar, S.; Li, B.; Meza, J.; Song, Y. Dynamo: Facebook’s Data Center-Wide Power Management System. In Proceedings of the 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), Seoul, South Korea, 18–22 June 2016. [Google Scholar]
- Zafar, S.; Chaudhry, S.A.; Kiran, S. Adaptive TrimTree: Green Data Center Networks through Resource Consolidation, Selective Connectedness and Energy Proportional Computing. Energies 2016, 9, 797. [Google Scholar] [CrossRef]
- Barroso, L.A.; Hölzle, U. The case for energy-proportional computing. Computer 2007, 12, 33–37. [Google Scholar] [CrossRef]
- Ryckbosch, F.; Polfliet, S. Trends in server energy proportionality. Computer 2011, 9, 69–72. [Google Scholar] [CrossRef]
- Sen, R.; Wood, D. Energy-Proportional Computing: A New Definition. Computer 2017, 8, 26–33. [Google Scholar] [CrossRef]
- Jiang, C.; Wang, Y.; Ou, D. Energy Proportional Servers: Where Are We in 2016? In Proceedings of the 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, GA, USA, 5–8 June 2017; pp. 1649–1660. [Google Scholar]
- Malladi, K.T.; Lee, B.C.; Nothaft, F.A.; Kozyrakis, C.; Periyathambi, K.; Horowitz, M. Towards energy-proportional datacenter memory with mobile DRAM. In Proceedings of the 39th Annual International Symposium on Computer Architecture (ISCA), Portland, OR, USA, 9–13 June 2012. [Google Scholar]
- Malladi, K.T.; Shaeffer, I.; Gopalakrishnan, L.; Lo, D.; Lee, B.C.; Horowitz, M. Rethinking DRAM Power Modes for Energy Proportionality. In Proceedings of the 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture, Vancouver, BC, Canada, 1–5 December 2012; pp. 131–142. [Google Scholar]
- SPECpower_ssj2008. Available online: https://www.spec.org/power_ssj2008/ (accessed on 26 December 2018).
- Jiang, C.; Wang, Y.; Ou, D.; Qiu, Y.; Li, Y.; Wan, J.; Luo, B.; Shi, W.; Cerin, C. EASE: Energy Efficiency and Proportionality Aware Virtual Machine Scheduling. In Proceedings of the 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD2018), Paris, France, 25 September 2018. [Google Scholar]
- Jiang, C.; Wang, Y.; Ou, D.; Li, Y.; Zhang, J.; Wan, J.; Luo, B.; Shi, W. Energy efficiency comparison of hypervisors. Sustain. Comput. Inform. Syst. 2017. [Google Scholar] [CrossRef]
- STREAM. Available online: https://www.cs.virginia.edu/stream/ (accessed on 26 December 2018).
- Jiang, C.; Han, G.; Lin, J.; Jia, G.; Shi, W.; Wan, J. Characteristics of Co-allocated Online Services and Batch Jobs in Internet Data Centers: A Case Study from Alibaba Cloud. IEEE Access 2019. [Google Scholar] [CrossRef]
- Meisner, D.; Gold, B.T.; Wenisch, T.F. PowerNap: Eliminating server idle power. In Proceedings of the 14th international conference on Architectural support for programming languages and operating systems, Washington, DC, USA, 7–11 March 2009; pp. 205–216. [Google Scholar]
- Isci, C.; McIntosh, S.; Kephart, J.; Das, R.; Hanson, J.; Piper, S.; Wolford, R.; Brey, T.; Kantner, R.; Ng, A.; et al. Agile, efficient virtualization power management with low-latency server power states. In Proceedings of the 40th Annual International Symposium on Computer Architecture, Tel-Aviv, Israel, 23–27 June 2013; pp. 96–107. [Google Scholar]
- Liu, Y.; Draper, S.C.; Kim, N.S. SleepScale: Runtime joint speed scaling and sleep states management for power efficient data centers. In Proceedings of the 2014 ACM/IEEE 41st International Symposium on Computer Architecture, Minneapolis, Minnesota, 14–18 June 2014; pp. 313–324. [Google Scholar]
- Chiaraviglio, L.; Cianfrani, A.; Listanti, M.; Liu, W.; Polverini, M. Lifetime-Aware Cloud Data Centers: Models and Performance Evaluation. Energies 2016, 9, 470. [Google Scholar] [CrossRef]
- Ferreto, T.C.; Netto, M.A.; Calheiros, R.N.; De Rose, C.A. Server consolidation with migration control for virtualized data centers. Future Gener. Comput. Syst. 2011, 27, 1027–1034. [Google Scholar] [CrossRef]
- Speitkamp, B.; Bichler, M. A mathematical programming approach for server consolidation problems in virtualized data centers. IEEE Trans. Serv. Comput. 2010, 3, 266–278. [Google Scholar] [CrossRef]
- Beloglazov, A.; Buyya, R. Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers. Concurr. Comput. Pract. Exp. 2012, 24, 1397–1420. [Google Scholar] [CrossRef]
- Liu, H.; Jin, H.; Xu, C.; Liao, X. Performance and energy modeling for live migration of virtual machines. Cluster Comput. 2013, 16, 249–264. [Google Scholar] [CrossRef]
- Xu, C.; Zhao, Z.; Wang, H.; Shea, R.; Liu, J. Energy efficiency of cloud virtual machines: From traffic pattern and CPU affinity perspectives. IEEE Syst. J. 2017, 11, 835–845. [Google Scholar] [CrossRef]
- Lagen, S.; Pascual-Iserte, A.; Munoz, O.; Vidal, J. Energy efficiency in latency-constrained application offloading from mobile clients to multiple virtual machines. IEEE Trans. Signal Process. 2018, 66, 1065–1079. [Google Scholar] [CrossRef]
- Jiang, C.; Duan, L.; Liu, C.; Wan, J.; Zhou, L. VRAA: Virtualized resource auction and allocation based on incentive and penalty. Cluster Comput. 2013, 16, 639–650. [Google Scholar] [CrossRef]
- Belabed, D.; Secci, S.; Pujolle, G.; Medhi, D. Striking a balance between traffic engineering and energy efficiency in virtual machine placement. IEEE Trans. Netw. Serv. Manag. 2015, 12, 202–216. [Google Scholar] [CrossRef]
- Yan, S.; Xiao, S.; Chen, Y.; Cui, Y.; Liu, J. GreenWay: Joint VM placement and topology adaption for green data center networking. In Proceedings of the 26th International Conference on Computer Communication and Networks, Vancouver, BC, Canada, 31 July–3 August 2017; pp. 1–9. [Google Scholar]
- Lago, D.G.; Madeira, E.R.; Medhi, D. Energy-aware virtual machine scheduling on data centers with heterogeneous bandwidths. IEEE Trans. Parall. Distr. Syst. 2018, 29, 83–98. [Google Scholar] [CrossRef]
- Liu, X.; Zhan, Z.; Zhang, J. An Energy Aware Unified Ant Colony System for Dynamic Virtual Machine Placement in Cloud Computing. Energies 2017, 10, 609. [Google Scholar] [CrossRef]
- Bai, Y.; Gu, L.; Qi, X. Comparative Study of Energy Performance between Chip and Inlet Temperature-Aware Workload Allocation in Air-Cooled Data Center. Energies 2018, 11, 669. [Google Scholar] [Green Version]
- Wibron, E.; Ljung, A.-L.; Lundström, T.S. Computational Fluid Dynamics Modeling and Validating Experiments of Airflow in a Data Center. Energies 2018, 11, 644. [Google Scholar] [CrossRef]
- Tsirogiannis, D.; Harizopoulos, S.; Shah, M.A. Analyzing the energy efficiency of a database server. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, Indianapolis, Indiana, 6–10 June 2010; pp. 231–242. [Google Scholar]
- Lang, W.; Patel, J.M. Energy management for mapreduce clusters. Proc. VLDB Endow. 2010, 3, 129–139. [Google Scholar] [CrossRef]
- Wong, D. Peak efficiency aware scheduling for highly energy proportional servers. In Proceedings of the 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture, Seoul, South Korea, 18–22 June 2016; pp. 481–492. [Google Scholar]
- Leverich, J.; Kozyrakis, C. On the energy (in) efficiency of Hadoop clusters. ACM SIGOPS Oper. Syst. Rev. 2010, 44, 61–65. [Google Scholar] [CrossRef]
- Schall, D.; Hudlet, V. WattDB: An energy-proportional cluster of wimpy nodes. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of data, Athens, Greece, 12–16 June 2011; pp. 1229–1232. [Google Scholar]
- Jiang, C.; Fan, T.; Qiu, Y.; Wu, H.; Zhang, J.; Xiong, N.; Wan, J. Interdomain I/O Optimization in Virtualized Sensor Networks. Sensors 2018, 18, 4395. [Google Scholar] [CrossRef] [PubMed]
- Jiang, C.; Qiu, Y.; Shi, W.; Cerin, C.; Xiong, N.; Wan, J. Escope: An Energy Efficiency Simulator For Data Centers. In Proceedings of the IEEE 39th International Conference on Distributed Computing Systems (ICDCS), Dallas, Texas, USA, 7–10 July 2019. submitted. [Google Scholar]
Performance | Power | Performance to Power Ratio | ||
---|---|---|---|---|
Target Load | Actual Load | ssj_ops | Average Active Power (W) | |
100% | 99.70% | 11,725,627 | 944 | 12,424 |
90% | 90.00% | 10,580,169 | 851 | 12,427 |
80% | 80.00% | 9,411,437 | 716 | 13,151 |
70% | 70.10% | 8,241,170 | 598 | 13,779 |
60% | 60.00% | 7,056,523 | 510 | 13,845 |
50% | 50.00% | 5,876,594 | 431 | 13,621 |
40% | 40.10% | 4,709,344 | 373 | 12,614 |
30% | 30.00% | 3,527,435 | 324 | 10,895 |
20% | 20.00% | 2,352,157 | 277 | 8498 |
10% | 10.00% | 1,173,811 | 228 | 5154 |
Active Idle | 0 | 82.9 | 0 | |
∑ssj_ops/∑power = | 12,120 |
Utilization Spot Where Peak Energy Efficiency Occurs | Count | Count | Total |
---|---|---|---|
100% | 334 | 65.49% | 510 |
90% | 16 | 3.14% | |
80% | 65 | 12.74% | |
70% | 81 | 15.88% | |
60% | 14 | 2.75% |
Metrics | Above the Ideal Curve | Intersect Once | Intersect Twice |
---|---|---|---|
count | 6 | 113 | 56 |
avg. EP | 0.76 | 0.886 | 0.860 |
med. EP | 0.76 | 0.891 | 0.855 |
avg. EE | 5264 | 6781 | 6756 |
Med.EE | 3849 | 5146 | 5318 |
avg. PEEP | 1.003 | 1.080 | 1.050 |
med. PEEP | 1.003 | 1.073 | 1.043 |
total | 175 |
Utilization | Power (Normalized to Its 100% Utilization) | Power (Normalized to Ideal Server) | Peak Energy Efficiency Spot |
---|---|---|---|
40% | 39.5% | 98.8% | |
50% | 45.7% | 91.3% | |
60% | 54.0% | 90.0% | Yes |
70% | 63.3% | 90.5% | |
80% | 75.8% | 94.8% |
Utilization Where Peak Energy Efficiency Occurs | Count | Total |
---|---|---|
60% | 14 | 510 |
70% | 81 | |
80% | 65 | |
90% | 16 | |
100% | 334 |
Type | CPU Utilization | Memory Utilization |
---|---|---|
Computing Intensive Workload | >70% | <30% |
Memory Intensive Workload | >20% and <50% | >60% |
Hybrid Workload | >30% and <60% | >30% and <60% |
No. | Platform | Year of Manufacture | CPU | Total CPU Core | CPU TDP (Watt) | Memory (GB) | Hard Disk |
---|---|---|---|---|---|---|---|
1 | Sugon A620r-G | 2012 | 2*AMD Opteron 6272 | 32 | 115 | 64(8G*8) DDR3 1600MHz | 4*SAS 300GB 10K rpm (RAID10) |
2 | ThinkServer RD640 | 2014 | 2*Intel Xeon E5-2620 v2 | 12 | 80 | 160(16G*10) DDR4 2133MHz | 1*SSD 480GB |
3 | ThinkServer RD450 | 2015 | 2*Intel Xeon E5-2620 v3 | 12 | 85 | 192(16G*12) DDR4 2133MHz | 2*HDD 4TB |
Number of VMs | Power Consumption | Prime Search Complete Time | STREAM | ||||
---|---|---|---|---|---|---|---|
Whole Running | Concurrent Phase | Bandwidth | Average Time | Min Time | Max Time | ||
3 | −49.98% | −46.28% | 0.31% | −22.45 | 28.44% | 29.00% | 28.84% |
6 | −49.13% | −45.02% | 7.53% | −27.46% | 67.80% | 48.15% | 47.10% |
8 | −47.23% | −45.11% | 7.52% | −22.34% | 4.80% | 35.15% | −7.96% |
12 | −40.56% | −37.07% | 8.49% | −19.72% | 16.75% | 42.54% | 56.48% |
Server | VM Scheduling Output | ||
---|---|---|---|
Initial (no EASE) | Packing Scheduling | EASE | |
#1 | 8 | 0 | 0 |
#2 | 2 | 8 | 4 |
#3 | 6 | 8 | 12 |
power(watts) | 554 | 317 | 309 |
average completion time (s) | 3476 | 1611 | 1606 |
Server | VM Scheduling Output | ||
---|---|---|---|
Initial (no EASE) | Packing Scheduling | EASE | |
#1 | 16 | 6 | 0 |
#2 | 6 | 12 | 6 |
#3 | 8 | 12 | 24 |
Power | 636 | 604 | 329 |
Average completion time | 3767 | 2370 | 2618 |
Server | VM scheduling output | ||
---|---|---|---|
Initial (no EASE) | Packing Scheduling | EASE | |
#1 | 8prime+8stream | 0 | 0 |
#2 | 3prime+3stream | 8prime+8stream | 4prime+8stream |
#3 | 6prime+6stream | 9prime+9stream | 12prime+9stream |
Power | 612 | 352 | 325 |
Prime average completion time | 3366 | 1610 | 1606 |
STREAM average maximum bandwidth | 574 | 514 | 513 |
STREAM average completion time | 16 | 22 | 25 |
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Qiu, Y.; Jiang, C.; Wang, Y.; Ou, D.; Li, Y.; Wan, J. Energy Aware Virtual Machine Scheduling in Data Centers. Energies 2019, 12, 646. https://doi.org/10.3390/en12040646
Qiu Y, Jiang C, Wang Y, Ou D, Li Y, Wan J. Energy Aware Virtual Machine Scheduling in Data Centers. Energies. 2019; 12(4):646. https://doi.org/10.3390/en12040646
Chicago/Turabian StyleQiu, Yeliang, Congfeng Jiang, Yumei Wang, Dongyang Ou, Youhuizi Li, and Jian Wan. 2019. "Energy Aware Virtual Machine Scheduling in Data Centers" Energies 12, no. 4: 646. https://doi.org/10.3390/en12040646