Impact of Secure Container Runtimes on File I/O Performance in Edge Computing
Abstract
:1. Introduction
- We analyze the detailed file I/O operations of three secure container runtimes and describe the entire file I/O stack of the runtimes thoroughly.
- We present the experimental results on both file I/O performance and CPU usage, which demonstrate the performance and efficiency of the runtimes.
- We offer symbol-level profiling results that point out the root cause of the differences in performance and CPU usage, which can suggest research directions for improving the file I/O performance of the runtimes.
2. Related Work
2.1. Performance Comparison and Analysis
2.2. Runtime Optimization
3. Background
3.1. Kernel-Based Virtual Machine with Intel Virtualization Technology
3.2. Linux Secure Computing Mode (Seccomp)
4. File Operations in Secure Container Runtimes
- Handling EPT: Operations that deal with EPT violation and the reconstruction of the EPT.
- Memory processing: Miscellaneous memory operations such as memory mapping and copy, except the handling EPT.
- Filesystem processing: The file I/O processing operation conducted in the host OS (e.g., vfs_read).
- Scheduling: Symbols related to the process scheduling.
- Application: Operations executed by the application running on containers or VMs.
4.1. Kata Containers
4.2. gVisor
4.3. Firecracker
5. Performance and CPU Usage Analysis
5.1. Experimental Setup
5.2. Sequential Read/Write
5.3. Random Read/Write
6. Bottleneck Analysis Using Symbol-Level Profiling
6.1. Kata Containers
6.2. gVisor
6.3. Firecracker
7. Discussion
8. Concluding Remarks
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Soltesz, S.; Pötzl, H.; Fiuczynski, M.E.; Bavier, A.; Peterson, L. Container-based operating system virtualization: A scalable, high-performance alternative to hypervisors. In Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, Lisbon, Portugal, 21–23 March 2007; pp. 275–287. [Google Scholar]
- Merkel, D. Docker: Lightweight linux containers for consistent development and deployment. Linux J. 2014, 2014, 2. [Google Scholar]
- Felter, W.; Ferreira, A.; Rajamony, R.; Rubio, J. An updated performance comparison of virtual machines and linux containers. In Proceedings of the 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Philadelphia, PA, USA, 29–31 March 2015; pp. 171–172. [Google Scholar]
- Li, Z.; Cheng, J.; Chen, Q.; Guan, E.; Bian, Z.; Tao, Y.; Zha, B.; Wang, Q.; Han, W.; Guo, M. RunD: A Lightweight Secure Container Runtime for High-density Deployment and High-concurrency Startup in Serverless Computing. In Proceedings of the 2022 USENIX Annual Technical Conference (USENIX ATC 22), Carlsbad, CA, USA, 11–13 July 2022; pp. 53–68. [Google Scholar]
- Hong, C.H.; Varghese, B. Resource management in fog/edge computing: A survey on architectures, infrastructure, and algorithms. ACM Comput. Surv. (CSUR) 2019, 52, 1–37. [Google Scholar] [CrossRef]
- National Vulnerability Database. CVE-2019-5736 Detail. 2019. Available online: https://nvd.nist.gov/vuln/detail/CVE-2019-5736 (accessed on 14 April 2021).
- Walsh, D.J. Are Docker Containers Really Secure? 2014. Available online: https://opensource.com/business/14/7/docker-security-selinux (accessed on 23 March 2021).
- Sultan, S.; Ahmad, I.; Dimitriou, T. Container security: Issues, challenges, and the road ahead. IEEE Access 2019, 7, 52976–52996. [Google Scholar] [CrossRef]
- Agache, A.; Brooker, M.; Iordache, A.; Liguori, A.; Neugebauer, R.; Piwonka, P.; Popa, D.M. Firecracker: Lightweight virtualization for serverless applications. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), Santa Clara, CA, USA, 25–27 February 2020; pp. 419–434. [Google Scholar]
- gVisor: Application Kernel for Containers. Available online: https://github.com/google/gvisor (accessed on 6 May 2021).
- Randazzo, A.; Tinnirello, I. Kata containers: An emerging architecture for enabling mec services in fast and secure way. In Proceedings of the 2019 Sixth International Conference on Internet of Things: Systems, Management and Security (IOTSMS), Granada, Spain, 22–25 October 2019; pp. 209–214. [Google Scholar]
- Barr, J. Firecracker—Lightweight Virtualization for Serverless Computing. 2018. Available online: https://aws.amazon.com/ko/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ (accessed on 14 April 2021).
- Bellard, F. QEMU, a fast and portable dynamic translator. In Proceedings of the USENIX Annual Technical Conference, FREENIX Track, Anaheim, CA, USA, 10–15 April 2005; pp. 41–46. [Google Scholar]
- virtio-fs. Available online: https://virtio-fs.gitlab.io/ (accessed on 25 May 2021).
- Russell, R. virtio: Towards a de-facto standard for virtual I/O devices. ACM SIGOPS Oper. Syst. Rev. 2008, 42, 95–103. [Google Scholar] [CrossRef]
- Klimovic, A.; Wang, Y.; Stuedi, P.; Trivedi, A.; Pfefferle, J.; Kozyrakis, C. Pocket: Elastic ephemeral storage for serverless analytics. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), Carlsbad, CA, USA, 8–10 October 2018; pp. 427–444. [Google Scholar]
- Kang, J.; Hu, C.; Wo, T.; Zhai, Y.; Zhang, B.; Huai, J. Multilanes: Providing virtualized storage for os-level virtualization on manycores. ACM Trans. Storage (TOS) 2016, 12, 1–31. [Google Scholar] [CrossRef]
- Young, E.G.; Zhu, P.; Caraza-Harter, T.; Arpaci-Dusseau, A.C.; Arpaci-Dusseau, R.H. The true cost of containing: A gVisor case study. In Proceedings of the 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19), Renton, WA, USA, 8 July 2019. [Google Scholar]
- Anjali; Caraza-Harter, T.; Swift, M.M. Blending containers and virtual machines: A study of firecracker and gVisor. In Proceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, Lausanne, Switzerland, 17 March 2020; pp. 101–113. [Google Scholar]
- Debab, R.; Hidouci, W.K. Containers Runtimes War: A Comparative Study. In Future Technologies Conference (FTC) 2020, Proceedings of the Future Technologies Conference, Vancouver, BC, Canada, 5–6 November 2020; Springer: Cham, Switzerland, 2020; pp. 135–161. [Google Scholar]
- Goethals, T.; Sebrechts, M.; Al-Naday, M.; Volckaert, B.; De Turck, F. A functional and performance benchmark of lightweight virtualization platforms for edge computing. In Proceedings of the 2022 IEEE International Conference on Edge Computing and Communications (EDGE), Barcelona, Spain, 10–16 July 2022; pp. 60–68. [Google Scholar]
- Armbrust, M.; Fox, A.; Griffith, R.; Joseph, A.D.; Katz, R.; Konwinski, A.; Lee, G.; Patterson, D.; Rabkin, A.; Stoica, I.; et al. A view of cloud computing. Commun. ACM 2010, 53, 50–58. [Google Scholar] [CrossRef]
- Ngenzi, A.; R, S.; Nair, S.R. Dynamic Resource Management in Cloud Data Centers for Server Consolidation. arXiv 2015, arXiv:1505.00577. [Google Scholar]
- Kumar, R.; Thangaraju, B. Performance Analysis between runC and Kata Container Runtime. In Proceedings of the 2020 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT), Bangalore, India, 2–4 July 2020; pp. 1–4. [Google Scholar]
- Viktorsson, W.; Klein, C.; Tordsson, J. Security-Performance Trade-offs of Kubernetes Container Runtimes. In Proceedings of the 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), Nice, France, 17–19 November 2020; pp. 1–4. [Google Scholar]
- Park, J.; Kim, H.; Lee, K. Evaluating Concurrent Executions of Multiple Function-as-a-Service Runtimes with MicroVM. In Proceedings of the 2020 IEEE 13th International Conference on Cloud Computing (CLOUD), Beijing, China, 19–23 October 2020; pp. 532–536. [Google Scholar]
- Barcelona-Pons, D.; García-López, P. Benchmarking Parallelism in FaaS Platforms. arXiv 2020, arXiv:2010.15032. [Google Scholar] [CrossRef]
- Cadden, J.; Unger, T.; Awad, Y.; Dong, H.; Krieger, O.; Appavoo, J. SEUSS: Skip redundant paths to make serverless fast. In Proceedings of the Fifteenth European Conference on Computer Systems, Heraklion, Greece, 27–30 April 2020; pp. 1–15. [Google Scholar]
- Ustiugov, D.; Petrov, P.; Kogias, M.; Bugnion, E.; Grot, B. Benchmarking, analysis, and optimization of serverless function snapshots. arXiv 2021, arXiv:2101.09355. [Google Scholar]
- Thomas, S.; Ao, L.; Voelker, G.M.; Porter, G. Particle: Ephemeral endpoints for serverless networking. In Proceedings of the 11th ACM Symposium on Cloud Computing, Virtual Event, 19–21 October 2020; pp. 16–29. [Google Scholar]
- Dukic, V.; Bruno, R.; Singla, A.; Alonso, G. Photons: Lambdas on a diet. In Proceedings of the 11th ACM Symposium on Cloud Computing, Virtual Event, 19–21 October 2020; pp. 45–59. [Google Scholar]
- Koller, R.; Williams, D. An ounce of prevention is worth a pound of cure: Ahead-of-time preparation for safe high-level container interfaces. In Proceedings of the 11th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage 19), Renton, WA, USA, 8–9 July 2019. [Google Scholar]
- Neiger, G.; Santoni, A.; Leung, F.; Rodgers, D.; Uhlig, R. Intel Virtualization Technology: Hardware Support for Efficient Processor Virtualization. Intel Technol. J. 2006, 10, 167–177. [Google Scholar] [CrossRef]
- Uhlig, R.; Neiger, G.; Rodgers, D.; Santoni, A.L.; Martins, F.C.; Anderson, A.V.; Bennett, S.M.; Kagi, A.; Leung, F.H.; Smith, L. Intel virtualization technology. Computer 2005, 38, 48–56. [Google Scholar] [CrossRef]
- Zhang, B.; Wang, X.; Lai, R.; Yang, L.; Wang, Z.; Luo, Y.; Li, X. Evaluating and optimizing I/O virtualization in kernel-based virtual machine (KVM). In Network and Parallel Computing, Proceedings of the IFIP International Conference on Network and Parallel Computing, Zhengzhou, China, 13–15 September 2010; Springer: Berlin/Heidelberg, Germany, 2010; pp. 220–231. [Google Scholar]
- Yang, S. Extending KVM with new Intel Virtualization technology. In Proceedings of the KVM Forum, Napa Valley, CA, USA, 11–13 June 2008. [Google Scholar]
- Dong, Y.; Xue, M.; Zheng, X.; Wang, J.; Qi, Z.; Guan, H. Boosting GPU Virtualization Performance with Hybrid Shadow Page Tables. In Proceedings of the 2015 USENIX Annual Technical Conference (USENIX ATC 15), Santa Clara, CA, USA, 8–10 July 2015; pp. 517–528. [Google Scholar]
- Kunwar, B. Disk I/O Performance of Kata Containers. Available online: https://www.stackhpc.com/images/IO-Performance-of-Kata-Containers-TheNewStack.pdf (accessed on 12 May 2021).
- Wilcox, M.; Zwisler, R. Linux DAX. Available online: https://www.kernel.org/doc/Documentation/filesystems/dax.txt (accessed on 8 April 2021).
- Szeredi, M. FUSE: Filesystem in Userspace. 2010. Available online: http://fuse.sourceforge.net (accessed on 8 April 2021).
- Kim, H.; Kim, S.; Jeong, J.; Lee, J. Virtual asymmetric multiprocessor for interactive performance of consolidated desktops. In ACM SIGPLAN Notices, Proceedings of the 10th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments—VEE’14, Salt Lake City, UT, USA, 1–2 March 2014; ACM Press: New York, NY, USA, 2014; pp. 29–40. [Google Scholar] [CrossRef]
- gVisor—Ptrace. Available online: https://gvisor.dev/docs/architecture_guide/platforms/#ptrace (accessed on 20 May 2021).
- gVisor—KVM. Available online: https://gvisor.dev/docs/architecture_guide/platforms/#kvm. (accessed on 20 May 2021).
- Fio. Available online: https://linux.die.net/man/1/fio (accessed on 8 April 2021).
- mpstat. Available online: https://linux.die.net/man/1/mpstat (accessed on 8 April 2021).
- Brown, N. Overlay Filesystem. Available online: https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt (accessed on 6 May 2021).
- Laurén, S.; Memarian, M.R.; Conti, M.; Leppänen, V. Analysis of security in modern container platforms. In Research Advances in Cloud Computing; Springer: Berlin/Heidelberg, Germany, 2017; pp. 351–369. [Google Scholar]
- Skarlatos, D.; Chen, Q.; Chen, J.; Xu, T.; Torrellas, J. Draco: Architectural and Operating System Support for System Call Security. In Proceedings of the 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Athens, Greece, 17–21 October 2020; pp. 42–57. [Google Scholar]
- Yang, J.; Tak, B. Security Assessment Technique of a Container Runtime Using System Call Weights. J. Korea Soc. Comput. Inf. 2020, 25, 21–29. [Google Scholar]
Kata Containers | gVisor | Firecracker | |
---|---|---|---|
Architecture | Containers on VMs | User-space kernel | Firecracker VMM-based microVM |
VMM technology | KVM | KVM | KVM |
I/O processing | virtio-fs | System call filtering | virtio-blk |
Programming language | Go | Go | Rust |
Component | Configuration | Component | Configuration |
---|---|---|---|
Processor | Intel E5-2650 [email protected] GHz (10 cores) | Container/VM OS | Ubuntu 18.04 LTS |
RAM | 256 GB | Docker container | v19.03.6 |
Storage | Intel 400 GB PCIe 3.0 x4 NVMe SSD | Kata containers | 1.12.0-alpha0 |
Operating system | Ubuntu 18.04 LTS | gVisor | release-20201208.0 |
Linux kernel | v5.4.0 | Firecracker | v0.21.0 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Lee, K.; Kim, J.; Kwon, I.-H.; Park, H.; Hong, C.-H. Impact of Secure Container Runtimes on File I/O Performance in Edge Computing. Appl. Sci. 2023, 13, 13329. https://doi.org/10.3390/app132413329
Lee K, Kim J, Kwon I-H, Park H, Hong C-H. Impact of Secure Container Runtimes on File I/O Performance in Edge Computing. Applied Sciences. 2023; 13(24):13329. https://doi.org/10.3390/app132413329
Chicago/Turabian StyleLee, Kyungwoon, Jeongsu Kim, Ik-Hyeon Kwon, Hyunchan Park, and Cheol-Ho Hong. 2023. "Impact of Secure Container Runtimes on File I/O Performance in Edge Computing" Applied Sciences 13, no. 24: 13329. https://doi.org/10.3390/app132413329