1. Introduction
The inclusion of communications on computer systems has allowed for new dynamics between devices and users, the deployment of new services, and the demand for new features. The constant evolution of hardware and communication systems enables new kinds of technologies that increase the productivity of traditional systems such as the Internet, education, surveillance, monitoring systems, networks, and others.
Therefore, there are more wireless technologies embedded in a mobile device and supporting different cellular networks such as 2G (GSM CSD, GPRS); 3G (UMTS/HSDPA); 4G (WiMAX, LTE, LTE-A (HSPA + LTE)); and 5G technologies. These rely on a central base station and a set of cells to provide services and coverage. However, these systems are prepared to support short-range communications that have rapidly evolved by reducing the size and increasing the computing capacity of microchips, allowing for overlapping networks within the same device. Two technologies are outstanding in that evolution: Bluetooth and IEEE802.11.x. When using wireless mobile devices, it is possible to build networks known as MANET (Mobile Ad-hoc Networks), which can be configured autonomously, do not require centralized control, and can automatically recover in case of failure. An alternative to exploit the features offered by MANET is to use mobile clouds [
1] as a platform to sharing resources in a cluster lead to take advantage to MANET features.
The massive number of users and services linked together brings new challenges due to the dynamic and stochastic behavior of distributed systems such as the Internet. Users consume applications and computing resources that are eventually shared on the network. The newly developed architectures provide users with Information Technologies services (IT), applications, platforms, information, monitoring, control systems, etc., to manage services and administer resources between the edge and the core network.
Currently, the devices and systems deployed on the edge layer can be classified in three types [
2]: Mobile Edge Computing (MEC), Fog Computing (FC) and Cloudlet Computing (CC). The MEC includes interactions with cellular networks and offers some cloud services in the cellular cell; the FC has a computing layer before the Cloud layer to store and process data; finally, the CC is deployed on dedicated devices with robust computing capacities, commonly called micro data centers.
The main contribution of this paper is the design and implementation of a distributed wireless operating system, dedicated to the dynamic management of computing resource in MANET devices [
3]. The goal is to find a solution to dynamically (re-)organize computation tasks on available network resources and node status. The operating system uses two elements to create and deploy virtual resources: abstraction and the primitives, which are managed by the Local Agent (LA) that resides as an instance on each physical node and the Orchestrator (OR) that work on one or more nodes to monitor and deploy policies on the distributed operating system.
This paper is organized as follows:
Section 2 presents a review of the existing solutions similar to the one proposed here.
Section 3 introduces the reference models adopted in the debate for the system architecture and application workload.
Section 4 describes the implemented prototype and the experimental demonstration of the proposed solution, showing its effectiveness. Finally,
Section 5 presents the final observation and future work.
3. S.O.V.O.R.A. Wireless Distributed Systems
One of the main problems within the architectures and structures presented so far is data management, i.e., how can you control and improve the operation of a system that has a distributed behavior? It is relevant to highlight that a distributed system operates at different stations. This means that the resources are in different machines and users are not aware of the interactions between them [
35]. These distributed characteristics bring a variety of challenges for information management and monitoring the entire status of the system. It is key to identify which services require information consistency, availability, and partitions tolerance [
36].
3.4. Network
In the proposed empirical model, the MANET was deployed with the B.A.T.M.A.N. protocol [
37] that works on an ad hoc mode in the IEEE 802.11 standard set on the wireless NIC. The self-configuration is based on the avahi-daemon for setting an IP address dynamically. The discovered node uses the ALFRED messaging service [
38] that publishes in the MANET and the information of each node as messages with an ID. Only nodes in the same cluster or ESSID can reach the messages on the network.
While each node has a label to link and register with the OR, the LA enables the ALFRED protocol to send information over the MANET (
Figure 4). The messages that pass between the nodes are through MANET control messages, while the ALFRED protocol messages are used as a json file with information about the node label, local states, available resources, application states, and network status. The information is collected by the local agent and sent to the OR to be analyzed and saved in its repository. The information collected is used to manage resources, applications, and tasks in the distributed wireless OS. The B.A.T.M.A.N. routing protocol is useful to reduce the latency and to know the location of the nodes according to the number of jumps between the nodes and the MAC address of the wireless NIC.
The second phase is controller-LA communication, as shown in
Figure 5. This communication is made from the LA to the distributed application; in this case, the application is located in a Docker container with the controller, which links the applications to monitor the performance. Thanks to this communication, it is also possible to adjust the operation speed of the applications, and in this sense, the number of resources used by the application. This scheme allows for monitoring at three levels: the hardware, the state of the mapped resources, and the applications deployed in the system.
The model proposed by S.O.V.O.R.A. is modular, despite being deployed on top of the operating system, and there is a hierarchy of layers (
Figure 3). The first modules are necessary for the implementation of the wireless ad hoc communication mode with the B.A.T.M.A.N. routing protocol. The node discovery module works on the ad hoc mode using the messaging service provided by ALFRED. For deploy applications, the Community Edition (CE) docker engine, and the micro-services architecture are used to create, deploy, and monitor applications. The modular model allows for the transition from edge networks to cloud computing.
The second layer is composed of three elements: the communication module that connects to the ad hoc kernel module and generates multi-thread servers and client services in the MANET; the load balancer that defines the abstractions of the control and data planes through the flow control strategies; and the local agent that manages the other two elements of the layer are present in each node (minimum one instance per node), and this also performs the execution and monitoring of tasks and service requests from users.
Monitoring local resources and application states is one of the basic services, and the OR executes this feature. The working layer is deployed in a single physical node for the control of the operating system (it is possible to deploy one or more ORs for MANET cluster). The last layer creates a simple distributed surveillance application with a monitoring service inside with some micro-services such as monitoring and web services deployed in docker containers.
The prototype contains two artifacts for its operation, the LA agent and the OR agent, both working over a MANET. Considering that the nodes can work as routers and clients, the artifacts use the Better Approach To Mobile Adhoc Networking (B.A.T.M.A.N.) as routing protocol, giving this model the ability to mitigate problems associated with coverage and hidden nodes, as well as allowing node discovering.
The LA acts with two roles: first, as a reactive agent, which executes the resources control policy, this model allows for controlling the applications to be deployed. In this case, each application is in a container and has a controller for its deployment. The model allows for having one or more local agents in one device to control resources and to monitor the applications. Second, as a deciding agent, the agent decides to apply the policies by sending them across the MANET. The local agent decides the amount of resources in the application container for the applications. The second element is the OR is an agent that can be deployed on unique node or each node on the network. It decides if an application is executed and continues the load balancing policy for the devices on the network. The agent has information about the global and local states of each device (CPU throughput, memory consumption, available bandwidth, network availability, and apps performance). This information is periodically received from local agents that inform the status of each node. In this environment, the OR model offers the possibility of monitoring different nodes and applications, allowing for the control of variables such as energy consumption, application distribution, and computing resources. These experiments only use one instance of an orchestrator agent due the limited physical nodes to manage the network (the experiment has six physical nodes).
The OR also allows for the control of distributed wireless operative systems and their interactions, e.g., communication between nodes, allocation, and assignment of computing resources on all cluster, distribution and management of throughput of the applications distributed on each node.
Author Contributions
Conceptualization, H.Z.C. and J.E.O.T.; methodology, H.Z.C.; software H.Z.C.; validation, J.E.O.T.; investigation, H.Z.C.; resources, H.Z.C. and J.E.O.T.; data curation, H.Z.C.; writing—original draft preparation, J.E.O.T. and H.Z.C.; writing—review and editing, J.E.O.T.; visualization, H.Z.C.; supervision, J.E.O.T.; project administration, H.Z.C. All authors have read and agreed to the published version of the manuscript.
Funding
This research received external funding from Systems and Industrial Department, Engineering Faculty, Universidad Nacional de Colombia-Bogotá.
Conflicts of Interest
The authors declare no conflict of interest.
References
- Fitzek, F.H.; Katz, M.D. Mobile Clouds: Exploiting Distributed Resources in Wireless, Mobile and Social Networks; John Wiley & Sons: Hoboken, NJ, USA, 2013. [Google Scholar]
- Dolui, K.; Datta, S.K. Comparison of Edge Computing Implementations: Fog Computing, Cloudlet and Mobile Edge Computing; Global Internet of Things Summit; IEEE: Dublin, Ireland, 2017; pp. 1–6. [Google Scholar]
- Ceballos, H.Z. Diseño de un Sub-Sistema de Cómputo Distribuido que permita implementar virtualización inalámbrica para gestionar recursos (Procesamiento, memoria, almacenamiento y dispositivos E/S) distribuidos en una Red Ad Hoc, mediante el modelo de pseudo Estado. Línea de Investigación: Computación Aplicada. Ph.D Thesis, Universidad Nacional de Colombia, Bogotá, Columbia, January 2018. [Google Scholar]
- Shahzadi, S.; Iqbal, M.; Dagiuklas, T.; Qayyum, Z.U. Multi-access edge computing: Open issues, challenges and future perspectives. J. Cloud Comput. 2017, 6, 30. [Google Scholar] [CrossRef]
- Bittencourt, L.F.; Diaz-Montes, J.; Buyya, R.; Rana, O.F.; Parashar, M. Mobility-aware application scheduling in fog computing. IEEE Cloud Comput. 2017, 4, 26–35. [Google Scholar] [CrossRef] [Green Version]
- Datla, D.; Chen, X.; Tsou, T.; Raghunandan, S.; Hasan, S.S.; Reed, J.H.; Dietrich, C.B.; Bose, T.; Fette, B.; Kim, J.H. Wireless distributed computing: A survey of research challenges. IEEE Commun. Mag. 2012, 50, 144–152. [Google Scholar] [CrossRef]
- Gao, M.; Addis, B.; Bouet, M.; Secci, S. Optimal Orchestration of Virtual Network Functions. arXiv 2017, arXiv:1706.04762. [Google Scholar] [CrossRef] [Green Version]
- Giotis, K.; Kryftis, Y.; Maglaris, V. Policy-Based Orchestration of NFV Services in Software-Defined Networks. In Proceedings of the Conference Network Softwarization, London, UK, 13–17 April 2015; pp. 1–5. [Google Scholar]
- Sun, C.; Bi, J.; Zheng, Z.; Hu, H. HYPER: A Hybrid High-Performance Framework for Network Function Virtualization. IEEE J. Sel. Areas Commun. 2017, 35, 2490–2500. [Google Scholar] [CrossRef]
- Santoro, D.; Zozin, D.; Pizzolli, D.; De Pellegrini, F.; Cretti, S. Foggy: A Platform for Workload Orchestration in a Fog Computing Environment. In Proceedings of the Cloud Computing Technology and Science, Sydney, Australia, 11–13 December 2017; pp. 231–234. [Google Scholar]
- Asnaghi, A.; Ferroni, M.; Santambrogio, M. DockerCap: A Software-Level Power Capping Orchestrator for Docker Containers. In Proceedings of the Computational Science and Engineering/Embedded and Ubiquitous Computing, Paris, France, 24–26 August 2016; pp. 90–97. [Google Scholar]
- Hui, F.; Mohapatra, P. Experimental Characterization of Multi-Hop Communications in Vehicular ad Hoc Network. In Proceedings of the 2nd ACM international workshop on Vehicular ad Hoc Networks, Cologne, Germany, 2 September 2005; pp. 85–86. [Google Scholar]
- Akyildiz, I.F.; Wang, X.; Wang, W. Wireless Mesh Networks: A Survey. Comput. Netw. 2005, 4, 445–487. [Google Scholar]
- Akyildiz, I.; Su, W.; Sankarasubramaniam, Y.; Cayirci, E. Wireless sensor networks: A survey. Comput. Netw. 2002, 38, 393–422. [Google Scholar] [CrossRef] [Green Version]
- Hu, P.; Dhelim, S.; Ning, H.; Qiu, T. Survey on fog computing: Architecture, key technologies, applications and open issues. J. Netw. Comput. Appl. 2017, 98, 27–42. [Google Scholar] [CrossRef]
- Bonomi, F.; Milito, R.; Zhu, J.; Addepalli, S. Fog Computing and Its Role in the Internet of Things. In Proceedings of the MCC Workshop on Mobile Cloud Computing, Helsinki, Finland, 17 August 2012; pp. 13–16. [Google Scholar]
- Bonomi, F.; Milito, R.; Natarajan, P.; Zhu, J. Fog Computing: A Platform for Internet of Things and Analytics. In Big Data and Internet of Things: A Roadmap for Smart Environments; Springer: Berlin/Heidelberg, Germany, 2014; pp. 169–186. [Google Scholar]
- Ning, H.; Li, Y.; Shi, F.; Yang, L.T. Heterogeneous edge computing open platforms and tools for internet of things. Future Gener. Comput. Syst. 2020, 106, 67–76. [Google Scholar] [CrossRef]
- Negash, B.; Rahmani, A.M.; Liljeberg, P.; Jantsch, A. Fog Computing Fundamentals in the Internet-of-Things. In Fog Computing in the Internet of Things; Springer: Berlin/Heidelberg, Germany, 2018; pp. 3–13. [Google Scholar]
- Mahmud, R.; Kotagiri, R.; Buyya, R. Fog computing: A taxonomy, survey and future directions. In Internet of Everything; Springer: Berlin/Heidelberg, Germany, 2018; pp. 103–130. [Google Scholar]
- Brogi, A.; Forti, S.; Ibrahim, A. How to Best Deploy YOUR FOG APPLICATIONS, Probably. In Proceedings of the International Conference Fog and Edge Computing, Valencia, Spain, 8–11 May 2017; pp. 105–114. [Google Scholar]
- Brogi, A.; Forti, S. QoS-Aware Deployment of IoT Applications Through the Fog. IEEE Internet Things J. 2017, 4, 1185–1192. [Google Scholar] [CrossRef] [Green Version]
- Sood, S.K. SNA based QoS and reliability in fog and cloud framework. World Wide Web. 2018, 6, 1601–1616. [Google Scholar] [CrossRef]
- Nan, Y.; Li, W.; Bao, W.; Delicato, F.C.; Pires, P.F.; Zomaya, A.Y. A dynamic tradeoff data processing framework for delay-sensitive applications in Cloud of Things systems. J. Parallel Distrib. Comput. 2018, 112, 53–66. [Google Scholar] [CrossRef]
- Santos, J.; Wauters, T.; Volckaert, B.; De Turck, F. Fog Computing: Enabling the Management and Orchestration of Smart City Applications in 5G Networks. Entropy 2017, 20, 4. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- De Sousa, N.F.S.; Perez, D.A.L.; Rosa, R.V.; Santos, M.A.; Rothenberg, C.E. Network Service Orchestration: A Survey. arXiv 2018, arXiv:1803.06596v4. [Google Scholar] [CrossRef] [Green Version]
- He, S.; Shin, H.S.; Xu, S.; Tsourdos, A. Distributed estimation over a low-cost sensor network: A review of state-of-the-art. Inf. Fusion 2020, 54, 21–43. [Google Scholar] [CrossRef]
- Zikria, Y.B.; Kim, S.W.; Hahm, O.; Afzal, M.K.; Aalsalem, M.Y. Internet of Things (IoT) Operating Systems Management: Opportunities, Challenges, and Solution. Sensors 2019, 8, 1793. [Google Scholar] [CrossRef] [Green Version]
- Baccelli, E.; Gündog˘an, C.; Hahm, O.; Kietzmann, P.; Lenders, M.S.; Petersen, H.; Schleiser, K.; Schmidt, T.C.; Wählisch, M. RIOT: An open source operating system for low-end embedded devices in the IoT. IEEE Internet Things J. 2018, 5, 4428–4440. [Google Scholar] [CrossRef]
- Al-Jaroodi, J.; Mohamed, N.; Jawhar, I.; Mahmoud, S. CoTWare: A Cloud of Things Middleware. In Proceedings of the 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW), Atlanta, GA, USA, 5–8 June 2017; pp. 214–219. [Google Scholar]
- Ullah, R.; Rehman, M.A.U.; Kim, B.S. Design and implementation of an open source framework and prototype for named data networking-based edge cloud computing system. IEEE Access 2019, 7, 57741–57759. [Google Scholar] [CrossRef]
- Benomar, Z.; Longo, F.; Merlino, G.; Puliafito, A. Enabling Container-Based Fog Computing with Openstack. In Proceedings of the 2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), Atlanta, GA, USA, 14–17 July 2019; pp. 1049–1056. [Google Scholar]
- Struhár, V.; Behnam, M.; Ashjaei, M.; Papadopoulos, A.V. Real-Time Containers: A Survey. In Proceedings of the 2nd Workshop on Fog Computing and the IoT (Fog-IoT 2020). Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Sydney, Australia, 21 April 2020. [Google Scholar]
- Li, J.; Jin, J.; Yuan, D.; Zhang, H. Virtual fog: A virtualization enabled fog computing framework for Internet of Things. IEEE Internet Things J. 2017, 5, 121–131. [Google Scholar] [CrossRef]
- Haddad, S.; Kordon, F.; Pautet, L.; Petrucci, L. Distibuted Systems: Design and Algorithms; John Wiley & Sons: Hoboken, NJ, USA, 2013. [Google Scholar]
- Lynch, N.A. Distributed Algzorithms; Elsevier: Amsterdam, The Netherlands, 1996. [Google Scholar]
- Neumann, A.; Aichele, C.; Lindner, M.; Wunderlich, S. Better approach to mobile ad-hoc networking (BATMAN). IETF Draft 2008, 1–24. Available online: https://tools.ietf.org/pdf/draft-wunderlich-openmesh-manet-routing-00.pdf (accessed on 8 December 2020).
- Mesh, P. ALFRED. Available online: https://www.open-mesh.org/projects/alfred/wiki (accessed on 8 December 2020).
- Guthaus, M.R.; Ringenberg, J.S.; Ernst, D.; Austin, T.M.; Mudge, T.; Brown, R.B. MiBench: A Free, Commercially Representative Embedded Benchmark Suite. In Proceedings of the International Workshop on Workload Characterization, Austin, TX, USA, 2 December 2001; pp. 3–14. [Google Scholar]
- Odroid, U.K. Smart Power 2. Available online: https://odroid.com/dokuwiki/doku.php?id=en:acc:smartpower2 (accessed on 8 December 2020).
- Ongaro, D.; Ousterhout, J. In search of an understandable consensus algorithm (extended version). Retrieved July 2016, 20, 2018. [Google Scholar]
Figure 1.
Orchestrator technologies [
26].
Figure 2.
Local Agent-Orchestrator Model.
Figure 3.
S.O.V.O.R.A. proposed Model.
Figure 4.
Local Agent-Orchestrator Communication.
Figure 5.
Local Agent Applications Communication.
Figure 6.
Experiment Model.
Figure 7.
Experiment Model.
Figure 8.
Orchestrator Info.
Figure 9.
Local Agent Log.
Figure 10.
Orchestrator Log.
Figure 11.
Odroid power consumption.
Figure 12.
Application performance over Odroid node 0.
Figure 13.
Application performance over each node.
Figure 14.
RPI node 2 power consumption.
Figure 15.
RPI node 1 power consumption.
Figure 16.
RPI node 2 power consumption.
Figure 17.
RPI node 3 power consumption.
Figure 18.
Throughput for node; workload controlled by OR.
Table 1.
S.O.V.O.R.A. terminal commands.
Local Agent |
is_alive()[options-node, ip] | Verify the nodes on line and linked with SOVORA |
node()[options-node,ip] | show information about the node resources, IP and UUID |
process()[options-node,ip] | Show the process running on execution time on local node |
Orchestrator |
sstatus [options][ip, port] | Shows the system status |
nstatus [options][ip, port] | Show the MANET status |
SOVORA |
server [options][ip, port] | Create a multi thread server on specific node |
global()[options-ssid] | Shows the node member on the MANET cluster |
org()[options-ssid] | Shows the network organization |
gglobal()[options-nodes] | Create a communication group between local agents |
disk()[options-app] | Provision disk space |
cpu()[options-app] | Provision a thread |
mem()[options-app] | Provision distributed memory |
io()[options-app] | Provision I/O devices connection |
pidg()[options-nodes] | Shows the process on all systems nodes |
dprocess()[options-node,ip] | Create a process on specific node |
mess()[options] | Create a broadcast message |
clock()[options] | Set the systems clock |
Table 2.
Board information.
Device | Architecture | Processor Fam. | Chip | Cores | Arm_Freq (MHz) | Gpu_Freq (MHz) | Mem (MB) | Wifi_Chip |
---|
RPi 3 | ARM | ARM Cortex-53 | Broadcom BCM2837 | 4 | 1200 | 400 | 1024 | CypressCYW43438 |
RPi zero w | ARM | ARM11 | Broadcom BCM2835 | 1 | 1000 | 400 | 512 | CypressCYW43438 |
Rpi 2 | ARM | ARM-Cortex-A7 | Broadcom BCM2836 | 4 | 900 | 250 | 1024 | External |
Odroid | ARM | ARMV8 | Samsung Exynos5422 | 8 | 2000 | 533–295 | 2048 | External |
Table 3.
Selected applications for the use case.
Application | Algorithm | Resources |
---|
Security | SHA | CPU, Mem, Bandwidth |
Image Processing | JPEG | CPU, Mem, Bandwidth |
Audio | ADPCM | CPU, Mem, Bandwidth |
Image Processing | SUSAN | CPU, Mem, Bandwidth |
Table 4.
Test Ad hoc-batctl/batman-adv RPi 3.
RPi_3batctl | 14 MB Rpi_Zero | 14 MB Odroid |
---|
%CPU | 100.00 | 100.00 |
Time (s) | 10.19 | 10.54 |
Power (mW) | 2211.88 | 2230 |
Energy (J) | 2211.88 | 2230 |
Throughput (MB/s) | 0.8893 | 1.7 |
MB Send | 7.605 | 18.5 |
Table 5.
Power Consumption RPI 3 + Wifi.
POWER + WIFI RPi 3 |
---|
APP | IDLE | 50%THR_MIN | 75%THR_MAX | THR_MAX |
---|
SHA | 1689.95 | 1839.75 | 1938.41 | 1991.14 |
ADPCM | 1689.95 | 1856.52 | 1906.49 | 1995.51 |
SUSAN | 1689.95 | 1850.2 | 1902.81 | 1942.54 |
JPEG | 1689.95 | 1852.39 | 1905.52 | 1929.54 |
| Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).