Approach for Designing Real-Time IoT Systems

Abstract

Along with the rapid development of Internet of Things (IoT) technology over the past few years, opportunities for its implementation in service areas that require real-time requirements have begun to be recognized. In this regard, one of the most important criteria is to maintain Quality of Service (QoS) parameters at an appropriate and sufficiently high level. The QoS level should ensure the delivery of data packets in the shortest time possible while preventing critical parameters relevant to real-time transmission from being exceeded. This article proposes a new methodology for designing real-time IoT systems. The premise of the proposed approach is to adapt selected solutions used in other types of systems working with real-time requirements. Some analogy to embedded systems with a distributed architecture has been noted and used in this regard. The main differences from the concept of built-in systems can primarily be seen in the communication layer. The methodology proposed in this article is based on the authors’ proposed model of real-time system functional specification and its mapping to the IoT architecture. In addition, the developed methodology makes extensive use of selected IoT architecture elements described in this article, as well as selected task scheduling methods and communication protocols. The proposed methodology for designing RTIoT systems is based on dedicated transmission serialization methods and dedicated routing protocols. These methods ensure that the time constraints for the assumed bandwidth of IoT links are met by appropriately prioritizing transmissions and determining communication routes. The presented approach can be used to design a broad class of RTIoT systems.

1. Introduction

For a number of years, there has been a continuous and rapid development of products and services in the area of Internet of Things (IoT) technology [1,2,3,4]. Research in this arena is focused on introducing new solutions in the form of dedicated, smart, specialized and autonomous systems [5,6,7,8]. The main reason for the widespread use of IoT solutions is usually the desire to improve people’s lifestyle and comfort, the optimal use of natural resources, and the low energy consumption of these solutions [9,10,11,12]. IoT solutions in this area often provide previously unattainable levels of efficiency while minimizing the contribution of the human factor. This trend is being forced by the ever-increasing demands of profit optimization in all industries and economies, as well as the quality and safety of life demands of specific groups of consumers and even entire global communities [13].

The most up-to-date literature describes numerous examples of implementations of classical IoT solutions for various purposes. As is well known, the so-called “smart” systems, e.g., smart homes, smart factories, smart cities, and smart grids, have gained great popularity [14,15]. In addition, a number of systems have been developed for the medical [16], automotive [17], agricultural [18], property protection [19], logistics [20], and sports and recreation [21] industries, as well as many other areas related to the immediate needs of the public. Smart systems for managing amusement parks [7,8,22]; automated customer service solutions based on speech recognition techniques and voice assistants [23]; facial recognition and motion detection systems used for security and access control [24]; smart systems for managing [25,26], controlling [27], and optimizing production processes [28,29,30]; and real-time object location systems [12,31,32] are being implemented. The aforementioned solutions are just a few examples of the known applications of IoT technology. The possibilities of implementing traditional solutions are very wide and mainly depend on the needs of the recipients and the ingenuity of their designers, thus influencing the development of a certain area of the service market.

At the same time, it should be noted that services and products where real-time transmission requirements play a key role are still major constraints in the implementation of IoT solutions. The need to design real-time applications for IoT (RTIoT) solutions was recognized more than 10 years ago [33]. The first technologies and standards in this area also began to emerge at that time. Examples include: ITCC (Intel Time Coordinated Computing) [34,35], TSN (Time Sensitive Network) [36], M2M (Machine to Machine Communication) [37], SaaS (Sensing as a Service) [38], and a multi-layered reference model dedicated to IoE (Internet of Everything) technologies [3]. However, most work to date in the field of RTIoT systems has only been concerned with trying to develop selected elements of the IoT architecture [39] or applications and technologies [40,41] optimized for the highest possible QoS (Quality of Service). These mostly deal with issues related to real-time transmission and protocols, server applications, or real-time databases. As previously mentioned, there was a need for real-time transmission early in the development of the internet, mainly for the purpose of transmitting multimedia data. The RTP/RSTP [42] or XMPP [43] protocols in use at the time did not allow for the parameters associated with time constraints to be met. It was not until the development of the IEEE 802.1 standard in the form of the TSN solution that some conditions were created for the development of data transmission methods that considered time parameters. However, this standard only defines basic mechanisms for time synchronization, transmission scheduling or resource reservation. It does not specify specific communication protocols or algorithms for scheduling and routing data transmission [44]. In terms of server applications, there is ongoing work on cloud computing. This work focuses on developing real-time cloud services, scheduling tasks according to real-time requirements, and developing web servers that handle requests with time constraints [45,46,47]. Real-time databases, on the other hand, define real time as the fastest execution of queries to a database. Examples of such databases are VoltDB [48] and Firebase [49].

Although it is possible to find numerous publications presenting examples of IoT applications running in real time, these solutions are not fully compatible with the concept of a real-time system. Typically, these solutions are highly optimized in terms of data processing and data transmission performance so the time requirements under typical operating conditions are met. However, such solutions do not provide the required QoS level when operating conditions change (e.g., with higher network load). Moreover, providing high performance is often associated with significant costs of computing and communication resources. Additionally, the methodologies of designing RTIoT systems that take cost and QoS optimization into account are unknown. Constraints related to real-time requirements are currently one of the main challenges faced by IoT system research, and they were also the main motivation for the authors of this study to carry out the presented research.

The purpose of this work was to identify the problems and needs that exist in the design of RTIoT systems, to identify research directions to solve these problems, and to propose a design methodology that considers the requirements of RTIoT systems. The contribution to the body of knowledge of this article is:

• The proposal of a methodology for designing real-time IoT systems considering the application of edge computing, fog computing, SDN (Software-Defined Networking), and NFV (Network Function Virtualization) concepts.
• An overview of the components, methods and technologies of distributed real-time systems, with particular emphasis on scheduling methods and communication protocols in the context of their applications in RTIoT systems.
• The proposal of a general RTIoT system architecture covering a broad class of applications.

The starting point for the RTIoT system design methodology proposed in this article is an analysis of approaches used in other classes of real-time systems. In particular, one can see a certain analogy with embedded systems, or rather, IoT systems can be seen as a certain generalization or development of the concept of embedded systems with a distributed architecture. The differences are particularly evident in the communication layer. This article presents a proposed methodology based on analogous approaches to the synthesis of embedded systems.

The rest of this article is organized as follows. Section 2 describes the functional specification model and synthesis process for real-time systems, and Section 3 considers selected components of IoT systems in relation to real-time requirements. Section 4 is devoted to an analysis of task scheduling methods and communication protocols possible to apply in RTIoT systems, and Section 5 describes the new methodology for designing RTIoT systems proposed by the authors. The summary and conclusions are provided in Section 6.

2. Functional Specification Model and Synthesis of Real-Time Systems

The design methodology for complex IT systems usually consists of two stages:

• Developing a high-level model: The system model allows for an analysis to validate and optimize the proposed solution. This is particularly important in the case of distributed systems, where the verification of the communication and synchronization mechanisms used between processes is of significance.
• Mapping the model to the target architecture: It is possible to map to a standard architecture (e.g., a multi-core processor) or to synthesize a specialized architecture optimized for a given application.

2.1. IoT Application System Model

The first stage of design is to create a system model of the application. The system model represents the application function at the highest abstraction level, where tasks (processes) and interactions between tasks are distinguished. At this level, what a task does is not as important as when it is activated, what the synchronization between tasks is, and how the tasks communicate with each other. Typically, system models are graph models. Existing methods use various system models. We can distinguish, for example:

• Task Graph (TG): one of the simplest and most popular methods of representing functions at the system level. It is a directed acyclic graph in which nodes represent tasks and edges represent the order in which tasks are performed (usually representing transmissions). The task is activated when all preceding ones are completed. In this way, sequential dependencies between tasks are presented. Transmission volumes are represented by edge weights. A sample task graph is shown in Figure 1. The graph describes a six-task system. Tasks on the same path in the graph are sequentially executed, while tasks from parallel paths can be executed in parallel. Extended versions of task graphs can also be found in the literature, e.g., conditional task graphs and multimodal task graphs [50,51]. Such models allow for the specification of special cases, such as the conditional or alternative performance of certain tasks.

• SDF (Synchronous Data Flow) or SSDF (Statically Schedulable Data Flow): models representing data flow (often used in modeling telecommunications applications). Like the TG, SDF is a directed graph. Unlike the TG, cycles can occur in SDF. The synchronization mechanism is described by determining the number of tags generated by the execution of a given task (for each output edge). The number of tags taken from each input is also specified for each task. The task is activated if the required number of tags is available on all inputs.
• STATECHARTS: a model based on a FSM (Finite State Machine), models based on a description in the form of various forms of automation enable the specification of control flow. Unlike a classic FSM, state charts enable parallel descriptions and task specifications. Tasks are related to transitions between states.

There are also a number of other methods that often target specific classes of applications. The system model should not impose any constraints on the target system architecture if this is not directly implied by the design assumptions. The architecture should be the result of system optimization. Therefore, the first step in the methodology should be to propose a method for specifying the application at the system level. This model will be the starting point for further design (scheduling, mapping into architecture, and optimization). Therefore, it should contain all the information so that the above tasks can be performed. Each method has limitations (e.g., the task graph offers no way of describing the control functionality). In the case of RTIoT, the model should allow for the specification of both computational and transmission tasks (communication tasks). The model should also target so-called reactive systems, i.e., systems in which particular sequences of actions/tasks are activated as a result of reactions to specific events.

2.2. Mapping Functional Specifications to Real-Time IoT Architecture

In a typical approach to mapping a system specification into a distributed information system architecture, three main steps are performed [52]:

• Resource allocation: At this stage, the target hardware architecture of the system is determined. Resources can be processors, specialized hardware modules, and communication channels (buses, communication processors, etc.). At the same time as the allocation, the connections between system components are determined.
• Assignment of tasks to resources: This step involves assigning individual tasks to resources. Tasks are assigned to computing modules, and transmissions are assigned to communication channels. Task allocation is closely related to resource allocation. For specialized resources, only tasks corresponding to the function performed by the resource can be assigned. Transmissions must only be assigned to communication channels between resources with assigned communicating tasks. When two tasks are assigned to the same resource, transmissions between them are ignored.
• Task prioritization: Task scheduling is necessary to determine the order in which tasks are performed when more than one task is assigned to a resource. Scheduling must consider the sequential relationships between tasks. During scheduling, speed optimization is performed. In the case of real-time systems, the main goal is to arrange the execution of tasks and transmissions in such a way that all timing constraints are met (hard-constrained systems) or that timing constraint overruns are minimized (QoS maximization).

IoT systems are characterized by a specific architecture, and for this reason, it is not possible to directly apply existing methods of synthesizing real-time systems to IoT applications. The main distinguishing features of these systems are:

• A distributed architecture based on internet infrastructure.
• Communication through internet links, thus not ensuring that time constraints are met.

From the above-mentioned properties of IoT systems comes the need to develop a specialized method for mapping specifications to the target architecture, as well as the need to develop solutions to ensure the implementation of transmission that meets real-time requirements.

3. Selected Elements of IoT System Architecture

There are many proposals for IoT system architectures. In order for a given architecture to enable the implementation of a real-time system, it must meet certain requirements in terms of the predictability of operation and the possibility of optimization in terms of response time. This section presents an overview of the various elements of the IoT architecture, with a focus on how they can be used in real-time systems.

3.1. Edge and Fog Computing

Today, the amount of data that are generated by IoT systems and then transmitted over the network is increasing at a very high rate. Therefore, it is important to use the throughput of IoT systems as efficiently as possible. The concepts of edge computing and fog computing [53] can be used to optimize this problem. One of the main features of edge computing is the clear separation of the physical devices that make up the edge infrastructure from the core network devices [54]. Edge devices can perform data preprocessing functions at the network edges, which significantly improves the ability to maintain the required transmission time parameters in the context of real-time systems. If the tasks performed by edge devices are subject to time constraints, then these devices are implemented as real-time embedded systems enabling real-time data processing [55,56]. Therefore, it is becoming increasingly desirable to use edge computing for smart data processing and performing specific optimization tasks [57,58]. The partial offloading of centralized cloud systems is also possible through fog computing mechanisms [59]. Solutions of this class can perform services by computation at the level of a distributed, decentralized network infrastructure (e.g., using local IoT networks/systems) [60]. Fog computing and edge computing also provide some opportunity to apply function virtualization techniques on a massive scale [61,62,63,64,65], which consequently reduces the demand for resources and computing power of a data center. Both of the solutions under consideration significantly reduce the amount of data that must be transmitted and processed by the core network/server infrastructure, thus relieving the burden on traditional cloud computing and data center systems. If the target resources are less loaded, then it is easier to meet the assumptions that guarantee throughput and latency at levels appropriate for real-time solutions [66]. The primary purpose of edge infrastructure and the use of distributed computing is to solve a number of critical tasks affecting the latency, throughput, energy consumption, and scalability of IoT systems [67]. Moreover, these solutions allow for a high degree of autonomy in managing the optimal configuration of a system [68,69] and increasing the level of security [70]. All of the above aspects favorably affect the ability to implement IoT services with time requirements.

3.2. Programmable Networks and Virtualization Techniques

In the context of real-time IoT tasks, new developments in Software-Defined Networking (SDN) and mass-scale Network Function Virtualization (NFV) methods are important elements to consider. SDN networks are commonly associated with IoT technology because the architecture provides a more flexible and manageable environment. In addition, more and more research (for example, on blockchain) is focused on implementing solutions that combine SDN and IoT technologies with NFV methods. Therefore, it is anticipated that NFV will be a very important tool to support the automation of network solutions, including in real-time transmission issues.

In SDN, management tasks are separated from data transmission tasks. In this approach, network devices solely act as relays that transmit sequentially routed packets from the source to the destination, with all policies in place for both communication and security [71]. The management space in SDN is a highly centralized area, while the data space operates on a distributed plane. Potential SDN applications related to real-time transmission are shown in [72], where a four-tier cloudlet architecture (based on SDN) in combination with a WBAN (Wireless Body Area Network) architecture was described. The paper described a cloudlet—a mobile, cloud-based data center system that is located at the edge of the network. This approach was shown to enable efficient traffic processing, optimal resource utilization, and reduced latency, which is of great importance for real-time systems. The approach proposes that all network controllers be located in a cloudlet, resulting in better optimization in terms of QoS mechanisms, as well as a complete view of the network. On the other hand, the authors of [73] proposed the concept of a three-layer SDN application based on a WBAN scheme called an SDWBAN (Software-Defined Wireless Body Area Network). In this solution, network traffic prioritization is performed. Higher-priority packets are analyzed and sent through the network first, which is made possible by optimal resource allocation (e.g., throughput), path allocation and the use of cloud computing. In this case, transmission latency is reduced by formulating a heuristic allocation model for minimizing the service time of individual services. Security issues for IoT systems are also an important issue. In this area, solutions using SDN are known, such as artificial intelligence mechanisms that automatically analyze network traffic while taking real-time transmission requirements into account [74].

In today’s vast IT world, technology is beginning to play an increasingly important role in the move away from physical infrastructure building and solutions based on dedicated physical network devices to virtualization technology. An example of virtualization technology in relation to the IoT area is the NFV solution. The implementation of virtualization technology in IoT solutions provides flexibility for applications regardless of the platform on which this solution is based or built while maintaining the full management of logical network functions and the modeling of the full technology stack. An additional advantage of implementing NFV technology in IoT solutions is the ability to fully manage and configure individual virtualized network elements at all levels (from physical machines to virtualized machines to sublime network services) [75]. From the point of view of the IoT and real-time requirements, the use of NFV virtualization technology has begun to play a key role due to the requirements of real-time transmission, i.e., transmission with the shortest possible packet delivery and processing time while maintaining an appropriate QoS and meeting the deadline value requirements of individual packets [76]. The biggest advantage of virtualization technology is its versatility in application to various IoT architectures, although due to its definition and similar mechanisms, NFV technology perfectly complements IoT solutions based on the SDN model architecture.

3.3. Real-Time Database Systems

To ensure the best possible QoS performance in real-time transmission, the solutions described in the previous sections are crucial from the points of view of hardware technologies and transmission and data processing concepts. At the same time, as the data volume generated by IoT systems continues to grow, there is also a steady increase in the need for solutions to store that data while supporting mechanisms that consider real-time requirements [77]. One of the directions is the concept “fast data”, which involves developing database technology to ensure the fastest possible execution of queries. Examples of such databases are Druid [78] and Volt Active Data [79].

Some of the most well-known approaches for storing large datasets are SDDS (Scalable Distributed Data Structure) structures, which enable the virtualization of distributed memory by aggregating resources at network nodes [80]. Considering real-time requirements and the need for QoS parameters, an extension of this concept in this direction are SDDS LH/RP (Scalable Distributed Data Structure Linear Hashing/Range Partitioning) approaches dedicated to real-time cloud applications [81]. An SDDS server schedules queries using real-time queuing methods that consider importance attributes, which enables the achievement of a higher level of execution of queries in a certain time in the case of SDDS LH/RP approaches than in the case of FIFO (First In, First Out) queuing approaches [82]. With such an implementation, QoS metrics in real-time IoT systems can be significantly improved [83]. The steady increase in demand for big data processing environments has led to the development of more dedicated data storage models, such as NoSQL [84] and NewSQL [85]. These IoT system solutions are beginning to gain value from the point of view of time requirements due to the need to address the processing of large volumes of data while maintaining QoS parameters.

In terms of real-time database systems, issues remain to be addressed in terms of ensuring that hard real-time policies are met for queries that update data that are already recorded/stored and for bucket splitting operations [86,87].

4. Task/Transmission Scheduling and Communication Protocols

The real-time system design methodology proposed in this paper assumes the appropriate scheduling of calculations and transmissions. In classic system solutions, scheduling can be implemented through FIFO queuing or other static or dynamic scheduling methods [88]. Static scheduling methods are mainly concerned with the design of embedded systems, which are most often used to control various devices when there are time constraints. These systems feature a dedicated architecture and perform predetermined functions. Therefore, it is possible to estimate the execution time of each function and to perform a ranking, ensuring that all time constraints are met. IoT systems can be viewed as a generalization of embedded systems with a distributed architecture. The main difference is the communication between system components. In the case of embedded systems, the components are directly connected and the transmission time is predetermined. In the case of IoT systems, transmission time also depends on other factors, e.g., on the traffic and throughput of links and on the route determined by routing processes. Thus, it is not possible to directly apply known static task scheduling methods to IoT systems, though these methods can be considered from the point of view of ensuring optimal cooperation with relevant communication protocols. Therefore, this section examines the scheduling methods and selected communication protocols used in data transmission tasks that assume the need to meet real-time assumptions.

4.1. FIFO

Depending on the application, FIFO queuing mechanisms can be implemented as a hardware sliding register or using different memory structures. Such solutions often use network devices such as switches or routers. These devices queue data packets as they travel from source to destination. Typically, one FIFO structure is used per connection [89]. Some network devices also have multiple FIFO structures/implementations for the simultaneous and independent queuing of different types of data. Most software implementations of FIFO queuing do not meet security conditions and require an additional security mechanism to verify the chain of data structures [90]. Achieving good performance in terms of throughput and scalability while maintaining a high level of security in IoT systems has always been a major challenge. The packet queuing process strategy represented by the FIFO algorithm contributed to the later development of algorithms such as SAGA-PBFT (Security-Aware Genetic Algorithm-based Practical Byzantine Fault Tolerance), which was developed on the basis of the PBFT (Practical Byzantine Fault Tolerance) algorithm. The SAGA-PBFT algorithm is used in solutions such as the block chain, where security aspects and transmission time requirements are extremely important [91].

4.2. Static and Dynamic Task Scheduling Methods in Real-Time Systems

Static task scheduling algorithms assume that before the target scheduling process, a set of input tasks is processed according to one of the specified sorting rules. These rules define how the input task array is sorted. With this approach, it is possible to determine which packages, criteria, or parameters in the IoT architecture will have the highest priority at the outset. Static task scheduling algorithms include LS (List Scheduling) mechanisms. In this group, we can distinguish the following sorting rules: LPT (Longest Processing Time), SPT (Shortest Processing Time), and RPT (Reverse Processing Time). In this case, the drawback of using list scheduling in real-time solutions is the additional overhead of executing the processes implemented in the sorting rules before the data packet is forwarded to the next network layers. This adversely affects packet delivery times, so meeting the goals of RTIoT solutions is difficult to implement [92]. Another popular static task scheduling algorithm is the Rate Monotonic Scheduling (RMS) method. This algorithm assigns priorities to individual tasks before they begin execution that do not change during the algorithm’s run time. Priorities are assigned to each task by taking the frequency of their appearance into account (the task with the shortest period will have the highest priority). RMS is typically used to schedule single-processor tasks [93], which is a major limitation of it for the design of extended RTIoT solutions. On the other hand, its expropriation feature (the highest priority task is performed) is decent in terms of maintaining the best possible QoS performance and meeting real-time conditions.

For dynamic scheduling in the context of the research presented in this article, it is worth considering the EDF (Earliest Deadline First) and LLF (Least Laxity First) algorithms. Real-time systems distinguish between two types of input data. The first type comprises data generated over certain fixed periods of time (e.g., traffic resulting from the functionality of the communication protocols used), and the second type comprises data generated as a result of actual events (e.g., data from various types of sensors). In IoT systems, which deal with data generated as a result of interactions taking place, the EDF algorithm [46] may find application in scheduling processes. In this solution, the value referred to as the deadline for each task is taken as the priority value, so the lower it is, the closer the deadline for a given task is. An additional advantage of the EDF algorithm over others is its good utilization of CPU resources and fairly fast response times, which, from the point of view of designing real-time IoT systems, can have key roles in ensuring adequate QoS parameters and other real-time requirements. Another example of a dynamic scheduling algorithm that can find application in a real-time IoT system is the LLF algorithm. It uses the deadline value, which, along with the time left to complete a given task, is used to calculate the laxity parameter used to prioritize the execution of individual tasks. In this case, the lower the value of laxity, the higher priority the task will be and the faster it should be completed [94]. A certain disadvantage of this algorithm is when there are many tasks with the same or similar laxity value simultaneously, the demand for computing power increases [95]. This represents a kind of limitation from the point of view of applications for RTIoT systems, since one of the priorities of IoT systems is to optimize their energy intensity [96]. On the other hand, however, the advantages of this algorithm include resistance to overloading, to which systems using the EDF algorithm are prone, which translates into the possibility of using it in the design and implementation of real-time systems [83]. Considering the above, it can be concluded that both EDF and LLF can be used for certain tasks in the design and implementation of RTIoT solutions.

4.3. Communication Protocols

In real-time IoT systems, in addition to the described task scheduling techniques, the use of appropriate types of transmission media and the implementation of proper communication protocols, with particular emphasis on routing protocols, also play important roles. Some well-known routing protocols to consider when designing RTIoT solutions are: RSTP (Real Time Streaming Protocol), WebRTC (Web Real Time Communication), XMPP (Extensible Messaging and Presence Protocol), MQTT (Message Queue Telemetry Transport), CoAP (Constrained Application Protocol), WebSocket, and 6LoWPAN (IPv6 over Low-Power Wireless Personal Area Networks).

The RSTP is an application layer protocol for the control, transmission and delivery of data while maintaining real-time transmission criteria. The most common implementations of this protocol are systems used to transmit voice and video in real time. Data are continuously delivered, even when RTSP requests are not received by the server. A new area in which this protocol is being applied is the technology area of the teaching of language, which includes listening, speaking, reading and writing [97]. Another protocol implemented in real-time IoT systems is WebRTC. Its most common applications are solutions for transmitting video images or streaming data in the broadest sense while achieving the lowest possible levels of latency and data transmission loss, as exemplified by applications for voice transmission and videoconferencing. Research [98] conducted in the area of the WebRTC protocol indicates that it yields better response times and connection stabilization times in web applications compared with other streaming protocols such as the RTSP [99]. In addition, from the point of view of QoS parameters and energy consumption, the implementation of the WebRTC protocol allows one to achieve better values, which translates into meeting the real-time transmission requirements of solutions implemented in IoT systems. The XMPP protocol, on the other hand, is based on XML [100]. The protocol is commonly available and can be freely modified, and many XMPP servers are available [101]. This translates into its main use being implementations in systems and applications for exchanging text-based content such as popular instant messaging applications. The simplest and lightest protocol for real-time data transmission is the MQTT protocol, which is based on the publish/subscribe pattern. It is designed for transmission for devices that do not require a high throughput. By limiting the transmission speed, the protocol provides a greater reliability. The protocol is perfect for IoT machine-to-machine connections, mobile devices, and where throughput and energy savings are required. By using this protocol, one can easily transfer data during a conversation from one device to another and continue the conversation [102]. An example of a protocol that uses the UDP protocol for communication and is also applicable to RTIoT systems is the CoAP protocol. It has been optimized for constrained devices and networks (used in wireless sensor networks) [103]. The CoAP protocol extends the paradigm of ubiquitous web services in mobile applications to IoT and M2M solutions, which can then be developed using shared and reusable RESTful web services. In doing so, this protocol considers IoT constraints and requirements or QoS requirements [104]. A two-way data exchange channel over a single TCP connection is provided by the WebSocket protocol. The protocol is designed to operate on ports 80 and 443, similarly to HTTP, allowing for interactions between a web browser (or other client application) and a web server at a lower load than alternatives while making it much easier to transfer real-time data to and from the server, which is an asset when considering its use in IoT systems [105]. Thanks to the previously mentioned interactions between the web browser and the server, the protocol has found applications in the development of online games, particularly [106]. With the development of technology, the need for IPv4 address allocation has gradually increased, which has contributed to basically exhausting the entire pool of this addressing. In response to the introduction of the new IPv6 addressing standard and the need to handle packets containing such addresses, the 6LowPAN routing protocol, which is characterized by a low power consumption, was created. The shortcomings of this protocol include the large size of the header relative to the data themselves; from the point of view of application considerations, the implementation of this protocol in RTIoT solutions can contribute to an increase in the level of latency or packet loss with a direct impact on the deterioration of QoS parameters. Today, further research is underway to address the challenges of the 6LowPAN protocol, which include fragmentation, header compression and security [107].

5. RTIoT System Design Methodology

The implementation of the functional model of the real-time system proposed in Section 2 requires the implementation of selected elements included in the IoT solution architecture. Typical IoT systems primarily consist of sensors providing data to the system, end devices (usually also equipped with sensors), a server application, and a client application [5]. The sensors can be directly connected to the server via internet links or can be an integral part of the end device. Indirectly, there may also be edge devices whose job is to pre-process data. Communication is usually carried out via low-energy technologies such as BLE (Bluetooth Low Energy) [108], ZigBee [109], and LoRaWAN (Long Range Wide Area Network) [110]. End devices using sensors communicate with an IoT system via internet links. They can also include actuators and controllers, so an IoT system can remotely control such devices. Server applications are usually installed in cloud computing or on a dedicated server, while the client applications used to operate the system can come in the form of mobile, desktop or web applications. All of the aforementioned components communicate using the relevant protocols.

In the case of real-time systems, there is a time limit on obtaining a response, which means that once a request is sent, a response is expected to be obtained within an assumed maximum time (deadline). Exceeding this time can be treated as an error (systems with hard constraints) or as a reduction in service quality (systems with soft constraints) [111]. Requests can be sent by either end devices or server or client applications. In order to reduce the occurrence of the aforementioned errors or to maintain the required level of quality of service in IoT systems, one can use the solutions described earlier, such as edge computing, fog computing, SDN, NFV, real-time databases, scheduling methods, and relevant communication protocols. These elements are described in Section 3 and Section 4 of this article, and based on them, an illustrative architecture for a real-time IoT system can be proposed, as shown in Figure 2. In the architecture, the following computing layers that can perform various tasks of the IoT system can be distinguished:

• Sensor and actuator layer (SL): the layer consisting the interface between an IoT system and its environment that enables the collection of data from the environment and the control of the elements of the environment.
• Edge layer (EL) (optional): an intermediate layer that enables the distributed processing of data without the need to send them to a central system server.
• Cloud layer (CL): the layer that contains the system’s servers and databases and that usually controls the operation of the entire system.
• User layer (UL): the layer that uses user applications (mobile, web or desktop). These applications allow a user to interact with the rest of the IoT system.

There are communication links between different layers that enable data transfer.

5.1. System Specifications

The starting point for designing an RTIoT system is a system specification in the form of an attributed set of task graphs. Each task graph describes one function of the system. The following attributes are defined for each graph:

• Maximum frequency of graph activation: This attribute specifies the maximum frequency of appearance of input events that cause the execution of functions represented by a given TG.
• Maximum number of TG instances: This attribute specifies the maximum number of simultaneous instances of the task graph. This corresponds to the maximum number of simultaneous events that cause the activation of the functions described by the TG.
• A set of time constraints: The time constraints are associated with the selected paths in the task graph and define the maximum time in which all tasks must be completed from the activation of the task that starts the path to the completion of the task that ends the path.
• For each task, an attribute is specified that assigns the task to a specific layer of the architecture. This attribute is defined by the designer.

Example task graphs are shown in Figure 3, Figure 4, Figure 5 and Figure 6. These graphs describe nine basic functions of a smart city system for managing parking spaces. Figure 3 shows a user-activated task graph for the following system functionalities: searching for the parking space closest to the user’s current location, the function of finding the user’s car in the parking lot based on their license plate number, the function of reserving any free parking space based on the entered search criteria, the function of charging a parking fee for the used parking lot, and the function of retrieving information on weather conditions.

For all system functionality, the starting point, or trigger (T0_0), is the end-user activity in a dedicated application made by selecting the appropriate function at the UL layer level. Depending on the selected function, a further sequence of events follows. When the user selects the parking space search function (T0_1), the user’s current location is retrieved using the GPS module (T0_4). If the car search function (T0_2) is selected, the user must enter the license plate number of the searched vehicle (T0_10) into the application. On the other hand, when the function of reserving a free parking space is selected from the user’s application (T0_03), the end user enters data/criteria for the search function, such as city and street (T0_12). The user also has the ability to download information on weather conditions (T0_17). In the next step (T0_5), the central server (a dedicated cloud application running in the CL layer) receives queries from the UL layer and starts the processes necessary to perform the called functions. For example, databases can be searched to find free parking spaces while considering information from the GPS tracker. A search for license plate numbers can be run, along with the retrieval of the vehicle location data assigned to them (parking/parking space number). Taking the user’s criteria into account, free parking spaces can be searched for, or the fee due can be calculated based on the time the parking space is occupied and the parking rate for the given parking space. The (T0_6) task runs functions in the user application responsible for receiving and processing data coming from the CL layer. In the next step, the user, based on the generated data, selects a parking space by activating the “Reserve” function (T0_7); in turn, for the process of searching for a car based on the entered license plate number (T0_10), the end result is the presentation of the results of the query processing in the end-user application (T0_11) or the start of the payment process using a debit/credit card or bank transfer (T0_14). It is also possible to visualize data on weather conditions (T0_18). Once the task is completed (T0_7), the server-level procedure for processing a parking space reservation request (T0_8) is started, and the state of the sensor is changed; this is realized, for example, by changing the color of the marking of a given space from green to red (T0_9). The task (T0_14) is followed by the payment process through data exchange with the bank/payment card architecture (T0_15). At the end of this path, the result is received in the form of information on the payment in progress (T0_16).

Figure 4 shows a task graph for the functionality of monitoring and updating the status of individual parking spaces.

The starting point/trigger for system function aimed at monitoring or managing parking spaces is to register a change in sensor state/status (T1_0) at the sensor (sensor) layer level. Then, via the network infrastructure, such information is transmitted to the servers that make up the parking system (T1_1). In the next step (T1_2), the central server (a dedicated cloud application running at the CL layer) receives queries from the EL layer and runs the processes necessary to execute the called functions—in this case, a function to update the database with the latest parking sensor states/statuses. In order for the process to be considered successful, the end result at the edge layer is to run a procedure that receives a confirmation of the completion of cloud processing for the database update (T1_3).

Figure 5 presents a task graph for the following functionalities: the function of reserving an available parking space for given time criteria (T2_1) and the function of plotting a route leading to a vehicle in the parking lot (T2_2). As for the task graph in Figure 3, the trigger for these functionalities is also the end-user activity of selecting the appropriate function from the user application (T2_0). Depending on the user’s selection of a specific function, they may be prompted in the next step of the process to provide the data necessary to perform the search function (T2_3). If the function of routing to a vehicle is implemented, the results of calling the cloud processing presentation function may be displayed in the user’s application (T2_9). In the next step (T2_4) the central server (a dedicated cloud application running in the CL layer) receives queries from the UL layer and starts the processes necessary to perform the called functions. For example, a database can be searched for free parking spaces given the criteria set by the user or a process can be run to search the database to determine the location of a vehicle with a specific license plate number. Then, the function responsible for receiving and processing data coming from the CL layer (T2_5) is run in the user’s application. In the next step, based on the generated data, the user selects a parking space by running the reservation function (T2_6). Alternatively, from the user’s application level, a function for the presentation of the results of cloud query processing (T2_9) is run. After the task (T2_6) is completed, the procedure for processing a parking reservation request (T2_7) is started from the EL layer. The next step could be a task to start the navigation function in the user’s application interface (T2_10). Finally, a change in the state of the sensor is made; this can be visualized, for example, by changing the color of the marking of the given parking space from green to red (T2_8). For another process, it could be running a function designed to retrieve current position data from a GPS module.

The last graph shows the tasks for the functionality of reading and processing information on the value of the outdoor temperature, as well as reading and processing information on the state of atmospheric pressure.

Similar to the task graph shown in Figure 4, the tasks described for the problem of collecting/updating data on weather conditions also have sensor readings (T3_0) as the starting point/trigger for system operation. Then, through the network infrastructure, the temperature (T3_1) or the atmospheric pressure (T3_2) information is transmitted to the servers that make up the CL layer level of the system. In the next step (T3_3), the central server receives queries from the EL layer and starts the processes necessary to perform the called functions. Processes can be considered successfully completed when the end result at the EL layer is the launch of a procedure that receives confirmation of the completion of cloud processing for database updates (T3_4).

5.2. Mapping Specifications to RTIoT System Architecture

The assignment of tasks to the RTIoT architecture layer is performed by the designer at the system specification stage. The next step is to map the specifications to the different layers of the RTIoT architecture. Figure 7 shows the results of the specification mapping introduced in Section 5.1.

For simplicity, only one instance of each graph was assumed. This corresponds to a system with one user and single sensors (one parking lot and one parking space). In a real system, one instance is created for each user and one instance is created for each sensor that can activate the TG. For this purpose, it is necessary to assume the maximum load on the system, i.e., the maximum number of simultaneous users and the maximum number of parking lots and stands served by the system. For clarity, the figure also does not indicate transmission volumes and time constraints. Based on the mapping, it is possible to obtain information about the transmissions between the layers and the sequence dependencies between these transmissions. For example, there may be 10 transmissions between the UL and CL layers: T0_4→T0_5, T0_10→T0_5, T0_12→T0_5, T0_17→T0_5, T0_5→T0_6, T0_14→T0_15, T0_15→T0_16, T2_3→T2_4, T2_9→T2_4, and T2_4→T2_5. It can also be noted that there are sequence dependencies between some transmissions, which means that these transmissions will not compete with each other for the communication link. This is important information from the point of view of transmission scheduling.

Assuming the maximum frequency of activation of individual tasks and the size of individual transmissions, it is possible to estimate the throughput requirements of communication links. These requirements can be reduced by appropriately scheduling transmissions so that time-constrained transmissions are executed in an order that ensures the maximum QoS.

5.3. RTIoT System Optimization

In the RTIoT system, each function, represented by a task graph, can be simultaneously activated by different sources (users and sensors). This can be represented by multiple instances of a given graph. When designing an RTIoT system, one specifies the requirements for the system by identifying the following constraints:

• Maximum number of instances of a given graph: This parameter determines the maximum load on the system in terms of the number of simultaneously activated tasks.
• Maximum processing time (deadline): This parameter specifies the maximum time that can elapse from the start of the t_start task to the completion of the t_stop task. For a given system, there can be multiple time constraints that define different paths in task graphs. For soft-real-time systems, a soft constraint is defined as ${\mathrm{d}}_{\max }^{\mathrm{s}}$ and a hard constraint is defined as ${\mathrm{d}}_{\max }^{\mathrm{h}}$.

The goal of optimizing RTIoT systems is to achieve the highest possible QoS. The QoS for a single constraint can be defined as:

$${\mathrm{QoS}}_{\mathrm{i}}=\{\begin{array}{cc}0& \mathrm{if}{\mathrm{t}}_{\mathrm{i}}>{\mathrm{d}}_{\max }^{\mathrm{h}}\\ 1& \mathrm{if}{\mathrm{t}}_{\mathrm{i}}<{\mathrm{d}}_{\max }^{\mathrm{s}}\\ 1-\frac{{\mathrm{t}}_{\mathrm{i}}-{\mathrm{d}}_{\max }^{\mathrm{s}}}{{\mathrm{d}}_{\max }^{\mathrm{h}}-{\mathrm{d}}_{\max }^{\mathrm{s}}}& \mathrm{in}\mathrm{other}\mathrm{cases}\end{array}$$

(1)

where t_i is the current execution time of tasks covered by the i-th constraint.

Then, the total QoS for the system can be determined as the average value of all QoS_i:

$$\mathrm{QoS}={\displaystyle \sum }_{\mathrm{i}=0}^{\mathrm{n}}\frac{{\mathrm{QoS}}_{\mathrm{i}}}{\mathrm{n}}$$

(2)

where n is the number of all constraints in all instances of task graphs.

While intending to obtain the highest possible QoS value, the optimization of the system consists in prioritizing the execution of tasks and transmissions in such a way as to minimize any exceedance of time constrains. For this purpose, the scheduling methods described in Section 4 can be used. However, in order to be able to perform tasks and transmissions according to the optimized prioritization, it is necessary to consider the following properties of the RTIoT architecture:

• Transmissions are carried out over shared internet links: To ensure predictable transmission times, it is necessary to develop routing methods that consider the required order and priority of transmissions.
• Individual tasks and transmissions are assigned to resources distributed over the internet. Therefore, in order to execute them in the required order, it is necessary to use time synchronization mechanisms. This can be accomplished by developing appropriate communication protocols.

Assuming that the aforementioned solutions are available, the presented methodology can enable the automatic synthesis of QoS-optimized RTIoT systems. The quality of a given solution is evaluated by calculating the QoS value defined by Equation (2). This value is calculated for the worst-case scenario, i.e., assuming the maximum system load and maximum computation and transmission times.

6. Conclusions

This article presents a methodology for designing RTIoT systems that includes the following elements:

• Four-layer generic RTIoT system architecture model.
• Functional specification method in the form of a set of task graphs with assignment of tasks to IoT architecture layers.
• A method for mapping functional specifications into an RTIoT system architecture.
• Requirements for communication protocols and routing methods used in RTIoT systems.

For the proposed RTIoT architecture model, a method is presented to ensure that the functional specification is mapped into an optimized IoT architecture. The starting point is the specification of the functions of a given system in the form of task graphs. For each task, their assignment to the different layers of the IoT architecture and time constraints are specified. The specification is then mapped into the target architecture by globally prioritizing tasks and transmissions. The scheduling method optimizes the schedule to achieve the highest possible QoS. The presented methodology is based on analogous methodologies used in the synthesis of embedded systems. However, due to the nature of the IoT architecture, it was necessary to consider specific features of the IoT architecture such as the lack of the centralized control of all elements of the system, the sharing of communication and computing infrastructure with other systems, and the dynamic number of tasks in the system. This paper also provides an overview of existing methods and technologies relevant to RTIoT systems. The purpose of this review was to present current solutions that can be applied to RTIoT systems, as well as to indicate the missing elements that require development in order to create a complete design environment for RTIoT systems. In this way, this paper can also be a valuable resource for designers and others interested in RTIoT.

The proposed methodology was illustrated with an example of designing a practical RTIoT system that performs the functions of a smart system for handling a city’s parking lots. The example demonstrated the practical usefulness of the proposed methodology in the design and optimization of RTIoT systems. To the best of our knowledge, this is the first paper to present a complete methodology for the design of a broad class of RTIoT systems while considering the QoS optimization of such systems.

Further work will involve the development of routing methods and communication protocols to enable the implementation of systems designed according to the methodology presented in this work. The result will be a complete RTIoT system design and implementation environment that ensures the creation of systems with a high level of QoS.