Horizontal IoT Platform EMULSION

Abstract

This article presents an overview of an Internet of Things (IoT) platform design based on a horizontal architectural principle. The goal in applying this principle is to overcome many of the disadvantages associated with the default design approach which, within this context, could be classed as “vertical” in that the IoT system and service are usually designed as stand-alone “silo-like” entities on their own autonomous platform. In a pure sense, each new IoT system and service is a new design ab initio. With the “horizontal principle”, the goal is that in the creation of a new IoT system and service, the provider needs only provide or adapt relevant architectural elements within a horizontal slice of an existing IoT architecture to enable the delivery of the desired IoT service. This article shows how embedding the horizontal principle into an IoT platform design brings the benefits of system design efficiency, effectiveness, and flexibility, together with at least the same scalability attributes inherent in the existing platform, an easily accessible adjustment, fine-tuning, and an openness to new use cases and application scenarios. The vision is the enabling of the realization of a potential multipurpose use of the IoT systems and services built on top of such platforms. The article presents a selective survey on the state of the art in IoT domains of application and in IoT platform architectural design solutions and lines of development from a vertical–horizontal categorization perspective. It presents examples of both IoT platform design solution types in use today. Within the context of strongly recommending the application of the horizontal design principle, the multitiered structure of the authors’ own EMULSION IoT platform based on this horizontal principle is presented in detail in the final part of the article.

1. Introduction

Internet of Things (IoT) systems and services have become a key integral component of global economic and social organization development today. In 2022, the number of connected IoT devices reached 14.4 billion and some USD 202 billion were spent in the IoT area [1]. Combining the artificial intelligence (AI) with IoT (also known as AIoT) is adding further to the growth of the IoT market, while the global industrial AIoT market alone may reach USD 102 billion in 2026 [2]. The IoT research literature covers a wide breath from theoretical to technological building blocks such as platform architectures, near-autonomous scalable sensor and actuator technology, communication, connectivity, and big-data infrastructure solutions; from applications, operational considerations, experimentation, and experiences from deployments to the impacts of the IoT on consumers, the public sector, and government, industrial, economic and commercial sectors.

Generally speaking, the IoT involves ubiquitous, connected, uniquely identifiable things/objects/devices that possess the ability to autonomously collect data about their environment (physical world) and exchange these, either directly or indirectly, via communication gateways [3]. Typically, the IoT things/objects/devices consist of embedded computational hardware and software, with some form of network connectivity to a remote computing resource. With the (intended) integration of IoT devices into automated systems, it is becoming ever more possible to capture an increasingly wider range of information about the physical world, and then to transmit, ingest, transform, filter, and enrich it by different system entities, forwarding it for further processing to the cloud, in order to make a decision, generate a prediction, trigger an action, create a value, help someone with a particular task, or learn from a process. IoT services thus complement AI and serve as a key means to enable the potential evolution of AI. AI has the capacity to realize an integration of IoT services resident on a variety of IoT platforms, to create and yield even more enhanced services, in a sense, as a whole is greater than its parts. In the provision of IoT services, the main challenges relate to ensuring a respective security and privacy [4,5], energy efficiency [6,7], and reliability [8,9]. Many solutions tackling different aspects of these exist, such as employing blockchain techniques for IoT security [10,11], integrating hardware security modules with permissioned blockchain as a new security layer in the IoT system architecture [12], using trust similarity for IoT security [13], utilizing federated learning for IoT security and privacy [14], using natural language processing (NLP) techniques for detecting automation rules that may potentially violate IoT security and privacy [15], combining IoT security with energy-efficiency solutions [16], applying good security–reliability tradeoff solutions [17], etc. However, the full integration of such solutions into a single IoT platform is more difficult than might be initially seen.

Bringing the “horizontal principle” to the platform architectural design—the focus of this article—may be shown to yield benefits for the whole IoT system design such as efficiency, effectiveness, and flexibility, including the scalability attributes already inherent in the existing base IoT platform designs, and enabling easy accessibility, system and service adjustment, and openness attributes to new use cases and application scenarios. This progresses the realization and development of potential multipurpose use of the IoT systems built on top of such platforms. Moving IoT platform design on to this development track will also help accelerate the evolution of AI-IoT integration.

However, by way of introduction, with the article’s focus as indicated, here it is appropriate firstly to review the status of state-of-the-art IoT systems and platforms from that vertical–horizontal binary architectural perspective. As one might expect, most IoT platform designs today may be classed as adhering to a vertical design principle. Looking at such IoT systems according to their domains of application, as follows, is as good a place as any to start.

• Industrial IoT (IIoT), involving the use of industrial end-devices such as pumps, conveyers, turbines, and motors; industrial machinery; industrial assets in the field such as pipelines; robots, containers, trucks, and ships in logistics; warehouses; etc. [3], generally for process quality control and management. For example, in January 2021, China announced its 3-year plan for creating 30 fully connected fifth-generation (5G) factories based on IIoT platforms, e.g., with an ability to perform quality control using high-definition cameras supported by AI [2].
  Recently, PTC and Microsoft have proposed the so-called “industrial metaverse” for describing mixed and augmented reality (AR) scenarios for IIoT-based product development and manufacturing [2].
• Energy production and distribution, involving smart grids, peak load management, distribution automation, advanced metering infrastructures, smart metering, etc.
• Transportation, supply chain, and logistics (the IoT domain which seemed to grow the fastest in 2022 [1]), utilizing IoT devices and visibility software for the tracking and tracing of assets transported by road, air or sea vehicles from factory manufacture through to customers, along with intralogistics robots, autonomous and connected cars/fleets, inventory and warehouse management, storage condition monitoring, cargo/shipment integrity monitoring, etc. [18].
• Smart healthcare, based on wearables (e.g., smart watches, health-status monitors, location trackers, etc.) and implants, e.g., for remote patient/patient-implant-device monitoring, detection of emergencies (e.g., in case of falling elderly people or the activation of an implanted heart pacemaker or defibrillator) and first-aid assistance, ambient assisted living (AAL), well-being applications, medical equipment tracking, etc. [18].
• Smart environment monitoring and control (based on environmental sensors), including also the structural-health monitoring of bridges, buildings, ground and foundation displacement profiles, etc.
• Smart agriculture, livestock breeding, and forestry, utilizing sensors, robots, unmanned aerial vehicles (UAVs), etc., in cropland, livestock, fisheries, and forests, e.g., for soil and crop monitoring, pest, irrigation, and fertilization management, livestock tracking and health monitoring, wildlife management, etc. [18], with the application goals to increase the production quality and quantity, and optimizing the human labor, while also tackling food security and climate change related problems.
• Smart cities, involving city traffic and road management, connected public transport operation, shared mobility (e.g., e-bikes, e-scooters, e-cars), smart parking, street lighting management, infrastructure management, environmental sensing and monitoring, etc. [18]. For example, the IoT strategy for New York City builds on already existing IoT initiatives (e.g., 23,000 connected public-transport vehicles, a myriad of connected smart speed cameras, 800,000 connected water meters, numerous air quality index (AQI) monitoring stations, and e-bike counters), and includes plans for the further training, funding, consultancy, and coordination of the deployment of the supporting communication networks, e.g., for connecting temperature and humidity sensors in an attempt to analyze citizens’ impact of the effect of various citywide running initiatives [2].
• Smart buildings and homes, involving heating, ventilation, and air conditioning (HVAC) management, energy usage displays, access and security management, water/gas leak detection, fire/smoke detection, connected fire extinguishers, elevator status monitoring, structural monitoring, indoor environmental monitoring, cleaning and garbage collection management, space occupancy monitoring, contact/proximity tracing, etc. [18].

Apart from suitable IoT platforms, the provision of IoT services requires supporting the interoperability and interconnection of heterogeneous IoT objects, electronic devices, communication modules and networks, and supplying the required operation, administration, and management (OAM) functionalities with respect to the provided data, services, and applications [19], along with the required consumers’ customization and personalization [20]. The realization of different IoT use cases and application scenarios depends on the flexibility of the underlying IoT platforms and their capacity to provide stable, efficient, effective, and secure solutions that meet the architectural and business requirements today [21]. The main benefits of using proforma IoT platforms include [3]: (i) bringing connected products to market more quickly and cheaply along with supporting monitor operations; (ii) a much simpler coding and deploying of applications for IoT solutions; and (iii) efficient edge-to-cloud communications, especially in the last few years.

The IoT platforms today in domains such as the above-mentioned ones are mostly of a vertical type, meaning they are focused on the service provision within a single IoT domain. By utilizing such platforms, IoT providers can operate in a vertical manner, involving separate applications/services, network connections, and things/objects/devices for the provision of a particular service, as shown in Figure 1. The result of this design approach could be described as leading to the formation of an “Internet of IoT-Silos”! While accepting their service provision effectiveness, describing them as such helps highlight all the corresponding drawbacks of this design approach, such as the fragmentation of the IoT domains, problematic interoperability and integration, difficult data exchange challenges, increased operational expenditure (OPEX), obstacles to achieving sufficient scaling and openness to new IoT services, and to address cross-domain use cases and application scenarios. It is not surprising then to find a significant growth in the number of IoT-platform OEMs and vendors, from 450 in 2017 to 613 in 2021. However, of these, a small few recognizable popular names such as Microsoft’s Azure IoT Hub (https://azure.microsoft.com/en-us/products/iot-hub, accessed on 4 April 2023), the Amazon Web Services (AWS) IoT Core (https://aws.amazon.com/iot-core/, accessed on 4 April 2023), the Google Cloud IoT Core (https://cloud.google.com/iot-core, accessed on 4 April 2023), and a few others dominate, taking around a 65% market share in 2021, which was up from 44% in 2016 [19]. However, Google announced the discontinuation of its IoT Core service, starting from 16 August 2023.

The horizontal architectural design approach for the creation of IoT platforms, shown in Figure 2, overcomes many drawbacks of the vertical approach. It facilitates service/application/network providers to contribute products within a horizontal slice in the delivery of IoT services which may then be utilized in multiple IoT domains. This design approach is reflective of the layered architectural approach in communication protocol design. In this, it also calls for international standards, albeit not as strongly as communication network architectures. Good designs will facilitate a seamless interaction between IoT applications and devices, even across industry verticals. As indicated above, this yields the benefits of system design efficiency, effectiveness, and flexibility, together with the same scalability attributes inherent in existing platforms, an easily accessible adjustment and openness to new use cases, new application scenarios, and a multipurpose use of the IoT platforms. In addition, it ensures the simplification of the IoT environment by removing the duplicate solutions, intertechnology operation, easy integration, full (or at least much greater) interoperability possibilities, the opening of new IoT business opportunities, and an efficient OAM of the whole IoT ecosystem throughout its lifespan.

The aim of this article is to point to these attractive benefits of taking the “horizontal design” approach in the development of IoT platforms. To this end, the state of the art in the area of IoT platform architectural design is presented first, followed by some sample horizontal IoT platform solutions used today. Finally, the authors’ own IoT platform, EMULSION [22], developed by following the horizontal approach, is briefly described.

2. IoT Platform Architectural Design: State of the Art

The commercial IoT platforms can be categorized into five types [19], briefly described here.

2.1. Communications Management/Connectivity IoT Platforms

These platforms (constituting 7% of the IoT platforms in 2021 [19]) are used for managing the connectivity and controlling the traffic to/from IoT devices, via different communication networks, such as:

• 2G–5G cellular networks: There is a recognized trend of users moving away from the legacy generations of cellular communications (2G/3G) toward the new generations (4G/5G). Because of the higher adoption of long-term evolution (LTE) cat. 1/4/6-based chipsets, the 4G IoT connections grew by 24% in 2021 [23]. In addition, for many IoT implementations, LTE cat. 1 became an alternative to the low-power wide area network (LPWAN) option (listed next) [23].
  It is notable that competition for IoT networking business between cellular operators is increasing as more and more IoT devices are equipped with an embedded subscriber identity module (eSIM), which allow their users to remotely change the current operator [24]. They might consider doing this, for instance, if they are not satisfied with the cost/quality aspect of the provided connectivity service.
• LPWANs, utilizing either licensed radio spectrum technologies, e.g., narrow-band IoT (NB-IoT) and LTE for machines (LTE-M), or unlicensed radio spectrum technologies, e.g., LoRa, Sigfox, and less popular ones such as mioty (https://mioty-alliance.com, accessed on 4 April 2023), Wize (https://www.allwize.io, accessed on 4 April 2023), RPMA (https://www.ingenu.com/technology/rpma/, accessed on 4 April 2023), Weightless (https://www.weightless-alliance.org/, accessed on 4 April 2023), NB-Fi (https://waviot.com/technology/nb-fi-specification/, accessed on 4 April 2023), ELTRESS (https://www.sony-semicon.com/en/eltres/index.html, accessed on 4 April 2023), Taggle (https://taggle.com/technology/, accessed on 4 April 2023), Стриж (https://strij.tech/, accessed on 4 April 2023), etc. [18]. The most popular of these, i.e., NB-IoT, LTE-M, and LoRa, can be used to set up either public or private networks. For instance, by May 2021, there was already a significant number of NB-IoT, LTE-M, and LoRa public network operators providing their services in 59, 34, and 37 countries, respectively, in addition to the Sigfox public network operators working in 72 countries worldwide [18].
  Communication over LPWANs is typically: (i) power-efficient (i.e., IoT devices can operate on small batteries for up to 10–15 years); (ii) long-range (from several kilometers in urban areas to 10+ km in rural areas, with a good signal penetration in underground and deep indoor environments); and (iii) low-cost (simplified communication protocols reduce the hardware complexity and cost of IoT devices, while the long range and star topology lessen the network infrastructure requirements). LPWANs aim to support: (i) massive numbers of IoT devices (single LPWAN gateway alone typically provides connectivity to thousands of IoT devices distributed over large areas); (ii) the simple, quick, and inexpensive deployment of both IoT devices and network infrastructure; and (iii) a high autonomy of IoT devices that are able to operate for several years without human intervention. In the IoT area, LPWANs are primarily used for smart metering, asset tracking and tracing, and large sensor networks operation [18].
• Wireless local area networks (WLANs), also known as Wi-Fi networks, established on the IEEE 802.11 standard, especially Wi-Fi 6 (IEEE 802.11ax) with its target wake time (TWT) power-saving techniques.
• Wireless personal area networks (WPANs), such as Bluetooth Low Energy (BLE), Bluetooth mesh, Zigbee, Z-Wave, etc., utilized in low-rate wide-range applications with a relatively high device density, e.g., for smart utility networks and smart cities. The network can span over a few kilometers [18].
• Wireless neighborhood access networks (WNANs), utilizing non-short-range mesh such as Wi-Sun (IEEE 802.15.4g), Wirepass (https://www.wirepas.com/, accessed on 4 April 2023), JupiterMesh (https://csa-iot.org/all-solutions/jupitermesh/, accessed on 4 April 2023), NeoMesh (https://neocortec.com/, accessed on 4 April 2023), etc., for creating utility networks or large sensor networks. The network can span over a few kilometers [18].
• Fixed communication networks, e.g., LANs (IEEE 802.3/Ethernet) and Fieldbuses (connecting industrial programmable logic controllers, PLCs, or I/O modules).
• Satellite networks: Through providing ubiquitous (nano)satellite low-power and low-cost connectivity, these have successfully aimed at covering the IoT networking market for IoT devices located in remote, isolated, or inaccessible areas [2]. In 2021, there were more than 5 million global satellite IoT subscribers, and it is expected that by 2026, the value of the market will grow to USD 1 billion. This estimate includes the growth of low earth orbit (LEO)-based satellite constellations for IoT networks, supplemented by hybrid cellular–satellite solutions realized through firmware upgrades on existing IoT devices with minimal hardware changes, and the entry of big technological companies as LEO-based broadband operators [25]. In the IoT area, satellite networks are primarily used for asset tracking and tracing, and remote telemetry. Key technologies include HiberBand, Myriota, and Astrocast [18].

A majority of cellular operators rely on these types of platforms to expand their IoT endeavors and offer services in new IoT domains [26]. Examples include Ericsson’s DCP, Cisco’s Jasper, Huawei’s IoT Connection Management Platform, and Verizon’s network + ThingSpace. Standard platform functionalities are the charging and billing OAM, connectivity orchestration/OAM, and service provisioning [21]. Add-ons may also be included as part of the subscription to a communication service, e.g., in the form of bill analyzers, usage anomaly detectors, etc. For the “smart homes” IoT domain, an important standard, released in 2022, is the IP-based home-automation Matter standard, which supports Ethernet, Wi-Fi, Thread, and BLE for the configuring and over-the-air (OTA) updating of home devices, and in addition, it provides connection to devices supporting other protocols, such as Zigbee, via communication bridges [1].

2.2. Device Management/Enablement IoT Platforms

These platforms (forming 35% of the IoT platforms in 2021 [19]) are utilized for remotely configuring, monitoring, controlling, and managing IoT devices. Typical platform functionalities include OTA firmware updates, deployment configuration, device monitoring, command and control, security, etc. [21].

2.3. Data Management/Enablement IoT Platforms

These platforms (representing 43% of the IoT platforms in 2021 [19]) are used for ingesting, storing, and analyzing data collected from IoT devices. Standard platform functionalities are: data storage in data bases/lakes/warehouses; data analysis by means of rules engines, data preparation, and extraction, transformation, and load (ETL); data analytics by utilizing AI/machine learning (ML)/deep learning (DL) techniques; and southbound data ingest/egress provision through data acquisition drivers, interfaces, IoT hubs, SDKs, and data brokers [21].

2.4. Application Management/Enablement IoT Platforms

These platforms (totaling 58% of the IoT platforms in 2021 [19]) are utilized for the rapid development, testing, verification, validation, and management of IoT applications. Standard platform functionalities include: IoT application management covering the marketplace and lifecycle; IoT application development, e.g., based on digital twins, IDEs, etc.; northbound data ingest/egress provision via suitable application programming interfaces (APIs), involving also corresponding alert/notification services [21]. IoT platform vendors (and some third parties) make money out of applications created on top of such IoT platforms. Typical examples are Siemens’ Edge2Web’s Director (costing USD 288 per month) and Closed-Loop Foundation applications, which are sold for use with Siemens’ IoT platform MindSphere [21].

2.5. IoT-Based Infrastructure as a Service (IaaS)

Despite the relatively small number of this type of IoT platforms (constituting only 3% of the IoT platforms in 2021 [19]), cloud hyperscalers (with their platforms) make an increasing IaaS revenue by also hosting on their infrastructures IoT platforms of other vendors [21]. For instance, Microsoft’s Azure is housing Uptake and Walmart (aimed at connecting HVAC and refrigeration units for reducing the energy usage, and applying ML techniques for the routing of thousands of trucks in the “supply chain” IoT domain [3]). Siemens’ MindSphere (https://siemens.mindsphere.io/en, accessed on 4 April 2023) uses platform as a service (PaaS) and IaaS services from AWS, Alibaba, and Azure [27]. This also facilitates the great involvement (and the huge increase of the use) of AI technologies, which are able to find patterns and discover valuable insights in data gathered from IoT devices [24]. Hyperscalers are gaining more and more significance in the delivery of industrial AIoT [27].

Today, the majority of IoT platform providers offer vertical solutions alongside their platforms [19]. As of yet, there is no evidence to indicate the take-up in these platform designs listed above of aspects of the horizontal principle dealt with above and in more detail below. Non-hyperscaler IoT platform providers, in particular, progressively deepen their focus on more vertically oriented specific applications (GE Digital, for example), services (Accenture), or solutions (for instance, Siemens) [21]. Many firms increasingly seem to be offering more vertical and/or use-case specific solutions by utilizing some underlying usually proprietary IoT platform. Large enterprises and multinational companies seeking IoT solutions to particular requirements committed at an early stage to a selected few IoT platform(s) providers, e.g., Walmart chose Microsoft’s Azure, while, in addition to the latter, Volkswagen chose Siemens’ MindSphere and AWS IoT Core [21]. However, many SMEs cannot afford to use these big-vendor platforms.

3. Horizontal IoT Platform Solutions

Applying the horizontal IoT platform principle to the building of IoT platforms should mitigate the silolike isolation of vertical IoT-platform solutions, in effect enabling the equivalent of interoperability between vertically based IoT solutions as a whole or at various virtual architectural layers. In other words, for instance, a horizontal IoT platform’s goal is to contribute to facilitating the gathering of IoT data from two or more different IoT domains, each otherwise realized and implemented following the vertical platform design and enabling a sharing of these data with all interested players. Most desirable is that this sharing would happen in an efficient, scalable, reliable, and secure way. In this way, horizontal IoT platforms should enable the creation of possibilities for unproblematic logic and functionality exchanges between different public/business activities [28]. For instance, the EMULSION IoT platform, presented in Section 4, can provide solutions which integrate the benefits of a “smart environment monitoring and control” IoT domain and its services, with those of a “smart healthcare” IoT domain and its services, yielding in effect a new IoT domain from this merging which, for instance, might be called a “personal health environment” and which would offer services related to smart healthcare monitoring and notifications for persons moving through personal-health-dependent risk-laden environments. Such IoT services would immediately find applications in locations where regular, natural, accidental, or man-made environmental pollution might occur and such pollutants might be a health hazard for all or a sector of inhabitants, man and/or animal. In more general terms, IoT service design, based on horizontal IoT platform principles, opens those services to the possibility of drawing on other IoT services, already present or yet to be designed, to enrich and expand their service. Hence, to such IoT services, the horizontal design principle brings the attributes of flexibility and adaptability, enabling their easy applicability to a wide range of real-life use cases and application scenarios. The “agnostic” foundations of such platforms typically offer generic tools and services that can be customized and extended to meet the specific needs of different users. Examples of horizontal IoT platform solutions are presented in the following subsections.

3.1. oneM2M

This is a general-purpose IoT standard (https://onem2m.org, accessed on 4 April 2023) for interoperable and scalable systems, ensuring a high degree of reuse and the interoperation of vertical applications. A horizontal architecture is specified in the form of a three-layer model consisting of an application layer, a connectivity layer, and a common services layer. The latter defines a common middleware technology for use between IoT devices, communications networks, and IoT applications, over standardized links going through communication gateways to cloud infrastructures, which allows one to mix and match components from different vendors and aggregate data from multiple silos. While oneM2M does not provide a full-featured IoT platform or set of tools, its focus on interoperability and standardization makes it a valuable component of many horizontal IoT solutions. By facilitating different IoT applications to discover and interact with different IoT devices in a common language, the standard allows IoT solutions to interoperate across different industry verticals, thus reducing fragmentation and complexity, increasing reusability, and improving the cost base.

3.2. DeviceHive

This is a generic, scalable, open-source IoT platform (https://devicehive.com, accessed on 4 April 2023), distributed for free use and change under Apache 2.0 license, and designed to be flexible and extensible, allowing developers to customize it in order to meet their specific needs. It helps in the communication and management of smart devices, connected via REST API, WebSocket, or MQTT. It supports Android and iOS libraries written in various programming languages, which makes it a device-agnostic platform. Easy integration with any other device, cloud, or platform is possible by using supported protocols and employing plug-in service features. The DeviceHive behavior can be customized by running custom JavaScript code. Batch analytics and ML can be run on top of device data by leveraging different “big data” solutions, such as ElasticSearch, Apache Spark (with Spark Streaming support), Apache Cassandra, and Apache Kafka for real-time batch processing. DeviceHive is well-suited for building horizontal IoT solutions that can be applied across a wide range of real-life use cases and application scenarios.

3.3. Distributed Services Architecture (DSA)

Allowing purpose-built products and services to interact with one another in a decentralized manner, this open-source IoT platform (http://iot-dsa.org, accessed on 4 April 2023) enables the distribution of functionalities among discrete computing resources, based on a network topology formed by multiple DSLinks between the edge devices and the DSbrokers’ tiered hierarchy, and availing of the computing resources available on the edge, fog, or cloud. It is designed to simplify the IoT application development and deployment, by providing a common framework for device connectivity, data management, and application services.

3.4. M2MLabs Mainspring

This is an open-source application framework (http://www.m2mlabs.com/, accessed on 4 April 2023), written in Java, for building and managing machine-to-machine (M2M) applications, which, after prototyping, can be transferred to a high-performance execution environment, built on top of a Java 2 Platform, Enterprise Edition (J2EE) server and a scalable Apache Cassandra data base. It supports the flexible modelling and configuration of IoT devices, communication between devices and IoT applications, and data validation, normalization, long-term storage, and retrieval for external applications. Designed to be flexible and adaptable, Mainspring can be applied to a variety of IoT use cases and application scenarios.

3.5. Nimbits

Targeting the provision of IoT-related services by constrained embedded systems, this solution (http://www.nimbits.com/, accessed on 4 April 2023) includes an open-source data-logging cloud, used for sensor data recording and free sharing among users by creating data points on the cloud, fed by numeric/text/XML-based values for performing calculations on the obtained sensor data, generating and managing alerts, sending data to social networks, spreadsheets, and websites, etc. [29].

3.6. Open-Source Internet of Things (OS-IoT)

This is an open-source platform (https://www.os-iot.org/, accessed on 4 April 2023), developed in C++, running under a variety of Linux operating systems, and supporting the oneM2M IoT standard. It allows the simplification of the process of connecting IoT devices hooked into an open, interoperable oneM2M ecosystem. With the provided support for the oneM2M network- and protocol functions, the platform allows application developers to interact with a system over a resource-oriented API, thus reducing the time spent and effort made to achieve that. Hence, application developers are freed to focus on the unique, value-added aspects of their applications, instead of having to deal with different communication networks and protocols. The OS-IoT’s main focus is on supporting lightweight client applications operating on constrained battery-powered devices. A further extension of the Linux OS-IoT library is the development of a bridge between the Open Connectivity Foundation (OCF) standard and the oneM2M IoT standard.

3.7. SiteWhere

This is an open-source application enablement IoT platform (https://sitewhere.io/en/, accessed on 4 April 2023) for the creation of IoT infrastructures and applications, supporting high-throughput, reliable, and low-latency processing, and dynamic scalability. The platform is built by means of a framework approach, facilitating the easy addition of new concepts. The platform has a powerful multitenant distributed architecture, built with Java microservices, which allows it to perform resilient high-performance stream processing and provide the required key features for building and deploying IoT applications. The main functionalities include device state management, “big data” event ingestion and persistence, large-scale command delivery, the export of device data to external systems, REST APIs, etc. Specializing in specific tasks, the microservices (Spring Boot applications) can self-assemble themselves into a platform instance, orchestrated as a distributed system using Kubernetes, which allows SiteWhere to run on almost any existing cloud platform as well as on on-premises installations. In addition, the platform provides Helm charts, allowing one to bootstrap an instance by means of a single command, thus hiding the complexities of the system configuration. An Electron-based application allows the easy administration of the platform instances. Infrastructure technologies include Apache Zookeeper and Apache Kafka, a variety of data bases (MongoDB, InfluxDB, Cassandra), and MQTT brokers.

3.8. Yaler

This is a relay infrastructure (https://yaler.net/, accessed on 4 April 2023) for secure access from any browser or mobile phone to embedded systems, located behind a firewall, a network address translation (NAT) device, or a mobile network router, with premium pay-per-use support. The web-based accessible and addressable devices (via a TCP socket) can be integrated with existing web applications or third-party services. In addition, YalerContrib contains various programming language examples, libraries, contributions, and binary downloads for selected platforms released as open source.

3.9. IoTConnect

This is a full-fledged PaaS horizontal IoT platform (https://www.iotconnect.io/, accessed on 4 April 2023) that allows IoT device communication and management, data storage, and applications enablement, supported by relevant security protocols. The platform consists of various components in the form of tools, smart SDKs (e.g., IoT portal, dashboard, rule engine, gateway edge analytics, command execution), APIs, and protocols (e.g., CoAP, MQTTS, HTTPS, AMQP), which allow the creation of multiple IoT solutions for the specific business goals of each supported industry vertical. The price offers from Microsoft (excluding the Azure infrastructure costs) range from USD 3000 per month (for small-scale IoT usage with around 150 million messages per month) to USD 8610 per month (for large-scale IoT usage with around 500 million messages per month).

3.10. EMULSION

This is a horizontal IoT platform, developed jointly by the University of Plovdiv (Bulgaria) and the North China University of Science and Technology, which is presented separately in the following section.

An underlying protocol/technology-based comparison of the presented horizontal IoT platform solutions is provided in

Table 1. Comparison of horizontal IoT platform solutions, based on underlying protocols and technologies used.

IoT Solution	Network/ Transport Protocols	Application Protocols	Data Processing/ Aggregation	Data Storage	Edge/ Fog/ Cloud Computing	Application Development
oneM2M	TCP/IP, UDP/IP	HTTP, MQTT, CoAP	-	Container type	Edge/ fog/ cloud	oneM2M on Stack Overflow, OS-IoT, OCEAN, OpenMTC, oneM2M Tester
DeviceHive	TCP/IP	MQTT, WebSocket	ElasticSearch, Spark, Kafka	Cassandra	Cloud	Java, Python, Node.js
DSA	SSL/TCP/IP, TCP/IP, (TCP, UDP)/IP, UDP/IP, IP	HTTPS, WebSocket, LDAP, CoAP, KNX	Spark, Kafka	MongoDB, RethinkDB, Redis, DynamoDB	Edge/ fog/ cloud	DSA Development Tool, Java, JavaScript, Python, Scala, Dart
M2MLabs Mainspring	TCP/IP, (TCP, UDP)/IP	HTTP, SOAP	scripts written in JVM and/or rule sets written in Drools	Cassandra	Cloud	J2EE
Nimbits	TCP/IP	XMPP	-	H2 (Java SQL DB)	Edge/ cloud	J2EE
OS-IoT	(SSL/)TCP/IP	HTTP(S)	-	-	Cloud	C++, HAIM
SiteWhere	(TLS/)TCP/IP	AMQP, MQTT, WebSocket, XMPP, Stomp	Zookeeper, Kafka	MongoDB, InfluxDB, Cassandra	Cloud	Java Spring, Kubernetes
Yaler	TLS/TCP/IP, (TCP, UDP, SCTP)/IP	HTTPS, SSH, SFTP, RFB	-	-	Cloud	Java SE, C, C#, Python
IoTConnect	TLS/TCP/IP, TCP/IP, UDP/IP	HTTPS, AMQP, MQTT, CoAP	Hadoop, Kibana/ ElasticSearch	CosmosDB, DocumentDB, NoSQL	Edge/ cloud	R, Python, D3.js and Power BI
EMULSION	TLS/TCP/IP(sec), TCP/IP	HTTPS, MQTT, WebSocket, XMPP	Zookeeper, Kafka, Storm, Camus	Redis, InfluxDB, TDengine, HazelCast, Hive, Impala	Edge/ cloud	Jakarta EE, Spring, Nacos

.

4. IoT Platform EMULSION

EMULSION is a horizontal IoT platform of a combined type (hardware and software), targeting the SMEs that are focused on the development of specific or regional solutions for niche use cases and application scenarios. Low-cost electronics and open-source software are integrated into a multitier IoT architecture (Figure 3 and Figure 4). In the sensor & actuator tier, different types of sensors (S), environment monitoring stations (MS), and location trackers (T) operate to capture the changes that occur in the physical world and send the corresponding information towards the cloud tier through data/remote transfer units (D/RTUs) and smart communication gateways, through different wireless communication networks, e.g., 2G–5G cellular networks, long-range wide area networks (LoRaWANs), WLANs (especially utilizing the Wi-Fi 6 standard), WPANs (e.g., utilizing the BLE standard), etc. For extending the communication range when trying to reach the corresponding gateway(s), wireless sensor networks (WSNs) can be used, if needed. After analyzing the data sent by the sensor & actuator tier, the cloud tier makes appropriate decisions, generates suitable recommendations for consumers, and sends the necessary configuration information and/or commands to the controllers (C), actuators (A), and guards (G), located in the sensor & actuator tier, for enforcing the required OAM actions needed for the realization of the imposed changes in the physical world.

EMULSION consists of seven tiers, as shown in Figure 4 and described in the subsections below.

4.1. Sensor & Actuator Tier

This tier involves: (i) IoT nodes used for detecting changes occurring in the physical world and sending notifications about that; and (ii) IoT nodes used for imposing the required changes in the physical world. For instance, in the “smart environment monitoring and control” IoT domain, the focus of the tier is on the AQI monitoring for daily live reporting (and forecasting) of AQI values in the target area(s). The multiparametric AQI monitoring stations used include both commercial ones (https://smartcity.shieldo.space/, accessed on 4 April 2023), shown in Figure 5, and self-made ones, shown in Figure 6.

The AQI monitoring stations are used to capture and save (and possibly also analyze) the data about the CO₂, NO₂, SO₂, O₃, PM_1/2.5/10, humidity, temperature, luminosity, and pressure values of the atmospheric environment in particular area(s). The AQI data obtained by these monitoring stations are used for elaborating short-/long-term forecasting techniques with respect to changes in the atmospheric environment. The produced forecasts are utilized for smart proactive route planning. The AQI monitoring stations are located at geogrid points in the area(s) of interest and used for conducting field-trial experimental testing, validation, and verification of the AQI monitoring system, built on top of EMULSION. The grid consists (predominantly) of fixed AQI monitoring stations. In addition, mobile AQI monitoring stations, mounted on participating vehicles, are also used in order to obtain the actual AQI values on roads of particular interest (Figure 7).

Another field of current interest, which is part of the “smart environment monitoring and control” IoT area, is displacement monitoring. For the corresponding prototype IoT system, built on top of EMULSION, a low-cost, robust, global navigation satellite system (GNSS)-based sensor unit (Figure 8) was designed, whose performance is close to that of the commercial (but much more expensive) sensor units available on the market today. Its design came in with a total cost for the printed circuit boards (PCB) and bill of materials (BOM) of only USD 221.

Initially, proprietary MCU-based boards were designed and developed, such as the low-cost, generic, core board, utilizing a robust, low-power consuming MCU (based on ARM Cortex-M3 STM32F103RE), Winbond W25Q32JVTCIQ serial flash memory, GPS/GPRS, NB-IoT, and LTE cat. 1/4 communication modules, and an open-source RT-Thread real-time operating system (RTOS). More than 200,000 boards were developed to date and successfully used in different prototype IoT systems, built on top of EMULSION. The production cost of the GPS/GPRS-type core board is less than EUR 4, which is much less than open-source (Linux, Android) hardware boards.

Later, a proprietary high-end, Android-based board (Figure 9) was elaborated for utilization by the EMULSION platform. Despite a “PCB+BOM” cost of only USD 30, it is very close in performance to open-source high-end boards sold on the market.

4.2. Communication Tier

This tier is used for ensuring seamless and transparent heterogeneous internetworking by means of smart communication gateways that “intelligently” operate between different communication networks, such as: (i) LPWANs, e.g., LoRa; (ii) WLANs, e.g., Wi-Fi 6; (iii) WPANs, e.g., BLE; (iv) cellular networks, including all their current and future generations (2G–5G); but also all LPWANs based on the 3rd Generation Partnership Project (3GPP) standards, such as LTE-M and NB-IoT.

Self-designed and self-made, low-cost, and ultralow-power D/RTUs were used in this tier. Each produced D/RTU was experimentally tested, and its use successfully demonstrated in prototype IoT systems, built on top of EMULSION. For instance, a low-cost (EUR 20) LTE cat. 1 based D/RTU was designed for use by EMULSION, operating in an ultralow-power consumption mode (Figure 10), with a size of only 4 × 10 cm, allowing it to be easily inserted and used in small spaces.

For NB-IoT water applications, a special ultralow-power, low-cost D/RTU was elaborated (Figure 11), consisting of an ARM MCU (based on the ultralow-power STM32L151), an NB-IoT module, a DC–DC power supply module, two LDOs, a TIA-485 module, and an optical-coupling isolation input module.

The communication tier also includes smart communication gateways, operating as intermediaries in the communication between the IoT nodes in the sensor & actuator tier and information centers in the cloud tier, and ensuring the required interoperability. Each smart communication gateway may also contain an edge computing node for the initial analysis of data, collected from the sensor & actuator tier and generating knowledge close to the source, and for establishing a secure communication with the cloud tier, where the relevant information centers collect and analyze all aggregated data to provide respective services to consumers via the service tier and client tier of the platform and/or obtain relevant commands/decisions/recommendations, which are sent back to the sensor & actuator tier and the client tier. The overall goal of the communication tier is to allow simultaneous communication of up to 1,000,000 IoT devices (of heterogeneous nature) within a single EMULSION cluster.

4.3. Modelling & Simulation Tier

This novel architectural element is proposed here for the integration into IoT platforms in order to perform the following two tasks:

• The modelling of IoT objects and services, along with their inherent attributes and temporal/spatial/event characteristics, in order to try and test the possible integration of (new instances of) these into the platform.
• The simulation of the provision of IoT services by solving different optimization tasks to determine the optimal configuration of a system, built on top of the platform, for each particular use case and application scenario, in order to achieve an efficient provision of IoT services and the best QoE for consumers, measured by standard metrics, such as the mean opinion score (MOS), standard deviation of the opinion score (SOS), h-acceptability, acceptance, percentage of (dis-)satisfied consumers judging a service as “good or better” (%GoB) or “poor or worse” (%PoW), etc. [30].

Sample optimization tasks include the determination of:

• The optimal deployment of IoT devices in the sensor & actuator tier;
• The optimal deployment of gateways in the communication tier;
• The optimal configuration of the cloud tier.

4.4. Client Tier

This tier facilitates the consumers’ access to services, by taking into account the current consumer-, service-, and (access) network context in order to make use of the “best” service instances available, as per the ABC&S communication paradigm [31].

The most important among different software applications here is the smart multiserving client application. In addition, supplementary client applications are also being developed. A cloud-based client–server design pattern and corresponding software application model are followed for the development of these applications along with the corresponding server applications (Figure 12).

4.5. Service Tier

This tier provides two types of services:

(1)

Regular IoT services (initially developed to serve the “smart environment monitoring and control” and “smart healthcare” IoT domains). Each service of this type is delivered in an always best connected and best served (ABC&S) way to the consumers. For instance, in the “smart healthcare” IoT domain, multiple healthcare services are being developed, ranging from daily activity and community services, to health-status monitoring services [32,33], and to even more sophisticated services, focused on preventing life-threatening situations by utilizing GNSS tracking of patients.

(2)

Supplementary services, which are used to find and recommend to consumers “best” instances of regular IoT services. A sample prototype recommendation service, which is being developed, assists consumers in proactively planning (in advance) driving/biking/jogging/walking routes through poor-air-quality/polluted areas, in accordance with their personal health status, e.g., imposing minimum health risk to them. Based on live AQI data provided by a geogrid of deployed monitoring stations, operating in the sensor & actuator tier, the routes are preplanned by means of AQI forecasting techniques and are dynamically changed depending on the current environment conditions. The incorporation of such health-related criterion into existing navigation systems and applications for route generation/recommendation is seen as an important functionality extension of the latter. Suitable models, algorithms, and techniques are being elaborated [34,35] for intelligent service recommendations by EMULSION.

The service tier consists of two subtiers:

(1)

An application-enabler subtier, which deals with modelling, formatting, encoding/decoding, and managing service descriptions, allowing the automatic discovery of IoT services by consumers;

(2)

A service-enabler subtier, which is based on the “all-IP” NGN paradigm and the IPDC technology.

4.6. Middleware Tier

This tier allows one to solve problems associated with the integration of heterogeneous IoT devices deployed and IoT applications developed, to achieve the required interoperability [36]. For this, a bridge functionality was first developed (in Java) between IoT devices and applications, operating as a transparency server. A robust HashedWheelTimer module was implemented to enable each such server to serve up to 50,000 IoT devices (Figure 13).

The other main components of this tier are the Message Queuing Telemetry Transport (MQTT) brokers distributed and operating over multiple servers. Middleware-based components were utilized to build a custom MQTT broker with pure Java. A project reactor design pattern in the broker’s design ensures a low latency and high throughput. The ultimate goal is the successful participation of each MQTT broker in the context of millions of simultaneous Transmission Control Protocol (TCP) connections. A four-layer architectural design of the MQTT broker is depicted in Figure 14. At the protocols layer, in addition to MQTT, two other communication protocols are supported, namely WebSocket and Hypertext Transfer Protocol (HTTP). The component layer supports clustering, configuration, topics, persistency, transport, and rule engine. The function layer supports device monitoring, connection management, subscribe/publish operation, and logs management. In addition, Zookeeper and Kafka modules provide data pipelining and data integration functions. The kernel layer contains the Java virtual machine (JVM) and the Vert.x cluster manager based on Hazelcast and Netty. The open-source distributed cache database Hazelcast and the asynchronous nonblocking network framework Vert were selected to enable the MQTT brokers to operate in a distributed fashion within the JVM environment.

4.7. Cloud Tier

Supplemented by a Lambda-based [37] Data Management Platform (DMP) core, this tier performs “big data” processing and analytics, to convert the raw sensor data gathered from the sensor & actuator tier into actionable analytic datasets for use by the IoT services provided.

The cloud tier has a three-cluster architecture, as shown in Figure 15. As a load-balancing cluster performing data loading into Hadoop, the high-throughput distributed messaging platform Kafka is utilized. This produces topics (via Kafka Brokers), which are then consumed (in real time) by the second cluster (Storm). At the end, the useful data are serialized to HBase in the third cluster (Hadoop), where Apache Hive and Cloudera Impala are utilized for data-mining purposes.

For the creation of common patterns, communication with services, and the collection of data by applications, the widely used in distributed systems Spring Boot tool was utilized. On a single cloud node of a typical IoT system, more than a decade of applications must work together in order to provide different IoT services. The corresponding processing diagram for the latter is shown in Figure 16. The MQTT clients “publish” their messages to the corresponding MQTT broker. Kafka applications consume the predefined topics, post the RCTM dataset to the decoding API, and the results are serialized to a distributed Redis database, a time-slice database, and a Hadoop HBase.

The EMULSION IoT platform has been successfully used as a basis for the development of various IoT prototype systems for different purposes, such as smart environment monitoring and control, rented bicycles OAM, operation and control of smart electric boilers, and pure water monitoring and control.

Future research work will be focused on further elaboration of novel techniques, algorithms, and models for service recommendation to consumers, provided by the prototype systems, built on top of EMULSION, and accessible anytime-anywhere-anyhow through any kind of mobile device via the “best” available wireless access network, in accordance with the ABC&S communication paradigm [31]. This will be followed by the development of corresponding software (run on multiple mobile platforms) ensuring the best quality of experience (QoE) is delivered to consumers when using IoT services supplied by EMULSION.

5. Conclusions

While a variety of elements of IoT were touched on in this article, such as theoretical and technological building blocks, operational considerations and deployment experiences, the impact on various actors from consumers to OEM commercial suppliers, and applications which are driving growth, our focus was rather on the evolution of IoT platforms’ structural architecture and drawing out the benefits and feasibility of incorporating a horizontal design principle. To contextualize this, we focused on presenting a selective survey of the state-of-art IoT platform architectures according to their attributes of being vertically oriented, that is, tending to be closed, proprietary, silolike structures, albeit in quite a variety of ways, or “horizontal oriented”, that is, being open, interoperable, flexible, scalable and more readily adaptable and moldable for the creation of new IoT services efficiently and effectively. The article sought to point to the attractive benefits of taking the “horizontal design” approach in the development of IoT platforms. The survey included dealing with a variety of IoT architectural elements such as the very important communications’ management and connectivity solutions, but also covered the management/enablement of the IoT devices, data, applications, and IoT-based IaaS. Many examples of “vertical” IoT platform solutions were presented, followed by platforms which adhered to the horizontal principle in varying degrees. The state of the art in regard to these latter platforms was particularly relevant given the terminal focus of the article, the presentation of the authors’ own EMULSION horizontal platform. Multiple examples of the horizontal IoT platform solutions were presented as well. The creative ideas encompassed in these examples were exploited and added to in creating the authors’ own EMULSION horizontal IoT platform architecture and design. The seven-tier architectural structure of the platform, including the sensor & actuator-, communication-, modelling & simulation-, client-, service-, middleware-, and cloud tiers, was described at length in the final section. The article sought to show the way EMULSION could convert collected sensor data and gather information from consumers about their IoT service activities into rich analytic datasets. The tangible example given brought together two IoT domains with their specific applications built in accordance with the horizontal design principle, i.e., the “smart environment monitoring and control” and “smart healthcare”; then, through EMULSION’s architectural attributes, a new third IoT domain, a “personal health environment”, could be readily created, emanating largely from an integration of these two IoT domains.

In conclusion, the article claims that the horizontal approach to IoT platform design, and EMULSION in particular, addresses many of the disadvantages of the vertical approach in the delivery of IoT services. The article claims that EMULSION can meet today’s requirements for multidimensional flexibility, interoperability, scalability, and adjustment to real-life use cases and application scenarios. By taking into account the current consumer-, service-, and (access) network context, it is able to provide highly personalized, customizable, and contextualizable IoT services to consumers by utilizing distributed real-time big-data processing and analytical techniques in the IoT’s edge and cloud.