1. Introduction
The Internet of Things (IoT) is an emerging global network affecting almost every domain by allowing the development of systems through intelligent devices and diverse application layer protocols according to user demand. Such sustainable growth of IoT technologies is also occurring in industrial application domains, namely the Industrial IoT (IIoT), which is leading industrial automation and communication infrastructures to undergo a profound advancement [
1]. The advancement in industrial networking facilitated by the union of IoT and IIoT technologies reinforces the concept of the fourth industrial revolution (IR 4.0), standardized in (DIN SPEC 91345:2016-04 2016 IEC 2017) [
2]. The IR 4.0 paradigm migrates current production systems into smart manufacturing, with a high degree of interconnection through intelligent networking and cognitive automation between the participating agents, and by shifting information into cloud-based applications for real-time data management [
3]. This complex evolutional concept provides scalability to increase manufacturing flexibility, offer comprehensive information exchange, and enable virtual information representation in everyday internet-enabled devices as endpoints [
4]. The evolution of the IIoT paradigm in industrial networks incorporates intelligent reconfigurable systems to process the production plant data flow based on various sets of requirements in a decentralized distributed manner. Distributed systems in shared cooperativity are holistic characteristics of modern factory systems, where data acquisition and information are processed locally in nodes of distant fields for a given industrial process. Hence, communication networks in such scattered applications enable the exchange of aggregated data among these nodes, and achieve mutual synchronization for controlling the overall production process.
However, industrial communication networks in the IR 4.0 context are currently a blend of both wired and wireless network technologies. Wired communications mainly include Modbus TCP/RTU, Fieldbus, EtherNet/IP, EtherCAT, TTEthernet, PROFINET, etc. [
5]. Furthermore, the wireless networks are derived from the IEEE 802 protocol family such as IEEE 802.11 (Wi-Fi) and wireless personal area networks (WPANs) [
6]. These infrastructures come from various manufacturers in different architecture and dissimilar hardware specifications, which often cause incompatibility issues and make the communication subsystems heterogeneous [
7]. Collaborative integration of such heterogeneous systems to accomplish reliable data exchange and industrial processes management leads to communication interoperability issues among the protocols used and increased complexity of heterogeneity management challenges at various automation levels. Due to the different standards and language-based underlying models, data aggregation from heterogeneous sources at reduced latencies is a crucial challenge. Hence, the primary intention of this study is to incorporate an open-source unified model across multiple industrial networks to address communication interoperability issues at different automation levels and management of aggregated data from the heterogeneous sources in a standardized manner. Open-source technologies have been chosen over proprietary infrastructures, as they offer greater configuration options with necessary information, enabling the development of custom scientific tools and the recreation of physical artifacts without concerns about license expenditures. Incorporating open-source software is also economically beneficial and can provide, on average, 87% and even 94% of substantial economic savings involving both the open-source hardware and software development tools compared to the commercial proprietary equivalents [
8].
To alleviate the issue in the communication network mentioned above, various IIoT protocols such as Modbus, PROFIBUS, and Distributed Network Protocol 3 (DNP3) can used named in the industrial domain to address network heterogeneity [
9]. However, in the IR4.0-based context, the Reference Architecture Model for Industry 4.0 (RAMI4.0) [
10] and Industrial Internet Reference Architecture (IIRA) [
11] specify the reference models for next-generation factories to align with IR4.0 compliant standards. These reference models recommend Open Platform Communication Unified Architecture (OPC UA) as the major connectivity standard to implement underlying communication layers for interoperable information exchange between diverse industrial applications and to enable seamless information exchange across heterogeneous networks in the IR4.0 context [
12]. According to the discussions above, OPC UA, as a promising IEC-62541 standard, has been chosen in this study to establish a unified platform among system components to aggregate heterogeneous data and automate interoperability among the system components in real-time. OPC UA is a vendor-neutral service-oriented-architecture (SOA) for industrial automation and Machine-to-Machine (M2M) communication [
13]. It exhibits client–server or publisher–subscriber model-based communication, where data exchange takes place over encrypted TCP/IP through SSL, TLS, and AES [
14]. OPC UA offers several combinations to map the client–server interactions, such as TCP-transported, binary-encoded, Hypertext Transfer Protocol (HTTP)-transported, Extensible Markup Language (XML)-encoded, and Simple Object Access Protocol (SOAP)-based services [
15]. It defines object-oriented data descriptions through the group of nodes referred to as the information model. This reusable model allows the integration of other standards and provides transport and semantic interoperability to exchange information across the heterogeneous network [
16]. Such an object-oriented information model provides great potentiality for OPC UA applications in various domains [
17]. OPC UA exhibits the flexibility to adapt from embedded devices with small service sets to Windows, macOS, and Linux platforms with full functionalities [
18].
IR4.0 compliant infrastructures experience syntactic interoperability to manage domain-specific data and semantic interoperability while exchanging data with shared meaning [
19]. The OPC UA paradigm is lacking in the cross-domain era to interact with the external services. Hence unlocking the network capabilities of OPC UA is well demanded to share the resources with other domains. Besides, OPC UA applications are largely missing the granularity of web and AI technologies for developing adaptive platforms to virtually represent information and perform remote process supervision. OPC UA involves SOAP/TCP binary-based services whose complexity is incompatible with the resource-constrained devices that lead to energy and latency overhead [
20]. Hence, OPC UA is not suitable for these devices without significant modification. Although OPC UA exhibits bulk data, it still lacks storage infrastructure. Small-scale traditional industries usually involve supervisory/monitoring systems for resource management. However, modern smart factories produce bulk data to process, which ask for structurally-accumulated storage systems for resource allocation [
21]. The cloud paradigm is an excellent choice in this aspect, which offers data logging, location-independent bulk resource management, and ubiquitous content access. As addressing interoperability is a prerequisite requirement of IR4.0 systems, another intention of this study is to overcome the semantic and syntactic interoperability issues of the OPC UA-enabled applications in order to seamlessly integrate heterogeneous-aggregated data with other domains without any restrictions. Such phenomena can enable interoperable M2M communication from manufacturing execution systems (MESs) to enterprise resource planning (ERP) [
22]. The path towards bridging OPC UA with other domains needs to involve a middleware protocol such as the Constrained Application Protocol (CoAP), Message Queue Telemetry Transport (MQTT), Advanced Message Queuing Protocol (AMQP), Representational State Transfer (REST), etc. [
16]. Among them, the REST approach reaches almost every domain as a universal communication solution to harmonize communication and shows relatively better services in unstable networks [
23]. Indeed, REST is a free and open-source software (FOSS) architectural style that can share information seamlessly with different agents and support open-source and proprietary systems. It is a protocol-independent, client–server architecture that exhibits a set of constraints for stateless application development. REST also provides uniform interfaces, mostly HTTP for resource representation via Uniform Resource Locator (URL) [
24]. Such resource representations are compatible with web browsers and arguments are added with the request URL [
25]. These features motivated us to use REST for establishing communications between OPC UA with different technologies.
Several proprietary solutions allow the REST interface to bridge OPC UA with the other domains. For instance, KEPServerEX [
26] offers three predefined REST URLs to read, write, and browse OPC UA data. However, the browse service of KEPServerEX provides all the configured “tags” without node attributes. HyperUA [
27] offers non-standard mapping by encoding OPC UA resources into the URL and provides an HTTP interface to access OPC UA node credentials for web clients. This approach asks for client-side HyperUA API implementation to interact with the server. The data FEED OPC suite [
28] also incorporates the RESTful API to access OPC UA resources. These proprietary solutions lack customizability according to user requirements and configurability facilities to fabricate developed infrastructure with a deeper understanding and better limit awareness. Moreover, various adaptation approaches have also been taken with the OPC UA binary protocol to achieve the RESTful extension. Grüner et al. [
29,
30] proposed resource-oriented information models for RESTful sessionless extension of OPC UA. They enabled stateless service invocation and modified caching headers to reduce handshake signals and exchange stateless requests, and evaluated OPC UA over UDP to minimize the communication overhead. A similar concept was taken in [
31], through the HATEOAS approach for interconnecting OPC UA nodes. However, these approaches do not meet the fundamental postulates defined by Fielding [
24], and are therefore unable to integrate loosely-coupled distributed systems with other domains. Another study in [
32], which follows the definition of REST postulates, proposed a Java OPC UA stack facilitating data integration between the OPC UA and HTTP/REST services. This approach provides access to the OPC information in a REST-based server to the RESTful clients. This approach also involves OPC UA sessionless service to achieve statelessness, does not have mapped interfaces to exchange OPC UA resources with the web domain. Although the OPC UA specification introduced a sessionless invoke method from Version 1.04 (OPC Foundation. Scottsdale, AZ, USA, 2018), most of the stacks still have not implemented the full functionalities for this version. Taking account of these studies, it can be commented that the protocol modification approach to making OPC UA adaptable with web technologies is not sufficient. OPC UA cannot be considered fully stateless due to it maintaining the established connection status with the clients, which is not compatible with the web domain.
A more appropriate approach will involve a gateway module in the middle to bridge the OPC UA with other domains. Incorporating REST is the best choice, as this paradigm is more compatible with the web domain than the other options, and can map the OPC UA resources into REST representation under the required specification. In [
33], mapping of OPC UA server credentials as REST resources was proposed to investigate the conditions of heterogeneous industrial devices from monitoring applications. This approach involves a Java SDK-based OPC UA implementation that causes a processing burden for resource-constrained embedded devices. Furthermore, the solution involves proprietary protocols and enterprise applications to monitor the condition of heterogeneous field devices. In [
34], an OPC UA-based gateway with an information server was proposed that integrates available production systems in an Industry 4.0 factory to allow supervision from the cloud, MES, or HMIs consumers. However, the functionalities of this gateway are limited to pre-defined templates that lack scalability and require custom configuration per cite. In another study in [
35], an OPC UA server–client architecture-based gateway was proposed for the oil and gas industry. Here, the server collectively gathers Oracle simulator data and the client transfers these data to the NoSQL Oracle cloud server through the RESTful API to be store and share with enterprise applications. The configurability of this proposed system is limited by the Oracle IoT Cloud-based proprietary software, which lacks adaptability with other vendors. In [
36], a declarative control concept through a REST-based OPC UA client decoupling approach was proposed to improve automation in a distributed system. Although this attempt enabled interoperability through the REST-based decoupling approach, the control functionalities were designed in a highly complex graph model through the OPC UA Browse-Service. Another attempt in [
20] was made to integrate OPC UA with web technologies to address interoperability. In this approach, a lightweight middleware was proposed that maps the OPC UA resources into HTTP requests to monitor them from the RESTful web platforms, and also incorporates an HTTP frontend to execute the web platform requests to the OPC UA server. However, this attempt exhibits two crucial drawbacks. First, the absence of uniform interfaces lacks the scalability that is an important aspect of stateless communication. Second, mapping the OPC UA server responses into JSON or XML to achieve interoperability makes the situation more brittle.
Although the previous literature makes valuable contributions to RESTful extension of the OPC UA to integrate data with web technologies, it still exhibits certain limitations, such as the involvement of proprietary software limited by configurability, the mapping of server responses into JSON or XML-based representation causing processing burden for resource-constrained devices, the absence of uniform interfaces to integrate decentralized distributed system components, and the lack of experimental analysis to validate the proposed solutions. The proposed architecture overcomes these limitations through concrete strategies, and its implementation was validated by integrating with proprietary/open-source platforms and performing extensive performance analysis. To alleviate these limitations and achieve the research goals, the following questions are investigated in this research:
RQ1: What is the appropriate stack to develop an open-source vendor-neutral full-stack OPC UA platform for data aggregation in the heterogeneous distributed system?
RQ2: How do we extend the OPC UA functionalities with other domains through the uniform REST interfaces, in order to be compatible with the web and embedded devices?
RQ3: What is the most suitable open-source software framework to develop a web service client for remote supervision of the OPC UA resources and perform interoperable M2M communication with a low communication overhead?
RQ4: How do we integrate open-source cloud service functionalities with the OPC UA environment to achieve cost-effective aggregated bulk data management?
This study implemented an open-source cross-platform OPC UA server as the communication wrapper to abstract the underlying networks, aggregate data from the heterogeneous sources, and address technical and communication interoperability of distributed systems. The proposed framework also developed a cost-effective REST-based OPC UA middleware module to bridge the OPC UA resources with other domains through uniform resource representations to address syntactic and semantic interoperability. This inexpensive middleware module acts as an OPC UA client to obtain the server address space node credentials and map these collected resources into the prescribed web requests, in order to share the aggregated resources with the external services. As a proof of concept, we developed a RESTful standalone web service client and cloud platforms to collect the address space node credentials from the middleware, present these received resources virtually to perform remote process supervision, and enable aggregated bulk data management by allocating the received data in the cloud database. The middleware module can also share its resources with other existing open-source and proprietary platforms for data management, storage, and visual analysis. Furthermore, the Telegram Bot API functionalities were integrated with the middleware script to perform OPC UA tasks by invoking messages as commands from Telegram messenger. The overall architectural framework of the developed system is depicted in
Figure 1.
Current industrial practice involves OPC UA-based propriety infrastructures, usually encapsulated for commercial purposes, and caters to the specific data transfer protocol. The proposed framework shows salient suitability in the IR4.0 context due to its simplified data flow, ubiquitous content accessibility, and open-source low-cost integration capability. The open-source architecture of the proposed system enables high design flexibility, allowing it to adapt from embedded devices with small service sets to Windows/Linux platforms and industrial automation systems with full functionalities. This scalable architecture has been tailored for engineering purposes, where the targeted users are students, researchers, and small and medium scale industries, to support seamless aggregated heterogeneous data management with several application domains for wider resources usability. The contributions of this proposed system summed up as:
We implemented an open-source vendor-neutral cross-platform OPC UA server to address communication interoperability and enable seamless aggregated data exchange across the heterogeneous distributed system at different automation levels;
We proposed an open-source REST-based OPC UA middleware to unlock OPC UA network capabilities to address semantic and syntactic interoperability to integrate aggregated data with other domains such as web, Telegram, and cloud services;
We developed a standalone web service client to receive OPC UA resources from the middleware, enable OPC UA address space remote supervision, and provide end-user application support to perform interoperable M2M communication;
We implemented a cloud platform named ThingsSentral
TM (
http://thingssentral.io:443/) to collect and store the middleware data in a dedicated database for further visual data analysis, and aggregated bulk data management. The proposed framework also synchronizes Telegram messenger as the remote commander unit to perform remote supervision and notify users about the event occurrences;
The rest of this paper is structured as follows. The contextualization of the analysis framework and the systematic development procedure of the proposed system is described in
Section 2.
Section 3 validates the adaptability of the developed system in real-time scenarios by integrating the aggregated data from a proprietary DeviceXPlorer and Prosys OPC UA server to the web service and the ThingsSentral
TM and ThingSpeak cloud platforms. In
Section 4, the performance analyses of the developed system deployed in a Raspberry Pi and Intel NUC PC are discussed in terms of platform resource utilization and operational time requirements. Finally, detailed discussions on the experimental results, the identified limitations, and future research scopes of the proposed framework are mentioned in
Section 5.
3. Performance Evaluation of the Proposed System
Figure 14 shows the data flow sequence in the entire system framework. Upon initialization, the OPC UA server creates an address space with the desired number of variable nodes, periodically calls the helper function to obtain sensor values (
Figure 14(1)), and updates the values returned from the helper function in the dedicated address space variable nodes (
Figure 14(2)). The connected OPC UA clients make requests (
Figure 14(3,5)) to the OPC UA server to obtain these address space resources, and the server responds to the valid client requests (
Figure 14(4,6)). The middleware, as an OPC UA client, also sends periodic requests to the OPC UA server (
Figure 14(7)) to obtain these node credentials. Upon receiving the server response (
Figure 14(8)), the middleware script then constructs web requests concatenating the received address space resources and sends the requests to the ThingsSentral
TM cloud (
Figure 14(9)) and web service (
Figure 14(12)). The ThingsSentral
TM cloud (
Figure 14(10)) and web service (
Figure 14(13)) receive and parse the payload from the request and send Response Code 200 to the middleware for successful request completion. ThingsSentral
TM cloud platform later stores these parsed data in a dedicated database table (
Figure 14(11)). The GUI of both the web and cloud platforms allows the investigation of selected sensor data graphically and statistically by retrieving the data from the database server. It also offers to download the data in several formats (
Figure 14(14–17)). The web service client (
Figure 14(18)) and the Telegram messenger (
Figure 14(20)) can request that the middleware perform OPC UA server-side tasks. Upon successful completion of the requested tasks, the middleware responds to these services for each request individually (
Figure 14(19,21)).
The OPC UA paradigm provides a discoverable endpoint URI to share the server address space credentials with the OPC UA clients. As an OPC UA client, the developed middleware module has sufficient realization of this paradigm to integrate with proprietary OPC UA servers such as DeviceXPlorer from Takebishi, NI LabVIEW from National Instruments, SIMATIC WinCC from Siemens, etc. Open-source OPC UA server simulators from Prosys, Integration Objects, Matrikon OPC, Unified Automation, etc. are also supported for OPC UA-based application testing, simulation, and validation purposes. Incorporating REST and the open-source design of the presented middleware module can feed the aggregated data from these OPC UA servers to the developed web service, the ThingsSentral
TM cloud, and several RESTful platforms such as ThingsBoard, IoTivity, ThingSpeak, etc. In this sense, different proprietary and open-source technologies can exchange information, which enables high design flexibility for affordable seamless aggregated heterogeneous data integration and real-time monitoring. As a proof of concept, the proprietary DeviceXPlorer OPC UA server (DXP) and the open-source Prosys OPC UA simulation server were integrated with this middleware module to aggregate data from heterogeneous sources. The ThingSpeak open-source IoT platform [
49], the developed web service client, the ThingsSentral
TM cloud, and Telegram messenger were configured to integrate data from these aggregated OPC UA servers to the web domain. The open-source Prosys OPC UA simulation server address space constitutes an object class, with one node class of several simulation variable nodes such as counter, random, sawtooth, sinusoid, square, and triangle, with necessary attributes. A Science, Technology, Engineering, and Mathematics (STEM) trainer kit was integrated with the proprietary DXP OPC UA server deployed in a Windows-based Intel NUC PC. Two Delta DVP-12SE Programmable Logic Controllers (PLC) from this STEM kit were connected with the DXP OPC UA server to create a dynamic heterogeneous network. One of them included an analog extension module to interface with analog devices and communicate with the DXP server through the COM port via a Modbus RTU RS485 interfacing link (PLC-1). The other PLC interfaced with some discrete devices and communicated with the DXP server through Ethernet port via ModbusTCP protocol (PLC-2). Upon establishing the connections, two object nodes, named ‘Analog’ for PLC-1 and ‘Digital’ for PLC-2, were created in the DXP server address space. Each of these object nodes was configured with the necessary variable nodes for accessing and storing the PLC data through the corresponding memory register address. Running these servers provided discoverable an endpoint URI, [opc.tcp://{platform address}/OPCUA/SimulationServer] for the Prosys simulation and [opc.tcp://{platform address}/52240] the DXP server to share the semantic information of their corresponding address space node credentials with the OPC UA clients.
Figure 15 depicted the evaluation setup for investigating the performance of the developed system with the proprietary and open-source software. The middleware periodically interacts with both the DXP and Prosys OPC UA servers as an OPC UA client to access their resources in two different threads through the established TCP/IP sessions in a synchronous manner. It also shares these collected server credentials asynchronously as a REST client with the web service, ThingsSentral
TM, and the ThingSpeak platform through the prescribed request format, as mentioned in
Table 3. A version of this middleware is currently being used as a gateway node in several IoT-based final-year undergraduate student projects at Universiti Kebangsaan Malaysia (UKM). Another modified version is currently supporting factory data management for several factories in Malaysia [
56].