*1.2. Paper Organization*

As illustrated in Figure 3, the rest of this paper is organized as follows. Section 2 discusses the proposed IoTactileSim structure and setup, along with the topological view.

Section 3 presents the scenarios and case studies to demonstrate the potential of the proposed testbed. Finally, Section 3 concludes the paper and offers some future avenues.

**Figure 3.** Diagrammatic view of the structure of the paper.

## **2. Proposed Framework**

In this section, we describe the proposed IoTactileSim testbed to support a broad range of tactile IIoT services. At first, we will present the network emulator to mimic the real-world communication network, followed by a detailed discussion on the industrial robotic simulator. Finally, we present the topologic view of the proposed simulator, along with the basic parametric settings.

#### *2.1. Simulator Structure and Setup*

The structure of the proposed testbed IoTactileSim as depicted in Figure 4, following the IEEE P1918.1 TI standard architecture. In general, the TI use cases are comprised of three key domains: master domain, network domain, and slave or controlled domain [5]. The master domain consists of operators (human or control algorithms) that exploit haptic devices. The slave domain deals with the slave robots or teleoperators that the master side operator directly controls via control signals. The network domain connects the master and slave sides to enable bi-directional communication. To control the slave side teleoperator, the master side sends the control signal, and in return receives the feedback information including haptic and audio-visual signals. The master and slave domain creates a global control loop over communication network infrastructure. To maintain stability for the tactile IIoT services and provide a real-time haptic sensation to the users, this global control loop demands a haptic packet sampling rate of ≥1 kHz, a packet loss rate between <sup>10</sup>−3–10−5, and latency ranging from 1–10 ms. The proposed IoTactileSim helps the users to investigate these strict requirements and evaluate their newly developed strategies for emerging industrial applications.

**Figure 4.** Indepth overview of the proposed virtual testbed IoTactileSim.

#### 2.1.1. Network Emulator Environment

As can be seen in Figure 4, the control signal is captured through the controller on the master side and forwarded to the network domain in a specifically encoded format. SDN, network function virtualization (NFV), and mobile edge computing (MEC) are employed with 5G technology in the core network of the proposed testbed to overcome the 1 ms latency challenge and to provide support for the next-generation industrial applications [19]. The network domain receives these packets and forwards them to the slave domain to perform the required task. The feedback in form of the haptic data is sent from the slave teleoperator to the master side human operator. The proposed IoTactileSim utilizes the Mininet emulator for the network design, resembling a real-world network operations and hardware in a virtual environment [20]. Mininet employs process-level virtualization to develop a virtual communication network with virtual hosts and connects them via virtual Ethernet pairs.

The proposed IoTactileSim enables the evaluation of large custom topologies with actual application traffic traces by deploying them into the physical network. It also enables the utilization of emerging technologies such as SDN, NFV, and MEC. In the SDN framework, control planes are separated from the forward plane in the network. The emulator is written in python language and freely available at the Mininet official website (http://mininet.org/, accessed on 10 November 2021). An overview of the basic architecture of the Mininet with open-source virtual switches Open vSwitch (OVS) and SDN standard protocol OpenFlow is depicted in Figure 5a. Mininet by itself is a network emulator that allows users to mimic real network topologies. It also enables users to build such network topologies in SDN architecture. This is what Mininet is capable of in a nutshell. It does not provide any support for integrating tactile Input/Output (I/O) modules or any other modules for that matter. It only provides us with a virtual environment where all network nodes are present (virtually) on a single physical device. The contribution of the IoTactileSim over Mininet is defined as:

• IoTactileSim allows users to integrate several tactile I/O modules with the Mininet environment.


**Figure 5.** Overview of the basic framework for Mininet and CoppeliaSim. (**a**) represents the Mininet network emulator with Open vSwitch and SDN based OpenFlow protocol, (**b**) illustrates the CoppeliaSim simulator scene environment along with control architecture.

#### 2.1.2. Robotic Simulator Environment

To develop the smart industry with human-in-the-loop and human-robot interaction haptic-driven teleoperation use cases like remote maintenance, inspection, industrial management, we utilized one of the famous industrial robotic simulators CoppeliaSim. This industrial robotic simulators CoppeliaSim is formerly known as (V-REP: Virtual Robot Experimentation Platform) [21]. The reason to use the CoppeliaSim is that it provides a range of emerging industrial applications, including factory automation, remote monitoring, safety monitoring, telerobotic operations, etc. Figure 5b illustrates the basic control architecture and scene environment of the CoppeliaSim simulator. As can be seen from Figure 5b, the simulation loop consists of the main and child script. The main script controls all child scripts attached to the specific object in the simulation environment. The remote Application Programming Interface (API) allows the user to interact with the simulator from outside the system through socket communication. The remote API client and server are responsible for providing these services through different programming languages like C/C++, Python, Matlab, Java, etc.

In the proposed IoTactileSim, we utilized the python remote API client to interact with the smart industrial application in the CoppeliaSim environment through socket communication. The control code of the developed IIoT applications is executed in the same computing machine where the network emulator was employed. The CoppeliaSim simulator is connected with a network to represent network-slave interaction that was designed utilizing Mininet. Additionally, the network is linked with the master domain and makes the master-network relationship. The overview of the connection between master, network, and slave domain using Mininet and CoppeliaSim in a single computing machine (personal computer) is illustrated in Figure 6, and an in-depth discussion on each module is presented in the next section.

3K\VLFDO0DFKLQHZLWK/LQX[268EXQWX

**Figure 6.** The flowchart illustrating the overall flow of the IoTactileSim testbed dynamics and relationship between different parts.

#### *2.2. Topological View of the IoTactileSim*

This section focuses on the architecture of the proposed IoTactileSim testbed as depicted in Figure 6. At first, we discuss the parameter initialization, settings, and user

interface to interact with the proposed testbed. Second, the topological view of the core network architecture is presented. Third, an in-depth discussion on application-agnostic design with the application and network connectivity is reported.

Initialization: In the initialization module, the simulator reads the parametric configuration files. It sets packet size, packet rate, Internet Protocol (IP) suite, IP address, link bandwidth, and link latencies. This module facilitates the users to provide the parameter settings as per their experiment need. If the user does not provide the parametric settings, then it automatically uses the default values of the parameters, as defined in Table 2.

**Table 2.** Summary of parameters and settings used for Simulation.


Start Network Emulator: In order to design a real-world network, a Mininet emulator creates a custom topology with five OVSs and three hosts. The hosts act as a standard computing machine and are responsible for the master domain, slave domain, and tactile support engine. The OVSs are connected with a single SDN controller. The SDN controller decides to handle the data plane and allows the network operator to control and manage the whole network via API provided by the Mininet. Therefore, we utilized the Mininet Python API in IoTactileSim, so that the users can change the network settings as per their experiment demands to evaluate their newly developed approaches. The users can change the parametric values from the configuration files as discussed earlier in the initialization step.

Simulation Cycle: After creating the network topology with three hosts that work as a master, slave, and tactile support engine, the simulator enters into the simulation cycle. In the simulation cycle, the actual experiments are performed as per defined conditions by the users through parametric settings or default values. The network host that acts as the master side, connects to the haptic device that the human operator uses to send the control signals to the salve side manipulator. The other host acting as a slave, connects to the CoppeliaSim simulator for executing the control commands in the designed virtual/physical tactile IIoT applications. After conducting a desired control experiment, the haptic feedback is sent to the human teleoperator at the master domain. This loop runs until the simulator reaches the defined threshold values like 1000 packets, etc. Finally, at the end of the simulation cycle, the simulator stores the experimental results into a file to investigate various network impairments.

Performance Analysis: This module receives the stored experimental data file and compiles result graphs to understand the strengths and weaknesses of the conducted experiments from the QoE and QoS perspectives. After representing the experimental result in the visual form, it stores these results, and enables the users to perform new experiments or to exit the simulator. The list of the default parameter values and settings used in the proposed IoTactileSim are summarized in Table 2. The proposed IoTactileSim is publicly available at Github (https://github.com/zubair1811/IoTactileSimV1.git, accessed on 30 November 2021) to the interested researchers to conduct extensive experiments to evaluate their suggested approaches.

#### **3. Result and Discussion**

In this section, we demonstrate the effectiveness of the proposed IoTactileSim with two different tactile industrial scenarios using the simulation environment and parameters setting defined in Table 2. These two realistic applications belonging to tactile industrial use cases define as follows


Moreover, these scenarios can be classified into two categories, like offline and realtime applications. On the one hand, offline (teleoperation with 3DoF) experimentation means that we already have a static dataset of some real-world teleoperation applications and utilize previously collected data for analytical analysis. On the other hand, real-time or online (haptic-driven remote operations) experiments indicate that interaction data between operator and teleoperator are collected in real-time to make a suitable decision to ensure stability and transparency. The real-time scenario is complex compared to offline because it deals with more data under time constraints. Most of the existing studies on industrial testbeds just utilized the offline methodology, while the proposed IoTactileSim considered both scenarios. The discussion considering offline and real-time scenarios is presented in detail in the following subsections.

#### *3.1. Scenario I: Teleoperation with 3DoF*

Scenario I considers the offline experimentation, where publicly available 3DoF haptic traces in [22] on teleoperations were utilized. To record the haptic traces, a human operator employs a haptic device (Phantom Omni) at the master to interact with the virtual environment, which acts as a slave domain. The virtual environment is comprised of a rigid movable cube lying on a wooden, smooth surface. The human operator makes an interaction (static and dynamic) with the rigid cube via the haptic device and receives force feedback. Figure 7 illustrates the 3DoF position and velocity control by the human operator via the haptic device and the received force feedback of the used haptic dataset [22] for experimentation. In the proposed IoTactileSim testbed, we transmit the control signals (position/velocity) from the master side to the slave side and ge<sup>t</sup> the haptic feedback (force). To minimize the E2E communication delay, a common practice is to transmit the haptic traffic packets instantly after receiving sensors' data, resembling a real-time tactile industrial IoT application.

**Figure 7.** Dynamic interaction of the human operator with virtual application via haptic device. ( **a**) positions of the human operator's hand at master side device, (**b**) velocity traces of the operator (**c**) force data traces of the teleopertor x, y and z-axis.

We adopt the same method in our experiments; after reading the sensors data, the system makes packets of the haptic traces as following:

$$PacketSize = Entropy / \text{LDP} / \text{IP} / 8 \times N\_{DoF} \tag{1}$$

where *Ethernet*/*UDP*/*IP* indicates the header of the ethernet, user datagram protocol, and internet protocol layer, respectively. *NDoF* is the number of DoF in the experimental data. We are using 3DoF, so the formulation can be evaluated as:

$$\begin{aligned} \text{Packet Size} &= 14/8/20/8 \times 3 \\ \text{Packet Size} &= 14 + 8 + 20 + 24 = 78 \end{aligned}$$

The interface of the IoTactileSim during communication between master and slave domain is depicted in Figure 8. At the master domain, control signals from utilized 3DoF haptic traces are selected and transmitted to the slave domain through the network domain. Similarly, after receiving specific control signals, the slave domain returns the corresponding force feedback to the master domain. The overview of the data flow interfaces between master and slave for the scenario *I* is depicted in Figure 8.

The performance analysis for a scenario I in terms of round trip delay is presented in Figure 9. To investigate the effect of the number of haptic data packets on round trip delay for IoT applications, the scenario I was simulated for the number of haptic data traces = 10 to 10,000. The latency investigation using IoTactileSim for a scenario I with the number of packets = 10, 100, 1000, 10,000 is depicted in Figure 9a–d, respectively. It can be seen clearly from the results, the tendency with an increase in the number of haptic data packets the round trip delay decreases from 5 to 2 ms. In Figure 9a, with the number of packet = 1, the packet latency approaches 6 ms as compared to Figure 9b–d, where packet latency is below 5 ms. To elaborate this latency decrement in detail, Figure 10 illustrates the packet delay histogram for a scenario I.

**Figure 8.** IoTactileSim interface for scenario I experiment.

**Figure 9.** Packet latency investigation for scenario I haptic data transmission; (**a**) data packets = 10, (**b**) data packets = 100, (**c**) data packets = 1000, ( **d**) data packets = 10,000.

**Figure 10.** Packet latency histogram for scenario I haptic data transmission; (**a**) data packets = 10, (**b**) data packets = 100, (**c**) data packets = 1000, (**d**) data packets = 10,000.

Contrary to Figure 9, the results in Figure 10 reveal the latencies of the most frequent haptic data packets. Similar to the results presented in Figure 10a–d, the simulation results in Figure 10a–d also indicate the decrease in packet latencies from 5.8 to 2.1 ms. From Figure 10a–d, it can be seen clearly that most of the haptic traces latencies centered between 1 to 2 ms, which is one of the stringent requirements for the tactile IoT services. Figure 10b–d, depicts the improvement in the communications network impairments (delay, jitter) with the number of packets = 100, 1000, 10,000. The reason behind this is that at the beginning, the proposed testbed IoTactileSim understands and fine-tunes the simulation parameters to support delay-sensitive and loss-intolerant applications. The efficacy of the scenario I regarding reliability characterization is summarized in Table 3. The reliability of the transmitted haptic data packets is evaluated in terms of delayed/lost and out-of-ordered packets.

**Table 3.** Summary of the reliability characterization for haptic datset and real-time haptic drive teleoperation experiment.



**Table 3.** Cont.

#### *3.2. Scenario II: Haptic-Driven Remote Operations*

In this section, we will present the real-time control of the teleoperator in the virtual environment to mimic the real-world tactile industrial remote operations. Similar to the scenario I, II also consists of the master, slave, and network domain where virtual teleoperator developed in CoppeliaSim acts as salve domain. In the master domain, physical haptic devices (haptic computer mouse, glove, and hapkit) are used to interact with the virtual environment, as illustrated in Figure 11. These haptic devices are easy to develop because their supplementary material is available publicly for the research community. The tactile computer mouse was presented in [14], while the study in [23,24] provide the design and development detail on a haptic glove and Hapkit, respectively. However, in this experiment, we only employed the (computer mouse and glove) to interact with the teleoperator as slave side. We mapped the physical computer mouse and glove X and Y direction to the XY coordinates of the developed virtual teleoperator in CoppeliaSim. The key focus of this experiment is to investigate the communication network parameters (latency, reliability) that affect the TI services. Additionally, it also demonstrates the potential of the proposed IoTactileSim to provide TI services under TI QoS/QoE requirements (1–5 ms). Human operators in the master domain use the haptic device to interact with the teleoperator at the slave side and receive haptic feedback. The interface of the IoTactileSim during direct controlling of the teleoperator in the virtual environment is depicted in Figure 11.

To observe the effect of the number of data packets on latency for scenario II, the simulation results are summarized in Figure 12. These results also demonstrate that with the increase in the number of packets from 10 to 10,000, the network communication latency tends to decrease. In this experiment, we directly control the teleoperator in the virtual environment via a computer mouse in real-time. The control signals from the computer mouse are packetized as defined in (1), sampled as per haptic system requirement, and transmitted to the teleoperator using default parameters values as listed in Table 2. The teleoperator at slave side receives the control commands to perform the required task and backward the force feedback to the human operator. Figure 12a–d, indicates the results of the packet latency for the number of data packets 10, 100, 1000, 10,000, respectively. In Figure 12a with the number of packets = 10, the value of latency lies between 9–7 ms.

In Figure 12b, up to 40 packets, latency value remains higher than 5 ms, and after that, the system gets convergence round trip latency around 2.5 ms. Similarly, the results in Figure 12c, gain a minimum latency value of 2.5 ms from the 20th data packet to the 1000th packet. To continue on a similar line as mentioned above, Figure 12d, exhibits a quick decreasing trend in round trip latency from 6 to 2.2 ∼2.0 ms , as the number of data packets increases from 10, to 10,000.

6ODYH'RPDLQ

**Figure 11.** IoTactileSim interface for scenario II experiment.

**Figure 12.** Packet latency investigation for scenario II real time haptic-driven remote operation; (**a**) data packets = 10, (**b**) data packets = 100, (**c**) data packets = 1000, (**d**) data packets = 10,000.

To elaborate this packet latency convergence in a better way, Figure 13 illustrates the histogram of the frequent data packets regarding packet latencies. The results in Figure 13c,d indicate that the packet latency reduces for an increase in the number of packets compared to results in Figure 10c,d. In addition, these results depict that the latency is more concentrated between 2 to 2.3 ms. It is also interesting to observe that, for the higher number of the data packet with a higher sampling rate, the proposed IoTactileSim is capable of reducing congestion and maintaining the application latency requirement. In addition Table 3 presents the in-depth reliability analysis for the scenario II experiment. As it can be seen clearly from Table 3 for scenario II the percentage of delayed or dropped packets decrease from 100% to 0.40% (10 to 39 data packets) with 100% to 0.30% out-of-order sending packets from 10 to 10,000 number of data packets.

**Figure 13.** Packet latency histogram for scenario II real time haptic-driven remote operation; (**a**) data packets = 10, (**b**) data packets = 100, (**c**) data packets = 1000, (**d**) data packets = 10,000.

In summary, the packet latency convergence analysis in Figures 9 and 12 and periodic packet variation analysis Figures 10 and 13 for use case scenarios I and II indicates that the proposed virtual testbed IoTactileSim provides the facility to the users to implement complex tactile industrial use cases, evaluate their proposed strategy, and investigate the QoS and QoE requirement of the implemented tactile IoT services. Some of the complex tactile IIoT use cases are illustrated in Table 4. The proposed IoTactileSim concentrate on providing QoS and QoE provisioning by taking different network parameters into account. Based on the mentioned complex tactile IIoT use cases along with requirement specification (delay, packet size, packet rate, packet loss rate, etc.) in Table 4 it indicates that IoTactileSim can ensure strict QoS-based traffic. The main objective of this paper is to provide a tool to minimize the network development cost while realizing the stringent QoS/QoE requirements for tactile IIoT applications. Moreover, it also offers to implement edge intelligence to the designed tactile support engine, which can be leveraged to improve QoS and QoE provisioning in highly dynamic network environments. The users can deploy machine learning, specifically reinforcement learning models, to track the frequently dynamic network environment states and make online decisions to improve network conditions and support time-varying user demands.


**Table 4.** Tactile IIoT use cases requirements specifications and characteristics supporting by proposed IoTactileSim.

#### **4. Discussion and Future Work Directions**

In our previous work [23], we analyzed the different haptic gloves and investigated how data processing increased the latency in the haptic communication loop and proposed a low-latency haptic open glove (LLHOG). Contrary to previous work, the focus of this paper is to provide network infrastructure to transmit haptic traffic between operator and teleoperator and simulate delay-sensitive and loss-intolerant tactile IIoT applications. However, there are various industry 4.0 applications under use cases class C, such as fleet management, tactile-driven logistics, cooperative robotics, and motion control, which demand higher QoS and QoE. To allow these real-time applications, the utilization of edge computing is required. Therefore, there is a need for edge-based network systems with native machine learning parts to provide the QoS and QoE requirement provisioning for these applications. In this regard, as a future, an edge-based ITE is developed as a tactile support engine to enable the ability for the user to train and deploy machine learning models at the edge to ensure QoS and QoE. The conceptual diagram to design and deploy the trained model on ITE is illustrated in Figure 14.

**Figure 14.** Conceptual architecture of ITE in IoTactileSim.

In future work, more practical challenges regarding tactile IIoT in the real-world scenario need to be considered. As discussed above, providing required QoS and QoE in the real-time complex industrial application is more challenging than simulation analysis. Therefore, we indented to test the proposed IoTactileSim in real-time physical IIoT scenarios and demonstrate the real-world experiment design overview in Figure 15. On the master side, we utilized the LLHOG, which consists of the rotary position sensors with a minmax scaling filter to send haptic data. The bionic robot hand, which consists of Arduino

and servo controllers, is used at the slave side. The specification, sample code, and documentation are available at the official website (https://wiki.dfrobot.com/, accessed on 30 November 2021 ). The proposed IoTactileSim connects the LLHOG and bionic robot hand to develop a closed control loop. The ITE is also integrated with the proposed IoTactileSim to monitor the network dynamics and guarantee the QoS and QoE requirements for tactile IIoT applications.

**Figure 15.** Modular representation of ITE in real-world scenario.
