**Collaborative Augmented Digital Twin: A Novel Open-Source Augmented Reality Solution for Training and Maintenance Processes in the Shipyard of the Future †**

**Aida Vidal-Balea 1,2,\*, Oscar Blanco-Novoa 1,2, Paula Fraga-Lamas 1,2,\*, Miguel Vilar-Montesinos <sup>3</sup> and Tiago M. Fernández-Caramés 1,2**


**Abstract:** Large companies use a lot of resources on workshop operator training and industrial machinery maintenance since the lack of this practice or its poor implementation increases the cost and risks of operating and handling sensitive and/or hazardous machinery. Industrial Augmented Reality (IAR), a major technology in the Industry 4.0 paradigm that may enhance worker performance, minimize hazards and improve manufacturing processes, could be beneficial in this situation. This paper presents an IAR solution that allows for visualizing and interacting with the digital twin of a critical system. Specifically, the augmented digital twin of an industrial cooler was developed. The proposed IAR system provides a dynamic way to perform operator training with a full-size model of the actual equipment and to provide step-by-step guidance so that maintenance processes can be performed more safely and efficiently. The proposed system also allows several users to use devices at the same time, creating a new type of collaborative interaction by viewing the model in the same place and state. Performance tests with many simultaneous users have been conducted, with response latency being measured as the number of connected users grows. Furthermore, the suggested IAR system has been thoroughly tested in a real-world industrial environment.

**Keywords:** industrial augmented reality; collaborative augmented reality; mixed reality; Industry 4.0; maintenance; training

### **1. Introduction**

The introduction of new technologies that optimize production processes has increased growth to the point of being considered a fourth industrial revolution. Thus, the term Industry 4.0 was established to name this new stage in which industry integrates manufacturing processes, information systems and communications technologies. In particular, Industrial Internet of Things (IIoT) and Industrial Augmented Reality (IAR) are key for the Industry 4.0 paradigm [1].

In addition, cyber-physical systems enable the interconnection between the real and virtual worlds. Although it is not a new term [2], digital twins [3] have experienced an increase in interest in recent years. In the case of the shipbuilding industry, processes are constantly evolving by introducing new technologies to adapt to Industry 4.0.

The aim of this paper is to demonstrate in a practical scenario the capabilities of Industry 4.0 for shipbuilding by mixing three key technologies (digital twin, IAR and IIoT) with the objective of optimizing maintenance, production and operator training tasks. To this end, an application has been developed for Microsoft Hololens Augmented Reality glasses [4]. It consists of a digital twin of a cooler and it allows the visualization of

**Citation:** Vidal-Balea, A.; Blanco-Novoa, O.; Fraga-Lamas, P.; Vilar-Montesinos, M.; Fernández-Caramés, T.M. Collaborative Augmented Digital Twin: A Novel Open-Source Augmented Reality Solution for Training and Maintenance Processes in the Shipyard of the Future. *Eng. Proc.* **2021**, *7*, 10. https://doi.org/ 10.3390/engproc2021007010


Academic Editors: Joaquim de Moura, Marco A. González, Javier Pereira and Manuel G. Penedo

Published: 30 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

animations and the interaction with the model in a virtual way. In addition, the application displays real-time information obtained from sensors embedded in the cooler.

#### **2. Design of the System**

The architecture of the designed system is depicted in Figure 1, where the multiple modules of the system can be distinguished clearly. Both the AR/MR and IoT layers make use of the Service Layer to communicate with each other. In addition, the Service Layer is in charge of coordinating all the processes as well as managing the different protocols used by the heterogeneous devices involved in the system.

**Figure 1.** Diagram of the architecture of the system.

In order to show the capabilities of the proposed system, an application was developed for Microsoft HoloLens smartglasses. Such an application allows for visualizing the 3D model of a cooler, as well as to receiving indications on repair and maintenance processes and operating parameters measured in real time by embedded sensors. The application is shown in Figure 2, which has been captured by one of the users, who watches in real time how the other user interacts with the digital twin of the cooler. Thus, both users share the same collaborative experience.

**Figure 2.** Expanded view of the parts of the digital twin of the cooler (**left**) and user performing a repair procedure on the real cooler, being guided by the developed AR application (**right**).

#### **3. Experiments**

In order to measure the performance of the system, a series of tests were carried out in a real environment. Taking advantage of the fact that Unity [5] allows the development of applications for different environments and operating systems, a mock-up version of the IAR application was developed to emulate its behavior when running multiple instances on a computer. Thus, a desktop computer can perform operations as if it were multiple users using the IAR application. In this way, the network load can be simulated in a realistic scenario without needing to purchase multiple pairs of HoloLens smart glasses.

The performed test cases were carried out for scenarios with 5, 10, 15 and 20 simultaneous clients. Average latencies were measured using 2.4 GHz and 5 GHz IEEE 802.11 ac networks. In all cases, the measured latencies included the processing and rendering times of the information contained in the packets. In other words, the latency is measured since a packet is generated at the source device and until it is processed and the information is rendered on the screen of the destination device. This includes waiting times related to the frame rate limits imposed by the smart glasses graphics engine. Measuring times in such a way ensures that the results provide an accurate representation of the waiting times that would be experienced by a real user that executes the application.

The average obtained latencies are shown in Table 1. As it can be observed, latency increases with the number of users. Additionally, the variance is notably low, especially for a lower number of users, due to network stability (i.e., users generate less interference). The differences between the 2.4 GHz and 5 GHz networks are approximately 3–5 ms, always higher in the case of the 2.4 GHz network due to the saturation of the spectrum in terms of the frequency and the characteristics of the physical layer.

**Table 1.** The 2.4 GHz and 5 GHz average latency measured for a device on the network (ms).


#### **4. Discussion**

The developed digital twin application was created and tested to develop shipyard assembly instructions and support. The proposed features allow for the detailed visualization of virtual elements in their full size, positioned precisely in the location where the real elements would be positioned. Moreover, the proposed application facilitates IIoT real-time interactions, collaborative training, and assistance tasks, all of which help to speed up manufacturing operations.

**Funding:** This work was supported by the Plant Information and Augmented Reality research line of the Navantia-UDC Joint Research Unit (IN853B-2018/02). The authors would like to thank CITIC for its support. CITIC, a research center accredited by Galician University System, is funded by "Consellería de Cultura, Educación e Universidades from Xunta de Galicia", with 80% of funds coming from ERDF Funds, ERDF Operational Programme Galicia 2014-2020, and the remaining 20% from "Secretaría Xeral de Universidades" (Grant ED431G 2019/01).

**Conflicts of Interest:** The authors declare no conflict of interest.

