Remote Service for Production Machines Using a 3D Collaborative Environment and Digital Twins ^†

Abstract

As digital twins are bringing increasing benefits to the product development process, their growing popularity is noticeable. While they are already very popular within the development process, they are not very common in service applications, although they have great potential for usage in remote maintenance, remote training, and optimization tasks, without the need to stop physical production. This article aims to describe a structure of a digital twin in a collaborative environment that enables service experts to guide field technicians through maintenance tasks using an interactive simulation of a production machine and immersive technologies.

1. Introduction

As the adoption of digital twins by engineering teams is becoming more popular, researchers and enterprises are starting to explore the use cases and benefits of remote service applications based on digital twins. It is difficult to find qualified service staff willing to constantly travel to customer sites for maintenance of forming machines. Since these machines are quite complex, special knowledge is needed to complete service tasks. Working with digital twins and immersive technologies can drastically improve remote service capabilities. If a company uses remote services, the travel time of service experts decreases while productivity increases, because they can work on more projects within the same timeframe and thus increase their work–life balance. However, so far, digital twins are often only used in special departments; therefore, they do not cover every aspect of a machine, making them less suitable for service tasks. Thus, digital twins are not generally used in all areas of the product life cycle. To achieve full adoption and interoperability of a digital twin, it is important that there is one common database for all departments that use a digital twin. An attempt to creating an IT system accessible to all disciplines is presented in ref. [1]. This article describes a project to enable simultaneous real time collaboration of a digital twin and the usage for remote service purposes.

This project aims to use immersive technologies and networking to create an interactive and realistic simulation of the actual machine, which enables it to be used for service tasks. The benefit of immersion is that it allows the user a natural perspective, which allows for a better awareness of design and function [2]. The system should also be usable by “non-professionals” who are not familiar with items such as CAD software and numerical control interfaces. With this project, the researchers hope to lower inhibitions and help ensure that digital twins are increasingly used in remote service applications. This article does not include a fully operational commercial system but rather a proof of concept for an accurate digital twin framework that will enable simultaneous collaboration across multiple sites.

2. Relevant Work

The use of immersive technologies like Virtual Reality (VR) and Augmented Reality (AR), summarized in the generic term Extended Reality (XR), for technical purposes is not a new idea. There have been many attempts to use XR technologies to interact with digital copies of the actual technical product. However, for remote service applications, there needs to be some form of networking where people can interact with each other as well as with the digital twin across multiple locations. An approach to implement a real-time multi-user XR application for face-to-face reviews, training and discussion is presented in ref. [3]. A small experiment was conducted to test how VR applications affect the performance of teams in finding design flaws. The results show that the teams that used a XR application to find design flaws performed better than the teams that used a traditional design review process with CAD and 2D drawings. This approach can also be used for service purposes, where service technicians can collaborate with engineers to find the root cause of a problem using a machine’s digital twin, since production does not have to cease to look for design flaws. A study examining the impact of immersive technology and collaboration on engineering is described in ref. [4]. This study reports that virtual reality is helpful for global engineering teams because it allows users to use their own perspective, which makes the software more valuable for collaboration than text messaging, voice calls, or on-screen visualizations.

3. Demands and Use Cases for Collaboration for Remote Service Applications

To be an effective tool for remote service applications, it is important that all relevant information for service is present within the digital twin. This means that it is essential that the engineering teams in the development process are working on the same database. Only if they work on the same platform is it possible for the service team to trace what they did when they designed and programmed the machine. So, to cover all necessary aspects, the requirements of the following disciplines will be considered [5] (p. 2):

• Mechanical Engineering
• Information Technologies
• Sales

The following section will describe the use cases for each discipline.

3.1. Mechanical Engineering

The main purpose of a digital twin for mechanical engineers is to visually evaluate their design. Immersion can bring the greatest benefits to mechanical engineers because it allows them to view a model from a human perspective. This makes it very easy for them to get an idea of the actual dimensions, design, and ergonomics of the machine. In order for mechanical designers to discuss the project, an intuitive user concept is required so that, for example, production workers or service technicians can also provide direct feedback on the design and ergonomics of the model [5] (p. 3). Another use case for mechanical engineering is modifications on existing machines. These can be test fitted and tested with the digital twin, which reduces the error rate of the actual machine and therefore reduces the downtime of a working machine.

3.2. Information Technologies

The IT department will use a digital twin mostly for developing and testing their programs, which will be used in the production process later. A digital twin will add value here, because it enables IT engineers to start development of their software earlier and do more testing, which will result in a lower error rate at the start of production. Another advantage for the IT department is that IT engineers will have a better understanding of the actual production process. In addition, the service department can test changes on the machine to optimize the running process without any interruptions [5] (p. 3).

3.3. Service

The service department can also make use of a virtual walk-in digital twin. Service staff can be trained on the machine (operation, troubleshooting, maintenance, assembly, and disassembly) in VR by simulating various events and faults. They can also look for design flaws by trying to reconstruct errors in the field to find a solution. By wearing an augmented reality (AR) headset, a service technician in the field can be assisted by a video stream from an expert at headquarters showing the work steps on the digital copy of a machine. The expert can show the steps in VR, which can be live-streamed to the augmented reality headset [5] (p. 4). In addition, the camera of the augmented reality headset can stream the view of the service technician back to the expert, so they can guide the technician and interrupt if something is done wrong. If there is no augmented reality headset available, it would be also helpful to use a mobile device because it can display a video stream from the simulation, and the camera can be used to monitor the work of the service technician.

4. Functions of the Collaborative Environment

As discussed in the previous paragraph, a digital twin that can be used for remote maintenance of forming machines has many requirements. The following are the essential features that need to be present to make it usable for day-to-day business [5] (p. 4):

• Powerful visualization
• Interface for the control software
• Accurate physics

An IT system that provides these functions is described in ref. [1]. The main purpose of this article is to describe what kinds of tools are needed to make a digital twin accessible for remote service purposes. A special focus also needs to be on the user concept for the different use cases. While VR and AR are useful in searching for mechanical problems, they do not present benefits to a service programmer who wants to optimize or debug the control software, because the programmer is in front of a personal computer and wants to test the changes simultaneously. Another issue is the availability of AR or VR headsets. In the field, there might be only a mobile device available. The hardware that could be used to join a session with the digital twin is as follows:

• AR/VR headsets
• PC/laptop with screen, mouse, and keyboard
• Mobile device with camera

The functions and required features for each kind of hardware are shown in

Table 1. Overview of hardware and features [5] (p. 5).

Device	Used for	Required Features
VR Headset	General inspection Interact with digital twin Collaborate in meetings	Good visual quality of the model Interfaces to interact with the model in VR, assembly studies Streaming user movements/voice chats/display avatars of other users
Personal Computer with Screen	General inspection Software engineering Attend meetings	Good visual quality of the model interface for industrial control software Move within the model, video streaming, voice chat
Mobile Devices	General inspection Attend meetings	Move within the model, video streaming, voice chat
Augmented Reality Headset	Supporting/ training service staff on site without physical presence of an expert	Video streaming in both directions, voice chat

.

5. Architecture of the Collaborative Digital Twin

The base of the collaborative environment is Unity3D (Unity Technologies, San Francisco, USA). With its open architecture and the C# API, which can extend and adjust basically every function of the development platform, it has become popular for industrial purposes that require interactive visualizations. With all the tools and assets that already exist, the software can easily be extended with an interface to basically every other type of software. It is also easy to implement both immersive and non-immersive user concepts withing one 3D platform. The architecture of the digital twin is described in ref. [2]. The Unity3D-based application is extended to commercial software for simulation and control. Virtuos (ISG Industrielle Steuerungstechnik GmbH, Stuttgart, Germany) is a software to create digital twins using an accurate mathematical model for each component of the machine, such as hydraulic cylinders, motors, and frequency converters. As control software, Kuka OfficeLite (KUKA AG, Augsburg, Germany) is used for robot programming, and TwinCat (Beckhoff Automation GmbH & Co. KG, Verl, Germany) is used to program and simulate PLC (Programmable Logic Control) units.

To enable collaboration in this research project, a simple server–client structure, which is displayed in Figure 1, was chosen. The hosting participant provides the resources for all calculations of the digital twin, such as the physics, movement, PLC, and robot control software. The other participants join the session and only stream their movements, camera feed, and voice feed. By centralizing the expensive calculations, the clients can also join with light hardware (e.g., mobile devices, AR headset, etc.). Of course, the environment could also be hosted in a cloud with scalable infrastructure, but for simplicity and proof of concept, an architecture with a simple host on a powerful workstation was the appropriate choice.

6. Use Case Example of Remote Service with Augmented Reality and Virtual Reality

This particular use case is important for remote service application. As stated above, service experts are rare, and extensive travel times are inefficient and costly. By using a digital twin in a collaborative environment, the service expert can stay in the head office and assist the field technician remotely, while being able to control the work. To quickly set up a VR session in an office, a previously developed custom rack that contains everything necessary for VR makes this a quick and easy process. In this use case example, a maintenance task at the automation demo facility of Albstadt-Sigmaringen University is simulated. A mechanical error occurs that prevents the machine from producing. Normally, if the machine is new and the error cannot be fixed by the operating staff on site, the machine would not be able to be used until a qualified service technician from the manufacturer arrives to fix the error. This can take several days, until a service expert is free and able to travel to the customer site.

With a remote maintenance system, a collaboration session can be hosted immediately. Figure 2a shows the environment for the service expert, with the VR rack where the expert can join the session via the VR headset. The field of view is displayed in Figure 2b.

The less experienced service technician then joins the session with an AR Headset. Figure 3a shows the technician at the machine wearing a Microsoft HoloLens (Microsoft Corporation, Redmond, USA). Figure 3b shows the field of view in AR. The video feed of the first-person view of the expert interacting with the digital twin is streamed to the field of view. This way, the technician can be guided to safely shut down the machine and perform the maintenance task. The camera feed of the AR headset is also streamed back to the expert, so they can monitor the activities of the field technician and interrupt if the technician is about to make a mistake. The live video feed from the field is also important for the expert to validate the result of the manufacturing process, especially in machines with processes that are not easy to simulate, such as forming machines.

If the field of view of the VR headset is not ideal, it is also possible to switch to a mixed reality stream that shows the expert interacting with the digital twin, if there is a green box and a dual lens camera available where the VR session is hosted. The creation of the mixed reality video is shown in Figure 4. Alternatively, if there is no possibility to use an AR headset, this might also work with a smartphone, with the display showing the video stream of the expert and the camera filming the actions of the technician in the field. This would have the downside that the phone needs an additional tripod or needs to be held by the technician, which makes it more complicated to handle.

If there is no stable permanent internet connection available, a video of the expert can also be recorded and then be sent, so the technician can download it to a device whenever a stable internet connection is available.

7. Discussion

The experiments have shown that digital twins can be a great addition to remote service. The fact that a service expert can show the steps in real time on a digital twin of a production machine makes it easier for the technician in the field to follow the steps of the expert. Immersive technologies have shown to be superior to a simple phone call in this case, especially for service tasks on more complicated machines, such as forming machines. They have the potential to reduce the travel time of service experts and therefore make them more efficient while still improving the work–life balance of the individual. However, to be able to adopt digital twins in service applications, it is mandatory that other departments work with the same database and the digital twin be accepted as the single source of truth across the entire product lifecycle. Only if there is a common database for all aspects of a digital twin can it contain all the information needed for the service department to find errors or optimize existing processes.

One problem of the digital twin is that its construction remains complicated, there is no commercially available platform that enables accurate simulation as well as collaboration in a 3D environment. It is challenging to set up a working digital twin within this environment, because every connection must be set up manually, and for each type of device a dedicated app is needed. The barriers to actively using digital twins in daily business activities are therefore still numerous, because the development effort remains substantial and there is no out-of-the-box solution that addresses all the needs described within this article. The environment developed in this project is a great proof of work, but it is com- plicated to set up and run while also being computationally expensive. For commercial use, deploying the simulation in a scalable cloud environment should be considered to overcome bottlenecks in computational resources. For the special use case for forming machines, simulation data of the forming process could be added to the collaborative environment.