**4. Implementation**

As far as software is concerned, the first step for the development of the framework is the development of the cloud platform. From a hardware point of view, the architecture of the framework has been developed so that a variety of platforms can be supported including PCs, HMDs, and handheld devices. In its current form, the framework has been developed so that the technician uses a Microsoft Hololens HMD and the expert engineer operates from a desktop PC.

For the platform, a remote server is utilized. The server has a dedicated domain for the storage of files, where 3D geometries are also uploaded. The supported filetypes for the 3D geometries are .fbx, .dae (Collada), .3ds, .dxf, .obj, and .skp. Besides the 3D geometries, the storage domain supports any form of files, providing enough degrees of freedom to the engineers in exchanging files with the technicians during a remote maintenance session.

As far as the development of the software is concerned, the Unity 3D game engine has been utilized. Regarding the AR application for the technicians, Unity 3D in conjunction with the Vuforia and MRTK (Mixed Reality Tool Kit) libraries were used. The Vuforia library is useful for the creation and handling of the AR content, whereas the MRTK library supports advanced user interaction functionalities for the Microsoft Hololens HMD. Regarding the expert engineer application for the remote support, a Windows Form Application has been developed. In order to enable the communication of the two applications, the integrated Unity UNet API (Application Programming Interface) was used. Since the above-mentioned API is targeted for the creation of multiplayer games, several functionalities had to be adjusted so as to align with the requirements of the framework. UNet can be realized through different layers. In Figure 5, the functionalities per layer of the UNet are presented.

**Figure 5.** Functionalities per Layer of Unity 3D UNet API.

As far as the communication module is concerned and, more specifically, the teleconference functionality, a variety of solutions has been examined, including Skype, Agora.io [34], GIGA Video Streamer [35], and the development of a custom teleconference tool. Among the available solutions, the latter was selected, i.e., the development of a custom teleconference interface. To begin with, the integration of Skype, although it is an appealing solution, and the quality of the service itself is sufficient. It required that it was run as an external application on the Microsoft HoloLens HMD, which had a negative effect on the performance of the app, as well as being unpleasant for the end-users to switch over to two different applications including one for the communication and one for the AR remote support. On the other hand, the Agora.io and the GIGA Video Streamer are very capable SDKs (Software Development Kit) that could be integrated in the proposed framework. However, these two SDKs are not free. Therefore, the development of a custom module based on the UNet API, discussed above, was the only choice satisfying the framework's requirements. Concretely, from the expert engineer side, the video stream from the camera and the microphone of the PC was captured and transmitted to the Microsoft Hololens HMD. Form the technician side, a pop-up window appears where the technician can view the expert engineer and, through the embedded speakers of the HMD, to hear the expert's voice.

From a hardware point of view, for the on-site technician, a Microsoft HoloLens [36] device has been utilized in order to take advantage of its four environment-understanding cameras and the mixed reality capture in terms of sensors, whereas, for the development of the framework and the validation experiments, a laptop PC has been utilized. More specifically, the laptop PC is equipped with an Intel core i7 processor clocked at 2.20 GHz, 8 GB DDR4 RAM, and a NVIDIA GeForce 1060 GPU with 6 GB dedicated memory.

#### **5. Case Study and Results**

The applicability of the proposed framework is tested and validated in-vitro in a laboratory-based machine shop as well as in a real-life industrial scenario. The industrial scenario involves the machine tools used in an existing machine-shop. Taking into consideration the technological level of the machines, this use case is ideal since the machine shop consists of machines coming from different technological eras. Thus, it is of grea<sup>t</sup> importance to examine the actual impact of the proposed framework in varying machine tools. In Figure 6, the validation of the framework in the real-life

scenario are presented. More specifically, in Figure 6a, the malfunctioning machine is visible along with the shop-floor technician who is wearing the HoloLens HMD in order to perform the maintenance operations, as instructed by the expert engineer. Similarly, in Figure 6b, another engineer is testing the developed framework. It is stressed out that, in the background, the expert engineers using the laptop PC in order to communicate with the technician are visible.

**Figure 6.** (**a**) Validation of the proposed framework by shop-floor technician; (**b**) Another technician tests the framework.

Similarly, in Figure 7, the practical implementation and validation of the framework in the lab-based scenario is presented. Concretely, in Figure 7a, the expert engineer, from his desktop, can view the user's FoV, which contains the malfunctioning machine tool. As can be seen, the spatial recognition algorithm of the Microsoft HoloLens device has recognized the technician's physical environment. Thus, the engineer based on that feature can place the 3D augmentations on the user's physical environment. On the other hand, in Figure 7b, the shop-floor technician is presented. What is worth noticing is that, in the top right corner of that figure, a snapshot from the HoloLens head up display is presented. The shop floor technician, after requesting assistance from the engineer, the connection has been established and a video teleconference has begun. Therefore, a communication window has opened in the technician's FoV, where the audio and video stream from the engineer are received and displayed.

**Figure 7.** (**a**) Expert engineer visualizes video stream from technician; (**b**) Technician is performing maintenance tasks based on expert's instructions.

#### *Validation Strategy—Design of Experiments*

In the following paragraphs, the steps followed for the validation of the proposed framework in the real-life industrial scenario will be presented. Since maintenance is a very important aspect regarding the flawless and continuous operation of manufacturing systems, the validation strategy should be carefully designed and also should provide a quantified result of the impact. Therefore, for each of the machine-tools examined in this process, two basic metrics will be used. According to the guidelines given by Chryssolouris in Reference [21] regarding the mathematical programming of manufacturing systems, the operating and maintenance costs for a machine can be calculated by the equation below.

$$\text{Operating and Maitenance Costs} = \sum\_{t=1}^{T} (P/F, I, t) \sum\_{i=1}^{N} m\_{it} \mathbf{x}\_{it} \tag{3}$$

where (*P*/*F*, *I*, *t*) ≡ Present a worthy interest factor for discrete compounding when the discount rate is *I%* per period and the discount interval is *t* periods. *mit* ≡ The cost of operating and maintaining a machine of type *i* during period *t.* Is the number of machines, on one hand, in the work center *i* for period *t*.

In order to quantify the outcome of the validation process for the proposed framework, each of the above-mentioned values will be calculated in order to conclude on the efficiency of the real-time remote maintenance.

Therefore, the cost for operation and maintenance for a CNC milling machine has been calculated, based on data derived from a real-life machine-shop. The cost of operation and maintenance has been calculated on an annual basis for time horizon of five years, and the results are depicted in Figure 8. Additionally, in Figure 8, a prediction of the cost is made for the oncoming five years based on statistical regression, presented by the red dot trendline. It is stressed out that, due to the non-linear relation between the individual costs of operation and maintenance of machine tools during their lifecycle [37], a 2nd degree polynomial has been calculated. The same calculations were made for the cost estimation based on the proposed methodology and, similarly, a prediction is made for future maintenance costs. As a result, with the proposed methodology, a reduction of approximately 100 k euros is feasible in a time span of 10 years, which indicates that the framework could eventually be implemented in manufacturing systems and improve their operation.

**Figure 8.** Cost of operation and maintenance analysis.

Besides the cost for operating and maintenance, another metric has been calculated and compared regarding the downtime of the above-mentioned machine. In manufacturing systems, time resources are equally important for the financial resources. Therefore, a maintenance scenario has been investigated. For this repair scenario, the machine tool breaks down at an unpredicted time. At that time, the production process for this machine stops and the expert engineer is called to come on-site, inspect the machine, and repair it. The same scenario has also been investigated, but instead of requesting an expert engineer to visit the machine shop, a remote support session is established, and the repair is conducted by the technicians available in the machine shop. In Figure 8, the distribution of time per maintenance task is displayed for each of the scenarios examined. From Figure 6, it is undeniable that a significant amount of time can be saved with the adoption of the proposed framework, as the time for travelling and inspection are considerably longer, which affects the overall time the machine tool has been off the production schedule. Through the use of Figure 9, it is intended to reflect the difference of time distribution between the two scenarios examined during the validation process of the proposed framework. Therefore, from Figure 9, it can be concluded that, in the first scenario, i.e., the "AS-IS-SITUATION", the most time spent was on inspection and travelling, as the expert engineer had to travel on site and physically inspect the malfunctioning machine. On the contrary, in the second scenario, i.e., "PROPOSED METHODOLOGY", a totally different time distribution has been observed with the most time spent on activities relative to the acquisition of spare parts for the machine repair. In addition to that, it can be concluded that the actual repair time, which is the black portion of the pie graph, in the second scenario is bigger. This observation can be considered normal as, in the second scenario, the maintenance operations have been carried out by a field technician, whereas, in the first scenario, the same operations have been carried out by an expert engineer with solid practical experience on such machines.

**Figure 9.** Distribution of time AS-IS situation vs. Proposed Methodology.

In Figure 10, a side-by-side comparison of the specific times for the two maintenance scenarios is presented. It is stressed that, with the adoption of the proposed framework, the time for inspection of the machine can be shorter as no travelling time is involved in order to reduce *MTTR*. However, due to the inexperience of the technician carrying out the maintenance activities, the actual repair operation time is longer, as the expert engineer had to perform additional checks in order to ensure that the repair was successful.

**Figure 10.** Time per maintenance action AS-IS situation vs. Proposed Methodology.
