**1. Introduction**

Equipment maintenance is one of the key elements consisting of manufacturing systems. In their research work, Mourtzis et al. in Reference [1] highlight the importance equipment maintenance as part of the production lifecycle, which reaches 60%–70% of the total production cost. Therefore, being able to forecast machine maintenance operations and perform them in a short time period, can lead to successful troubleshooting and simultaneously increase machine tools availability. Additionally, since the replacement of damaged components can be as high as 70% of the total maintenance cost [1], it is one of the top priorities among manufacturing firms to discover alternative policies for cutting maintenance costs as a means of increasing their revenues. As a result, the overall production performance is optimized [2].

Besides the failure forecast, a major issue identified is the completion of accurate and error-free maintenance operations, and, thus, the reassurance of fully functional machines in the earliest time possible. Based on that concept, a considerable amount of research effort has been put on the design and development of real-time maintenance support tools and applications based on mobile devices to prevent unnecessary errors from happening [3–6]. With the evolution of Information and Communication Technologies (ICT) as well as the massive break out of augmented reality (AR), the research community extended that approach by exploiting the advantages of AR for data projection during the maintenance operations [6–11]. Generally, under the framework of Industry 4.0 in an attempt to bridge the physical world with virtual worlds, new technologies have emerged, such as

Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR) [12]. These digital technologies lie under the umbrella term Extended Reality (XR). The term XR is often used by practitioners and scientists in order to describe the three immersive technologies mentioned above [13].

However, most of the research works available on the web cover basic aspects of Maintenance and Repair Operations (MRO) and provide flexible and accurate troubleshooting [3–6]. The constant innovation of cutting-edge technologies unveils new opportunities for remote maintenance support. Therefore, there is an apparent need for the development of real-time AR frameworks for the support of MRO.

More specifically, the proposed research deals with the design and development of a novel framework for real-time remote maintenance support based on AR. Inspired by the concepts of Cloud manufacturing and Product Service Systems (PSS), the framework was built in a cloud-enabled environment aimed at bridging the di fferent working groups involved in the production lifecycle. Considering that companies seek increased operational availability, also known as availability (A) of their machines, engineers are constantly developing new maintenance policies that can ensure decreased Mean Time To Repair (*MTTR*) and increased Actual Machining Time (AMT). The following formula reflects the correlation between the above-mentioned variables and characteristics of actual machines.

$$ROTBF = MTBF - AMT\tag{1}$$

where *ROTBF* is the Remaining Operating Time Between Failures. *MTBF* is the Mean Time Between Failures. *AMT* is the Actual Machining Time.

$$A = \frac{MTBF}{MTTR + MTBF} \tag{2}$$

where *A* is the machine availability. *MTTR* is the Mean Time To Repair.

From Equation (2), it is clear that, in order to increase the operational availability of the machine, represented by *A* in Equation (2), there are two possible alternatives. The first alternative is to increase the numerator, i.e., the *MTBF*, and the second alternative is to decrease the denominator, i.e., (*MTTR* + *MTBF*). However, under the scope of this research work, the aim is to develop a framework that will facilitate the maintenance process of machine tools and, by extension, it will result in the minimization of *MTTR*. Therefore, if *MTTR* is decreased, then the denominator of Equation (2) will also decrease and, thereby, the operational availability of the machine tool will eventually increase.

It is stressed out that the above-mentioned indexes are the cornerstones of maintenance contracts offered either as a service by maintenance contractors or OEMs (Original Equipment Manufacturers) as part of their sales program. Although, a maintenance plan may seem attractive due to the low *MTTR* offered by the maintenance contractor, e.g., 24 h. The Mean Time To Recovery is another index that is highly a ffecting the quality of the service, as it directly a ffects Operational Availability of the machine. The Mean Time To Recovery is an aspect of technical maintenance, dependent on a variety of factors, such as part lead times, administrative delays, and technician transportation delays. Aimed at minimizing the *MTTR* and eliminating the factors mentioned above, this research work is dedicated to the modeling, development, and practical implementation of a real-time, AR-based framework for remote maintenance support.

The reminder of the paper is structured as follows. In Section 2, an extensive literature review is carried out, focusing on topics relevant to technical maintenance and AR. Then, in Section 3, the proposed system architecture is presented and, in continuation in Section 4, the steps leading to the practical implementation of the framework are discussed. Section 5 deals with the discussion of a real-life industrial scenario used for validating the developed framework. Lastly, the paper concludes in Section 6, arguing the results and the future work of this research work.

#### **2. Literature Review**

#### *2.1. Literature Review Methodology*

In order to conduct a substantial literature review, a research methodology has been adopted by Reference [14], restructured so as to meet the requirements of the current research work. For the research, the most well renowned publication databases, namely Google Scholar, ResearchGate, ScienceDirect, Scopus, and Web of Science, were used. a total of 34 research papers has been tracked and examined. The investigation of the available literature was performed in two levels based on the year of publication. More specifically, publications older than a decade have been examined in order to track down the evolution of AR throughout past years and the newer publications were examined so as to conclude on the current trends in the field of Remote Maintenance Support and, by extension, to compare similar frameworks for the proposed framework.

#### *2.2. Maintenance as Part of the Modern Manufacturing System*

As stated in the previous paragraphs, it is clear that maintenance besides being a crucial part of the production line, as it ensures the continuous and flawless operation of the physical assets, is also a very complex process. Therefore, there is a considerable amount of research work trying to cover all of its aspects. However, maintenance activities can be further categorized regarding the status of the physical assets. Concretely, the most common types of technical maintenance found are: (i) Corrective Maintenance, (ii) Preventive Maintenance, (iii) Risked-based Maintenance, and (iv) Condition-based Maintenance [15]. Throughout recent decades, many types of technical maintenance policies have been developed in an aim to cover the majority of the industrial needs. However, with the big diversity in the general policies that firms adopt, there is no optimal solution regarding the equipment maintenance plan. With that said, the maintenance policy adopted a company, and is highly dependent on the general business model and, by extension, to the general goals to be achieved. As a result, all of the above-mentioned policies are equally important for the research community and the industrial domain as well. Regardless, the scope of this research work is to present the design and development of a framework that supports field technicians mainly in corrective scenarios, where a malfunction has already occurred and, in many cases, standard procedures cannot be applied. Therefore, the research work of this paper could be classified as an enhanced corrective maintenance policy, which is based on the provision of service for on-demand guidance.

The current industrial era is characterized by the immense decentralization of the manufacturing networks. Thus, the actual manufacturing plant may be in a di fferent region than the firm's headquarters, which makes communication of the two departments more di fficult [3,16]. However, the ongoing technological advances in the ICT have enabled the remote monitoring, remote operation, and the remote maintenance of the modern manufacturing systems [17,18]. Modernized maintenance under the term of Smart Maintenance have occurred over the last decade.

#### *2.3. Augmented Reality Based Remote Maintenance*

The concept of Digital Reality has been under the research scope during recent decades. With the advent of the fourth industrial revolution, such cutting-edge digital technologies have met an increasing advance, as a result of the technological advances in the ICT. More specifically, Augmented Reality (AR) is a very popular digital technology, which o ffers the advantage of enhancing the user's perception by augmenting their physical environment with computer-generated information. In the field of AR, Azuma is considered a pioneer, with the first survey on AR being published two decades ago [19], concluding that AR back then was a very immature technology. a huge potential for future development was expected. In a more recent publication [20], the challenges that must be overcome by researchers in order to make AR a mainstream form of media are discussed. Among others, the most important challenge the research and development community have to overcome, is the creation of highly intuitive AR applications and tools.

In the available literature, there can be found many publications covering the AR-based aspect of maintenance. The majority of the works is focused on the remote assistance of technical maintenance operations through the various AR-based frameworks. a typical application of remote maintenance support is presented in Reference [21], where the technicians are capable of performing maintenance operations in robotic arm manipulators with the use of AR-projected instructions. Another unique AR framework for the support of MRO is presented in Reference [22]. In Reference [23], the authors cover another aspect of AR-based remote maintenance under the framework of Product Service Systems (PSS). Therefore, the AR tools developed can facilitate manufacturers to add value to their line of products by transitioning to the PSS philosophy.

In the research work presented in Reference [4], the authors have carried out research on the recent trends around the Smart Maintenance paradigm by investigating recent publications as well as by extracting useful empirical knowledge. In Reference [5], a collaborative remote maintenance framework has been suggested. Although the research work is inspiring and of good quality, the framework implementation is based on technologies and equipment that currently can be considered as obsolete, as the technician has limited mobility since a PC is required by their side so as to visualize the AR instructions. Moreover, the instructions are based mostly on the projection of textual information. Lastly, in this research work, the use of frame markers is not suitable for the maintenance of bigger and more complex machines. Therefore, what is needed is more robust and more compact framework enhancing the technician's mobility when working on the machine. In a more recent paper, a similar remote maintenance support system based on AR is presented [24]. It is remarkable that the authors have put a considerable amount of effort on providing a mobile tool, based on the utilization of tablet PC. However, similar to the previous work presented, the use of frame markers is inevitable, which can highly affect the overall performance of the tool and, by extension, the performance of the shop-floor/field technician.

Although there is a continuous development of the AR-based tools proposed by the research community during the last two decades, there is a lack of added functionality. The above-mentioned development as well as the practical implementation of these frameworks in real-life industrial scenarios and the in-vitro experimentation in laboratory environments has led to the conclusion that AR has to treated as a back-bone technology. By extension, in the near future, the development of AR tools has to be further enhanced by the addition of more functionalities. a representative example is the framework presented in Reference [25], where an adaptive AR framework for machine operation as well as technical maintenance is presented. The adaptivity of the framework is based on the skill level of the technicians. Similarly, in Reference [26], an innovative framework based on a wrist-haptic tracking methodology is introduced.

#### *2.4. Maintenance under the Framework of Industry 4.0*

The modern manufacturing world is undergoing a digitalization phase under the framework of Industry 4.0. During the current industrial revolution, digital technologies have tremendously advanced, which enables the improvement of other scientific fields, including that of technical maintenance. With the recent development in the ICT, and the integration of Internet of Things (IoT), the manufacturing domain has passed in a whole new phase by relying on the utilization of data, i.e., Big Data Analytics. For the successful completion of MRO besides the technical knowledge/expertise of the stu ff carrying out the operations, it is crucial that all the needed components are acquired. Therefore, companies are maintaining an inventory of components. However, under the framework of Industry 4.0, and with the utilization of digital technologies, the existing inventory practices can be further improved through proper classification of the MRO components while additive manufacturing can facilitate in reducing lead times to produce several components [6–10,26–28].

For Industry 4.0 applications, two key objectives are to ensure maximum uptime throughout the production chain, and to increase productivity while reducing production costs. In Reference [29], a systematic approach to analyze the strengths and weaknesses of current open-source Big Data and stream processing technologies and assess their use for Industry 4.0 use cases is presented. In addition, the model of Industry 4.0 is gradually being implemented in worldwide development, distribution, and marketing chains. As such, the design and integration of a stainless-steel predictive maintenance system is presented in Reference [29]. This case study uses data from machinery involved in producing high-quality steel sheets. Following the predictive maintenance field, the use of real-time detection and prediction algorithms regarding future failures has significantly benefited from the technological advances of Industry 4.0 era. There's also growing interest in decision-making algorithms triggered by predictions of failure over the past few years. a literature review on decision making in predictive maintenance in the context of smart manufacturing is done in References [30,31]. Additionally, a review on predictive maintenance as an integrated predictive platform for production systems, focusing on maintenance approaches, methods, and tools is described in Reference [32].

Another significant issue is the estimation of the maintenance time for a new maintenance project, which is among the main maintenance o fferings, is based solely on the experience and knowledge of the engineer. The research work presented in Reference [33] proposes a framework for knowledge-based calculation of maintenance time based on Key Performance Indicators (KPI) monitoring to support the capture and reuse of information in maintenance activities as well as to improve the performance of the given maintenance PSS. Next, an extendable and reusable scheduling approach in the context of predictive maintenance, which supports multiple heterogeneous inputs and outputs, is described in Reference [33]. In accordance with the current schedule, predictive maintenance indicators from the tracked equipment are used for scheduling maintenance operations. a web service architecture is implemented to accommodate highly di fferent use scenarios such as suppliers of equipment.

## **3. System Architecture**

#### *3.1. General System Architecture*

As discussed in the previous paragraphs, the scope of this research work is focused on the design and development of a real-time, remote maintenance assistance framework based on AR by enabling the creation of new communication channels between the expert engineers and the shop-floor technicians. Concretely, there is an opportunity to eliminate the need for preparing AR scenes, which considerably decreases the time and e ffort needed for creating such content. Moreover, certain limitations can be overcome as expert engineers, through the use of the proposed framework, are able to guide shop-floor technicians in less likely MRO scenarios. In order to meet the above-mentioned technical requirements, the methodology adopted by aiming at addressing the identified scientific gaps will be presented in detail in the following paragraphs. The general architecture flowchart is depicted in Figure 1.

**Figure 1.** Flowchart of the proposed system.

The sequence of actions begins as long as a malfunctioning machine tool has been noticed. Then, the shop-floor technician who is carrying a device, a Head Mounted Display (HMD), contacts the expert engineer for assistance. It is stressed out that the coexistence of the two involved parties under the same premises is not required. More specifically, through the use of dedicated communication protocols, the shop-floor technicians can be remotely connected to the technical support department of the machine tool manufacturer. As long as the connection between the two parties has been established, a live video stream from the shop-floor technicians' device is broadcast to the expert engineer. This process can be realized as a live teleconference between the two parties. Therefore, while the video stream is broadcast to the expert engineer, the shop-floor technician is capable of making annotations so as to ease the engineer to ge<sup>t</sup> a better understanding of the machine tool malfunction. As soon as the engineer is fully aware of the malfunction, they can create on-demand AR content, which will be projected in the shop-floor technician's FOV (Field Of View). It is stressed out that, during the live-video casting from the technician, the device is also capable to 3D scan the space around the technician. As such, the algorithm running in the background is fully aware of the user's surroundings. Therefore, the expert engineer can register 3D content in the technician's FOV more easily and more successfully. The framework architecture can be realized through a platform, where the shop-floor technicians can connect and request assistance from expert engineers in real-time. From the expert engineer point-of-view, the platform serves as a tool enabling the expert to guide the shop-floor technician with the use of simple 3D tools. Therefore, it is of grea<sup>t</sup> importance to initially present the platform architecture. In Figure 2, the architecture of the proposed framework is depicted. From the figure, the structure of the Cloud Platform is visible and consists of services and the toolbox.

**Figure 2.** Proposed system architecture and steps sequence.

The most important component of the platform architecture is the online connection and user interconnection framework. More specifically, the platform acts as a cloud platform, where users, either expert engineers or field/shop-floor technicians, can connect to and, most importantly, they can exchange digital information. It is stressed out that there are no limitations in the type and volume of the exchanged information. Additionally, the user experience can be further improved by the support of live teleconference of the involved actors. In order to enhance the communication experience between the involved parties, the framework supports a teleconference feature. When this feature is enabled, then the user is capable of having a live video feed from the expert engineer, projected in his FoV. As such, it is expected that the communication between the two parties is facilitated, as the technician can perform MRO simultaneously. As far as the information exchange is concerned, the platform is connected with a cloud database. In the database, there is dedicated space where certain 3D geometries are saved. These geometries are imported in the AR scene only by the expert engineers. The collection of these tools can be realized as a digital toolbox of the most common tools used in MRO. However, due to increasing machine tool complexity, specialty tools might be needed. As such, the expert engineer is capable of adding a digital copy of the specialty tool through the "Create tool" Graphical User Interface (GUI).

In Figure 3, the GUI for joining an online maintenance session is depicted. On the top left corner of the GUI, the user through the provided dialog can select a session to join. In case the session has reached its end, then the user has to leave the session. On the lower left corner, the provided buttons assist the master engineer to retrieve from the Cloud platform the required tools by performing a "drag and drop" action to position them in the technician's FOV. However, if the desired tool does not exist in the Cloud toolbox, then the engineer is capable of uploading a 3D geometry. More specifically, in Figure 3, the "plus" icon when hit triggers the "Create Tool" function, so that the engineer adds the geometry in a suitable filetype along with relevant info and a description of the tool. The filetypes accepted by the Cloud Database have been limited to the filetypes accepted by the AR tool. Therefore, since the framework is developed in the Unity 3D game engine, the supported model file formats for 3D geometries are, .fbx, .dae (Collada), .3ds, .dxf, .obj. Then the algorithm running in the background is responsible for uploading the geometry on the Cloud Database as well as to create a new record in the SQL Database. The SQL database is created and used in an ontological manner, so that a smart-searching algorithm can automatically generate suggestions to the engineer during the AR scene creation phase. In Figure 4, the set of the initial tools used for populating the Cloud Toolbox is presented. For this manner, the most common technical tools have been utilized.

**Figure 3.** GUI for online maintenance session.

**Figure 4.** 3D CAD geometries of tools for AR instructions.

## *3.2. Cloud Platform*

The proposed remote maintenance service is developed as a cloud-based platform. The Cloud platform enables the communication of the shop-floor/field technicians and the expert engineer as well as the distribution of information between the involved parties. The Cloud platform is comprised of two domains: (i) the communication domain and (ii) the data storage domain. As discussed in the previous paragraphs, the communication domain is used for matching shop floor technicians for expert engineers and for establishing the communication channels between them. On the other hand, the data storage domain in essence is the Cloud Database where all the data are saved in SQL tables as well as the 3D geometries uploaded.

#### *3.3. Remote Maintenance Sessions*

As long as the shop-floor/field technicians identify a malfunctioning machine, then, with the use of a mobile device, they use the cloud platform so as to communicate with an expert engineer. In the cloud platform, this form of communication is realized through the creation of online sessions. For each session created, a communication domain is created. Upon the establishment of the communication domain, the two parties are capable of bilaterally exchanging information.
