Augmented Reality Based Distant Maintenance Approach

Abstract

This paper focuses on the implementation of Industry 4.0-based maintenance approaches using AR technology. The primary objective is to highlight the benefits of AR in maintenance and provide a framework for implementing AR-based maintenance systems. AR can assist maintenance personnel in performing maintenance tasks more efficiently by providing real-time instructions and information. By overlaying virtual information on real equipment, AR can guide maintenance personnel through maintenance procedures, reducing human errors and improving the quality of the work. AR can also provide remote assistance to maintenance personnel, allowing experts to remotely guide less experienced personnel through complex maintenance procedures. The proposed application provides motorcycle service assistance using AR and telepresence technologies. It enables remote monitoring of the service flow and performs an automated identification of parts. The system provides instructions for disassembling more complex mechanical components, provides a 3D model of the object, and enables ordering of the necessary parts.

1. Introduction

Industry 4.0 has revolutionized the manufacturing industry by integrating advanced technologies such as artificial intelligence (AI), the Internet of Things (IoT), and Augmented Reality (AR) into production and maintenance processes. One of the significant applications of Industry 4.0 in maintenance is the use of AR to enhance the maintenance approach. AR is a technology that overlays virtual objects onto the real world, enabling operators to visualize and interact with virtual objects in real-time [1] (see Figure 1). The benefits of an AR-based maintenance system are numerous. First, it reduces the time required to perform maintenance tasks, which improves equipment availability and reduces downtime. Second, it improves the quality of maintenance work by providing real-time instructions and reducing human errors. Third, it reduces the risk of accidents by providing remote assistance to maintenance personnel in hazardous environments. Fourth, it improves the training of maintenance personnel by providing a hands-on, immersive experience.

The implementation of an AR-based maintenance system requires a step-by-step approach. The first step is to identify the maintenance processes that can benefit from AR technology [2]. This includes processes that involve complex equipment, hazardous environments, or where remote assistance is needed. The second step is to select appropriate AR devices and software that meet the requirements of the maintenance processes identified in the first step. The third step is to develop AR applications that are customized to the maintenance processes and equipment (see Figure 2). This includes developing 3D models of the equipment and creating virtual overlays that provide instructions and guidance to maintenance personnel. The presentation example of this article is the creation of an assembly guide using Augmented Reality, including the recognition of a Minarelli AM6 engine block with the help of artificial intelligence. As soon as the recognition has taken place, the motor assembly instructions and the 3D exploded view of the block become available to the user using the application.

The application is designed for the Android-based operating system, but it is easily portable to other operating systems. The purpose of this study is to demonstrate the effectiveness of the proposed AR system in achieving accurate and useful telepresence systems. The results of this study have significant implications for the development of maintenance management systems, which can greatly improve the efficiency and productivity in remote assistance using AR and telepresence technologies. The main contributions of this research are summarized as follows:

• An AR-based telepresence device was developed and tested for remote monitoring of the service flow and automated identification of parts.
• A novel identification algorithm was developed and tested in an AR environment.
• A Unity–Vuforia-based application was developed to provide motorcycle service assistance using AR and telepresence technologies.

The paper is organized as follows. Section 1.1 describes the related work. Section 2 presents the materials and method, followed by the description of the developed telepresence and AR system. Finally, the discussion and conclusions are presented in Section 5 and Section 6.

1.1. Related Work

In [1], the main maintenance areas supported and enabled by Industry 4.0 technologies were discussed, along with the introduction of an Industry 4.0-based predictive maintenance approach for the manufacturing industry. In [2], various Industry 4.0 enabling technologies that enhance operational flexibility in final assembly were presented, with proven examples of their application in increasing flexibility. In [3], the researchers proposed an Augmented Reality (AR)-based system that integrates the existing Manufacturing Execution Systems (MES) with Cyber-Physical Systems (CPS), creating a Cognition Level. The study highlights the opportunities offered by AR technologies to CPS through an application scenario analyzed in a real factory setting. The authors in [4] presented results from six case studies involving over two hundred respondents from academia and the industry, focusing on the usage and strategies for implementing different xR technologies. Paper [5] focused on the user experience (UX) of AR-based instructions for assembly. An evaluation matrix and an AR prototype were developed and assessed in a UX test, collecting data on both the hedonic and pragmatic qualities. In [6], the authors conducted a systematic literature review (SLR) to analyze I4.0 research studies in the automotive sector. The objective of [7] was to propose an original framework that investigates the impact of the Industry 4.0 principles on assembly system design, describing the traditional dimensions of this problem and the evolution of the industrial environment over the last three centuries. In [8], the authors provided a brief survey of current use cases in factory applications and industrial inspections that are actively being developed. In [9], an efficient AR system was proposed, which enables the locating of a component in the manufacturing plant and the visualization of maintenance instructions for the corresponding failure modes in a handheld and vivid manner on the operators’ mobile device. The activity within the industrial plant is recorded by a camera, which serves as the sensory input for the proposed AR system. The authors in [10] constructed a platform by integrating AR goggles, enabling the acquisition of data for the technical evaluation of rotary components and facilitating direct interaction with the user. The platform presented allows for the utilization of artificial intelligence to analyze vibrations generated by the rotary drive system in order to determine the technical condition of a wind turbine model. The image processing system, which measures frequencies generated by the machine, is used for monitoring purposes. A new system architecture was presented in [11] for controlling industrial devices using Mixed Reality (MR) applications. Additionally, a new method for measuring the distance between real and virtual points is proposed based on this system’s architecture. The research involves the use of a physical robot and its Digital Twin (DT). The paper demonstrates the feasibility of controlling the robot and its DT using gestures recognized by Mixed Reality goggles. The aim of [12] was to demonstrate the current state-of-the-art of AR in maintenance and highlight the most relevant technical limitations, as revealed by the results of a systematic literature review. The analysis involved filtering a large number of publications and identifying 30 primary studies published between 1997 and 2017. The findings indicated a high level of fragmentation among hardware, software, and AR solutions, resulting in complexity when selecting and developing AR systems. The study’s results shed light on areas where AR technology is still lacking maturity. The user, technical, environmental, and regulative requirements for an AR maintenance worker support system were presented in [13], based on the analysis of three diverse production sites.

1.2. The Role of AR in Industry 4.0 Maintenance Procedures

In the context of Industry 4.0, Augmented Reality (AR) has emerged as a transformative technology, revolutionizing maintenance procedures and enhancing the overall efficiency and effectiveness. AR plays a crucial role in empowering maintenance personnel by providing them with real-time information, intuitive guidance, and improved situational awareness. One of the key contributions of AR in Industry 4.0 maintenance procedures is the ability to overlay digital information onto the physical environment. By wearing AR-enabled devices such as smart glasses or using AR applications on mobile devices, maintenance technicians can access critical data, instructions, and visualizations directly in their field of view. This capability enables technicians to quickly identify and diagnose issues, locate components, and perform repairs or maintenance tasks more effectively. According to [14], AR also facilitates remote collaboration and knowledge sharing among maintenance teams. With AR, experts can provide remote assistance to technicians in real time, regardless of their physical location. Through AR-enabled communication platforms, technicians can share live video feeds, annotations, and 3D models, allowing experts to guide them through complex maintenance procedures. This collaborative approach not only improves the accuracy and speed of maintenance activities but also reduces the downtime and the need for on-site visits. AR technology finds extensive applications across various domains within the digital factory, enabling operators to remain updated on crucial events and engage in collaborative tasks with digital assets. In maintenance procedures, for instance, operators utilize their mobile devices to assign and track their tasks throughout the day, interact with production equipment, and access digitized step-by-step procedures via AR frameworks. Furthermore, the integration of artificial intelligence technologies facilitates mechanical intelligence, which plays a significant role in fostering collaboration between humans and machines. This aspect proves particularly valuable in modern manufacturing environments, as machines, despite their limited understanding and management of the surrounding environment, can still contribute effectively.

2. Materials and Methods

The aim of this study was to investigate the current state of integrating maintenance management systems into an Industry 4.0 environment and to explore how Industry 4.0 technologies and features impact the functions of maintenance management systems [3]. The study focused on identifying key integration enablers such as integration concepts and integration-assisting tools. The study also considered potential applications, future trends, and challenges associated with integrating maintenance management systems in an Industry 4.0 environment [4,9]. Industry 4.0 significantly influences the functions of maintenance management systems by introducing new technologies and concepts that enhance the efficiency, effectiveness, and automation of maintenance processes. Real-time data acquisition, predictive maintenance, remote monitoring and diagnostics, and Augmented Reality support are some ways in which Industry 4.0 impacts maintenance management systems. Industry 4.0 transforms maintenance management systems by leveraging data, connectivity, automation, and advanced technologies to enable more proactive, efficient, and informed decision-making in maintenance operations. The tools that are used to assist in the integration process within an Industry 4.0 environment are presented in Section 3. Industry 4.0 facilitates remote monitoring and diagnostics of equipment and systems. Maintenance management systems can access real-time data from connected assets, enabling the remote monitoring of asset health, performance, and operational parameters. Remote diagnostics capabilities allow maintenance teams to identify issues, troubleshoot problems, and provide guidance to on-site technicians, minimizing the need for a physical presence and improving response times. Maintenance management systems can store digital manuals, procedures, maintenance histories, and other relevant information. This facilitates easy access, retrieval, and the sharing of knowledge, improving the efficiency and effectiveness of maintenance activities.

3. The Developed Telepresence System

A telepresence system is a form of modern mechatronic device or actuator that is currently emerging in various fields, particularly in unmanned vehicle systems, remote communication, and field inspections. A schematic overview of the developed system is presented in Figure 3. Telepresence encompasses a wide range of technologies and human studies, making it challenging to systemize, implement, and analyze. It is often approached from different perspectives, including teleoperated construction systems, telepresence systems for unmanned vehicle systems, human–machine interfaces, visual aid and perception, and user operations. The construction of the system primarily focuses on its structure, which incorporates a series of sensors, actuators, and control mechanisms. Recent attention has been given to research on unmanned vehicle systems due to their high mobility and cost-effectiveness. Studies on human–machine interfaces have revealed common occurrences of situational awareness loss, inaccurate attitude judgment, and failure to detect obstacles. The operator plays a crucial role in the control loop and relies on video for perception; hence, direct interfaces require high bandwidth communications to ensure low delay interactions.

In complex or dynamic situations, the accurate perception of the remote environment and timely control decisions can be challenging for the operator of a telepresence system. To address these issues, the use of multimodal and multisensory interfaces is recommended to provide efficient tools for command generation and feedback. The research presented in this study focuses on the real-time control of a telepresence system driven by two motors. The primary objective of the control is to achieve the desired motion of the developed system in real time. A stable Bluetooth networking capability ensures that devices can communicate seamlessly, exchange data, and perform tasks without experiencing frequent disconnections or interruptions, as referred to in [15]. It is particularly important in scenarios where devices need to maintain real-time communication or rely on a constant exchange of information, such as in remote monitoring, control systems, or collaborative operations. To achieve stable Bluetooth networking, several factors come into play.

• Signal strength: The Bluetooth signal strength should be strong enough to maintain a reliable connection between devices, even when there are obstacles or interference present. A weak signal can lead to connection drops or a degraded performance.
• Interference avoidance: Bluetooth operates in the 2.4 GHz frequency band, which can be crowded with other wireless devices like Wi-Fi routers, microwaves, or cordless phones. To ensure stable networking, devices should employ interference avoidance techniques or adaptive frequency hopping to mitigate the impact of the interference.
• Power management: Efficient power management is important for stable Bluetooth networking. Devices should optimize their power consumption to maintain a stable connection while conserving battery life.
• Protocol robustness: The Bluetooth protocol stack, including the lower-level layers like the link layer and physical layer, should be robust and capable of handling potential challenges and variations in the wireless environment. This includes error correction mechanisms, packet retransmissions, and adaptive modulation techniques.
• Quality of Service (QoS): QoS mechanisms in Bluetooth help prioritize and manage data transmission to ensure that critical or time-sensitive information is given priority, improving the overall stability and reliability.

By focusing on these aspects, manufacturers of Bluetooth-enabled devices and software developers can enhance the stable Bluetooth networking capability, ensuring a consistent and reliable connection between devices. This is crucial for applications that rely on continuous data exchange, such as the use of Bluetooth in Industry 4.0 maintenance procedures, where reliable communication is essential for remote monitoring, control, and collaboration. In order to develop an augmented reality based remote maintenance system, the first step is to create a 3D model of the hardware structure as shown in Figure 4.

The commands are received through the Bluetooth network by utilizing a Raspberry Pi board. The Raspberry Pi board then interprets these commands and controls the motors. The motors used in this system are BGE3004-type motor controllers, which are manufactured by Dunkermotoren GmbH. These motor controllers require a 0–10 V analog signal for speed control, but the microcontroller lacks analog outputs. To overcome this limitation, an additional circuit is implemented between the Raspberry Pi and the motor controllers. This circuit converts the pulse width modulation output signal from the microcontroller into the required analog signal range.

As previously mentioned, the actuators in this system are BG42 × 30 motors, as depicted in Figure 5. These motors offer a range of rotation speeds from 500 rpm to 5000 rpm. The motor controllers provide the capability to set the direction of rotation and include a start/stop function. The control of these functions is achieved using the digital outputs of the microcontroller.

The telepresence system is controlled using a laptop, tablet, or smartphone as a human–machine interface. These devices are connected to the developed device via a Bluetooth network. In addition to serving as a human–machine interface, the laptop plays a crucial role in recording and transmitting the user’s input commands. The graphical user interface is designed as a webpage, where the user can input commands using the arrow buttons, as shown in Figure 6. The arrows correspond to different movement directions of the robot, including forward and backward linear movement, rotation in both directions, and stopping. Furthermore, the interface allows the user to preset the movement speed and acceleration.

4. Building Augmented Reality Applications

This research work centers on developing a real-time remote maintenance assistance framework utilizing AR technology. The primary focus lies in creating novel communication channels that connect expert engineers with shop floor technicians. The system aims to enhance the remote maintenance support and enable a seamless collaboration between the two parties. AR has emerged as a powerful technology that allows users to blend digital content with the real world. AR applications are being developed across various industries, including gaming, entertainment, education, and manufacturing. Unity and Vuforia are two popular tools that are used to develop AR applications, as shown in Figure 3. How to build AR applications with Unity and Vuforia will be discussed in this article. Unity is a game engine that provides a user-friendly interface for designing and developing 3D games and applications [17,18,19]. Vuforia is an AR platform that provides computer vision technology to create AR experiences. Vuforia recognizes and tracks target images and objects in real time, allowing the AR content to be anchored to them, as presented in Figure 7.

The first step in building an AR application with Unity and Vuforia is to create a new Unity project. After creating the project, import the Vuforia Engine package from the Asset Store [20]. Once imported, the Vuforia configuration window will appear, where the Vuforia license key can be added. The Vuforia license key is necessary to use the Vuforia Engine.

The next step is to set up the Vuforia image target. Image targets are the physical objects or images that Vuforia uses to recognize and track (see Figure 8). The area marked with a yellow line indicates the detected object. In the Vuforia configuration window, select the type of image target to be used, such as a single image, multiple images, or a cylinder. Then, import the image target into the Unity project.

After importing the image target, it is necessary to create an AR camera. The AR camera is the main camera that is used to view the AR content in the scene, as can be seen in Figure 9. In Unity, create a new camera object and add the Vuforia Augmented Reality script component to it.

After giving the specifications of the architecture, an algorithm to localize the motorcycle parts in the environment was defined as can be seen in the Algorithm 1.

Algorithm 1 Location of Objects in the Environment

Require: exist = 0; message = ””; i = 1 Ensure: String result//message while exist = 0 and i < length(DB) + 1 do if Partname == DB (i) then exist = 1 message = Partname + “FOUND” end if i + + end while if exist = 0 then message = Partname + “Not FOUND” end if return message

An algorithm was created to find specific parts on the engine, which, after detecting the object, generates a list of parts and displays a 3D model with disassembly instructions as well as an option to order spare parts. Once the AR camera is set up, it is time to add the AR content. The AR content can be any 3D object, a model that is anchored to the image target, or an exploded model with part descriptions and prices with online ordering option, as depicted in Figure 10. In Unity, import the 3D model that is to be used as the AR content. Then, add it to the scene and set its position and scale.

Finally, test the AR application on a device that has the Vuforia engine installed. To test the application, build it for the device and then deploy it [21]. The AR content should appear on the image target in real time when viewed through the device camera. Unity and Vuforia are powerful tools that can be used to develop AR applications. The process of building an AR application with Unity and Vuforia involves creating a new project, setting up the Vuforia image target, creating an AR camera, adding the AR content, and testing the application on a device. With these tools, developers can create immersive AR experiences that can be used across various industries [22].

5. Discussion

In this research, a comprehensive review of recent publications related to remote maintenance support was conducted. The review revealed that, while there are several approaches contributing to the field, only a limited amount of research focuses on providing real-time support. This discovery led to the identification of an opportunity for new research. Consequently, a detailed framework for real-time remote maintenance and repair operations based on AR was designed and developed. The primary objective of the framework was to establish interactive, intuitive, and collaborative communication channels between shop floor technicians and expert engineers. The developed application was thoroughly tested in an actual industrial setting to ensure that all functional requirements were met. The results of the study demonstrate the successful fulfillment of the functional requirements outlined during the problem modeling phase, and there are prospects for further improvement. Specifically, the content authorization methodology emerged as a crucial element in AR applications. Future endeavors could concentrate on integrating the developed framework with an automated content authoring methodology, thereby significantly reducing the effort and skills required from engineers when creating AR content. Such advancements would enhance the efficiency and effectiveness of the remote maintenance support system and pave the way for more streamlined maintenance processes in the future. As a verification of the system, the accuracy of the recognition of certain objects was measured according to the parameter of the number of defined markers and according to the distance of the object. To build a system for automatic part recognition, it is necessary to teach the system what a certain part should look like. This is realized by defining a certain number of characteristic markers on a part. The better the markers are defined, the better the degree of object recognition. Part of the research includes determining the optimal number of markers for the reliable recognition of parts, taking into account the distance from which recognition of a part is carried out. All authors in the reviewed articles reached the consensus that Industry 4.0 technologies and features facilitate modern maintenance concepts, resulting in terms such as “digitalized maintenance”, “smart maintenance”, “self-maintenance”, and even “Maintenance 4.0”. The predictive maintenance approach, along with corrective, preventive, and prescriptive maintenance strategies, serve as driving forces [23]. However, determining the optimal maintenance strategy in an Industry 4.0 environment is contingent on several factors, primarily the complexity of the employed technologies. These factors can be evaluated and selected based on specific objectives and constraints, such as maximizing the availability and reliability and minimizing the life cycle costs, through an appropriate multi-objective optimization method.

As can be seen in Figure 11, the object recognition converges continuously, and the accuracy continues to rise. After 300 points of defined markers, Vuforia object recognition has reached 0.96, achieving a high prediction accuracy.

A high level of coordination and resource allocation are necessary to ensure effective maintenance organization. In Industry 4.0, essential integration-assisting tools like supervisory control and data acquisition (SCADA) systems, enterprise resource planning (ERP) systems, and computerized maintenance management systems (CMMSs) are monitored [24]. They have a significant influence on the maintenance processes and associated costs. The allocation of resources and spare parts are managed effectively in the cloud by the ERP system. Maintenance control is handled by computerized systems and knowledge dashboards, and its effectiveness can be monitored and improved at different levels from the machine level to the overall plant. Maintenance control can be enhanced through data modeling, optimization, and behavior pattern detection by many Industry 4.0 technologies [25]. Unseen factors affecting the machine performance, product quality, and productivity can be detected by AI techniques like machine learning. Feedback passing through maintenance control is facilitated by software support systems, resulting in maintenance planning and rescheduling activities. Figure 12 shows the recognition capability versus distance graph for 284 marker points.

The traditional evaluation metrics of maintenance management systems govern and evaluate the performance of the overall system. An integrated sensory system and SCADA provide a new method for information collection for monitoring and controlling the performance of manufacturing systems for industrial organizations [26]. The collected data can be processed by AI-based predictive maintenance modules, and real-time measurements and information are shown by graphical user interfaces (GUIs) communicated between the CMMSs, ERPs, integration-assisting tools, and SCADA systems. Figure 13 shows recognition capability versus distance graph for 547 marker points.

By providing common means of communication between machines and operators, production departments, and different maintenance systems in different locations of a company’s plants, interoperability allows for seamless maintenance management. Various maintenance management systems, like ERP, CMMS, and SCADA, receive software support, ensuring compatibility and smooth report generation and data exchange by transmitting and pipelining the conditions and operational performance data in a standard format. A great tool for tracking 3D designs with a smartphone or tablet, especially when visualizing larger datasets, can be provided by using QR codes. A large factory layout or production line can be loaded and positioned over a QR code by Vuforia’s Image Tracking feature [27]. It is ideal for visualizing large designs on a tabletop to see all the data at once, and the ability to utilize this on a handheld device makes AR technology much easier and accessible for such use cases [28].

Flexibility when working with digital objects interacting with a physical environment can be provided by QR codes, and they are designed to be adaptable to a wide variety of use cases. QR codes are particularly useful in AR when consistent accuracy is required, such as precise placement in AR. Scaling is much easier within AR by having a point of origin in the real world. The point of origin will remain the same, but the model can be scaled larger or smaller with QR codes. System improvements compared to the first version are an almost complete reconstruction of the hardware, as can be seen in Figure 14, as well as the addition of new software tools with AR support.

6. Conclusions

The distant maintenance approach offers several benefits, including increased efficiency, reduced downtime, and improved safety. With AR, technicians can overlay digital information onto physical equipment, enabling them to quickly identify components, view step-by-step instructions, and access relevant data without the need for extensive training or physical manuals. This improves the accuracy and speed of maintenance tasks, ultimately leading to enhanced productivity and cost savings. The implementation of an AR-based distant maintenance approach does come with its challenges. Ensuring reliable and stable network connectivity, optimizing AR application performance, and addressing user acceptance and training are critical factors to consider. However, advancements in AR technology, coupled with the increasing availability of a robust network infrastructure, are continuously addressing these challenges and paving the way for widespread adoption. As AR technology evolves, more sophisticated features such as real-time data analytics, machine learning algorithms, and predictive maintenance capabilities can be expected. These advancements will further empower maintenance teams to proactively detect anomalies, optimize maintenance schedules, and improve the overall equipment reliability. In conclusion, the Augmented Reality-based distant maintenance approach represents a significant leap forward in the realm of maintenance operations. By harnessing the power of AR, organizations can transform their maintenance processes, maximize their uptime, and achieve higher levels of efficiency and productivity. As the technology continues to mature, embracing this approach will undoubtedly become a competitive advantage for forward-thinking industries seeking to stay at the forefront of maintenance excellence. The future plans for further advancements and expansions can be correlated in several key areas like integration with the Internet of Things, artificial intelligence integration, training and skill development, data analytics, and performance optimization. The integration of AR with the IoT technologies will enable real-time data exchange and seamless interactions between physical equipment and digital information. This integration can provide technicians with instant access to equipment performance data, diagnostics, and predictive maintenance insights, improving decision-making and efficiency. AI algorithms can enhance AR applications by analyzing vast amounts of data collected from maintenance operations. AI-powered AR can provide intelligent recommendations, automate routine maintenance tasks, and offer proactive alerts for potential equipment failures, optimizing maintenance strategies and reducing downtime. AR-based training simulations can provide immersive and interactive learning experiences for maintenance technicians. Future plans may focus on developing advanced AR training modules that offer realistic scenarios, virtual equipment simulations, and adaptive learning pathways to enhance technicians’ skills and knowledge. AR-generated maintenance data, combined with advanced analytics techniques, can provide valuable insights into equipment performance, maintenance patterns, and optimization opportunities. Future plans may involve leveraging AR-generated data to improve maintenance planning, resource allocation, and the overall equipment effectiveness.