Augmented Reality Support for Commissioning and Monitoring of Electromechanical Multipoint Die Cushion ^†

Abstract

Commissioning die cushions of a press is a highly complex task, and errors in the process must be avoided. Augmented Reality (AR) can help the engineer to overcome the encountered problems by visualizing the current situation with a superimposed 3D model, providing necessary instruction steps derived from the actual data of the press control and monitoring system, and checking the received data for errors. This paper describes an industrial AR-based concept to support an electromechanical multipoint die cushion commissioning process. It shows how an AR application can support workers in this production use case to avoid failures and reduce the needed commissioning time, resulting in higher productivity.

1. Introduction

Production engineering offers several fields of application for Augmented and Virtual Reality (AR/VR) such as design, planning, monitoring, maintenance, and training [1]. Especially in the technical sector, complicated issues impede manual processes, leading to errors and time-consuming error tracing [2]. In such cases, virtual technologies can serve as communication tools and enable, e.g., the visualization of technical issues or the support of assembly, repair, and training tasks [3]. They act as a human-machine interface (HMI) to facilitate manual work and help human workers to avoid failures and to accelerate the working task [4]. The support can be realized, e.g., by visualizing working steps with superimposed 3D models of machine parts or tools and by displaying and checking important online data of the machines directly on-site. The overall goal of the AR-based HMI is to improve manual working tasks in a user-centric approach.

This paper introduces an AR concept that supports a servo press’s commissioning with an electromechanical multipoint die cushion. The relevant process is complex and error-prone due to poor clarity of the situation. E.g., the die cushion bars are installed in the basement and are hard to see. To overcome this situation, an AR concept was developed and implemented which supports the operator on-site. One crucial aspect comprises linking live data from the press control and several sensors with the AR approach. Thus, it is possible, e.g., to capture position data which is necessary to provide the proper instructions to the user and to check specific values to avoid errors. The starting situation, the AR concept and the implementation are presented in the next chapters, followed by a discussion and conclusion addressing the achieved results, to illustrate the benefits of AR applications in industry-related use cases.

2. Current Use of Augmented Reality in Production

AR technology can support the operator of a machine tool by providing an extended human-machine interface and optimizing the production processes. It is thus a building block for mastering the increasing complexity in production [5]. As for current use of AR in production, an AR-based system for cyber-physical systems is described here to realize intuitive and efficient human-machine interactions for processes of, e.g., monitoring, controlling, simulation, and optimization [6]. Therefore, the CNC (computerized numerical control), sensors, and cameras are connected via network to the AR application. With this system, the results of controlling, simulation, and optimization are visualized in an AR view to achieve a better process supervision for the operator.

Ref. [7] presents an AR system for machining simulation to analyze, adjust and visualize the simulation process with a virtual work piece. The CNC of the real machine tool is used for this purpose. The system can be used by machinists and trainees for trial and learning. It enables the operator to experiment with different machining parameters on real machines without breaking tools or machines. The survey results show that the comprehension of machining and simulation processes is improved by this approach. By combining the online data of real machines with the virtual environment, the workers are issued with a uniform and user-oriented interface for complex processes.

Besides using online data, the AR-based visualization of manual and complex working steps can support the workers in the commissioning process of machines. Ref. [3] presents an AR system with a 3D model and text support for the training of commissioning of tools for machines. With step-by-step instructions, including superimposed 3D objects, the worker is supported in a vivid and user-friendly way. Virtual training with AR shows a better learning curve compared to conventional instructions, and errors can be avoided on real machine components [3]. Also, Ref. [8] shows the assembly of industrial components with the help of an AR system. The process was improved with superimposed 3D models in contrast to the conventional documentation that can be difficult to manage in some working situations due to unclear drawings or complex processes.

The 3D visualization of instructions is also used in the assembly and maintenance of machines [4]. The AR technology provides an outstanding presentation of complicated manual processes in a user-friendly way which conventional instructions cannot provide. Problem detection is improved using 3D superimposed visualization and detailed sequential instructions [4]. Also, Ref. [9] developed an AR application for maintenance and repair of a vehicle. The realized user study shows that humans can locate individual tasks in a maintenance sequence more quickly. Moreover, mechanics found the AR condition intuitive and satisfying. Another advantage is that AR applications can support the worker without expert intervention.

3. The Electromechanical Multipoint Die Cushion

Modern servo presses are characterized by applying the nominal force over the entire working stroke [10]. Despite the enormous energy density of the latest torque motors, even higher nominal force is required for numerous forming tasks. For this purpose, several torque motors are installed on the ram and the die cushion. The latter is located on the basis of the press and graduates the blank holder pressure by applying a counterthrust to control the material flow during the process. A multipoint die cushion’s number of cylinders can differ. A four-point die cushion enables it to influence the corners, whereas six-point and eight-point die cushions can be used to influence work piece contour.

The project comprised the development of an electromechanical multipoint die cushion with up to eight servo spindle direct drives [11,12]. Figure 1 shows the structure of the device. The direct drives are connected with quills that are in turn linked with the blank holder to apply the needed forces. The whole die cushion is located in the basement within the press structure and cannot be seen when controlling the press. When commissioning the drive chains or installing and removing the individual modules, there is no direct visual feedback to the commissioning engineer or the press operator. Furthermore, it is necessary to move between the press control located on the ground floor and the die cushion in the basement. This spatial separation complicates the commissioning process and increases the susceptibility to errors.

4. Aim and Approach

The die cushion is not used in every process. If it is to be used, the drives are first referenced using an external measuring system. With this, a drift in the measured values occurs with repeated referencing. If the resulting measurement error is too large, this leads to stresses in the die cushion plate, since the entire system is overdetermined by the eight connecting ducts. To avoid these stresses, the drives must therefore be recalibrated regularly. The AR application is designed to support this process by overcoming problems arising from the worker’s lack of understanding of the machine’s situation. The proposed solution implements a connection to the press control to obtain relevant machine and sensor values for monitoring and verification purposes, e.g., position and state values of die cushion components. These machine values enable real-time visualization of relevant information and processes, as well as verification of the work steps performed. For example, the AR-based visualization enables direct position referencing for the commissioning engineer by virtually displaying the direct drives. This feature is a significant advantage compared to the present state, where this context is difficult to comprehend. Furthermore, real-time visualization can detect and highlight stress situations and process errors can be detected and highlighted quickly.

The suggested approach is to design a user-centric AR system that guides the worker through the complex commissioning process while verifying and displaying online data to avoid errors. For this purpose, the current process must be analyzed and a real-time data workflow, visualization and interaction concept must be developed. The solution has to be easy and intuitive to use for the actual worker to ensure a high level of acceptance and a low barrier to use. Based on the above constraints, an AR application, including instructions and data connection to the press, was designed, implemented and tested in the present use case.

5. Augmented Reality Based Concept for Commissioning and Monitoring of the Die Cushion

To support the commissioning process optimally, the worker must gain an overview of each die cushion component’s state at any work time. Using the AR application, the state of the die cushion and the working steps are monitored, visualized, and checked for errors. Live data from the press control and all sensors is necessary to realize these functions. An additional monitoring system covers this task, being connected to the control and sensors, and it offers a unified and consistent interface for communication purposes. The system can also debug and log beneath the AR-based commissioning process.

The AR concept is based on a detailed analysis of the manual working process and the related conditions of the die cushion and the press. Figure 2 shows the complete sequence plan for the commissioning process with the related machine data. Descriptions of working steps have been derived to create step-by-step instructions for the workers. This process is now similarly mapped by the AR application. Thus, the manual working steps are linked with the connected live data, which is the fundamental element of the suggested approach, resulting in AR-supported commissioning.

The AR application uses two pillars to aid the process: visualization and validation. The first shows additional information with different graphical elements: 3D models, pictures, text, and interactive elements, e.g., live charts. This visualization part is used to display the current working step, visualize invisible parts in the correct positioning, and show monitoring information. In this context, it is essential to apply the guidelines for clear graphical user interfaces. It is necessary to show only the information relevant to the current task to achieve simple and fast usage. The second purpose of the application is to validate the received data against the expected values. Only if the validation result is positive, the next working step in the AR application can be started. This aspect helps prevent setting errors and serves as a protocol functionality.

A wireless connection between the monitoring system and the AR application is preferred to allow unrestricted movement of the workers. It can be realized via standard technologies, e.g., WLAN or Bluetooth. The use of 5G networks does not offer any benefits since the latency and data bandwidth of the older standards are sufficient for the present use case. The realized API is based on connection-oriented TCP/IP to prevent data losses. A proprietary interface was implemented for testing purposes. It is advantageous to rely on standardized formats, e.g., OPC UA to achieve better generality.

Figure 3 shows the final system structure for the AR application and the press systems. The data of the press and additional sensors is recorded by the monitoring system and transferred to the data interface of the AR application. Different components for monitoring and instructions, including 3D models and GUI elements, access and visualize the data in the AR application.

For users of the AR application, it is essential to move around the press and see the referenced 3D model of the die cushion in all positions. Therefore, marker-based tracking covers the entire press with its floor area of three-by-three meters and its symmetrical structure.

6. Implementation of the Augmented Reality Based Concept

The developed AR-based concept is implemented with the AR framework ARViewer [13], which was created to support human work in production environments especially, and to create AR applications more efficiently by encapsulating the technical aspects. Therefore, the framework includes fundamental AR components for camera handling, tracking, user interaction, visualization, various AR devices, and interfaces for external systems to connect, e.g., machine tools and other factory systems. These components can be customized to create new use cases and visualize the required production information in a user-friendly way [14].

Figure 4 shows the resulting AR application with the full view of the 3D visualization of the press, including the die cushion and monitoring charts. The 3D model is based on the CAD model adapted for the AR use case. The number of polygons is reduced, the kinematic structure is adjusted, and materials are created for clear real-time visualization. Based on the connected live data of the monitoring system via WLAN, the dynamic components of the 3D model move to synchronize the model with the real state of the press, resulting in a properly aligned AR overlay. Additionally, the live values are visualized in different tables and charts to provide detailed information during the work. The working steps are shown with texts and highlights in the 3D model. They depend on the current situation based on the live data.

The AR device is chosen based on the operator’s work and the criteria catalog of [15] that focuses on industrial applications. As a result, a tablet computer is used (see Figure 5) since it is sufficient for the given use case and integrates well into relevant companies’ typical IT infrastructure. Each single working step can be followed by a period of consultation—the operator can use the tablet computer before/after one working step and continue his/her work. Wearing AR glasses during the whole process is not a practical solution due to the type of work in this particular use case; displaying different 3D views, documents, and drawings in a user-friendly way requires an appropriatelyw large display.

The application requires a 360-degree tracking approach to enable the user to move around the whole press. The structural circumstances of the press allow marker-based tracking (see Figure 5). The user can look at the virtual press and all the inside die cushion components from all positions due to mounting four different markers, one on each side of the press.

7. Discussion

With the developed AR system, the worker can be supported during the commissioning process, but the AR system must always be adapted to the current machine and the new process. Existing AR systems, such as the used ARViewer [13], can assist this process, but setup work is still required.

For the described use case, a tablet computer was selected because the required data can be visualized very well and it is easy to use in certain situations. However, a device, such as AR glasses, in which the hands are free to work would be desirable; with the current wearing comfort and display quality, though, this option is limited in its suitability. With the further development of the existing hardware this should be possible.

To improve the commissioning process, an additional AR application was developed and an additional device was used. For a further development of the commissioning process, it would be helpful to have one single device for all required process steps, e.g., functions of the machine control.

8. Conclusions and Outlook

This paper presents an AR-based concept and implementation to support the workers in commissioning and monitoring of an electromechanical multipoint die cushion of a servo press. Using the AR solution directly on-site facilitates handling the complex die cushion with its control system and corresponding work processes. The summarizing visualization of all live data of the monitoring system is beneficial for the work process, including a 3D visualization of the current die cushion state and the correct instruction steps based on the current control data. The crucial point for the AR solution comprises the linkage between each manual working step and the connected live data of the monitoring system. Only if the received information has the correct value, can the next working step in the AR application be started. The usage of the developed AR-based human-machine interface at the press showed that there is a significant potential to improve the manual working process and to avoid failures.

While the current approach is unidirectional, bidirectional communication can be beneficial to controlling the press with the AR application. Such bidirectional communication can be a starting point for future improvements to the presented solution; however, safety aspects must be considered.