Development of Standalone Extended-Reality-Supported Interactive Industrial Robot Programming System

Abstract

Extended reality (XR) is one of the most important technologies in developing a new generation of human–machine interfaces (HMIs). In this study, the design and implementation of a standalone interactive XR-supported industrial robot programming system using the Unity game engine is presented. The presented research aims to achieve a cross-platform solution that enables novel tools for robot programming, trajectory validation, and robot programming debugging within an extended reality environment. From a robotics perspective, key design tasks include modeling in the Unity environment based on robot CAD models and control design, which include inverse kinematics solution, trajectory planner development, and motion controller set-up. Furthermore, the integration of real-time vision, touchscreen interaction, and AR/VR headset interaction are involved within the overall system development. A comprehensive approach to integrating Unity with established industrial robot modeling conventions and control strategies is presented. The proposed modeling, control, and programming concepts, procedures, and algorithms are verified using a 6DoF robot with revolute joints. The benefits and challenges of using a standalone XR-supported interactive industrial robot programming system compared to integrated Unity–robotics development frameworks are discussed.

1. Introduction

Robots are important in high-mix, low-volume manufacturing due to their versatility in performing manufacturing tasks [1]. There is a high demand for the development of robotics and automation solutions that are less expensive, safer, easier to install and re-program, where the operator can work as an assistant to robots to solve unpredictable problems, and that can be adapted by small- and medium-sized enterprises (SMEs) [2,3].

Programming of industrial robots using conventional robot programming methods is time-challenging and often demands expert knowledge [1,2,3]. The most widely used programming method for industrial robots is programming using teaching pendants [1]. This method, also known in the literature as the Lead-Through method [4], is an online programming method that typically takes considerable time, especially if a robot has more degrees of freedom than the joystick of the teaching pendant (due to the separate programming of the position and orientation of the robot’s end-effector). Using conventional Walk-Through online programming, the human operator directly interacts with the robot, i.e., physically moves the robot to the desired configuration [4]. This method allows intuitive robot programming; however, it is limited as only a few lightweight robots offer the necessary zero-torque control [4]. Online robot programming methods result in downtime (the robot must be present during programming), which additionally prolongs changeover times when users introduce new applications. Besides the aforementioned online programming methods, there are also offline programming methods that do not require the physical presence of the robot during programming [1]. Offline programming methods require high-fidelity 3D modeling of robot and its workspace in a virtual environment and are time-consuming [1]. For example, if a new object of manipulation is introduced into the robotic application, its 3D model has to be developed using appropriate software. The simulation module needs to be tested on real robots before actual offline programming can be performed, and these tests may require more effort to refine the models to achieve high fidelity [1]. Despite their efficiency, most commercial solutions are subjected to high-cost licenses and have limited flexibility and range of applications [1]. One of the significant factors prohibiting the widespread application of robots by small and medium enterprises (SMEs) is the high cost and necessary skill for robot programming and re-programming [1,5,6]. In the last several years, there has been an effort to increase robotic and automation technology in SME pipelines. Although some progress has been made, there are still challenges to be overcome [5,6].

The arising concept of Industry 5.0 (i5.0) refers to the human-centric approach in manufacturing systems and focuses on enabling technologies for improving human–machine interaction, such as extended reality (XR), which has become a popular multidisciplinary research field over the last decade. XR, comprising augmented reality (AR), virtual reality (VR), and mixed reality (MR), has been increasingly used in networks of connected physical systems and human–machine communication. XR tools are used in several aspects of the manufacturing process, some major examples being assembly assistance, product design, maintenance, operators’ training, service, repair, etc. [6,7]. Recently, XR-assisted software solutions have been proposed in the robot programming domain [1,2,4,7,8,9,10,11,12,13,14,15,16,17].

This article presents a comprehensive procedure for developing a Unity-based standalone XR-enhanced interactive industrial robot programming application. The system is designed to offer cross-platform solution allowing users with limited robot programming skills, as well as experts, to leverage the advanced visualization capabilities of XR technologies for programming complex robot trajectories and facilitating the debugging process.

The methodology proposed within the research involves a comprehensive approach involving elements of robot programming and control within a Unity-based XR environment. Its key components are as follows:

• Robot modeling in Unity.
• Control design in Unity (including solving inverse kinematics, developing a trajectory planner, and designing motion controllers).
• Development of XR-enhanced robot functionalities in Unity.
• Development of post-processor for offline robot programming languages.
• Integration and verification.

The conceptual design and implementation of the system are presented; it offers novel XR-supported robot programming approaches, such as AR/VR-assisted Linear-Path Lead-Through (AR/VR-LT), AR/VR-assisted Linear-Path Walk-Through (AR/VR-WT), and AR/VR-assisted Debugging Tool. Industrial 6 DoF robot RL15 [18,19] is used to verify developed modeling, control, and programming concepts, procedures, and algorithms.

The result of this study is an open-source application, L-IRL XR, release 1, which is publicly available for use and further development at [20] under a GPL license. The download links for the executable and source files are included in [20]. The proposed framework is a cross-platform solution that is executable with a uniform code that is compatible across various devices (PC, macOS, Android, and iOS).

This article is divided as follows. In Section 2, related work and the main contributions of the article within the domain of research interest are given. The robot modeling procedure and control design strategy in Unity are presented in Section 3. Section 4 details the design of the XR-enhanced robot programming and simulation framework, including the system architecture and a description of the developed robot programming functionalities. Discussion is presented in Section 5; finally, concluding remarks are given in Section 6 of the article.

2. Related Work and Key Contributions

Considering the literature and available commercial solutions [1,2,4,7,8,9,10,11,12,13,14,15,16,17,21,22,23,24,25,26,27,28], the overall employment of XR HMI in industrial robotics can be separated into three categories: (1) robot programming, (2) system engineering, and (3) personnel training (see Figure 1). In the domain of robot programming, XR-technology represents a significant aid via its visualization tools as it allows better physical context awareness for the programmers.

Researchers seek ways to design intuitive and safe robot programming tools using emerging technologies. Often, simulation technology advancements depend on budgets that are available to build simulators and train people to use them [29,30]. Game engines are considered a cost-effective and flexible solution for developing an XR application [8]; over the years, the expansion of game engines and their capabilities has led to their increasing utilization in research involving robotics and multibody system simulation. The significant advantage of game engines like Unity [31] and Unreal Engine [32] over dedicated robot simulation software—which include frameworks and libraries that are predominantly used as dynamics engines, such as MuJoCo [33], DART [34], and Simbody [35] and standalone simulators such as Gazebo [36], Coppelia Robotics [37], Cyberbotics Webots [38], etc.—is their integrability with different AR/VR SDKs. Compared to specialized software such as ROS-Gazebo [39], game engines lack some of the control design tools (inverse kinematics solvers, path planners, motion controllers); for this reason, developers typically use the integration of game engines with a dedicated robotics development framework [40].

Unity, one of the most popular game engines, is a leading platform for the development of XR applications with high-quality 3D rendering, which can be deployed across a variety of platforms and can integrate different XR SDKs such as Google’s ARCore, Apple’s ARKit, Vuforia, and Microsoft’s MRTK; these are some of the most prominent frameworks at the moment [33,41,42]. Conventional offline industrial robot programming can be performed in robot’s joint space or by programming the robot’s tool center point (TCP) motion in Cartesian space. In Cartesian space, TCP motion is programmed by using the following: (1) point-to-point (PTP) motions where a user defines the initial and final poses while the trajectory planner defines the path between the two poses, according to the specific criteria; (2) continuous-path (CP) motion, where a user defines a path to be followed by TCP [43], such as linear, circle, etc. In order to achieve the TCP motion programming in Cartesian space, several integrations of Unity with ROS [44] (ROS–Unity suite) and offline/online programming environments like RoboDK [45] have been proposed recently; these leverage the ROS/RoboDK-integrated trajectory planner and its integrated numerical inverse kinematics solvers. The authors of [1,8,11,12,13,14] used the ROS–Unity suite; the work presented in [16] is based on the RoboDK/Unity suite, and [15] is based on the Matlab/Unity suite.

Compared to using Unity–robotics development framework software suites, a standalone XR-supported industrial robot programming system delivers several advantages:

• Reduced latency and computational overheads;
• Easier customization for developers;
• Elimination of the need for additional computers running ROS or similar software, thereby reducing the necessary hardware requirement and enhancing user mobility.

A standalone app eliminates the need for inter-process communication between Unity and ROS or similar software, which can introduce additional latency and computational overheads. This results in more streamlined and efficient execution. At the same time, there is no need for additional software installations or their integration with different devices, nor is there a need for developers to become acquainted with additional software, such as ROS [44] (ROS2 v.humble), RoboDK [45] (v4.0.0.), and similar. The development process in only one IDE is simpler than that of two IDEs for reasons such as simpler debugging, simpler testing, and better customizability.

The authors of [17] present a framework enabling end-users to manage complex robotic workplaces using a tablet and AR. The article proposes two different system integrations: standalone and integration based on ROS. Meanwhile, coding is performed in Python. However, a standalone solution does not include an augmented robot; rather, it uses a real robot, which suggests that robot downtime is performed when programming in AR. The authors of [2] propose an AR-supported robot programming system developed in C/C++ language. Regarding the robot programming method, pick-and-place programming (PTP robot programming) is proposed in [2], while [1,2,4,8,11,13] propose emulated Walk-Through and emulated Lead-Through [4,16].

The Unity-based solutions proposed in [1,8,11,12,13,16] do not tackle the integration of the robot dynamics in an augmented/virtual robot model, nor the setting of the parameters of the X drive component for articulation body. Transformation of the left-handed Unity frames to end-user frames is not discussed.

Motivation for the development of the proposed XR-supported robot programming system includes the following aspects.

• Develop AR/VR-supported robot programming framework that enables the following:

  (1)

  Industrial robots programming using conventional offline robot programming methods such as CP robot programming in an online regime in augmented/virtual reality. In this way, the downtime of conventional offline robot programming is avoided.

  (2)

  Using the advanced visualization capabilities and spatial awareness that the XR environment provides (compared to 2D-display-based virtual robot programming frameworks) for verifying the programmed path and for the debugging of robot programs.

• Develop cross-platform solution for a more consistent and adaptable experience for both developers and end-users.
• Standalone solution which integrates augmented/virtual robots.
• Open-source and customizable solution for developers. Contribute to research and developer communities by making the solution open-source.
• The main contributions of the research can be summarized as follows:
• Providing open-source, cross-platform, standalone XR-supported interactive robot programming system prototype.
• Development of the method for closed-form inverse kinematics implementation in Unity based on frames transformations.
• Development of the method for motion controller set-up in Unity that avoids oscillatory response in robot joint motion and takes into account installed actuators’ capabilities.
• Development of novel tools which allow for linear motions robot programming in Cartesian space in an AR/VR environment.
• Development of a novel AR/VR-assisted debugging system.

3. Robot Modelling and Control Design in Unity

The design of standalone XR-enhanced robot programming tools in Unity requires the development of a robot’s 3D model, the implementation of inverse kinematics algorithm for a specific robot, and, depending on the application, may require the implementation of the trajectory planner. Coding is performed in C#, as Unity supports the C# programming language natively.

The main requirements that were set for the development are as follows:

• The developed system should be cross-platform or cross-platform-adaptable;
• The developed system should integrate robot dynamics;
• The developed system should be able to verify the programmed path in an extended reality environment;
• The developed system should integrate a trajectory planner, enabling the programming of CP motion along linear paths in Unity;
• The developed system should be open-source and customizable;
• The developed system should integrate a post-processor for exporting the programmed paths as offline instruction code readable by the robot controller.

The architecture diagram for the proposed framework is given in Figure 2.

The approach to robot modeling in Unity is to create a detailed 3D robot model in a 3D-design software and import it into Unity. Herein, the import is performed by using XML file formats such as URDF (Universal Robot Description Format) [46] and glTF (GL Transmission Format) [47] (used for textures), which are open-source standards equipped with elements and attributes specific to robot kinematic and dynamic structure. SolidWorks [48] is used to design a 3D robot model as it allows model exportation to the mentioned XML formats using free and open-source add-ons [49,50]. The creation of the robot model in Unity is structured into Procedure 1, Appendix A.

Unity’s built-in physics engine, Nvidia PhysX, is used in this study. Apart from Nvidia PhysX, there is a possibility of using the MuJoCo (Multi-Joint dynamics with Contact) engine as a plug-in library. However, MuJoCo computes convex-hull approximations based on the 3D model, which may cause difficulties for highly detailed robot models with concave surfaces regarding visual fidelity and collision detection, enforcing additional efforts with implementation. Herein, the Articulation Body class [51] from Nvidia PhysX is used for the robot dynamics integration and motion controller set-up. In this research, a method for setting motion controller parameters, taking into account the applied actuators’ capabilities, has been proposed.

Several AR-supported interactive robot programming tools have been developed and integrated within one marker-based Unity-developed application, which is presented herein. Vuforia [41], an AR software development kit for smart portable devices (smartphones, tablets, head-mounted see-through devices, or smart glasses) that provide essential tools for creating AR applications, is used for generating AR robot image overlay in a real laboratory environment. The developed AR app is cross-platform within the realm of PC, macOS, Android-based, and iOS-based devices. Due to the system’s cross-platform adaptability, implementing the developed robot programming framework into a virtual reality (VR) environment is straightforward: developers simply need to utilize the appropriate VR SDK instead of Vuforia. This study used the Oculus Rift S VR headset to experience VR; its SDK is Meta XR All-in-One SDK (UPM) [52]. GUI development in Unity is based on the mixed reality toolkit [42].

3.1. Implementation of the Inverse Kinematics and Frames Transformations

The connection between the Euler angles, typically used by robot programmers to define the orientation of the TCP with regard to the reference frame, is commonly realized via a rotation matrix. In Equation (1),${}^1\mathbf{R}_2$ is a 3 × 3 rotation matrix, also known as a cosine direction matrix, representing the rotational transformation of the first frame (herein named FRAME1) to the second frame (herein named FRAME2). Columns of ${}^1\mathbf{R}_2$ can be regarded as scalar products of unit vectors x₁, y₁, z₁ constituting FRAME1 and unit vectors x₂, y₂, z₂ constituting FRAME2. The rotation matrix and the position vector form a 4 × 4 homogeneous transformation matrix (HMT), which is widely used in the robotics community to define the poses (positions and orientations) of the frames attached to robot links.

$${}^1\mathbf{R}_2=\left[\begin{array}{ccc}\cos \left({\mathbf{x}}_2,{\mathbf{x}}_1\right)& \cos \left({\mathbf{x}}_2,{\mathbf{y}}_1\right)& \cos \left({\mathbf{x}}_2,{\mathbf{z}}_1\right)\\ \cos \left({\mathbf{y}}_2,{\mathbf{x}}_1\right)& \cos \left({\mathbf{y}}_2,{\mathbf{y}}_1\right)& \cos \left({\mathbf{y}}_2,{\mathbf{z}}_1\right)\\ \cos \left({\mathbf{z}}_2,{\mathbf{x}}_1\right)& \cos \left({\mathbf{z}}_2,{\mathbf{y}}_1\right)& \cos \left({\mathbf{z}}_2,{\mathbf{z}}_1\right)\end{array}\right].$$

(1)

The Unity engine was not originally designed for robotics applications; as a result, certain conventions and practices commonly used in robotics may require additional adaptation. Key conventions for kinematic modeling in robotics, contrary to Unity, use right-handed frame orientation. Poses of a TCP in Unity editor are defined using Cartesian coordinates x, y, z, and Euler angles X, Y, and Z, representing the angles of rotation (in the clockwise direction) about the x, y, and z axes with regard to left-handed Unity BASE reference frame (here denoted as BASE_U) [53]. Unity uses the z-x-y Euler angles sequence of extrinsic rotation [54]. This sequence corresponds to the y-x-z sequence of intrinsic rotation. In order to control the robot movements by a user in Unity in Cartesian space, a solution for the inverse kinematics problem (IK algorithm) for the selected robot has to be integrated. Integrating the IK algorithm defined with regard to reference frame BASE_IK, oriented differently to BASE_U frame, implies the mapping of the TCP pose from the BASE_U frame to the BASE_IK frame. The numerical values of the TCP rotation matrix and position vector with regard to the BASE_IK frame are the input into the IK algorithm. Transformation from any left-handed frame to any right-handed frame by application of three successive rotations is impossible. For this reason, it is advisable for the left-handed BASE_U reference to be selected to match the axes of frame BASE_IK as closely as possible in order to obtain the simplest TCP pose transformations using axes transformation. This is demonstrated in detail in a case study represented in Section 3.3.

The IK algorithm is executed for each rendering frame based on Unity’s frame rate. The IK algorithm includes checking whether the defined joint positions are within their limits based on the robot’s workspace. Speeds are calculated for each rendering frame using the Euler differentiation method to verify that the defined joint speeds remain within the motor limits.

3.2. Robot Dynamics Integration and Controller Design

Dynamics play a highly significant role in the synthesis of control algorithms, the mechanical structure design, and the motion simulation of robots. For the purpose of robotic research and with the intention of being used for realistic physics behaviors in the context of industrial applications simulation, in addition to standard rigid body dynamics, NVIDIA PhysX SDK offers the reduced coordinate articulations feature based on Featherstone’s Articulation Body Algorithm (ABA) for forward dynamics computation [51,55]. Energy in mechanical systems can be stored in a spring or an inertia. In NVIDIA PhysX, articulations are driven by “joint acceleration springs” [51]. ArticulationDrive is a struct component implemented in Unity Engine PhysicsModule, which simulates the application of forces and torques to the interconnected bodies. The drive component executes force/torque F using user-defined stiffness and damping components:

$$F=\mathrm{stiffness}\left(\mathrm{currentPosition}-\mathrm{target}\right)-\mathrm{damping}\left(\mathrm{currentVelocity}-\mathrm{targetVelocity}\right).$$

(2)

In the scientific literature related to Unity and within Unity tutorials, there is no general rule or recommendation to set the stiffness/damping parameters, so users usually tune them case by case to match the real robot as closely as possible. Herein, we propose a simple rule for setting parameters of the X drive component of the Articulation Body, which avoids oscillations in the robot links motion and takes into account motor possibilities.

Adopting $F=m\ddot{x}$ in Equation (2) and converting the equation into continuous error domain yields:

$$m\ddot{e}-b\dot{e}+ke=0,$$

(3)

where k is the spring stiffness constant, b is the damping coefficient, and e = e(t) is the error signal, e = target – current position. Second order systems in general have complex-valued natural responses. The characteristic equation of the system given in Equation (3) is:

$$m{s}^2-bs+k=0,$$

(4)

The second order polynomial Equation (4) has two solutions (pole locations of the system):

$$s_{\mathrm{1,2}}={\displaystyle \frac{b\pm \sqrt{b^2-4km}}{2m}},$$

(5)

If the roots have an imaginary component, i.e., $b<2\sqrt{km}$, then the response has an oscillatory component. To avoid oscillations in system response Equation (5), the damping coefficient $b\ge 2\sqrt{km}$ has to be adopted. The critically damped case where $b=2\sqrt{km}$, and the poles of the second order system coincide on the real axis, is the fastest response without oscillations.

Within X drive parameters selection, it can be adopted that the spring stiffness constant k and the damping coefficient b correspond to the proportional $K_P$ and derivative $K_D$ gains of the positional PD controller, respectively; mass m is the mass of the link, and the relation $K_{Di}=2\sqrt{K_{Pi}{m}_i}$ is adopted for $i=1,2\dots n$. In order to achieve realistic robot simulation, herein we assume that the maximum force/torque generated by an actuator (drive limit parameters in X drive component), the input of which is the output of the feedback controller; this corresponds to the assumed maximum allowed error in position $E_{Pmaxi}$ and to the assumed maximum allowed error in the velocity $E_{Vmaxi}$ of the joint i:

$${\tau }_{\max i}=K_{Pi}{\mathrm{E}}_{\mathrm{Pmax}i}+K_{Di}{\mathrm{E}}_{\mathrm{Vmax}i},$$

(6)

where the maximum output (after the gear transmission) driving torque/force generated by the motor in joint i, ${\tau }_{\mathrm{m}\mathrm{a}\mathrm{x}i}=r_i{\tau }_{M\mathrm{m}\mathrm{a}\mathrm{x}i}$, $r_i$ is the gear ratio, ${\tau }_{M\mathrm{m}\mathrm{a}\mathrm{x}i}$ is the maximum motor torque at the rotor shaft, ${\mathrm{E}}_{\mathrm{Pmax}i}$ is the maximum allowed position error, and ${\mathrm{E}}_{\mathrm{Vmax}i}$ is the maximum allowed velocity error. Solving Equation (6) for $K_{Di}=2\sqrt{K_{Pi}{m}_i}$, and adopted assumed maximum acceptable error in position and velocity for i = 1, 2, …, n values $K_{\mathrm{D}i}$, $K_{\mathrm{P}i}$, have been obtained:

$$K_{Pi}={\displaystyle \frac{2m{E}_{\mathrm{Vmax}i}^2+E_{\mathrm{Pmax}i}{\tau }_{\max i}-2\sqrt{m^2{E}_{\mathrm{Vmax}i}^4+m{E}_{\mathrm{Pmax}i}{\tau }_{\max i}{E}_{\mathrm{Vmax}i}^2}}{E_{\mathrm{Pmax}i}^2}},$$

(7)

from which the derivative $K_D$ gain is calculated as:

$$K_{Di}=2\sqrt{K_{Pi}{m}_i},$$

(8)

3.3. Case Study-6DoF Industrial Robot RL15

For the verification of the proposed procedures, algorithms, and programming concepts, the 6DoF serial robot with revolute joints RL15 is considered, Figure 3. Technical parameters of RL15 robot are given in Appendix B.

For the RL15 robot, the inverse kinematics solution in [18,19] uses a right-handed frame presented in Figure 4a, with an adopted notation that A is the rotation about the x-axis, B is the rotation about the y-axis, and C is the rotation about the z-axis (positive in a counterclockwise direction). The IK solution in [18,19] uses specific Euler angles sequence of intrinsic rotations to form the TCP rotation matrix with regard to the BASE_IK frame.

In order to obtain the TCP rotation matrix and the TCP position with respect to the BASE_IK frame (shown in Figure 4a)—this was used within the previously developed IK [18,19], a left-handed BASE_U reference frame (shown in Figure 4b) is selected to match the axes of the frame in Figure 4a as closely as possible, in order to obtain simplest TCP poses transformations. From Figure 4, it follows:

$$x_{\mathrm{I}\mathrm{K}}=x_{\mathrm{U}},y_{\mathrm{I}\mathrm{K}}=y_{\mathrm{U}},z_{\mathrm{I}\mathrm{K}}={-z}_{\mathrm{U}},$$

(9)

$$A=-X,B=-Y,C=Z,$$

(10)

An arbitrary Euler angle sequence can be used to calculate the rotational matrix, which is a numerical input into the IK algorithm for the RL15 robot developed in [18,19]. If the y-x-z sequence of intrinsic rotation is adopted, then the following is obtained:

$$\begin{array}{c}{}^{\mathrm{B}\mathrm{A}\mathrm{S}\mathrm{E}\_\mathrm{I}\mathrm{K}}\mathbf{R}_{\mathrm{TCP}}=\mathbf{R}\mathbf{o}\mathbf{t}\left(y,B\right)\mathbf{R}\mathbf{o}\mathbf{t}\left(x,A\right)\mathbf{R}\mathbf{o}\mathbf{t}\left(z,C\right)=\\ \left[\begin{array}{ccc}\cos B\cos C+\sin A\sin B\sin C& \cos C\sin A\sin B-\cos B\sin C& \cos A\sin B\\ \cos A\sin C& \cos A\cos C& -\sin A\\ \cos B\sin A\sin C-\cos C\sin B& \sin B\sin C+\cos B\cos C\sin A& \cos B\cos A\end{array}\right]\end{array}$$

(11)

The IK algorithm for the RL15 robot developed in [18,19] is implemented in Unity in C# language.

For the RL15 robot, the controllers’ gains (X drive class stiffness and damping parameters) using the presented method in Section 3.2.—, i.e., Equations (10) and (11) for i = 1, 2…, n links—are given in

Table 1. X drive class stiffness and damping parameters for robot RL15.

Joint k	EPmax [rad]	EVmax [rad/s]	ki = KPi	bi = KDi
1	0.02	0.02	95,283.3	11,516.7
2	0.02	0.02	221,489	8160.72
3	0.02	0.02	100,166	5734.751
4	0.02	0.02	21,165	1979.84
5	0.01	0.01	75,287.2	844.82
6	0.005	0.005	36,626.9	363.121

. Motors’ limits manually entered into the X drive class are given in

Table A2. Robot RL15 model parameters.

Joint k	Mass [kg]	Max. Motor Torque [Nm]	Gear Ratio	Max. Joint Torque [Nm]	Motor and Gearbox Inertia [kgm2 10−3]
1	347.95	17.8	120	2136	4.2
2	75.17	38.6	119	4593.4	9.0
3	82.34	17.8	119	2118.2	9.0
4	46.3	3.47	133.4	462.9	0.8
5	2.37	3.47	219.4	761.32	1
6	0.9	3.47	53.3	184.95	0.7

in Appendix B.

The motion of the RL15 robot based on the proposed modeling and controller set-up methods is shown in the video provided in the Supplementary Materials. In AR applications, the robot is scaled down for the purposes of the demonstration video. The user sets the desired path in the Cartesian coordinate system, and the inverse kinematics calculates the movement of the joints.

4. Design of XR-Enhanced Robot Programming and Simulation Framework

In this section, the interactive AR/VR-enhanced, standalone robot programming and simulation tools developed in this research are presented. Several robot programming concepts and tools are proposed:

• AR/VR standalone Linear-Path Lead-Through robot programming tool (AR/VR-LT);
• AR/VR standalone Linear-Path Walk-Through robot programming tool (AR/VR-WT);
• AR/VR standalone Robot Programming by demonstration tool (AR/VR-PbD);
• AR/VR standalone Simulator tool for offline robot programming languages scripts (AR/VR-S);
• AR/VR standalone Debugging tool for industrial robot arms in AR/VR environment (AR/VR-D).

There is some discrepancy regarding the terminology used for robot programming methods in the literature. To avoid confusion with the classifications and terminology, the terminology used herein is in compliance with that in [56,57].

To verify the proposed programming concepts requiring trajectory generation, a trajectory planner (TP) has been developed and integrated into Unity, Appendix C. The GUI for the developed AR/VR-enhanced robot programming tools is given in Figure 5.

4.1. Description of the Functionalities of the XR-Enhanced Robot Programming Tools

4.1.1. AR/VR Linear-Path Lead-Through

AR/VR-LT implies a Lead-Through programming method where the TCP motion is programmed using a teaching pendant in an AR/VR environment. GUI has been created with buttons and functionalities to emulate teaching pendants in AR/VR (Figure 6). The user can move TCP using developed GUI buttons and, after moving the TCP to the desired poses, they can choose to generate linear path segments between the selected waypoints in Cartesian space, or they can save the poses into the memory for later usage.

4.1.2. AR/VR Linear-Path Walk-Through

The AR/VR-WT robot programming method is conceptualized as an integration of the Walk-Through method and linear CP robot programming method in an AR/VR environment. This method harnesses the advantages of time-efficient and intuitive Walk-Through robot programming [1,4] and applies them to the linear CP method. AR/VR-WT implies the manual “dragging” of the TCP position and/or the orientation of a robot in an AR/VR environment by using an emulated robot handle—a VR headset joystick or a phone/tablet for an AR implementation. In the AR environment, to allow the user to move the robot’s TCP as intuitively as possible, the TCP coordinate frame becomes a child of the AR camera frame, implying that the TCP pose follows the motion of the phone/tablet executed by a user. In the VR environment, the user moves the joystick to change the TCP pose according to the position and orientation of the joystick. After moving the TCP to the desired poses, the user can choose to generate linear path segments between the selected waypoints in Cartesian space, or they can save the poses into the memory for later usage. The AR/VR-WT robot programming method is depicted in Figure 7a (AR) and Figure 7b (VR).

4.1.3. AR/VR- Programming by Demonstration

The AR/VR-PbD method is built upon the previously described AR/VR-WT tool, where the TCP is dragged by the operator in an AR/VR environment. Contrary to the AR/VR-WT method, where the linear trajectory between desired waypoints is defined and interpolated by the planner, here, the trajectory generation is performed by recording the motion of the TCP. In other words, a trajectory is generated directly by the operator’s hand movement, which the TCP follows. The validity of the execution of the programmed movement of the physical (real) robot is ensured by the inverse kinematics calculation (with accompanying IK checks on joints’ position and speed described in Section 3.1) for every rendering frame, as well as by the integration of dynamics and the control method using X drive from the Articulation Body class into the system (for a more detailed explanation, see Section 3.1 and Section 3.2). The AR/VR-PbD inherits the intuitiveness of the conventional Walk-Through robot programming method in an AR/VR environment (see Figure 8a (AR) and Figure 8b (VR)).

4.1.4. AR/VR-Simulator

In the simulator mode (AR/VR-S), Figure 9a,b, a visual playback of the programmed robot motion is conducted in an AR/VR environment, which is useful for the programmed trajectories advanced visual inspection and verification in a safe environment. Users can inspect the robot path from a close distance, from multiple directions, use the integrated slider, pause the operation, “inspect” programmed speed, etc. The attainable trajectory is defined in one of two ways:

(1)

Using previously described AR/VR-supported tools;

(2)

Reading poses from a text file obtained as the output from offline robot programming languages.

Regarding the latter, a parser for reading outputs of selected offline robot programming languages has to be developed. In this research, a simple parser specifically designed to interpret the output of the L-IRL robot programming language [18] has been developed.

4.1.5. AR/VR-Debugger

Industrial robots’ programming presents a unique set of challenges regarding the debugging problem. One of the challenging situations in this sense is a debugging process for continuous-path TCP motion due to the highly nonlinear and coupled nature of the inverse kinematics solution, which can make the cause-and-effect relationships between the robot’s joint motion and a TCP motion in Cartesian space difficult to interpret. AR/VR provides many opportunities for enhancing debugging by providing immersion and allowing the developer to visualize problematic TCP poses and path segments that are superimposed in situ in the real world, intuitively displaying the limitations and discontinuities in the robot’s real-world view [7,10].

This research introduces an AR/VR Debugger Tool (AR/VR-D), which is designed to provide straightforward motion application program debugging and trajectory correction to ensure achievability within the specific robot’s workspace parameters. Firstly, by using the AR/VR-D tool, the user is able to inspect the unattainable TCP path in the AR/VR environment. Figure 10a shows the example of the unattainable TCP path, i.e., an array of successional poses that the robot’s TCP should go through for a programmed robot motion with some of the poses classified to be unattainable by the trajectory planner due to the robot limits being exceeded within the robot’s workspace. The unattainable path is generated by implementing a modified IK algorithm, in which joint angle limits have been removed. The user can then examine the unattainable TCP trajectory, helped by the visual color presentation of the predefined TCP path and a vertical stack of numerical values of the robot’s joint angles, which are calculated using original IK algorithm (with accompanying checks on joint limits) (see Figure 10a). In Figure 10a, it can be seen that the TCP path can take a blue, yellow, or red color: unachievable poses are shown in red; poses nearing unattainability—specifically, those within a defined half-range of 10° to the joint limits—are shown in yellow. In the case presented in Figure 10a, the impossibility of achieving programmed TCP trajectory is a consequence of exceeding the maximum allowed angle of the 5th link (highlighted in red). The slider is added so the user can slide along the trajectory in order to visualize the mechanism between TCP pose change and joint position changes in critical path segments.

The user can correct unachievable TCP trajectory in EDIT mode by selecting and editing the unattainable TCP poses (Figure 10b). An unattainable pose is selected by bringing the TCP close to it and by pressing and holding the EDIT button. When the TCP pose snaps to an unattainable pose, the user is enabled to freely move the TCP to find a new attainable pose. The displayed robot joint angles and TCP Cartesian coordinates facilitate the editing of the problematic pose.

4.2. System Architecture and Workflow Diagram

Figure 11 shows the system components and information flow for each developed function. The general, more detailed flow chart of the system is given in Figure 12.

The integrated functionalities that have been developed and implemented as C# scripts in Unity for the tools presented in Figure 11 are given in Figure 13 and Figure 14 below.

AR/VR Walk-Through (AR/VR-WT) allows the user to move the robot TCP by “dragging”. Figure 13 shows the flow chart of the developed AR/VR-WT feature. Let us assume that the user is using an AR application, and that the AR-experiencing device is a smartphone/tablet. The Update() function is the main function of C# code in Unity, and it is executed once for each rendering frame. By holding the AR/VR-WT button (see Figure 5) at the beginning of the first frame, a new (temporary) GameObject called Temp is created; TCP coordinates are assigned to it and the TCP coordinate system becomes a child of a phone/tablet AR camera coordinate system. This part of the program code is executed every time when the AR/VR-WT key is pressed again. After that, still holding the AR/VR-WT button, Temp GameObject has the desired TCP coordinates in space at all times, so they are simply transferred to the TCP. In a similar way, the TCP robot moves in VR.

Figure 14 describes the basic implemented functions of the L-IRL XR application that allow programming in the AR/VR environment. The AddCoords() function allows the user to visually set a new frame (pose) in the robot workspace. Such poses are used to generate the trajectory. Programmed TCP poses in Cartesian space can be redefined by the user using the EditCoords() function, which is also used for unattainable trajectories. The LoadCoords(), SaveCoords(), and RemoveCoords() functions allow the user to load poses from a .txt file, save poses to a .txt file, or completely erase them from the robot’s workspace.

4.3. Post-Processor

A simple post-processor is developed for the exporting of programmed paths as offline instruction code; these are readable by the robot controller. Post-processing outputs are demonstrated in the video within the Supplementary Materials.

5. Discussion

For robotic applications, some of the prime features that need to be considered when selecting robot modeling, control, and simulation environments are simplicity, reliability, safety, and precision. Low cost and customizability are important as well. XR technology offers many advantages in enhancing human–robot interaction, which can be valuable for both novices and experts.

In this research, a standalone XR-supported robot programming framework for industrial robots is proposed. The absence of communication between the XR-experiencing device and the ROS-enabled computer contributes to minimizing latency and computational overheads in the system. Consequently, it could be argued that the proposed solution enables smooth running on less-expensive devices.

From a developer’s point of view, the absence of the need to develop system interaction between Unity and robotics development frameworks/environments (APIs) is advantageous. Achieving real-time synchronization between robotics frameworks such as ROS and Unity can be challenging, particularly when dealing with robotics applications that require precise timing and coordination. Delays or synchronization issues may impact the accuracy of simulations. Both ROS and Unity are complex systems with their own learning curves. Also, ROS and other robot frameworks evolve over time with new releases, and compatibility with different versions has to be ensured.

One of the shortcomings of the Unity-based standalone robot programming software at the moment is the simulation of motion controllers, as limited motion controller design is available compared to the ROS–Unity suite. Even though real robot motor controller settings are usually not available to users, it could be useful for research to simulate advanced motion control laws in an XR environment. Still, XR enhancement in robot programming is mostly used for superior visualization benefits.

The standalone feature enables better mobility of the end-user, as it is not necessary to connect devices on the same network, contrary to the solutions integrating ROS [1,8,11,12,13,14] and RoboDK [16].

The solution presented in [17] has a standalone mode in which the programming of a real—not augmented—robot is performed (which causes robot downtime). The solution presented in [2] is a standalone solution with an integrated kinematic and dynamics model offering pick-and-place (PTP) robot programming. It uses additional external libraries for robot dynamics and kinematics, as well as the plotting library. Compared to the solution presented in [2], which is developed in C/C++, the system presented herein is simpler to develop, as it is based on Unity and its supporting libraries.

The positive impact of XR-based HMIs in robot systems has been proposed as a solution for improving the time and cost efficiency of the robot programming process [1,2,4], which could positively influence the percentage of robot employment in SMEs. One important aspect of the proposed system’s applicability, particularly for SMEs, is the cost of integrating the novel XR-assisted system into the production line. Software development costs generally include licensing fees and development expenses. While Unity is mostly free for developers and entirely free for end-users, ROS is also free, and RoboDK is a commercial software requiring end-user licenses as it must run on an additional computer. As the proposed system is standalone, development costs could be higher compared to ROS–Unity suite-based solutions [1,8,11,12,13,14], but not significantly, whereas hardware costs are lower. Implementation, training, maintenance, and operational costs depend on the specific solution’s design and implementation details.

The ROS dependency of the ROS–Unity suite-based solutions in [1,8,11,12,13,14] makes cross-platform compatibility difficult since ROS is primarily targeted to certain Linux distributions. There is an ROS version for Windows, but it is not as well documented and supported as the Linux versions. The application presented in this article is standalone and exclusively Unity-based, which inherently facilitates cross-platform implementation. The solution proposed in this article is an AR cross-platform and AR/VR cross-adaptable solution. Cross-platform and cross-platform adaptability features offer a more adaptable experience for both developers and end-users.

Regarding motion programming functionalities, this research proposes two novel XR-robot programming concepts: AR/VR Linear-Path Lead-Through and AR/VR Linear-Path Walk-Through. AR/VR-LT and AR/VR-WT represent the integration of classical Lead-Through and Walk-Through online programming methods with offline continuous-path programming [43], while inheriting the simplicity of online Lead-Through [1] methods and the simplicity/intuitiveness of the online Walk-Through method [1,4], applying them to the domain of offline programming for continuous robot paths. One of the significant advantages of programming in an AR environment is that the robotic environment does not have to be modeled, including new robot manipulation objects or working surfaces.

All the presented functionalities of the XR-supported robot programming system are based on robot models with integrated dynamics, including motor capabilities with a novel method for controller set-up in Unity, which is one of the contributions of this paper.

The proposed open-source framework for XR-enhanced robot programming could be used as a complementary tool for commercial software, offering the advanced visualization capabilities of XR technology for trajectory verification and debugging purposes.

6. Conclusions

In this study, the design and implementation of a standalone XR-enhanced interactive industrial robot programming system using the Unity game engine is presented. The proposed open-source framework aims to deliver a cross-platform solution to minimize the effort and time invested by human–robot programmers, as well as system developers, and provide novel programming concepts leveraging the advantages of online robot programming methods to offline robot programming using augmented and virtual reality. The here-proposed development methodology includes a procedure for creating a robot model in Unity using CAD modeling; additionally, it includes the implementation of a solution to the inverse kinematics problem, appropriate frame transformations, and a method for motion controller design in the robot’s joint space, which avoids the oscillatory response in robot joint motion in Unity and takes into account the installed actuators’ capabilities. The robot programming system provides an XR-assisted simulation tool and an XR-assisted debugging tool for the enhanced visual trajectory verification and debugging process. Verification of the proposed modeling, control, and programming concepts, procedures, and algorithms is performed using a 6DoF robot. The methodology demonstrates a detailed and structured approach to developing a standalone, cross-platform, interactive, XR-supported industrial robot programming system; it leverages the strengths of Unity and XR technologies to enhance user experience and efficiency in robot programming.