Collision Risk Assessment and Operation Assistant Strategy for Teleoperation System

Abstract

Teleoperation robots remain superior to fully automated robots in complicated and unstructured environments (e.g., in-orbit assembly). However, the collision risk is also greatly increased in these environments. Therefore, the teleoperation robot should possess the capability of collision risk perception and be configured with security assistance strategy to improve safety and efficiency. With this objective in mind, this paper proposes a collision risk assessment system based on fuzzy theory, which comprehensively considers the effects of shortest distance, operation speed, and delay time. The introduction of fuzzy theory makes the risk assessment results more accurate. Furthermore, this paper also proposes a creative discrete expandable bounding box method to make the calculation of the nearest distance between complicated obstacles and the robot more efficient and faster. Secondly, to improve the safety and efficiency of teleoperation, this paper proposes a set of teleoperation assistance strategies for teleoperation robots based on collision risk. The strategies include partial view highlighting, variation in the motion mapping ratio, and haptic risk warning, all of which are achieved in the graphical interactive interface. Finally, this paper verifies the proposed collision risk model and the operation assistant strategy through experiments. The results show that the proposed fuzzy collision risk model has a correct trend with risk factors, and the proposed operation assistance strategies can effectively reduce the robot collision risk and improve the safety of teleoperation robots. In conclusion, this research contributes to the collision risk assessment and obstacle avoidance assistance strategy of teleoperation robots.

1. Introduction

With the development of in-orbit manufacturing technology [1,2], in-orbit printing-assembly has become a trend for large spacecraft. In-orbit assembly can be divided into automatic and teleoperated assembly. Fully autonomous robots have certain drawbacks in complicated and unstructured environments, such as in-orbit assembly, due to mechanical, sensor, and control technology limitations [3]. When compared with automated robots, teleoperation robots can combine the initiative of operators with the high precision of the robot, which remain superior to the fully autonomous approach for rapid decision making and the handling of emergency cases [4].

Safety is critical in the field of teleoperation assembly. Due to limited vision, deficient control technology, and the physiological limitations of the operator, the teleoperation assembly robot can easily collide with the surrounding assembly parts, and even slight negligence may cause unnecessary loss [5], especially for high-risk, high-value aerospace exploration tasks. Therefore, collision risk assessment and obstacle avoidance strategies become primary issues in teleoperation applications. This not only boosts the safety of teleoperation systems, but also enhances the efficiency of safely accomplishing the assembly task.

The most straightforward and brief method to avoid collisions with surrounding assembly parts is the virtual fixture, which assists a teleoperation robot to accomplish assembly task by restricting the robot’s movement to certain areas [6] or guiding the robot along a preset path [7]. The virtual fixture can be custom generated [8] or automatically generated [9]. The virtual fixture ensures the safety of the robot system while maintaining a certain degree of operational freedom, but it also constrains the robot from moving beyond the allowed range, thereby limiting the flexibility of teleoperation. In addition, artificial potential field-based control [10] may be a better option to address issues in path planning and real-time control [11,12]. Each surrounding assembly part is considered as the origin of the repulsive potential field; the repulsive force is inversely proportional to the distance from the obstacle, and the risk can be directly assessed by the repulsive force. A drawback of this approach is that the classical potential field cannot capture the relative motion between the robot and the obstacle and is mostly applicable in two-dimensional space.

The above-mentioned methods are more operationally oriented and, for that reason, cannot directly capture the collision risk of the operation. Therefore, teleoperation-assisted control methods based on safety assessment are attracting increasing attention. Zanchettin et al. [13] used the minimum separation distance criterion (MSDC) to evaluate the safety of the robot and to improve operational safety by constraining the robot speed. Sun et al. [14] estimated the collision probability by obtaining the distance between the robot and the obstacle and translating it into a conservative speed limit. A so-called distance-controlled velocity was proposed [15], the key idea of which is to adjust its speed on the specified path according to the robot’s distance relative to environmental obstacles. This concept was further clarified in [16]. These studies evaluated the collision risk of the robot by calculating the closest distance between components. The evaluation results were not sufficiently accurate, because only the influence of a single factor was considered.

To comprehensively assess the collision risk, Vatcha et al. [17] considered velocity in the collision assessment model and proposed the concept of a dynamic envelope, which is a region around the surface of an obstacle whose size depends on the risk assessment results. Nevertheless, this method is only applicable to known simple-shaped obstacles, and a similar concept was proposed in [18]. Rocco and Lacevic et al. [19,20] proposed a hydrostatic and electrostatic hazard field containing information about the robot’s attitude and velocity to assess the hazard level of the robot’s surroundings. Motoi et al. [21] generated a collision prediction map by using the collision time between a robot moving at a certain speed and the surrounding obstacles. A dynamic modified SSM (speed and separation monitoring) method was presented in [22] for ensuring the safety of human–robot collaboration and added some inherent characteristics of robots that had been quantified to the risk index; the safety of the assisting robot was improved by adjusting the speed. Pan et al. [23] not only took direct factors, such as speed and position, into consideration, but they also considered indirect indicators, such as application time and operational proficiency; their work was divided into three levels based on the calculation: safe, dangerous, and very dangerous. The kinematic inverse solution with high safety was selected through the redundancy of the robot arm. This method was also demonstrated in [24], despite causing significant joint movements of the robot arm.

Although these studies considered multiple factors, they mainly used a linear formula calculation approach for risk assessment with a certain percentage of each risk factor. But this is not reasonable, because the role of different factors is not necessarily fixed in different situations. Under such circumstances, the fuzzy theory has shown excellent outcomes as an intelligent method for speed control, temperature regulation, etc. [25]. It incorporates the wisdom of experts into the decision-making process, thus making it applicable to different situations and allowing decision makers to learn and maintain it in the decisions. Fuzzy theory was used to fuzzify the input risk factors in [10], but fuzzy inference methods were not used for crash risk assessment. In [26], fuzzy theory was applied to the collision risk assessment of ships, but, currently, no application has been reported in the collision risk assessment of teleoperation processes.

To address the above issues, this paper proposes a new collision risk assessment model to address the current incomplete and inappropriate issues of collision risk assessment. Based on this model, a series of teleoperation-assisted control strategies have been presented in this paper to improve the safety and efficiency of teleoperation. In order to make a more comprehensive assessment of the collision situation, this paper developed a fuzzy-theory-based integrated collision risk assessment model, which takes three factors into consideration: closest distance, operation speed, and delay time. Moreover, compared to traditional risk evaluation methods, the process of fuzzy reference for collision risk incorporates expert experience, thus helping decision makers in extracting and keeping expert knowledge. In the calculation process of risk factors, this paper puts forward a novel discrete expandable bounding box method in calculating the shortest distance between the robot arm linkage and the surrounding assembly parts; the mechanical linkage is enveloped by a series of bounding boxes of different radii, and the smallest radius of all bounding boxes colliding with the obstacle is considered as the relatively shortest distance, thus reducing the computational difficulty. In order to achieve more effective and safer work for teleoperated robots, this study proposed a teleoperation-assisted control strategy based on the assessed collision risk: The virtual feedback force was linearly mapped to the collision risk through the development of haptic gloves to achieve intuitive physical feedback on the safety of the current operation. The introduction of risk-assessment-based variable scale motion mapping and highlighted dangerous area display method make the robot operate more reasonably and effectively in obstacle avoidance during operation, thus reducing the possibility of collision. Finally, the accuracy and effectiveness of the proposed collision risk model and assisstant strategy were verified by simulation and experiments. In brief, this study provides an effective method for collision risk assessment and the obstacle avoidance of teleoperation robots.

Apart from Section 1, the paper is organized as follows: Section 2 describes the system overview. The detail of graphical modeling is described in Section 3. Section 4 presents the collision risk assessment model. Section 5 introduces the assistant strategy of teleoperation. Next, in Section 6, the designed collision risk assessment model and assistant strategy are verified and analyzed by experiments. Finally, Section 7 summarizes the research of this paper and puts forward future work.

2. Proposed System

As is shown in Figure 1, a typical teleoperation assembly system is constituted of a local side and a remote side. The local side is composed of an operator, a visualized interface of the remote environment, a haptic feedback glove, and a position locator. In the master subsystem, all the devices are wireless, which enables the operator to control the objects naturally and flexibly. On the remote side, a cooperative robot is installed to perform in-orbit assembly tasks. At the end of the robot, a force sensor is equipped to measure the contact force. The video streams are acquired using RGB depth (RGB-D) cameras. The communication between the local side and the teleoperation robot is performed by protocol/internet protocol (TCP/IP).

When an operator sends the command to the remote side through the position locator device, the virtual robot, as well as the teleoperation robot, follow the trajectory of the locator device. A wireless position locator device (VIVE Tracker, HTC Inc.) was adopted in this system to perform motion tracking. Following a communication delay, the teleoperation robot is controlled by the commands, and the visual, contact force information were fed back to the operator.

During teleoperation assembly, the robot must keep avoiding collisions with surrounding assembly parts. Even a minor collision can cause serious damage to the robot or workpiece. To improve the safety and efficiency of teleoperation assembly, this paper proposes a teleoperation-assisted system based on collision risk assessment. Two main procedures were implemented in the entire framework: First, a collision risk assessment model based on fuzzy inference was proposed, and motion security in the VR environment was evaluated. Secondly, assistant strategies were defined based on the evaluated collision risk to improve the manipulation safety and efficiency.

Figure 2 describes the entire framework of the teleoperation system. Based on the visual information of the depth cameras, the graphic model of the teleoperation robots and environment was built using the 3D reconstruction technique. The operator obtained the shortest distance relative to the obstacle, delay time, and operation speed in a graphical interface by controlling the virtual robot in real time in a virtual environment. In this system, the shortest distance was measured by the proposed discrete expandable bounding box method. Based on the aforementioned three elements, this paper designed a fuzzy-theory-based collision risk assessment model to evaluate the security of motion. Furthermore, three guidance strategies were defined in this paper according to the evaluated collision risk, including partial view highlighting, variation in motion mapping ratios, and haptic risk warnings. These assist strategies were implemented with the described physical interaction units.

3. The Establishment of the Virtual Environment

Due to the collision risk assessment and teleoperation assistance strategies were implemented in the VR environment. The reconstruction of the remote environment was critical for the designed teleoperation system.

3.1. The Graphical Modeling of the Robot

In the system designed in this paper, a 6-DOF lightweight robot (UR5, Universal Robot Inc.) was employed as the teleoperation robot. The master position, acquired from the localizer in a Cartesian coordinate system, was transmitted to control the virtual robot. The position of the localizer needed to be converted to a set of robot joint angles to allow an accurate description of the desired robot posture. In this paper, the D-H method based on the shortest path principle was adopted to solve the inverse-kinematics problem. Based on the calculated joints of the robot and geometric parameter, a visualized interface of the robot arm was realized as shown in Figure 3. OpenSceneGraph 3.4.0 (OSG), an object-oriented, cross-platform 3D graphics toolkit that offers the features and capability of OpenGL, was used to build the graphical modeling of the robot [27].

3.2. The Graphical Modeling of the Environment

In contrast to the teleoperation robot, there are usually no sensors arranged on the manipulated object to directly obtain location information. Therefore, we achieved the reconstruction of the geometry model by visual localization.

In this research, a camera was fixed on the table for identification and positioning of the object. Owing to the extremely low price and good performance, the Realsense (D455) vision sensor was employed in our system to locate the target objects. In order to realize accurate and stable posture detection, this research designed a marker-based posture detection method by integrating ArUco markers at the objects. The Sergio algorithm [28] was adopted to locate the posture of the markers.

The ArUco marker is a square marker composed of a wide black border and an inner binary matrix that determines its identifier (ID number), as shown in Figure 4. The main benefit is that their detection is robust, fast, and simple. By combining the ID number of the ArUco marker and the projection state of the black border in the camera image, the posture of the target relative to the camera can be calculated. Moreover, ArUco markers are not only used for component positioning, but also store and extract component assembly information for multi-component assembly task planning.

As shown in Figure 5, The following processes make up the positional estimation of ArUco markers:

(1)

Enhanced grayscale image boundary thresholding that uses a locally adaptive thresholding technique. Contour extraction was performed on the threshold image using the Suzuki algorithm [29], and, then, polygon approximation was performed using the Douglas–Peucker algorithm to detect possible candidate markers in the scene.

(2)

To evaluate if the aforementioned potential markers were “genuine” markers, the internal encoding of the markers was recovered.

(3)

To determine if the aforementioned candidate markers were actual markers or simply a byproduct of the environment, marker ID recognition and error correction were carried out.

(4)

To determine the marker’s position in relation to the camera, corner point optimization and pose estimation were performed.

(5)

Based on the relationship between the camera coordinate and the base coordinate, the posture of the marker relative to the base of the robot arm was calculated.

In this study, the 3D model of the object was constructed in SolidWorks, and then these models were assembled in OpenSceneGraph to determine the posture of the surrounding assembly parts by the acquired positioning information. Finally, the robot arm model and the reconstructed virtual environment were combined in one scene, as shown in Figure 6.

4. Collision Risk Assessment

A dynamic collision risk assessment model based on fuzzy theory was developed in this study to assess the collision risk of operational activities during teleoperation. Specifically, the collision risk assessment method consists of three steps: selecting the assessment factors, obtaining the assessment factors, and calculating collision risk in real time using fuzzy theory.

4.1. Selection of the Evaluation Factors

The accurate and appropriate selection of risk indicators is the basis for correct collision risk assessment. Currently, most studies only consider distance as a factor, which leads to less accurate assessment results. In this section, three basic factors that have a significant effect on collision risk were selected to build a fuzzy factor indicator set, which included three factors such as the relative shortest distance D, the delay time of teleoperation T, and the speed of operation V. These factors can accurately reflect the risk degree of robots. Moreover, these indicators are irrelevant to each other.

• The relative shortest distance D:

Since collision occurs at a distance of zero, the relative shortest distance has a direct impact on the collision risk. It is also the factor of most interest in the current research. The closest distance is the shortest physical distance between each obstacle nearby and the robot arm linkage. It is not just the distance from the robot arm’s end to the workpiece’s center, but is an object-to-object distance.

2.

The delay of teleoperation T:

For teleoperation robots, the time delay is one of the critical factors affecting operation risk. The time delay includes the signal transmission delay caused by limits on the speed of radio transmission and the data processing delay at sending and receiving stations and satellite relay stations. Furthermore, the delay time of teleoperation is not a definite value, and it is very susceptible to the quality of signal transmission. The uncertain time delay will cause some uncertainty [5] in the whole teleoperation system, which directly increases the risk of collision. Therefore, the delay time of the system should also be considered as one of the factors.

3.

The operation speed V:

Operation speed is closely related to collision risk. For example, the collision risk at a high speed is much higher than that in slow conditions. Especially in the case of large latency, fast speeds can cause large deviations between the remote operation and the visual display. Consequently, teleoperation robots are prone to colliding with non-target objects. As can be seen from the above, operation speed is an important factor that has a significant effect on the collision risk.

4.2. Calculation of the Evaluation Factors

• Calculation of the relative shortest distance

Due to the irregular shape of the obstacles and the robot, it is challenging to calculate the distance between the connecting rod of the robot and the surrounding assembly parts using the conventional method. To simplify this problem, the method of the bounding box [27] was applied in this paper. Additionally, when measuring the nearest distance with conventional techniques, the two closest positions of the robot and the obstacles must be obtained, which can be challenging to accomplish [23]. Therefore, in this study, the discrete expandable bounding box approach was proposed as a solution to tackle this issue.

To illustrate this method briefly, only one obstacle is introduced. As is shown in Figure 7, the connecting rods of the robot are wrapped with expanding capsule envelopes of various radii. The radii of the capsules are $R_0$, $R_0+d_1$, $R_0+d_2$, and … $R_0+d_n$, respectively, where $R_0$ denotes the smallest radius needed to completely enclose the robot arm. These preset envelope boxes move in tandem with the robot arm as it moves. Therefore, the shortest distance between the robot connecting rod surface and the surrounding obstacles can be converted into a collision detection problem between the bounding boxes and the surrounding obstacles. The smallest radius in all bounding boxes that collides with obstacles is considered as the relative shortest distance.

The aforementioned procedure was performed utilizing the designed digital interface whose image processing function can handle the mentioned bounding box collision detection and offer feedback on the radii of contact layers. Despite the discretization method’s limitations of precision, the issue is made more solvable, and the inaccuracy is within acceptable bounds.

The specific calculation process is shown in Equations (1) and (2):

$$R_{box-i}=R_0+d_i(d_i<d_{i-1})$$

(1)

$$D=\min (d_{i\text{\_}contacted})$$

(2)

where, $R_{box-i}$ represents the radius of the $i$ layer bounding box, $R_0$ denotes the smallest radius needed to completely enclose the robot arm, $d_i$ denotes the increase in the radius of the $i$-th bounding box compared to the minimum radius bounding box, and $d_{i\text{\_}contacted}$ indicates the increase in the radius of the bounding box where contact occurs.

2.

Calculation of delay time T

The delay time comprises the delay in signal transmission brought on by radio transmission speed restrictions as well as the delay in data processing at sending and receiving stations and satellite relay stations [30]. As a result, the delay time is the amount of time that passes between sending a response signal by the local side and getting it from the remote side. To gauge the delay time, a master signal was created. It was initially transmitted by the local side, if it was accepted by the slave, the response signal was sent back to the host. The time gap was used as the delay time of the current command period. The formula for calculating the delay time of the current command cycle is as follows:

$$T_i=T_{i\text{\_}recevie}-T_{i\text{\_}send}$$

(3)

where, $T_i$ represents the delay time of the $i$ command cycle, $T_{i\text{\_}send}$ is the the time of sending signal, $T_{i\text{\_}recevie}$ presents the time of receiving response signal, and $i$ is the command cycle.

3.

Calculation of operation speed V

The operating speed of the current state is directly proportional to the movement distance, so the speed is calculated by the movement distance, which is calculated as follows.

The operating distance of the current state is:

$$S_i=\sqrt{{(p_i-p_{i-1})}^2}+\sqrt{{[(r_i-r_{i-1})/6]}^2}$$

(4)

where, $S_i$ denotes the movement distance of the operator; $p_i$($p_x,p_y,p_z$) refers to the 3D position coordinate of the operator at the sampling period $i$; $p_{i-1}$ indicates the position of the operator at the sampling period $i-1$; $r_i$($r_x,r_y,r_z$) denotes the posture of the operator at the sampling period $i$; and $r_{i-1}$ indicates the posture of the operator at the sampling period $i-1$. The operating speed V at this point is:

$$V=S/T_{sample}$$

(5)

where $T_{simple}$ is the time of the sampling period.

4.3. Collision Risk Assessment Model

Fuzzy set theory was proposed by Zadeh in 1965 and is extensively applied to model uncertain systems when there is no complete and precise information [31,32]. As shown in Figure 8, A typical fuzzy system consists of three function layers: an input layer, a fuzzy inference layer, and an output layer. According to traditional set theory, an element either belongs to a particular set or not. However, in fuzzy set theory, the association of an element with a particular set lies between 0 and 1, which is called the membership degree. As with distance and collision risk, the degree of danger cannot be expressed precisely, but can be described in fuzzy terms, such as “extremely dangerous” or “somewhat dangerous”, Additionally, the fuzzy inference layer combines expert experience, which can help decision makers in extracting and keeping expert knowledge. Therefore, the fuzzy theory was introduced in this study.

• Fuzzy set and fuzzification

The purpose of fuzzification is to transform the exact input variables into fuzzy variables with linguistic meaning. Before the performance of fuzzification, a scale transformation of the input values was executed to convert them into their corresponding domain ranges. The output domain was [0–5], and the transformed input domains were T = [0–5], V = [0–5], and D = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] respectively. The transformation of the shortest distance is nonlinear, which is set according to the discrete radius of the envelope and experience. The transformation formula is:

$$\overline{D}=\left\{\begin{array}{l}\begin{array}{ll}1& (0 \mathrm{cm}<d\le 1 \mathrm{cm})\end{array}\\ \begin{array}{ll}2& (1 \mathrm{cm}<d\le 2 \mathrm{cm})\end{array}\\ \begin{array}{ll}3& (2 \mathrm{cm}<d\le 3 \mathrm{cm})\end{array}\\ \begin{array}{ll}4& (3 \mathrm{cm}<d\le 5 \mathrm{cm})\end{array}\\ \begin{array}{ll}5& (5 \mathrm{cm}<d\le 7 \mathrm{cm})\end{array}\\ \begin{array}{ll}6& (7 \mathrm{cm}<d\le 9 \mathrm{cm})\end{array}\\ \begin{array}{ll}7& (9 \mathrm{cm}<d\le 12 \mathrm{cm})\end{array}\\ \begin{array}{ll}8& (12 \mathrm{cm}<d\le 15 \mathrm{cm})\end{array}\\ \begin{array}{ll}9& (15 \mathrm{cm}<d\le 20 \mathrm{cm})\end{array}\\ \begin{array}{ll}10& (20 \mathrm{cm}<d\le 30 \mathrm{cm})\end{array}\end{array}\right.$$

(6)

where $\overline{D}$ is the transformed distance, and $d$ is the true distance.

In fuzzification, common membership functions include the Triangular membership function, the Trapezoidal membership function, the Gaussian membership function, the Sigmoid membership function, etc. [31]. The speed membership function was used here as an example to illustrate the setting up process. As the division of different speed fuzzy sets was symmetrical and uniformly distributed, the linear triangular membership function and linear trapezoidal membership function were utilized. The speed within the region was roughly divided into five parts, and the slope of the trapezoidal and triangular membership functions were established according to the requirement that the membership function of each speed point belonging to the fuzzy subset was greater than 0.5. The final established membership function for a uniformly symmetric distribution is shown in Figure 9, Figure 10 and Figure 11. The fuzzy sets of shortest distance in the form of linguistic rating variables were described as ‘Very close (NB)’ ‘Relatively close (NS)’, ‘Moderate (M)’, ‘Relatively far (PS)’, and ‘Far (PB)’. Similarly, as is shown in Figure 12, a set of fuzzy affiliation functions were created for the model output, which were represented as ‘Very dangerous (NB)’, ‘Relatively dangerous (NS)’, ‘Dangerous (M)’, ‘Relatively safe (PS)’, and ‘Safe (PB)’, respectively.

2.

Fuzzy rule and fuzzy inference

The basis of fuzzy inference is presented as output fuzzy variables, which are inferred from input fuzzy variables based on a set of fuzzy inference rules in linguistic terms [33], which are extracted from expert knowledge.

The fuzzy rule is expressed in the form of “If-Then”, and two components make up the “If-Then” rule: an antecedent that is compared to the input and a consequent. A fuzzy rule base contains a set of fuzzy “If-Then” rules, and these fuzzy “If-Then” rules are employed by fuzzy inference to define a mapping from input fuzzy sets to output fuzzy. In this paper, the fuzzy “If-Then” rules designed according to the expert experience were expressed as a language rule system with three inputs and one output:

R1: if “T” is NB and “V” is NB and “D” is PB, then Safety degree K is NB;

R2: if “T” is NB and “V” is NB and “D” is PS, then Safety degree K is NB;

……

Rn: if “T” is NB and “V” is PB and “D” is PB, then Safety degree K is PB;

The fuzzy reference rule tables, in the case when the delay time is NB, NS, PS, and PB, can be expressed in the forms of

Table 1. Fuzzy rule of collision risk in case of T is NB.

Speed	Distance
	PB	PS	M	NS	NB
NB	NB	NB	NB	NS	NS
NS	NB	NB	NS	M	M
M	NS	NS	M	PS	PS
PS	M	M	PS	PB	PB
PB	PB	PB	PB	PB	PB

,

Table 2. Fuzzy rule of collision risk in case of T is NS.

Speed	Distance
	PB	PS	M	NS	NB
NB	NB	NB	NB	NS	NS
NS	NB	NB	NS	M	M
M	NB	NS	M	M	PS
PS	NS	NS	PS	PB	PB
PB	PS	PB	PB	PB	PB

,

Table 3. Fuzzy rule of collision risk in case of T is PS.

Speed	Distance
	PB	PS	M	NS	NB
NB	NB	NB	NB	NB	NS
NS	NB	NB	NS	NS	M
M	NB	NS	M	M	PS
PS	NS	NS	M	PS	PS
PB	M	M	PS	PS	PB

and

Table 4. Fuzzy rule of collision risk in case of T is PB.

Speed	Distance
	PB	PS	M	NS	NB
NB	NB	NB	NB	NB	NB
NS	NB	NB	NS	NS	NS
M	NB	NB	NS	M	M
PS	NB	NS	M	M	PS
PB	NS	M	M	PS	PS

.

Mamdani [34] and Sugeno [35] are two frequently employed methods for fuzzy inference systems. Researchers rely heavily on Mamdani fuzzy inference method due to its practicality and efficiency. In light of this, the max-criterion Mamdani inference mechanism was implemented in this study for collision risk inference.

3.

Defuzzification

Defuzzification is used to the convert fuzzy outputs into clear outputs. There are several common defuzzification methods such as the center of the area (COA), the bisector of the area (BOA), the mean of the maximum(MOM), the smallest of the maximum (SOM) and the largest of the maximum (LOM) [25]. In the defuzzification layer, the output fuzzy numbers are converted to crisp numbers through the center of the area (COA) defuzzification method.

$$\mathrm{Center} \mathrm{of} \mathrm{area}: Z_{\mathrm{COA}}=\frac{{\displaystyle \underset{\mathrm{Z}}{\int }{\mu }_A(Z)Zdz}}{{\displaystyle \underset{Z}{\int }{\mu }_A(Z)dz}}$$

(7)

5. Teleoperation Assistant Control Strategy Based on Collision Risk Assessment

Virtual reality is a critical method to tackle the large time-delay problem [36]. The operator can control the robot directly in the virtual environment, yet safety and efficiency issues are still present. In teleoperation, the remote-end robot follows the trajectory of the local side. The robot will inevitably collide with the surrounding environment owing to factors such as planar visual errors and handshaking. Meanwhile, when approaching an obstacle (workpiece), the traditional teleoperation method applies a running step to observe and confirm a step for the sake of safety, thereby resulting in lower efficiency. To solve these problems, this paper proposed a teleoperation assistance strategy based on dynamic collision risk assessment to improve teleoperation safety and efficiency.

Based on the 3D modeling, this paper designed a graphical interface in which all the designed assistant operation methods were implemented. As is shown in Figure 13, the graphical user interface was segregated into four sub-modules: the virtual prediction sub-module, the local highlighting sub-module, the real-time image-returning sub-module, and the information-displaying and command-sending sub-module.

5.1. Graphical User Interface of Teleoperation

The top left corner of the interface is the predictive-simulation interface, which is built using the OSG based on remote environment information. In this virtual interface, the operator can control the virtual robot in real time through the wearable positioner. The “Status” sign in the lower right corner indicates whether the robot arm is malfunctioning: the green color indicates normal, and the red color indicates collision and other failures. The relative shortest distance between the robot and the surrounding obstacles can be obtained in real time during the movement by the proposed bounding box method. Moreover, the input parameters required for the dynamic collision risk assessment model, such as operation speed and delay time, can also be obtained in real time. The calculated collision risk assessment value is displayed in the information display column (lower left corner), which displays the angle information of each joint of the teleoperation robots at the same time. This column also contains several function buttons on the upper side, such as the start and end of the control program, the update of the simulation interface, etc. The video information of the remote environment acquired from the depth camera is displayed in the bottom right corner of the interface to facilitate the operator to grasp the real environment.

5.2. Assistant Strategy for Teleoperation

Based on the user interface, the teleoperation assisstant strategies include the following three aspects.

Strategy 1: Collision risk indication based on haptic feedback glove

Considerable studies have been conducted on virtual guidance force based on haptic feedback devices such as generating virtual guidance force based on artificial force field theory. This method synthesizes the integrated guiding force based on the repulsive force of the obstacle and the attractive force of the target so that the operator can intuitively feel the distance from the surrounding obstacles. This paper presented a haptic feedback strategy based on collision risk assessment. The risk assessment model not only includes the distance to the surrounding obstacles, but it also includes the operation speed and delay time. After completing the collision risk assessment of the current state, the haptic glove intuitively indicates the collision risk of the current operation to guide the operator to make reasonable decision. The feedback force exerted by the glove is proportional to the risk of collision; the higher the risk, the stronger the force. The glove used in this study is a force feedback glove developed by the laboratory [37], which can achieve a resolution of 0.02 N and a force feedback of up to 1.55 N.

Strategy 2: Variable motion mapping ratio based on collision risk assessment

Reducing the operation speed or lowering the motion mapping ratio of the master–slave is an effective means of improving operational safety. However, the continuous adjustment of the robot operating speed may also affect the task execution efficiency. Herein, the motion mapping ratio was adjusted to three levels according to whether the collision risk reached the set threshold. The motion mapping ratio of the master–slave end decreases as the risk index increases, thus improving the operating accuracy and reducing the collision risk. Risk thresholds and motion mapping ratios are obtained based on expert knowledge and experience.

The collision risk assessment index to motion mapping ratio can be expressed by the equation:

$$K=\left\{\begin{array}{l}{k}_1\begin{array}{ll}& \begin{array}{lll}& (s_d<s_{safe})& \end{array}\end{array}\\ k_2\begin{array}{llll}& & (s_{saf}{}_e\le s_d\le s_{danger})& \end{array}\\ k_3\begin{array}{llll}& & (s_d>s_{danger})& \end{array}\end{array}\right.$$

(8)

where $s_d$ denotes the collision risk value obtained by fuzzy theory.

The motion mapping ratio K of the master–slave is the ratio of the relative posture change of the master robot to the relative posture change of the slave robot. The relationship between the control position of the robot and the relative posture change of the operator is expressed by the following equation:

$$P_{robot\text{\_}i}=P_{robot\text{\_}i-1}+K{P}_{trac\ker }=P_{robot\text{\_}0}+K(P_{trac\ker \text{\_}i}-P_{trac\ker \text{\_}i-1})$$

(9)

where $P_{robot\text{\_}i}$ is the current control position, $P_{robot\text{\_}i-1}$ is the robot position at control period $i-1$, and $P_{trac\ker \text{\_}i}$ $\mathrm{and}$ $P_{trac\ker \text{\_}i-1}$ are the current and the last positioning position of tracker which is controlled by operator, respectively.

Strategy 3: Local highlighting display

This strategy is a further optimization based on the designed Strategy 2. In Figure 13, the top right corner of the user interface is the local highlighting sub-module whose function is to highlight the areas of high collision risk in order to assist the operator in observing the virtual environment more carefully. Objects with the highest collision risk are shown in red. The highlighted magnification of the view is aligned with the motion mapping ratio in Strategy 2 to improve coordination of the operation. The current magnification is shown as the value in the upper right corner.

6. Experiment and Discussion

A prototype of the system was developed. The correctness of the collision risk assessment model and the effectiveness of the operation assist strategies were verified by experiments. Experiments were carried out based on a physical environment that was composed of a 6 DoF UR5 robot arm, a Robotiq force sensor, a Robotiq adaptive parallel gripper, a Realsense camera, and a worktable. All of the experiments were performed on an AMD Core R5-4600 CPU and Windows 10 system.

6.1. System Evaluation

In this section, the accuracy of the designed localization method was measured. In addition, the setting of the discrete expandable bounding box method was illustrated and its effectiveness in distance measurement was verified.

• Localization accuracy

The UR5 robot arm can provide high-precision position information, and the repeated positioning precision in the x, y, z direction was ±0.02 mm, ±0.02 mm, and ±0.05 mm, respectively. Therefore, the accuracy of the designed positioning method can be evaluated by using the localization information of the robot in different positions. As is shown in Figure 14a, the markers were fixed to the end of the robot. The end of the robot was guided to 10 different points using a point-to-point motion form (Figure 14b). The position of the target marker can be obtained by the visual algorithm. The error between the position obtained by the localization method and the information read by the robot represents the accuracy of the localization algorithm.

As is shown in Figure 15, the positioning error along X, Y, and Z-axis did not exceed 0.5 mm, and the angular error around X, Y, and Z-axis was less than 1° in the whole working space. The positioning error was no more than 1 mm and 2°. Therefore, this method has high accuracy and good stability. In the future, the marker can be printed together with the workpiece through 3D printing.

2.

Validation of the discrete expandable bounding box method

This paper proposed a new method for calculating the nearest relative distance based on the discrete expandable bounding box, where the shortest distance can be obtained by pre-setting the bounding boxes with different radius. Empirically, the bounding boxes are divided into 15 layers, and the radius of each layer is $R_n=R_0+d_n$ and $d_n$ = [1 cm, 2 cm, 3 cm, 4 cm, 5 cm, 6 cm, 7 cm, 8 cm, 9 cm, 10 cm, 12 cm, 15 cm, 20 cm, 25 cm], respectively. $R_0$ is the minimum radius that can just achieve the robot arm enclosing. The smallest added radius of the bounding box ($d_n$) that is contacted with the surrounding obstacles represents the shortest distance. The design of the radius revealed that the radius was unevenly distributed, which is reasonable, because more precise information is needed when approaching obstacles, for example, the spacing of $R_1$, $R_2$, and $R_3$ was only 1 cm. When the distance was about 20–30 cm, the safety was high and no more effort was needed to focus on the actual distance. A screenshot of the shortest distance acquisition through this method is shown in Figure 16. As is shown in the right of the figure, the robot arm link was enclosed by multiple layers bounding box. The D in the upper right corner represents the shortest distance calculated theoretically, and the Dr represents the shortest distance calculated by the discrete expandable bounding box. Therefore, the error did not exceed $d_n/2$.

6.2. Validation of the Collision Risk Assessment Model

To validate the correctness of the proposed collision risk assessment model, three scenarios were designed in the virtual environment. A different assessment factor was kept for each scenario to verify the correct relationship between the collision risk and the factors. To illustrate the problem intuitively, one obstacle was placed. The coordinate value of the obstacle was (0 mm, 600 mm, 100 mm), and the radius was 50 mm. In each scenario, the robot moves from the starting point A to the end position B according to the red line shown in Figure 17 with the robot’s end-effector posture remaining constant, and the collision risk value of each scenario is recorded.

Scenario A. This scenario verified the positive correlation between the collision risk and the nearest distance. The robot moves to the terminal position B from starting point A with a fixed speed (v = 40 mm/s) and a delay time (T = 2 s). Figure 18a depicts the results.

Scenario B. In this scenario, the robot moves to the same terminal position from the same starting point with different terminal speed; the velocity of each movement is 10 mm/s, 20 mm/s, and 40 mm/s, respectively. The delay time is all set at 2 s. The results are shown in Figure 18b.

Scenario C. In this scenario, the delay time of each robot movement was set to 1 s, 2 s, 3 s, and 4 s, respectively, and the speed of each movement was 30 mm/s. The results can be observed in Figure 18c.

As can be seen in Figure 18a, the collision risk increased with the decrease in the shortest distance for the same speed and delay time, with a more pronounced trend as the robot approached the obstacle. Moreover, the risk value did not increase linearly, because the results combined the expert experience of risk assessment in different scenarios, which is also the advantage of this method. In addition, when the robot was at the same starting position, the collision risk value of the robot with an end speed magnitude of 40 mm/s was 0.6206, which is lower than the calculated value of 1.8438 for 10 mm/s. According to the fuzzy definition of collision risk and membership function (Figure 12) the smaller the value, the higher the collision risk. Therefore, the result shows that the risk increased with the increase in the speed magnitude. It can be proven that the collision risk assessment model is not only related to the shortest distance, but it is also related to the operation speed. Figure 10c illustrates that the delay time has a similar effect on collision risk as speed.

6.3. Effectiveness of the Operation Assist Strategy

6.3.1. Experiment Setup

For evaluating the proposed operation assistance strategy, a real teleoperation system was setup in the laboratory. The graphic virtual environment was constructed based on the remote environment, and the system setup is shown in Figure 19. Five obstacles were placed on the worktable, and the ArUco marker was fixed on the workpiece.

6.3.2. Task and User Study

One robot collisionless movement task was designed to evaluate the teleoperation assistance method. The task is described as follows:

The operator manipulated the virtual robot through the wearable localization devices, and the operation assistance strategies were generated in real time. As is shown in Figure 20, the start area was labeled as point A, and five different obstacles were labeled as O1–O5. The distances between obstacles were 20 cm, 16 cm, 13 cm, and 11 cm, respectively. The operator was asked to move the robot to the target area while avoiding obstacles. The delay of teleoperation was simulated by data delay transmission, and the delay time was a random number with a mean value of 2. During this task, there were no assistance strategies but only visualization in Environment I, while the designed strategies were provided in Environment II.

As observed from Figure 21a, during the operation, it was difficult to intuitively determine whether the robot arm had collided with surrounding obstacles due to visual errors and other factors; therefore, the operator had to spend a lot of time to switch perspectives to carefully observe whether a collision had occurred. In the same case, with the assistance strategies, as is shown in Figure 21b, by the collision risk warning through the haptic (strategy 1) and by highlighting the local danger area (strategy 3), the operator could clearly identify which obstacle was closer to the robot arm and required our attention. Some risks that were ignored due to visual errors were warned.

The task was tested 10 times. The number of collisions during each test was recorded as well as the time spent on reaching the target. The results of 10 parallel experiments verified the effectiveness of the proposed method, as is shown in Figure 22. The average number of collisions with the assistant strategy was significantly lower than the without strategies by 90%. Moreover, as is shown in Figure 23, the average time required to reach point B in Environment I and Environment II was 45.2 s and 33.6 s, respectively, which was reduced significantly by 36%. Therefore, it is obvious that the designed operation-assisted strategies can significantly reduce the possibility of collisions and enhance efficiency.

Due to the fact that the designed three assistant strategies are focused on identifying operation collision risk and improving the observation of risk points (strategy 1 and 3), the ability to avoid collisions was insufficient. Therefore, the performance of the teleoperation was still influenced by the operator, and collisions could still occur due to human physiological limitations.

7. Conclusions

This paper proposed a new collision risk assessment model and designed a collision-risk-based operational assistance strategy for teleoperation. The model took into account the effects of relative shortest distance, operation speed, and delay. In addition, a discrete expandable bounding box method was proposed in this paper to make the shortest distance calculation possible. Meanwhile, this paper introduced fuzzy theory into the collision risk assessment model, which can help decision makers in extracting and keeping expert knowledge. Moreover, based on the assessment of the collision risk, three assistant strategies, including partial view highlighting, variation in motion mapping ration, and haptic risk warning, were defined in this paper. The collision risk model and the teleoperation assistance strategies were evaluated experimentally. The experimental results indicate that the discrete expandable bounding box method designed in this paper could accurately obtain the shortest distance, and the proposed fuzzy collision risk assessment model had the correct relationship with the factors to accurately quantify the collision situation. Simultaneously, the designed operation assistance strategy could effectively reduce the risk of robot collision and improve the safety and efficiency of the teleoperation robot for in-orbit assembly.

The designed assistant strategies in this paper were focused on identifying operational risk and improving the observation of risk points, while the ability to avoid collisions was insufficient. Consequently, in the future, we will continue to improve our teleoperation system, including the study of autonomous obstacle avoidance methods, virtual modeling techniques in non-structural environments to adapt to the complex and changing space environment. Additionally, we will conduct assembly tests in microgravity to explore the effects of microgravity on assembly.