Improving Teleoperator Efficiency Using Position–Rate Hybrid Controllers and Task Decomposition

Abstract

In hazardous environments where direct human operation of machinery is not possible, such as in a nuclear power plant (NPP), teleoperation may be utilized to complete tasks safely. However, because teleoperation tasks are time consuming, complex, highly difficult, and need to be performed with incomplete information, they may increase the human operator’s cognitive load, which can affect the efficiency of the human operator as well as the completeness of the task. In this study, we propose a teleoperation system using a hybrid teleoperation controller that can increase operator efficiency for specific teleoperation tasks in complex sequences. First, we decomposed the task into a sequence of unit subtasks. For each subtask, the input space was allocated, and either position control or rate control by the hybrid controller was determined. Teleoperation experiments were conducted to verify the controller. To evaluate the efficiency improvement of the teleoperators, the completion time and NASA task-load index (NASA-TLX) were measured. Using the hybrid controller reduced the completion time and the NASA-TLX score by 17.23% and 34.02%, respectively, compared to the conventional position controller.

1. Introduction

Teleoperation is typically used in hazardous or unstructured environments where direct human operations or automatic controls are not available, such as nuclear power plants (NPPs). Therefore, many studies have attempted to use teleoperation in NPPs. Some teleoperation applications in NPP for power plant dismantling [1,2,3,4], object handling [5,6,7], pipe cutting [8], and rescue [9] have been studied. In general, teleoperation is performed by viewing only the image screens that appear on the monitor. As a result, this prevents the operator from accurately determining the distance or direction of the target object because the operator only sees the image of the target object displayed on the monitor.

Many techniques for performing sophisticated teleoperation tasks to overcome this problem are being studied. Force-rendering techniques are widely used methods in teleoperation to assist operators in controlling haptic devices. In [10], a method was developed to formulate the desired behavior in any number of task frames associated with the task to optimize motion control in a limited workspace. Furthermore, additional linear constraints could have been applied to each task frame.

In [11], the effects of force feedback on a blunt dissection task were examined. It was argued that force feedback is useful in this context when there is a large contrast in the mechanical properties along the dissection plane between adjacent areas of the tissue. A parallel robot-aided minimum surgical system with a force-feedback function was employed to study the practical force-feedback effects of artificial tissue samples [12]. The force feedback was provided using pre-programmed rules and provided to the user through the joystick to prevent the robot from falling into an unstable state or coming in contact with an obstacle [13]. In [14,15], interactive virtual fixture generation with shared control was proposed to harness the cognitive abilities of teleoperators.

A retargeting function was used to deal with complex kinematic constraints owing to the difference in topology, anthropometry, joint limits, and degrees of freedom between the human demonstrator and the robot [16,17,18]. In [19], a Cartesian-to-Cartesian mapping retargeting function using a data glove and wrist-tracking sensors to control the slave robot was proposed.

Overcoming the conspicuous workspace disparity between the master device and the slave robots is also a major problem in teleoperation. To solve this problem, some researchers have implemented hybrid control for teleoperation [20,21,22,23,24]. In [20], a control method for teleoperation by integrating hybrid position and velocity control with an interoperability protocol was proposed. The proposed method can overcome the difference in the workspace between the master device and the slave robot. Most of the hybrid position–velocity controllers are switched on manually with a specific button at the operator site [20,21].

The bubble technique, which is based on hybrid position–rate control was also implemented to overcome the disparity in workspace between the master and slave [25,26,27]. The control method applied depends on whether the haptic device is inside or outside of the bubble. When the haptic device is inside the bubble, position control is used for fine positioning, and when it is outside the bubble, rate control is applied for coarse positioning. The motion-scaling technique [27,28] is another well-known technique for dealing with the disparity between the master and slave workspaces. A motion-scaling factor is handled with a scale equal to the ratio of the maximum workspace of the haptic device workspace to the largest workspace of the virtual environment.

Another issue in teleoperation is its difficulty, as it is necessary to simultaneously manipulate the slave robot’s multiple degrees of freedom while viewing only two-dimensional (2D) camera images that do not give an accurate perspective of the machinery installed at a remote site. Neuroscience studies have shown that when a human operates an object in three-dimensional (3D) space and consciously adjusts the two degrees of freedom to three degrees of freedom, the remaining degrees of freedom are driven by unconscious subordinate commands, which can be acquired through human experience or training [29].

The teleoperation task that we deal with is called MADMOFEM (operating a manual drive mechanism of a fuel exchange machine) and is covered in Section 2. This task requires a complex sequence of clutch and driver operations performed with a dual-arm manipulator. Prolonged telemanipulation can cause heavy workload, and interpretation of the 2D camera images of the remote site can increase the cognitive load experienced by the operator. Therefore, we focus on increasing the efficiency of the operator by reducing the work time and the level of difficulty of a task. To do this, we first divided our objective task into specific subtasks and applied a position–rate hybrid controller that adjusts the input space and control mode of the slave robot according to the current subtask index.

The remainder of this paper is organized as follows. In Section 2, our objective dual-arm teleoperation task, called MADMOFEM, is described and demonstrates the sition–rat-rate hybrid controller that adjusts the input space and control mode of the slave according to the corresponding subtask index. The experimental setup and scenarios are explained in Section 3. Section 4 presents the experimental results, and Section 5 presents the conclusions.

2. Control Method

This chapter describes the operation of a dual arm manipulator called MADMOFEM. The task is to perform clutch and shaft work, which must be done in a series of sequences. In addition, both agile and fine movements of the robot are required. To solve this problem, the control method can be changed according to the subtask, and the difficulty of the operation can be lowered through appropriate constraints for the degrees of freedom of the master device.

2.1. Madmofem Task

In a fuel-handling machine—one of the main components of a pressurized heavy water reactor (PHWR) nuclear power plant (NPP) age—the machinery can become stuck in the pressure tube in front of the PHWR calandria. Although the machine has a means of troubleshooting the manual drive mechanism, accessing the manual drive mechanism remains a difficult problem. If the machine is stuck, the NPP becomes radioactive, and the machine can be located at heights of up to 9 m. Therefore, Shin developed a manual drive mechanism for a fuel exchange machine (MADMOFEM) to handle the machine remotely in an NPP that is not accessible to humans [30].

Owing to the complexity of the MADMOFEM task and considering the limited cognitive abilities of the human operator, a control scheme is required. There are two objective operations: driver and shaft operations. The operations contain a certain sequence of manipulation subtasks. The sequence starts with the insertion of the tool into the clutch of the right robot. Then, the shaft operation with the left robot is executed. After the shaft operation is completed, the tool inserted into the right robot is removed from the clutch as shown in Figure 1.

In Figure 1, the right part starting with “C” represents the task of inserting the tool into the clutch with the right slave manipulator, and the left part starting with “S” symbolizes the insertion of the tool into the shaft with the left slave manipulator. The items below are explained for each subtask. The MADMOFEM task is a complex task that requires sequential operation of the clutch and shaft.

• C1: Align the rotation axis of the right robot and the clutch.
• C2: Align the x- and y-axes of the right robot and the clutch.
• C3: Move the right robot in the z-axis direction to approach the clutch.
• C4: Insert the tool into the clutch through fine manipulation of the right robot.
• C5: Standby.
• S1: Align the x- and y-axes of the left robot and the shaft.
• S2: Move the left robot in the z-axis direction to approach the shaft.
• S3: Insert the tool into the shaft through fine manipulation of the left robot.
• S4: Remove the tool from the shaft with the left robot.
• S5: Standby.
• C6: Remove the tool from the clutch with the right robot.
• C7: Task finished.

During teleoperation, the slave robot is generally controlled by position or rate control. In the case of position control, a scaling factor is usually incorporated to overcome the different workspaces between the master and slave. A low scaling factor has advantages in areas that require precision control with high-resolution movement, such as inserting a peg into a hole, and has a considerable amount of indexing to approach the target. On the other hand, with a high scaling factor, the slave can approach the target at a high speed; however, low-resolution movement might occur in dangerous situations. When rate control is used, it is possible to approach the target proximity area conveniently; however, on the other hand, unlike position control, it has poor transparency that can cause risky situations near the target.

2.2. Position Control Mode

Using position control, the motion of the slave robot corresponds to the motion of the master device with superior transparency. Depending on the scaling factor, a trade-off between the safety of the operation and overcoming the workspace disparity occurs. Therefore, the scaling factor should be determined. In our system, because the task environment is at a height of 9 m, a relatively low scaling factor was determined in consideration of safety. The displacement moved by the master became the input of the slave (1).

$${\dot{x}}_{(m,p)}=\frac{K_p(x_{(m,t)}-x_{(m,t+\Delta t)})}{\Delta t}.$$

(1)

${\dot{x}}_{(m,p)}\in R^{(6\times 1)}$ represents when the master velocity when the position control mode is used, $x_{(m,t)}\in R^{(6\times 1)}$ is the position of the master at time t, and $K_p\in R^{(6\times 6)}$ is used as the scaling matrix. The velocity of the master is obtained as the displacement of the master during $\Delta t$.

2.3. Rate-Control Mode

When the slave is operated in the rate-control mode, significant stability and transparency issues are experienced when contact is made with the environment; however, rate control was proven to be suitable for the slave when it is operated in free motion [24]. In our system, therefore, rate control is applied in the initial condition where the slave was far from the target. The master velocity in the rate-control mode is a vector of magnitude away from the master’s origin position (2). The master’s origin position is set when the button on the master device is initially pressed.

$${\dot{x}}_{(m,r)}=K_r(x_{(m,t_0)}-x_{(m,t_0+\Delta t)}).$$

(2)

${\dot{x}}_{(m,r)}\in R^{(6\times 1)}$ indicates the value input to the slave when the rate-control mode is applied, $x_{(m,t_0)}\in R^{(6\times 1)}$ is the position of master when the gripper of the master is pressed for the first time, and $K_r\in R^{(6\times 6)}$ is used as the scaling matrix. The velocity of the master was determined by the displacement from the origin of the master. The farther away from the origin, the higher the velocity obtained as the output of the master. Our master system set the origin as the point at which the button on the master is initially pressed.

2.4. Position-Rate Hybrid Teleoperation Control

Figure 2 is a schematic of the proposed position–rate hybrid teleoperation control. The subscripts m, s, c, and d in Figure 2 represents the master device, slave robot, current, and desired, respectively. The position-based process recognizer outputs the corresponding subtask index based on the distance between the slave robot position and the desired position. With this result, the input space controller determines the input space ($\alpha$), and the control-mode selector determines the control mode ($\beta$) between the position and rate control.

$${\dot{x}}_m=diag\left(\alpha \right)(\beta {\dot{x}}_{(m,p)}+(1-\beta ){\dot{x}}_{(m,r)}),$$

(3)

$${\dot{q}}_s=J^{*}\left({\dot{x}}_m\right).$$

(4)

${\dot{x}}_{(m,p)}\in R^{(6\times 1)}$ represents the position controller, and ${\dot{x}}_{(m,r)}\in R^{(6\times 1)}$ represents the rate controller. ${\dot{x}}_m\in R^{(6\times 1)}$ is determined by the scalar value of $\beta$ in the tables below to apply the corresponding mode between the position and rate control for each subtask. $J^{*}\in R^{(7\times 6)}$ is the inverse of the Jacobian, and ${\dot{q}}_{(s,d)}\in R^{(7\times 1)}$ is the desired joint velocity. Our teleoperation system assigned position and rate-control modes based on the corresponding subtask index using a predetermined $\alpha$.

We selected an advantageous control method from the corresponding subtask index among the two control methods to complete the MADMOFEM tasks. For example, because of the distance between the target position ( the position of the shaft or driver) and the initial position, we used rate control to approach the target quickly and switched to position control when the target was close, allowing the task to be completed accurately and safely.

Predetermined input space vector $\alpha$ and control mode $\beta$ are listed in

Table 1. The predefined parameters related to the left robot.

	${\mathit{\alpha }}_{\mathit{r},\mathit{x}}$	${\mathit{\alpha }}_{\mathit{r},\mathit{y}}$	${\mathit{\alpha }}_{\mathit{r},\mathit{z}}$	${\mathit{\alpha }}_{\mathit{r},\mathit{roll}}$	${\mathit{\alpha }}_{\mathit{r},\mathit{pitch}}$	${\mathit{\alpha }}_{\mathit{r},\mathit{yaw}}$	${\mathit{\beta }}_{\mathit{r}}$
C1	0	0	0	0	0	1	1
C2	1	1	0	0	0	0	0
C3	0	0	1	0	0	0	0
C4	1	1	1	0	0	0	1
C5	0	0	0	0	0	0	1
C6	0	0	1	0	0	0	0
C7	0	0	0	0	0	0	1

and

Table 2. The predefined parameters related to the right robot.

	${\mathit{\alpha }}_{\mathit{l},\mathit{x}}$	${\mathit{\alpha }}_{\mathit{l},\mathit{y}}$	${\mathit{\alpha }}_{\mathit{l},\mathit{z}}$	${\mathit{\alpha }}_{\mathit{l},\mathit{roll}}$	${\mathit{\alpha }}_{\mathit{l},\mathit{pitch}}$	${\mathit{\alpha }}_{\mathit{l},\mathit{yaw}}$	${\mathit{\beta }}_{\mathit{l}}$
S1	1	1	0	0	0	0	0
S2	0	0	1	0	0	0	0
S3	1	1	1	0	0	0	1
S4	0	0	1	0	0	0	0
S5	0	0	0	0	0	0	1

. With the $\alpha$ vector, the input space for the slave robot is determined and is also used for haptic rendering. If the corresponding value is zero, then the input space is ignored; otherwise, the input space is allowed. For example, in the C1 state, the operator must handle the master device in the yaw direction. Therefore, only the value of ${\alpha }_{yaw}$ is set to 1.

When the value of the Euclidean distance between the robot and the target in the direction where the input space is allowed is below the threshold, the state is converted to the next state. For example, because the input space in the yaw direction is only allowed in the C1 state when the yaw component of the target and the robot are below the threshold, it is converted to the next state, which is the C2 state. In the C2 state, the ${\alpha }_x$ and ${\alpha }_y$ components are equal to 1; therefore, when the Euclidean distance in the corresponding direction is below the threshold, the state switches to the C3 state, and proceeds to switch repeatedly until the final state is achieved.

As the corresponding state is determined based on the slave and target positions, the algorithm has the disadvantage of not being available in an unstructured environment because it assumes that we know the location of the target. Therefore, in future work, we plan to improve the state conversion automatically by using vision-based deep learning that takes the egocentric view mounted on the end-effector as an input.

2.5. Minimizing the Operating Degree of Freedom

The haptic rendering technique was applied to the master device so that the operator could intuitively recognize the minimized input space. We constrained the freedom of the master device with each subtask. If the subtask is part of the process that aligns the x- and y-axes, a reaction force is generated by offsets in both directions of the z-axis, thus, constraining the master device to move only on the x–y plane (Figure 3). If the subtask is part of the process of inserting the tool into the z-axis, a reaction force is provided, which results in the manipulator being unable to move in the x- and y-axes so that the master device is constrained to move only along the z-axis.

3. Experiments

3.1. Experimental Setup

Two Force Dimension’s Omega7 devices (https://www.forcedimension.com/, accessed on 29 August 2022) and two KUKA lbr iiwa r820 manipulators (https://www.kuka.com/, accessed on 29 August 2022) were used as the master device and slave robot, respectively. The master PC acted as a TCP/IP server and performed the calculations for the master’s operation velocity and haptic rendering. The slave PC included a position-rate hybrid controller and a process recognizer.

The process recognizer output the corresponding subtask index for each slave robot and sent the results back to the master PC. All communication between PCs was based on the TCP/IP protocol(Figure 4). A total of seven healthy subjects participated in the teleoperation experiments. As shown in Figure 5, the subjects manipulated two slave manipulators (left arm and right arm) that were approximately 9 m high with two master devices (left-handed and right-handed) by viewing only the hand-eye camera images (left side in Figure 6) and the global camera view (right side in Figure 6). Sufficient practice time was given for the subjects to become accustomed to teleoperation.

3.2. Scenario

In the scenario, the objective was to remotely operate the MADMOFEM in a nuclear power plant with two-armed robots (Figure 6). The robot system consists of a mobile robot at the bottom, a mast capable of moving up and down to a height of 9 m in the middle, and two slave robots at the top. The experiment is assumed to begin after the mast reaches a height of 9 m. The sequence of the tasks is shown in Figure 1.

The tasks of minimizing the input space and constraining the degrees of freedom of the masters are automatically implemented by subtasks. In the C2 state, for example, a subtask aligns the x-axis and y-axis of the right robot and the clutch. Therefore, inputs other than the x and y directions were removed, and rate control was used. The haptic device generates reaction forces to be manipulated in the x–y plane through haptic rendering (Figure 3).

In the C3 state, in which the tool is inserted into the clutch in the z-axis direction by the right robot, inputs other than the z-axis direction are ignored, and rate control is used as in the C2 state. Haptic rendering was used to manipulate the slave in only the z-axis direction. In the C5 state, because the right manipulator is close to the clutch, the control mode is converted to position control, allowing precise insertion of the tool into the clutch. In this way, the subjects performed the experiment according to the sequence shown in Figure 1.

4. Results

To confirm the efficiency improvement of the operator, the distance operated by the master, task completion time, and fatigue level were measured using a NASA-TLX questionnaire.

4.1. Manipulated Distance

When the operator manipulated in the position control mode, the operator needed to index repeatedly to approach the goal position. In contrast, in the rate-control mode, the operator needed a lower amount of indexing to approach the desired position. For this reason, the operator manipulated the master device less when a hybrid controller was used. The manipulated distance was reduced by 22.69% (** p < 0.01) from 1226.2 to 948.02 mm as shown in Figure 7. This result indicates that, when the hybrid controller is applied, the workspace mismatch between the master and slave can be overcome by applying the rate control at a distance far from the target and the position control at a distance near the target.

4.2. Completion Time

The completion times were 212.78 and 257.08 s using the hybrid control and not using the hybrid control, respectively (Figure 8). Thus, the completion time decreased by 17.23% (** p < 0.01) when the hybrid control was used. This means that, when the slave is far away from the target, the slave can be quickly manipulated to approach the target with rate control.

4.3. NASA-TLX

The NASA-TLX was measured to evaluate the workload of the subjects during the teleoperation experiments. Subjects scored between 0 and 100 for six questionnaire measurements (mental demand, physical demand, temporal demand, performance, effort, and frustration). The subjects were evaluated after each experiment. As shown in Figure 9, the average NASA-TLX reduced from 56.17 to 37.06. In other words, with the hybrid control, the average NASA-TLX score was 34.02% (*** p < 0.001) lower than that achieved with conventional position control. Furthermore, when the hybrid control was used, we confirmed that each measurement score was lower than for the case where position control was used (Figure 10). In other words, minimizing the input space and constraining the degrees of freedom of the master with haptic rendering had a significant influence on the NASA-TLX score.

5. Conclusions and Future Work

We introduced a position–rate hybrid control that can increase the efficiency with which an operator completes a specific dual-arm teleoperation task called MADMOFEM, which is time-consuming, complex, and highly difficult. Furthermore, the operator is required to complete the task with incomplete information. To decrease the cognitive load and a heavy workload, we first decomposed the task into a sequence of subtasks. The process recognizer, based on the relative distance between the slave and the target, recognizes which subtask index each arm currently corresponds to.

Depending on the subtask index, the degrees of freedom for human manipulation are minimized by constraining the input space using the input-space controller, and either the position or rate control is selected by the control-mode selector. Experiments were conducted to compare general position control and hybrid controllers, including freedom constraints of master devices, to verify to what extent the proposed system increased the operator’s efficiency. By allocating a control mode method suitable for the situation due to the hybrid controller, the workload was reduced by reducing the working time and the distance the master moved. Furthermore, we confirmed that the cognitive load of the operator was reduced through constraining the degrees of freedom of the master device suitable for the subtask.

In our current teleoperation system, the process recognizer outputs the result of the subtask index based on the position of the slave and the target. If the mobile robot at the bottom of the system moves or the position of the target changes, the relative position of the target changes; as a result, the position of the target needs to be changed. Therefore, in future work, a robust process recognizer that uses a deep-learning model whose inputs are camera images taken at a remote site will be developed so that the recognizer can be used even when changes occur in an unstructured environment.