Visual Servo Control of the Macro/Micro Manipulator with Base Vibration Suppression and Backlash Compensation

Abstract

This study investigates the visual servo control of the space station macro/micro manipulator system. The proposed approach is based on the position-based eye-in-hand visual servo (PBVS) and takes advantage of the hardware sensors to overcome the macro manipulator’s base flexibility and joint backlash. First, a vibration suppression approach based on the reaction force feedback control is proposed, the deflection forces are measured by the six-axis force/torque sensor at the base of the micro-manipulator, and damping is injected into the flexible base in the closed-loop control to suppress the base vibration. Second, the small changes of joint backlash are compensated based on the macro manipulator joint angles sensor and converted to the desired motion of the payloads. Finally, PBVS with the lag correction is proposed, which is adequate for the precise positioning of large payloads with significant low-frequency oscillations. Ground micro-gravity experiment implementation is discussed, simulations and experiments are carried out based on the equivalent 3-DOF flexible base manipulator system and the macro/micro manipulator ground facilities, and results demonstrate the effectiveness of the proposed control algorithm.

1. Introduction

Manipulator is the most widely used automation equipment in the robot and has been widely applied to industrial production, surgical operation, humanoid robot, underwater search and rescue, and space stations. The space station manipulators perform a number of tasks on orbit, including cargo handling, equipment installation and maintenance, space vehicle rendezvous and assisted docking, and support for astronauts in extravehicular activities (EVA) [1].

The China Space Station will be built gradually, the core module arm (CMM, 10 m length) and experimental module arm (EMM, 5 m length) are currently in orbital service [2]. For some working conditions, the manipulator not only requires an extensive range of motion but also needs to ensure the accuracy of the local operation, a relative rigid robot (EMM) mounted serially on the tip of a long, flexible robot (CMM) is often used to increase reach capability, shown in Figure 1. It is a well-known macro/micro manipulator system (MMMS), which is a kind of flexible base manipulator system (FBMS), and is often used to perform tasks that are dangerous or may be incapable for astronauts [3]. Similar applications were in the International Space Station (ISS), such as the SSRMS-SPDM provided by CSA and JEMRMS provided by JAEA [4,5].

Flexibility and backlash are two crucial factors concerning the performance of FBMS. The large scale of CMM makes the manipulator flexible, and the planetary gear reducer of CMM causes backlash and extra flexibility [6]. The dynamic coupling between the manipulator and flexible base complicates the control problem. The reaction forces and moments caused by the motion of EMM will excite the joints sliding within the backlash, thus leading to offsets at the end-effector EEF. As deflection forces increase, the flexible modes excite vibration, leading to EEF tracking errors and performance deterioration.

The tip position sensing cannot be obtained from the forward kinematics due to the imperceptible nature of flexural link displacements. One feasible solution to improve the positioning accuracy is the usage of a camera mounted at the EEF of the manipulator, which is a kind of eye-in-hand (end-point closed loop) servo system [7]. Visual servo (VS) is classified into a position-based visual servo (PBVS) and image-based visual servo (IBVS) [8]. In IBVS, the visual information is direct feedback for control to minimize the image plane error by extracting image features; in PBVS, the relative pos between the end-effector and the target is estimated, which can be controlled in the unified architecture in Cartesian space. PBVS is designed in our system, and markers are configured on the grapple fixtures outside the space station [9], the obtained full pose information (6D) helps to assist the maneuvring of the operators.

The basic PBVS control structures have been discussed by Wilson in [10,11]. For better load capacities and high precision of the EEF, the flexible states sensing and vibration control are introduced, making the control law much more complex. Various strategies have been developed to cope with a flexible base. The trajectory planning based on the input shaping [12,13] is feedforward control, which is insufficient for the system with an initial excitation. To overcome this problem, the feedback controller combined with input shaping is proposed by Cannon in [14]. However, it depends on the joint space tracking and is not suitable for eye-in-hand VS. The passive analysis with large payload [15,16] has been addressed and PD+ control is applied with the desired rigid system feedforward and tracking errors feedback. Similar PID control with a feedforward term is applied in the precise positioning of payloads by flexible space robot systems [17], however, it is an eye-to-hand IBVS.

Most commonly, the vibration damping term introduced by the micro manipulator is a feedback. Wayne J. Book studied many different ways to damp the vibration, such as active damping control in [18] and inertial vibration damping control in [19], which is usually combined with the two-time scale control in [20]. It is assumed that the tracking control and base vibrations are separated in the frequency domain, and the dynamic model is decoupled into two-time scale based on the singular perturbation theory. This concept has been widely applied in the IBVS control [21,22,23,24,25,26], where the slow control torque vanishes the image feature error and fast control torque sufficiently dampen the vibrations of the elastic mechanical structure. However, the tracking errors affected by the base vibration damping and the controller may become deficient due to modeling errors.

The base vibrations should be suppressed during the visual servo task of EEF; it is a kind of dual-task control. Torres et al. [27] firstly propose the pseudo-passive energy dissipation, the idea is to choose the PD gains in the joint controllers to transfer the energy from base to manipulator actuators. For the redundant manipulator, the reaction null-space control framework proposed by Nenchev et al. in [28] is based on the coupling momentum. In an alternative way, the resolved solution based on the null-space of the redundant manipulator is studied by the author in [29], the damping term is added in the closed-loop by the successive null space projection. Recently, augmented projections based on coordinate transformation is proposed by Dongjun Lee in [30]. Different coordinate transformation methods are proposed by Gianluca Garofalo et al. in [31,32], where the dynamic consistency is satisfied with the decoupling of joint space and operation space.

The macro/micro manipulator is difficult to model. The joint of the EMM is flexible with backlash, and the joint stiffness in and out of the backlash differ. The dynamic model-based control, as mentioned before, is inadequate for this situation. However, the sensors of the manipulators will make it possible for precise deformation measurement. The six-axis force/torque sensor (FTS) is designed in [33], which can measure the three orthogonal forces and torques between EMM and CMM. The base force/torque sensing are discussed for different applications [34,35,36], but with a lack of macro/micro manipulator vibration suppression control.

The contribution of this paper is to deal with the vibration problem of the MMMS with PBVS, the requirements of the control strategy are related to the robot’s high speed and high precision operations with large payload. Firstly, the PBVS is performed effectively in cooperation with eye-in-hand camera and other types of sensors due to the multi-sensor data fusion. Secondly, the vibration control of the FBMS is studied by the FTS damping control and backlash compensation.

This paper is organized as follows. Section 2 discusses the measurement of the manipulator payload motion. In Section 3, an implementation of a visual servo is given, including base vibration and backlash compensation. In Section 4, simulation results are discussed for the macro-micro manipulator systems. In Section 5, experimental implementation and evaluation of a laboratory’s flexible base manipulator system is discussed. The paper ends with concluding remarks in Section 6.

2. Vision-Based Measurement System

2.1. EMM Vision System

The EMM is a redundant seven-degree-of-freedom manipulator symmetrical about the elbow joint [37], with an identical EEF at each end, one of which acts as a base and can be connected to the module alone or to the CMM in a macro-micro way, while the other end is used to operate the payload. Each EEF is equipped with a vision system (short for EVS) primarily used to measure the attitude of the cooperative target and provide live image data. The EVS includes three separate optical systems called hand camera (HC), load-operated camera (LC), and vertical camera (VC). HC is located at the bottom for VS when the EFF is unloaded; LC is located at the top for the VS of small or medium-sized payloads when the hand camera is obscured; VC is mounted in a direction orthogonal to the other two cameras for VS of lateral payloads on the exposed platform. In addition, when the manipulator operates large payloads, such as pallets, a payload camera (PC) is mounted at the front of the pallet, as shown in Figure 2. The LC and VC use the same visual marker (VM), the others have different visual markers for the corresponding camera due to the working distance.

The architecture of the EVS is shown in Figure 3 with redundant backup for each unit. The HC is the interface and control centre of the vision system, the external camera image data interface with PC and compressed image data interface with the manipulator controller are implemented here. Each of the cameras has an independent lighting system but they share one processing and compression unit. The imaging of HC and LC share the same image processing unit. The VC and PC have the independent imaging units which are located in their own camera. The image processing item in Figure 3 is to perform a measure function to recognize the marker and calculate the relative pose. In addition, the image processor has an internal 1553B bus function that allows it to receive and forward various control commands from the central control; the information received includes image data, temperature data, and control signal, such as lighting control and heating control. The interaction of the camera information with the manipulator controller via an internal 1553B bus at a rate of 8 Hz, through which the manipulator controller sends commands to the EVS and receives the measurements. The camera imaging and process are based on FPGA. Image compression is based on DSP and transmission via the Ethernet at 25 frames per second.

2.2. Measurement of Position and Orientation

The relative position and orientation from different targets to cameras are unified by the well-known intrinsic and extrinsic parameters. The intrinsic parameters matrix K satisfies:

$$Z_c\left[\begin{array}{c}\overline{u}\\ \overline{v}\\ 1\end{array}\right]=\left[\begin{array}{ccc}{a}_x& 0& u_0\\ 0& a_y& u_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\end{array}\right]=K{\mathit{P}}_{\mathit{c}}$$

(1)

which is $Z_c{P}_x=K{\mathit{P}}_{\mathit{c}}$, where $a_x=f_c/dx,a_{\mathrm{y}}=f_c/dy$, and $X_c,Y_c,Z_c$ are the coordinates of three dimensional points in space; $\overline{u},\overline{v}$ are the coordinates of the ideal imaging point on the image plane in the image coordinate system; $f_c$ is the focal length; and $a_x,a_y$ is the equivalent focal length of the camera in the x,y direction.

The position and orientation measured from the vision marker to the camera are expressed by transformation matrix ${}^{\mathrm{cam}}{\mathrm{T}}_{\mathrm{vm}}$. The cubic prism is fixed on the EVS and used for the calibration from camera to cubic prism, which is represented as ${}^{\mathrm{cp}}{\mathrm{T}}_{\mathrm{cam}}$. The EVS fixed on the EEF and matrices ${}^{\mathrm{EEF}}{\mathrm{T}}_{\mathrm{cp}}$ are calibrated from the cubic prism to the EEF tool coordinate. The pose information relative to the tool coordinate can be expressed as:

$${}^{\mathrm{tool}}{\mathrm{T}}_{\mathrm{vm}}={\left[{}^{\mathrm{EEF}}{\mathrm{T}}_{\mathrm{tool}}\right]}^{-1}{·}^{\mathrm{EEF}}{\mathrm{T}}_{\mathrm{cp}}{·}^{\mathrm{cp}}{\mathrm{T}}_{\mathrm{cam}}{·}^{\mathrm{cam}}{\mathrm{T}}_{\mathrm{vm}}$$

(2)

where ${}^{\mathrm{EEF}}{\mathrm{T}}_{\mathrm{cam}}{=}^{\mathrm{EEF}}{\mathrm{T}}_{\mathrm{cp}}{·}^{\mathrm{cp}}{\mathrm{T}}_{\mathrm{cam}}$ is the extrinsic parameters, which are measured accurately “before” flight and do not change significantly “during” the flight. The control is proposed to move the payload to the desired position, ${}^{{\mathrm{tool}}_{\mathrm{d}}}{\mathrm{T}}_{\mathrm{vm}}$ is given as the desired berthing position relative to the visual marker coordinate, which is listed in the central controller through external command. Then, the servo error can be estimated in Equation (3):

$${}^{{\mathrm{tool}}_{\mathrm{d}}}{\mathrm{T}}_{\mathrm{tool}}{=}^{{\mathrm{tool}}_{\mathrm{d}}}{\mathrm{T}}_{\mathrm{vm}}·{\left[{}^{\mathrm{tool}}{\mathrm{T}}_{\mathrm{vm}}\right]}^{-1}=\left[\begin{array}{cc}{\mathit{R}}_{\mathrm{e}}& {\mathit{e}}_p\\ 0& 1\end{array}\right]$$

(3)

The servo controller that minimizes the tracking errors is defined as $\mathit{e}={\left[{\mathit{e}}_p^{\mathrm{T}},{\mathit{e}}_o^{\mathrm{T}}\right]}^{\mathrm{T}}\in {\Re }^6$, where ${\mathit{e}}_p$ and ${\mathit{e}}_o$ are the position and orientation errors, respectively, where ${\mathit{e}}_o=\theta \mathbf{a}$ is computed by angle/axis parameterization for the rotation, which gives the minimum travel distance from the initial to the berth orientation and keep the marker at the center of the camera field of view.

3. Integration of Visual Servo Control

3.1. Model of Macro-Micro Manipulator

When the macro-micro manipulators are in series, the flexibility of the macro-manipulator is more important. Considering that the micro manipulator is stiff, the dynamic model is set up by the Lagrange equation with the generalized coordinates of the joints, as shown in Formulas (4) and ().

$$\begin{array}{cc}\hfill {\mathit{H}}_b{\ddot{\mathit{q}}}_b+{\mathit{H}}_{bm}{\ddot{\mathit{q}}}_m+{\mathit{C}}_b+{\mathit{D}}_b{\dot{\mathit{q}}}_b+{\mathit{K}}_b\Delta {\mathit{q}}_b=& 0\end{array}$$

(4)

$$\begin{array}{cc}\hfill {\mathit{H}}_{bm}^T{\ddot{\mathit{q}}}_b+{\mathit{H}}_m{\ddot{\mathit{q}}}_m+{\mathit{C}}_m+{\mathit{D}}_m{\dot{\mathit{q}}}_m=& \mathbf{\tau }\end{array}$$

(5)

where ${\mathit{q}}_b\in {\Re }^{nb}$ denotes the local coordinates of the macro-manipulator, ${\mathit{q}}_m\in {\Re }^{nm}$ stands for the joint coordinates of the micro-manipulator. The states of $\Delta {\mathit{q}}_b$ are the current angles minus initial states. $nb$ and $nm$ are the DOF of the micro/macro manipulator, respectively. ${\mathit{H}}_b\left({\mathit{q}}_b,{\mathit{q}}_m\right)\in {\Re }^{nb\times nb}$, ${\mathit{D}}_b\in {\Re }^{nb\times nb}$, and ${\mathit{K}}_b\in {\Re }^{nb\times nb}$ denote base inertia, damping, and stiffness, respectively. ${\mathit{H}}_m\left(q_m\right)\in {\Re }^{nm\times nm}$ is the inertia matrix of the arm. ${\mathit{H}}_{bm}\left({\mathit{q}}_b,{\mathit{q}}_m\right)\in {\Re }^{nb\times nm}$ denotes the so-called inertia coupling matrix. ${\mathit{C}}_b\left({\mathit{q}}_b,{\dot{\mathit{q}}}_b,{\mathit{q}}_m,{\dot{\mathit{q}}}_m\right)\in {\Re }^{nb}$ and ${\mathit{C}}_m\left({\mathit{q}}_b,{\dot{\mathit{q}}}_b,{\mathit{q}}_m,{\dot{\mathit{q}}}_m\right)\in {\Re }^{nm}$ are velocity-dependent nonlinear terms, ${\mathit{D}}_m\in {\Re }^{nm\times nm}$ denotes arm joint damping and torques, $\tau \in {\Re }^{nm}$ are considered the control inputs. It can be seen that no external forces are present, and this assumption holds throughout the paper.

The stiffness of the base joint ${\mathit{K}}_b$ is non-constant due to the gear backlash, as shown in Figure 4. The stiffness is not zero in the backlash due to the friction and lubrication. Outside the backlash, the stiffness is much more significant due to the flexibility of the gear reducer.

The working condition of the manipulator in orbit is complicated, the dynamic model given above is an ideal model, in order to specify the dynamic characteristic, the following two essential assumptions are needed.

Assumption 1.

No constrained motions are considered, which means the manipulator is under free space.

Assumption 2.

The joint friction is not considered, which means the deflection of the macro-manipulator will oscillate around zeros.

3.2. Reaction Force/Torque Feedback Control

The internal forces measured by the force/torque sensor ${\mathit{F}}_S$ is described in the installation coordinate and transformed to the EEF by:

$${\mathit{F}}_B=\left[\begin{array}{cc}{}^B{R}_S& 0\\ {\times }^B{\mathit{P}}_S^B{R}_S& {}^B{R}_S\end{array}\right]·{\mathit{F}}_S$$

(6)

where P and R are the position and rotation matrix expressed in the reference frame sensor to EEF, ${\mathit{F}}_B$ represents the interaction force and torque from the manipulator motion. The forward direction is defined from joint to EFF.

A special mechanical interface (MI) between CMM and EMM combined the macro and micro manipulator in series. It consists of two different grapple fixtures for each manipulator. Each end of the mechanical interface corresponds to the end-effector of the CMM and EMM, respectively. The coordinate frame defined for the End of CMM (EOC) and Base of EMM (BOE) is shown in Figure 5.

The relation expressed in the transfer matrix is:

$${\mathrm{T}}_{\mathrm{MI}}=\left[\begin{array}{cccc}-1& 0& 0& d\\ 0& -1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$$

(7)

The Jacobian of the CMM form BOE to the base of the macro-micro manipulator (BMM) is:

$${{}^{BMM}\mathit{J}}_{CMM\_BOE}=\left[\begin{array}{cc}{I}_3& -{{}^{BMM}R}_{CMM}·{\mathit{P}}_{MI}\times {{}^{BMM}R}_{CMM}^T\\ 0_3& I_3\end{array}\right]·{{}^{BMM}\mathit{J}}_{CMM}$$

(8)

where ${}^{BMM}{\mathit{J}}_{\mathrm{CMM}}$ and $R_{\mathrm{CMM}}$ are expressed in the base coordinate frame of CMM. The calculation method regards the mechanical interface as the tool of the CMM. The deflection torque of the joints due to the reaction force is:

$${\mathit{\tau }}_{ext}={{}^{BOE}\mathit{J}}_{CMM\_BOE}^T·{\mathit{F}}_B$$

(9)

where ${{}^{BOE}\mathit{J}}_{CMM\_BOE}$ is the Jacobian of CMM with the mechanical interface and expressed in the coordinate of BOE.

$${{}^{BOE}\mathit{J}}_{CMM\_BOE}=\left[\begin{array}{cc}{{}^{BOE}R}_{BMM}& 0\\ 0& {{}^{BOE}R}_{BMM}\end{array}\right]·{{}^{BMM}\mathit{J}}_{CMM\_BOE}$$

(10)

It is essential to study the manipulator with a stationary initial state. Ignoring the force sensor’s elasticity in the model and treating it as an ideal force sensing element, from the dynamic Equation (4), the deflection dynamic can be striped as:

$${\mathit{H}}_b{\ddot{\mathit{q}}}_b+{\mathit{K}}_b\Delta {\mathit{q}}_b+{\mathit{C}}_b=-{\mathit{\tau }}_{ext}$$

(11)

We can see from the base deflection dynamics that the base forces, which are measured by the FTS, reflect the motion of the base. The deflection forces depend on the robot’s state and the control input while performing the visual servo task. The desired closed-loop base deflection dynamic is:

$${\mathit{H}}_b{\ddot{\mathit{q}}}_b+({\mathit{D}}_b+G){\dot{\mathit{q}}}_b+{\mathit{K}}_b\Delta {\mathit{q}}_b=0$$

(12)

The base reaction forces are feedback to achieve the desired closed-loop dynamic, where a positive definite damping term G is added to damp out the base deflection:

$${\dot{\mathit{q}}}_{bd}={\mathit{\tau }}_{ext}/G$$

(13)

Due to the slow deflection of the flexible base, the dynamic inertial forces of the macro manipulator can be neglected, the contribution of the joint deflection in the coordinate of BOE denoted by the FTS is:

$${}^{BOE}{\mathit{v}}_{BOE\_d}={{}^{BOE}\mathit{J}}_{CMM\_BOE}·{\dot{\mathit{q}}}_{bd}={{}^{BOE}\mathit{J}}_{CMM\_BOE}·G^{-1}·{{}^{BOE}\mathit{J}}_{CMM\_BOE}^{\mathrm{T}}·{\mathit{F}}_B$$

(14)

Then, the base deflection can be converted to the EEF by:

$${{}^{Tool}\mathit{v}}_f={{}^{Tool}T}_{BOE}·{{}^{BOE}\mathit{v}}_{BOE\_d}$$

(15)

where ${{}^{Tool}T}_{BOE}$ is the velocity convert matrix from the tool of the EMM to the BOE.

$${{}^{Tool}T}_{BOE}=\left[\begin{array}{cc}{{}^{BOE}R}_{Tool}^T& -{{}^{BOE}R}_{Tool}^T·P_{tool}\times \\ 0& {{}^{BOE}R}_{Tool}^T\end{array}\right]$$

(16)

The application process of base FTS is needed. Firstly, the macro-micro model is detected compared to the single-arm mode. Then, the FTS is enabled to the low-pass filter. The static judgment is needed to ensure there is no movement of the manipulator system at the initial states. Due to the zero-point drift of the FTS, the sensor is needed to reach zero for some conditions. Besides, a threshold is set due to the sensitivity limitation and noises. Finally, the visual servo control is carried out based on the base FTS feedback.

3.3. Base Backlash Compensation

Due to the low stiffness of the gear backlash, the joint deflection is much more sensitive to the reaction force/torque, and the backlash is not to be ignored. However, the force/torque sensor at the base of the EMM is inoperative for the minor response. Considering the working mode is the motor of the macro manipulator is fixed, the perception of the backlash can be executed by the fusion angle of the joint. The effect of backlash to the EFF can be calculated by:

$${{}^{Tool}\mathit{e}}_{bl}={{}^{Tool}T}_{BOE}·{}^{BOE}{\mathit{J}}_{CMM\_BOE}·\Delta {\mathit{q}}_{bl}$$

(17)

where $\Delta {\mathit{q}}_{bl}$ is the angular difference by the initial joint angle minus the real joint angle.

It should be noted that the joint deflection feedback through FTS or directly through the joint angle are two different methods for different purposes. The FTS reveals the dynamic motion of the manipulator, it works when there is a wide range of disturbance due to the limited sensing accuracy, which is effective for the period of acceleration saltation. However, the backlash is a small deflection, and the compensation works when there is a small range of oscillation, this is essential for the period of residual oscillation when the visual errors approach zeros.

3.4. Integration of a Visual Servo

The visual servo control based on the base vibration suppression and base backlash compensation is introduced based on Equations (15) and (17).

$${{}^{Tool}\mathit{v}}_d=-k_p\mathit{e}+k_f{{}^{Tool}\mathit{v}}_f+k_{bl}{{}^{Tool}\mathit{e}}_{bl}$$

(18)

where the gain values $k_p$, $k_f$, and $k_{bl}$ are positive defined. $k_p$ is the coefficient that converges the tracking errors in the way of exponential convergent, $k_f$ and $k_{bl}$ are the coefficients that control the compensation errors in the task space. Then, the desired trajectory of the joints is resolved by the inverse kinematics of the EMM. Besides, the desired velocity ${{}^{Tool}\mathit{v}}_d$ expressed in the task space should be converted into the coordinates of BOE or use the Jacobian of ${}^{Tool}{\mathit{J}}_{EMM}$.

Due to the low stiffness of the flexible base and large payload-to-manipulator mass radio, the low-frequency property is significant and the phase margin is meager. A lag compensator with time constants $k_1,k_2$ is defined as $\frac{k_{2}s+1}{k_{1}s+1}$ to improve the closed-loop performance. The control block is shown in Figure 6.

3.5. Stability Analysis

The closed-loop visual servo system based on the proposed method can be written as:

$$\dot{\mathit{e}}=-k_p\mathit{e}+{\overline{k}}_f{\Delta \dot{\mathit{q}}}_b+{\overline{k}}_{bl}\Delta {\mathit{q}}_b$$

(19)

which:

$$\begin{array}{cc}\hfill {\overline{k}}_f& =k_f{·}^{Tool}{T}_{BOE}{·}^{BOE}{\mathit{J}}_{CMM\_BOE}·G^{-1}\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill {\overline{k}}_{bl}& =k_b{·}^{Tool}{T}_{BOE}{·}^{BOE}{\mathit{J}}_{CMM\_BOE}\hfill \end{array}$$

(21)

Assuming that the configuration of the manipulator is non-singular and the inverse kinetic solution of the redundant manipulator is closed-form without self-motion, which means $\mathit{e}=0$ is the only solution of ${\mathit{q}}_m\to {\mathit{q}}_d$. When the EEF reaches the desired berth position, the base deflection should be damped to zero. The steady state convergence value of the visual error is ${\mathit{e}}_{\infty }={\mathit{e}}_{0}exp\left(-k_{p}t\right)+({\overline{k}}_f{\Delta \dot{\mathit{q}}}_b+{\overline{k}}_{bl}\Delta {\mathit{q}}_b)/k_p$, in which it can be seen that the errors are exponential decay and the steady-states are affected by the base deflections. The final states of ${\Delta \mathit{q}}_b$ are influenced by motion of the manipulator under the Assumptions of A1 and A2. The isolated equilibrium of the closed-loop system is ${\left[\begin{array}{ccc}\Delta {\mathit{q}}_b^T& \Delta {\dot{\mathit{q}}}_b^T& {\mathit{e}}^T\end{array}\right]}^T={\left[\begin{array}{ccc}0& 0& 0\end{array}\right]}^T$. The configuration of the manipulators does not vary quickly compared with the elastic vibration, it is reasonable to treat the matrices as constant. Then, the closed-loop system can be described by the flowing state-space form:

$$\left[\begin{array}{c}{\Delta \dot{\mathit{q}}}_b\\ {\Delta \ddot{\mathit{q}}}_b\\ \dot{\mathit{e}}\end{array}\right]=\left[\begin{array}{ccc}0& I& 0\\ -{\mathit{H}}_b^{-1}{K}_b& -{\mathit{H}}_b^{-1}\left({\mathit{D}}_b+G\right)& 0\\ {\overline{k}}_{bl}& {\overline{k}}_f& -k_p\end{array}\right]\left[\begin{array}{c}{\Delta \mathit{q}}_b\\ {\Delta \dot{\mathbf{q}}}_b\\ \mathit{e}\end{array}\right]$$

(22)

The stability of the overall system can be analyzed by investigating the poles of the closed-loop system, the eigenvalues are determined by the roots of the characteristic polynomial:

$$p\left(\lambda \right)=\left(\lambda +k_p\right)\left({\lambda }^2+{\mathit{H}}_b^{-1}\left(D_b+G\right)\lambda +{\mathit{H}}_b^{-1}{K}_b\right)$$

(23)

It is sufficient to validate that all roots are contained in the left open complex half-plane when G is positive definite, it is globally asymptotically stable in the equilibrium.

4. Simulation Results

4.1. Simulation Model

This section presents the simulation results for the proposed control laws via the MMMS model. The kinematic models of the CMM and EMM are the same but with different link length parameters. The frames of the 7-DOF offset manipulator are shown in Figure 7, which are based on the modified D-H parameters in

Table 1. D-H Parameters of the CMM/EMM.

${Link}_i$	$a_{i-1}$	${\alpha }_{i-1}{(}^{\circ })$	$d_i$	${\theta }_i{(}^{\circ })$
Base	0	0	0	90
1	0	90	$l_0$	0
2	0	−90	$l_1$	0
3	0	−90	$l_2$	−90
4	$l_3$	0	$l_4$	0
5	$l_5$	0	$l_6$	90
6	0	90	$l_7$	0
7	0	−90	$l_8$	0
End	0	90	0	90

where the subscripts B and E are the base and end of the manipulator, respectively. The CMM and EMM are connected in series by the MI shown in Figure 5. Due to the symmetry of the architecture, both manipulators can crawl on the space station, and the base or end of the manipulator can be reversed. Link 1 is physically formed by the combination of joint 3, lower arm boom, and joint 4. Link 2 is physically formed by the combination of the elbow joint, upper arm boom, and joint 5. We only consider the flexible booms of the CMM, and the link of EMM, and payload are approximated by rigid bodies. The viscous damping and joint torsional springs are included in all joints, which are modularized designed. The joint backlash of CMM is considered as Figure 4, where the backlash is $L=0.2$ deg and stiffness and damping are listed in

Table 2. Joint properties of the CMM and EMM.

Manipulator	CMM		EMM
	Stiffness (Nm/rad)	Damping (Nms/rad)	Stiffness (Nm/rad)	Damping (Nms/rad)
kL	19	3	$5\times {10}^4$	210
k	$9\times {10}^5$	3183

. The flexible boom is modeled as a hollow rod, with an elastic modulus of 90 GPa and a shear modulus of 34.6 GPa. The simulation model is built by ITI-SimulationX, where the inertial parameters and payload are listed in

Table 3. Mass properties of the CMM/EMM.

Manipulator	Property	Mass (kg)	$I_x(\mathrm{kg}·{\mathrm{m}}^2)$	$I_y(\mathrm{kg}·{\mathrm{m}}^2)$	$I_z(\mathrm{kg}·{\mathrm{m}}^2)$
CMM	Joint 1	59.5	4.322	1.438	4.121
	Joint 2	53.6	2.468	1.329	2.267
	Link 1	108.4	2.782	384.756	384.327
	Link 2	98.7	4.527	341.956	339.871
	Joint 6	53.6	2.468	1.329	2.267
	Joint 7	53.5	4.322	1.438	4.120
	EEF	100.5	3.347	9.731	9.833
EMM	Joint 1	31.7	0.366	0.366	0.332
	Joint 2	31.7	0.366	0.366	0.332
	Link 1	74.7	0.881	70.702	70.554
	Link 2	57.5	0.730	45.040	44.918
	Joint 6	31.7	0.366	0.332	0.355
	Joint 7	31.7	0.366	0.334	0.358
	EEF	28.9	0.52	0.527	0.421
Payload	–	3000	722.5	1160	1062.5

.

The visual servo simulation of the MMMS with a payload is performed, where the pallet camera is in front of the payload, as shown in Figure 2. The task is to move the payload out of the cargo spaceship and away from the pallet with the support of the EVS. We assume that the light condition is ideal and the Assumption of A1 is considered, which means there is no external contact force. The desired berth place is 3.5 m away in the X-direction. The initial configurations of the CMM is ${\mathit{q}}_b={(-151,-20,-29,-37,-77,-72,-11)}^T$ deg and EMM is ${\mathit{q}}_m={(160,-86.096,-56.871,-89.166,71.999,87.643,-167.113)}^T$ deg with the joint order from base to end. The sampling time of the central controller and visual measure is 100 ms, the joint controller is 2 ms. Furthermore, an inherent time delay exists in the posture measuring and posture transferring due to the performance limitation of the EVS. A delay of 0.5 s is added in the visual feedback to simulate the time delay of the real physical system.

The velocity and acceleration limitations are imposed to avoid the sharp movement when the pose error is large at the beginning due to the fixed gain coefficient of $k_p$. The max planned velocity and acceleration of the visual servo is (0.02 m/s, 1 deg/s) and (0.002 m/s${}^2$, 0.1 deg/s${}^2$), respectively. The lag compensator is $\frac{s+1}{8s+1}$ for the large payload, and the gain matrices were set as follows:

$$\begin{array}{cc}\hfill k_p& =0.05\ast eye\left(6\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill k_f& =1\ast eye\left(6\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill k_{bl}& =0.2\ast eye\left(6\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill G& =1e5\ast eye\left(7\right)\hfill \end{array}$$

4.2. Simulation Results

The results from the simulation are shown in Figure 8, Figure 9 and Figure 10. In Figure 8a, we denote the relative position and orientation of the visual servo. (b) Denotes the joint angles of CMM and (c) denotes the joint deflections of CMM. Figure 9 denotes the deflection of the end of the CMM, which is described in the frame of the base of the MMMS. Assuming that the orientation of the EEF and object frames slowly vary with respect to time, we use the ZYX Euler angle to represent the orientations. From Figure 8a, it is seen that, under the visual servo control, the tracking errors are reduced at a relatively steady velocity in the long-range motion and converge to zeros when reaching the desired berth position within one vibration cycle. A low-frequency vibration appeared in the manipulator system due to the rather low natural frequency (0.007 Hz). As shown in Figure 8c, the joint of the CMM mostly moves within $\pm 0.1^{\circ }$, which is the boundary of the backlash. The joint motion of the CMM will reverse at the backlash boundaries due to a higher stiffness, which will work beyond the backlash. Even though the backlash is small, the deflection of the CMM is nonnegligible, which can be seen in Figure 10. However, with the backlash compensation, the deflection of the CMM converges to zeros when the EEF reaches the desired target. One can see that, acceleration during the start-up period and deceleration near the target period will stimulate the deflection in the backlash. From Figure 9, it can be seen that, at the beginning of the acceleration period, the motion of the EMM inevitably causes a disturbance to the base, leading to the joint oscillation of the CMM in the backlash, the base vibration based on the FTS works at this period due to the dead zone limitation of the FTS. The joint trajectory of EMM is performed with the joint two fixed inverse kinematics, as shown in Figure 8b.

5. Experimental Results

5.1. The Experimental Facility

In this section, the test-bed used in this research is described in detail. Considering the high experimental cost of the MMMS, a tunable planar flexible base with three degrees of freedom is designed, which includes a two-dimensional motion platform carrying the linear variable stiffness module (containing X and Y directions) and one rotation (Rz direction) variable stiffness module, as shown in the right picture of Figure 11b. The stiffness can be adjusted corresponding to different configurations of macro manipulators. An inherent slight backlash is added in the three directions due to the mechanical structure, but the real-time base backlash compensation is not included.

A prototypical manipulator of EMM is connected with the flexible base and is supported on the air-bearing table. The manipulator is constrained to move in the horizontal plane, only the three parallel joints of the manipulator can rotate, which is equivalent to a three-link manipulator. The configuration from 1 to 7 is ${[90,-90,wrist,elbow,shoulder,-90,-90]}^T$ deg, as shown in the sub-figure of Figure 11a. The joints are supported by the micro-gravity compensation system, the EEF (including FTS) near joint 7 (shown in left top picture of Figure 11b) is connected with the flexible base. The detail of the FTS used in the experiment has been exhibited in [33]. The EEF near joint 1 can be connected with the payload. Figure 11a shows the scenario of the empty load visual servo, where the payload is fixed on the ground. A grapple figure is mounted in the back of the payload, the VM of the HC is installed on the grapple fixture, which is shown in [9]. The left picture of Figure 11b shows the scenario of the 1500 kg load visual servo, where the EEF connected with the grapple fixture. HC is obscured by the large payload and the PC will work.

The software of the manipulator system includes communication, imaging, visual measurement, active thermal control, motion, power management, fault detection and handling, software maintenance and update, and task list management. The control system of the manipulator is distributed, where the central controller is fixed on the elbow joint, as shown in Figure 11a. Controllers of the joints and central controller are all based on the BM3803 and FPGA. The console, consisting of a server, display, and monitor, is connected based on Ethernet. The primary task processing process is that the central controller parses the control commands and control parameters on the manipulator’s dedicated 1553B bus, as well as the real-time calculation results, and controls the joints, effectors (including FTS), and end cameras, respectively, via the arm’s internal 1553B bus to complete the specified arm system tasks. After the task is completed, the arm software system returns to a non-working state or a safe or standby mode state, as required, pending the next task.

Interpolation is required between the camera, central controller, and joints due to the different performance of the processors. The frequency of the posture measurement, central controller, and joint controller is 8 Hz, 10 Hz, and 500 Hz, respectively. The posture measurements obtained from EVS are widened to 10 Hz. The desired joint angles calculated in the central controller are interpolated in the joint controller every 50 times. The joint controllers are PD-based motor control with a steady closed-loop control accuracy of better than 0.025 deg. The sensors’ data are stored in the server at a frequency of 10 Hz for the fast telemetry.

Two cases of experiments have been performed based on the flexible base manipulator testbed. First is the no-load visual servo based on the HC; second is the large payload visual servo based on the PC. Here, we assume no contact between payload and pallet. The visual marker is ahead of the camera, and two desired berth positions are planned to reach the final target. The manipulator approaches the target and moves away from it to demonstrate the in-orbitobit task, which is divided into four trajectories in different directions. As shown in the results, the green vertical dotted line is the beginning time, and the black one is the stop time. The visual measures and base FTS are monitored. The visual measure accuracy of HC is (0.5 mm, 0.3 deg) to (1.5 mm, 0.5 deg), and PC is (2 mm, 0.5 deg) to (5 mm, 1 deg), which is related to the working distance. The resolution of the cameras is (0.1 mm, 0.1 deg), the threshold of sensitivity of the FTS is (2 N, 2 N, 2 Nm). A comparison results of the two cases are listed in

Table 4. Visual servo results of the two cases.

Cases	Vibration Control	Max Visual Errors			Vibration Periods	Total VS Time (s)	Torque of FTS (Nm)	Stability
		X (mm)	Y (mm)	Rz (deg)
Without payload	No	0	7	0.2	8	374	7.7	Yes
	Yes	0	6	0.3	1	386	7.4	Yes
1500 kg payload	No	12	77	0.7	4	530	18	No
	Yes	0	77	0.8	1	406	13.5	Yes

.

5.2. Visual Servo Control without Payload

Case 1 is the visual servo without a payload mounted on the EEF. The stiffness in X, Y, and RZ of the flexible base is 1.6375 N/mm, 8.158 N/mm, 529 Nm/deg, respectively. The control gains are Kp = (0.032, 0.048, 0.05) and G = 17,000. The max planned velocity and acceleration of the visual servo are (0.02 m/s, 1 deg/s) and (0.001 m/s${}^2$, 0.05 deg/s${}^2$). No extra lag compensator is needed due to the small disturbance to the flexible base. The initial distance in the X-direction is 1000 mm, and the intermediate berth is 120 mm. The initial configuration of the wrist, elbow, and shoulder is (39, −96, −100) deg. The visual servo control will finish when the tracking errors in X, Y, and Rz are less than (3 mm, 3 mm, 0.3 deg), showing that the manipulator reaches the desired berth position.

The experimental results for case 1 are given in Figure 12, in Figure 12a,c there is no vibration suppression algorithm, while the right Figure 12b,d is based on FTS feedback. Figure 12a,b show the position error (x-axis and y-axis) and orientation error (Euler angles representation), respectively, with respect to the desired relative position to the HC markers. The visual measure results show that the EEF’s motion speeds up and then slows down exponentially, and the visual control is finished normally in each trajectory. The visual errors shown in Table 4 are similar due to the measurement accuracy and resolution limitation. The difference of the trajectory in the x-axis is not obvious because the motion of the EEF is slanted to the y-axis (expressed in the coordinate of BOE). As the stiffness of the base in the y-axis is larger than the x-axis, the higher fundamental frequency is as expected without payload, which means the vibration control does not promote the tracking accuracy under the higher fundamental frequency; the dominant factor is the measurement accuracy and resolution limitation.

One can notice that the visual error of the y-axis is significantly suppressed, and the time of the oscillation during each motion is shortened from the initial 8 cycles to about 1 cycle. Similar results of contrast are shown in Figure 12c,d, where the forces (x-axis and y-axis) and torque (Mz) of the base FTS, which is represented in the coordinate of BOE. From the comparison of Mz, it can be seen that the base vibration is quickly suppressed after one period of vibration, the FTS-based suppression works very well. One can notice, the reaction torque of Mz is relevant to the visual tracking errors of the y-axis, the base vibration suppression algorithm can effectively suppress the base disturbance and reduce the measured oscillation period of the EEF. Furthermore, we compare the total VS time under the zero payload condition, as shown in Table 4. It should be mentioned that the total servo time of the four trajectories is not necessarily related to whether or not the vibration control is used. When the servo control stability is guaranteed, the exponential convergence method of the tracking errors is the major contributors to the servo time.

5.3. Visual Servo Control with Large Payload

Case 2 is the large payload visual servo comparison with and without vibration suppression. In order to verify that the control effect is more significant and robust to different stiffness, the manipulator carries a 1500 kg load at a maximum planning speed of 50 mm/s in free space. A new stiffness in X, Y, and RZ of the flexible base is 1.046 N/mm, 2.3155 N/mm, and 120.91 Nm/deg, respectively. The frequency of the system in the configuration of (−155, 0, −90) deg is (0.0875, 0.035) in the X and Y directions, which is expressed in the coordinate of BOE. The control gains are Kp = (0.05, 0.05, 0.05) and G = 17,000. The max planned velocity and acceleration of the visual servo is (0.05 m/s, 0.5 deg/s) and (0.003 m/s${}^2$, 0.05 deg/s${}^2$). The extra lag compensator is needed, which is $\frac{s+1}{4s+1}$. The camera of HC is sheltered and the PC is needed, which is mounted in front of the payload. The initial distance in the X direction is 1100 mm, and the intermediate berth is 200 mm. The initial configuration of the wrist, elbow, and shoulder is (−55, −102, −89) deg. The visual servo exit threshold is set to (5 mm, 5 mm, 0.3 deg) due to the different accuracy of the cameras.

The experimental results for case 2 are given in Figure 13, the left Figure 13a,c has no vibration suppression algorithm, the right Figure 13b,d is based on FTS feedback. Figure 13a,b show the position error (x-axis and y-axis) and error (Euler angles representation), respectively, with respect to the desired relative position of the tool coordinates of payload to the VM of PC. The vibration reflected in the visual errors in Figure 13a is obvious compared to the empty load condition in Figure 12a, due to the lower fundamental frequency and higher motion speed. One can notice in the first trajectory, in the time from 7 s to 118 s, the servo control stopped abnormally after three swing periods, causing the visual measure errors to fluctuate until 190 s; it is obvious that the VS without vibration control will take a long time to wait for the vibration to fade away, as shown in Table 4. The total servo time of the four trajectories is much longer if there is no vibration suppression. It can be seen that the visual servo without vibration suppression has a large end oscillation at high speed and is difficult to attenuate, which may cause the camera to lose the target, leading to the termination of the visual servo motion, causing unnecessary disturbances, as shown in Table 4. The servo errors in the x-axis are 12 mm beyond the target, as a comparison, the overshoot in the X direction is avoided with the suppression algorithm, and dampen the non-forward direction (y-axis and Rz); thus, the vibration period basically decays from 4 to 1. Naturally, the time cost is lower, for example, it takes only 120 s to finish the first trajectory, reduced by 37%.

The vibration suppression-based visual servo algorithm reduces the disturbance torque on the base (Figure 13d) from 18 Nm to 13.5 Nm, decreases the swing frequency of the load, and reduces the magnitude of the lateral swing to a certain extent to ensure that the target is within the field of view of the camera and improves the reliability of the visual servo tracking algorithm. One can notice that the visual error of the y-axis is significantly suppressed and the x-direction is smooth in Figure 13b. It should be noted that the deflection of the visual errors in the first half period of vibration is inevitable, even though the vibration will be damped rapidly, as shown in the Y-axis and Rz-axis. A reasonable interpretation is that the system’s natural frequency is low, and the response bandwidth of the manipulator is not enough to instantly compensate for the visual errors, even though the manipulator has one redundant DOF for compensating for different directions of motion. In general, it can be seen that the swing period, which is reduced, enables the flexible base manipulator system to stabilize faster when the visual servo enters the exit domain and switches to standby mode, improving the positioning accuracy of servo control.

5.4. Visual Servo Control of MMMS

To further verify the effectiveness of the real time joint backlash compensation, we subsequently validated the visual servo algorithm on the planar MMMS test-bed. Comparing with the previous test-bed of FBMS, the flexible base is replaced by a prototypical manipulator of CMM [38], and the EMM is joined together through the mechanical interface on a larger air-bearing table. The power supply and communication is rebuilt in the “Macro-Micro Mode”, where the 1553B bus of the two manipulators are connected, and the manipulators are commanded through the operating consoles.

The experimental scenarios were designed in three trajectories to move between the “coarse” and “fine” position, where the coarse position is 200 mm ahead of the fine position. There are initial errors of −528 mm, −21 mm, and 2.3 deg in the X, Y, and Rz, respectively. The visual servo results shown in Figure 14 are tested without payload. It can be seen that the controller is stable in different trajectories, and the final berth accuracy is less than (3 mm, 3 mm, 0.3 deg). One can notice that the gradual deceleration works when the manipulator approaches the target; at the same time, the max overshoot of 12 mm in the Y-direction is apparent, which gradually converges to zero and no widespread fluctuations are generated.

6. Conclusions

In this paper, we analyzed the visual servo control problem of the macro-micro manipulator system with large payloads, in which the force/torque sensor (FTS) is mounted at the base of the micro-manipulator. An integrated controller was proposed, including exponential convergence of visual errors, backlash compensation, and vibration suppression. The base deflection can be perceived by FTS in the large deflection or by joint sensors in slight backlash, and a dynamic model is not needed. A prototypic implementation of the vision servo system of the Experimental Module Arm has been shown. A simulation model with a large payload was established for the macro-micro manipulator in-orbit operation task, and the integrated control strategy was verified. The planar flexible base manipulator system was set up for ground experimental verification, and the visual servo experiments were carried out. The effectiveness of the vibration suppression is demonstrated based on the proposed methodology under different stiffness and payload conditions. The vibration periods decay from four to one and the servo time was reduced 37%. Further, the joint backlash compensation algorithm is validated based on the ground MMMS test-bed. Test results confirmed the efficacy and robustness of the control approach, ensuring the reliability and improving the efficiency of the VS.

There are still some limitations due to the assumptions. We consider no constrained motions, in practical application, the contact may occur in the rapid and precise positioning of the payload, especially for the payload in the pallet. Moreover, the joint friction in the backlash is not considered, in practical, the static friction in the backlash will work if the reaction force is low enough. The static equilibrium will change and cause a steady error of the visual control. In future work, the isolated static equilibrium should be reconsidered, and visual impedance can be investigated further.