Constrained Image-Based Visual Servoing of Robot Manipulator with Third-Order Sliding-Mode Observer

Abstract

A new image-based robot visual servo control strategy based on a third-order sliding-mode observer (TOSM) model predictive control is proposed in this study. This new control strategy solves the problem of robot visual servo control with system constraints and time-varying disturbances when the camera and model of the robot manipulator are uncertain and the joint velocity is unknown. In the proposed method, the joint velocity and system centralized uncertainties are estimated simultaneously based on a third-order sliding-mode observer, and the image-based visual servoing problem is transformed into a nonlinear optimization problem based on a model predictive control method considering both visibility constraints and actuator constraints, which minimizes the predicted trajectory cost function to generate the control signal for each cycle. Simulations were carried out to verify the effectiveness of the proposed control scheme.

1. Introduction

To make robots more intelligent and flexible, they are often equipped with vision sensors as tools for interacting with the outside world. In recent decades, robotic visual servoing has been fully researched and widely used in various areas, such as robotic production logistic [1], robotic service [2,3], robotic navigation and exploration [4,5]. According to the different feedback information from the camera, the visual servoing system can be divided into image-based visual servoing [6,7], position-based visual servoing [8,9], and hybrid visual servoing [10,11]. Refs. [6,7] adopted the image-based visual servoing method, which takes the deviation between the current and desired image features as the control deviation. The robustness of different position-based visual servoing systems was discussed in [8,9], where the control deviations are the deviations of relative pose between the end-effector and the target in the Cartesian coordinate system. The hybrid visual servoing method combines the above two visual servoing methods, and the control deviation is composed of two-dimensional (2D) and three-dimensional (3D) deviations. Refs. [10,11] discussed the applications of the hybrid visual servoing method in mobile robots and parallel robots, respectively. Among these, image-based visual servoing has been the most widely studied and is discussed in this study.

In the image-based visual servo system, the hand-eye mapping relationship is represented by the dynamic relationship between the velocity of the feature points and the joints of the manipulator, which is usually expressed as a Jacobian matrix. Because the Jacobian matrix contains the robot kinematics model and internal and external parameters of the camera, the traditional Jacobian matrix requires a tedious calibration process, and accurate calibration results are difficult to obtain in practical application scenarios. Uncalibrated visual servoing directly defines the error in the image space according to the image features of the manipulator end effector and target and estimates the Jacobian matrix or other nonlinear mapping models during the real-time operation of the system. Yoshimi and Allen proposed an online Jacobian matrix estimation method to solve the jacking task on the plane. In each servo cycle, the manipulator moves through tentative motion and records the image feature changes at the end of the manipulator in the image plane, and uses the parameter identification method. Estimating the current-image Jacobian matrix value, due to its sampling nature, the real-time performance of the system cannot be satisfied [12]. Wang and Liu [13] put forward a neural network algorithm of mixed genetic majorization back-propagation with an aim of model compositing the Jacobian matrix. Zhong put forward an algorithm that combines an Elman neural network and robust Kalman filter to determine the interaction matrix online, taking into account composite noise [14]. The authors of [15,16] both utilize the limit learning machine to evaluate the interaction matrix pseudo inverse for avoiding noise interference and matrix singularity. A six-degree-of-freedom visual servo control method based on image-moments was proposed to solve the problem of underwater vehicle dynamic positioning [17]. In [18], the authors solved the problem of quality inspection of remote radio units (RRUs) through image-based visual servo control. A depth-independent interaction matrix was designed that relates the depth information with the area of the region of interest surrounding the power port in the controller. The stability of the proposed visual servoing controller was analyzed by Lyapunov’s theory. However, the aforementioned studies do not fully consider the linear and nonlinear constraints in visual servoing. For example, if visibility constraints are not considered, the image feature points may be out of the field range of the camera during visual servo control. If the torque constraints are not considered, the joint torque given by the controller may exceed the physical limitation of the robot manipulator. These behaviors will lead to the loss of control quality and failure of the visual servoing task. Therefore, it is crucial to consider the system constraints in designing the controller of visual servoing. One strategy for dealing with constraints is through the path planning technique. In [19], a visual servoing technique based on optimized trajectory planning was proposed to optimize the control process and determine the trajectory parameters by minimizing image-feature deviation. In this technique, the majorization problem is converted into the convex problem, and the trajectory tracking problem for a four-degree-of-freedom manipulator is realized by combining it with the depth-estimation method. The artificial potential field was used in [20] to obtain the initial path, then the initial path was checked and corrected based on the genetic algorithm, and finally, the visual servoing task under the constraint environment was completed. The intelligent control approach has also been utilized to deal with the visual servo system constraint problem. A finite-time optimal control framework was used in [21] to solve feature correspondence and control problems simultaneously, and a small unmanned quadrotor was taken as an example to verify the theoretical research. In [22], a deep reinforcement learning-based visual servo approach was put forward to deal with the problem of feature loss, and an adaptive adjustment controller gain based on DRL was designed to enhance the visual servoing efficiency in the context of meeting the visual field constraints. An approach based on Lyapunov was presented in [23] to realize the semi-global asymptotic (exponential) regulation of the HVS system under unconstrained conditions, and the system’s local asymptotic stability was verified considering the driving-speed constraint. In [24], a method based on the sliding-mode control method was advanced to meet the constraints of robot vision servoing, and 2D and 3D cases were simulated separately. The robustness and feasibility of this approach were confirmed by using the traditional six-axis manipulator. Model predictive control (MPC) methods can solve the constraints explicitly and are often used in constrained systems. Ref. [25] developed a novel visual servo-based model predictive control method to steer a wheeled mobile robot (WMR) moving in a polar coordinate toward the desired target, where kinematic and dynamic constraints are both considered. A predictive controller of visual servo based on the traditional image Jacobian matrix was designed in [26]. A depth-independent Jacobian matrix was used in [27] to verify the robustness of MPC in eye-in-hand (ETH) and eye-to-hand (EIH) visual servoing systems without considering the robot dynamics model. In [28], a quasi-min-max MPC method for visual servoing was presented, in which the depth value is a fixed constant in the Jacobian matrix. In [29], a constrained predictive visual servoing method for fully actuated underwater robots was introduced.

Motivated by the works mentioned above, this paper aims to discuss the visual servoing problem of the robot manipulator considering system model uncertainty, system constraints and time-varying disturbances with the joint velocity unknown using a third-order sliding-mode observer-based model predictive control (MPC-TOSM) method.

The main contributions of this paper are given as follows:

• A new MPC control strategy is proposed based on the TOSM observer. Considering the nonlinear dynamics of the robot, the MPC controller output the optimal sequence of joint torque with visibility constraints and actuator constraints, and the TOSM observer is employed to observe the system centralized uncertainties together with joint velocities.
• Compared with the classical traditional control method in [30,31], the proposed strategy in this paper can achieve better servo performance with model errors and system constraints in a 2-DOF robot manipulator. Compared with the recent visual servo control method proposed in [27], the proposed strategy in this paper provides faster convergence speed and more accurate control with time-varying disturbances in both 2-DOF and 6-DOF robot manipulators.
• The global stability of the system when combining MPC controller and TOSM observer is proved by the Lyapunov stability theory.

These contributions are validated by various comparative experiments.

The residual parts of the paper are organized as follows. Section 2 presents the visual servo model. In Section 3, the sliding-mode observer and MPC controller of the image-based visual servoing are designed. Simulation analysis results are detailed in Section 4. Finally, simulation results that could verify the effectiveness of the proposed image-based visual servo control method are presented in Section 5.

2. Visual Servoing System Modeling

In the current section, the kinematics of visual servoing and robot dynamics modeling are introduced.

2.1. Kinematics of Visual Servoing Systems

In accordance with the positional correlation between robot and camera, a visual servoing system can be divided to eye-in-hand (EIH) and eye-to-hand (ETH) configurations, as shown in Figure 1 and Figure 2, respectively. In both of these configurations, we give a unified coordinate mapping correlation between the image and camera coordinate system. The feature points in the image coordinate system are defined as $s_n={(u_n,v_n)}^T$ and can be formulated by

$$s_n= \frac{1}{z^i}\left[\frac{m_1^T}{m_2^T}\right]\Omega \left[\frac{y_c}{1}\right],$$

(1)

where $z^i$ stands for the depth information of a feature point existing in the camera coordinates, $m_i^T$ the ith row of the camera parameter matrix which is unknown $M\in {\Re }^{3\times 4}$, and $\Omega \in {\Re }^{3\times 4}$ the coordinate transformation matrix which is determined by kinematics, expressed as $\Omega =\left(\begin{array}{cc}R& P\\ 0_{1\times 3}& 1\end{array}\right)$, where $R\in {\Re }^{3\times 3}$ is a matrix representing the rotation relationship and $P\in {\Re }^{3\times 1}$ is a matrix representing translation relationship, and $y_c$ represents the unknown feature point in Cartesian coordinates.

Different configurations correspond to different matrices. In the configuration of Figure 1, $M=D{\Omega }_e^c$, $\Omega ={\left({\Omega }_e^b\right)}^{-1}$. In the configuration of Figure 2, $M=D{\Omega }_b^c$, $\Omega ={\Omega }_e^b$. The above symbols are defined as follows.

$D\in {\Re }^{3\times 4}$ is a matrix that involves the camera’s internal parameters, ${\Omega }_e^c$ stands for the coordinate transformation matrix of the robot base framework relative to the camera framework, ${\Omega }_e^b$ represents the coordinate transformation matrix of the robot base relative to the end-effector framework relative to the robot base framework, and it can be gathered from the forward kinematics of robot, and ${\Omega }_b^c$ the coordinate transformation matrix of the robot base framework relative to the camera framework.

The depth $z^i$ can be expressed as

$$z^i=m_3^T\Omega \left[\frac{y_c}{1}\right].$$

(2)

Differentiating the above formula relative to time, we can acquire the relationship between the joint velocity and variation of the feature point with time as

$${\dot{s}}_n=\frac{1}{z^i}C\dot{q},$$

(3)

where $\dot{q}\in R^{n\times 1}$ denotes the velocity of robot joints, n the freedom degrees of the robot, and $C\in R^{2\times n}$ the image Jacobian matrix where the depth is independent, which is expressed as

$$C=\left[\begin{array}{c}{m}_1^T-u_n{m}_3^T\\ m_2^T-v_n{m}_3^T\end{array}\right]\frac{1}{\partial q}\left(\Omega \left[\begin{array}{c}{y}_c\\ 1\end{array}\right]\right).$$

(4)

The unknown robot kinematic parameters together with the camera parameters identified the components of the depth-independent image Jacobian matrix. In the above formula, both the Jacobian matrix and depth information in it are nonlinear. Therefore, the following properties are introduced herein, and the nonlinear elements are linearized by the regression matrix and unknown parameter vector.

Property 1.

The product $D\eta$ for a vector $\eta \in {\Re }^{n\times 1}$ can be parameterized linearly as below equations:

$$D\eta =B\left(s_n, \eta , q\right)\theta ,$$

(5)

$$d^T\eta =b\left(s_n, \eta \right)\theta ,$$

(6)

where $B\left(s_n, \eta , q\right)$, $b\left(s_n, \eta \right)$ denote the regression matrices, while θ represents the associated unknown parameter vector identified through the products of the unknown robot kinematic parameters together with the unknown camera parameters.

Property 2.

The depth $z^i$ can be expressed as the unknown parameters linear form as below

$$z_i=Y\left(q\right){\theta }_z,$$

(7)

where $Y\left(q\right)$ represents the regression matrix, while ${\theta }_z$ stands for the unknown parameters influenced via the products of an unknown camera parameters together with feature position.

2.2. Robot Dynamics

The dynamic control system of robot manipulators can be regarded as a kind of second-order uncertain nonlinear system. The dynamic model of the robot manipulator is given in Lagrangian form as follows:

$$H\left(q\left(t\right)\right)\ddot{q}\left(t\right)+K\left(q\left(t\right), \dot{q}\left(t\right)\right)\dot{q}\left(t\right)+G\left(q\left(t\right)\right)=\tau \left(t\right)+{\tau }_e\left(t\right),$$

(8)

where $H\left(q\left(t\right)\right)\in {\Re }^{n\times n}$ denotes the inertia matrix, $K\left(q\left(t\right),\dot{q}\left(t\right)\right)$ denote the centripetal and Coriolis torque matrix, $G\left(q\left(t\right)\right)\in {\Re }^{n\times n}$ the gravitational torques, $\tau \left(t\right)\in {\Re }^{n\times 1}$ the input torque vector, and ${\tau }_e\left(t\right)\in {\Re }^{n\times 1}$ the slowly varying external disturbances.

The modeling deviation between the real and mathematical models $\Delta H_0\left(q\left(t\right)\right)$, $\Delta K_0\left(q\left(t\right), \dot{q}\left(t\right)\right)$, and $\Delta G_0\left(q\left(t\right)\right)$ can be expressed as

$$\Delta H\left(q\left(t\right)\right)=H\left(q\left(t\right)\right)-H_0\left(q\left(t\right)\right),$$

(9)

$$\Delta K\left(q\left(t\right), \dot{q}\left(t\right)\right)=K\left(q\left(t\right), \dot{q}\left(t\right)\right)-K_0\left(q\left(t\right), \dot{q}\left(t\right)\right),$$

(10)

$$\Delta G\left(q\left(t\right)\right)=G\left(q\left(t\right)\right)-G_0\left(q\left(t\right)\right),$$

(11)

respectively, where $H_0\left(q\left(t\right)\right)$, $K_0\left(q\left(t\right), \dot{q}\left(t\right)\right)$, and $G_0\left(q\left(t\right)\right)$ denote the corresponding nominal parts of the model. Then, Equation (8) can be written as

$$\begin{array}{cc}\hfill H_0\left(q\left(t\right)\right)\ddot{q}\left(t\right)& + K_0(q\left(t\right), \dot{q}\left(t\right))\dot{q}\left(t\right)+G_0\left(q\left(t\right)\right)=\tau \left(t\right)+{\tau }_e\left(t\right)\hfill \\ \hfill & -\Delta H\left(q\left(t\right)\right)\ddot{q}\left(t\right)-\Delta K(q\left(t\right), \dot{q}\left(t\right))\dot{q}\left(t\right)-G\left(q\left(t\right)\right)\hfill \end{array}$$

(12)

By defining variables $F(q, \dot{q}, t)$,

$$F(q, \dot{q}, t)={\tau }_e\left(t\right)-\Delta H\left(q\left(t\right)\right)-\Delta K\left(q\left(t\right), \dot{q}\left(t\right)\right)-\Delta G\left(q\left(t\right)\right),$$

(13)

Equation (12) can be rewritten as

$$H_0\left(q\left(t\right)\right)\ddot{q}\left(t\right)+K_0\left(q\left(t\right), \dot{q}\left(t\right)\right)\dot{q}\left(t\right)+G_0\left(q\left(t\right)\right)=\tau \left(t\right)+F(q, \dot{q}, t).$$

(14)

The joint acceleration can be expressed as

$$\begin{array}{c}\hfill \ddot{q}\left(t\right)=-H_0{\left(q\left(t\right)\right)}^{-1}{K}_0\left(q\left(t\right), \dot{q}\left(t\right)\right)\dot{q}\left(t\right)-H_0{\left(q\left(t\right)\right)}^{-1}{G}_0\left(q\left(t\right)\right)\\ \hfill +H_0{\left(q\left(t\right)\right)}^{-1}\tau \left(t\right)+H_0{\left(q\left(t\right)\right)}^{-1}F(q, \dot{q}, t).\end{array}$$

(15)

Considering the uncertain parameters and external disturbances, the approximate dynamic model can be defined as

$$\ddot{q}\left(t\right)=f\left(q\left(t\right), \dot{q}\left(t\right)\right)+\Delta f,$$

(16)

where $f\left(q\left(t\right), \dot{q}\left(t\right)\right)=-H_0{\left(q\left(t\right)\right)}^{-1}{K}_0\left(q\left(t\right), \dot{q}\left(t\right)\right)\dot{q}\left(t\right)-H_0{\left(q\left(t\right)\right)}^{-1}{G}_0\left(q\left(t\right)\right)+H_0{\left(q\left(t\right)\right)}^{-1}\tau \left(t\right)$, $\Delta f=H_0{\left(q\left(t\right)\right)}^{-1}F(q, \dot{q}, t)$.

3. Third-Order Sliding-Mode Observer

As a multi-variable complicated nonlinear system with multi-variable and strong coupling characteristics, the manipulator is easily affected by various internal and external factors during operation, and its own joint velocity measurement is often accompanied by noise. At the same time, it is difficult to ensure the modeling accuracy and hand-eye calibration accuracy of the manipulator in the visual servo system. Therefore, in the current section, a third-order sliding-mode observer is introduced to estimate the system speed and centralized uncertainty, and then the estimated information is input to the model predictive controller as a feed-forward signal. The robot dynamics model (15) is reconstructed into state space equations.

Variables $x_1=q$, $x_2=\dot{q}$ are proposed to rewrite the kinetic model as

$${\dot{x}}_1=x_2$$

(17)

$${\dot{x}}_2=f\left(x\right)+g\left(x\right)u\left(t\right)+\Pi \left(x, t\right)$$

(18)

where $f\left(x\right)=H_0{\left(q\left(t\right)\right)}^{-1}\left(-K_0\left(q\left(t\right), \dot{q}\left(t\right)\right)\dot{q}\left(t\right)-G_0\left(q\left(t\right)\right)\right)$, $g\left(x\right)=H_0{\left(q\left(t\right)\right)}^{-1}$,

$u\left(t\right)=\tau \left(t\right)$, and $\Pi \left(x, t\right)=H_0{\left(q\left(t\right)\right)}^{-1}F(\dot{q}, q, t)$.

The following TOSM observer is presented:

$${\dot{\widehat{x}}}_1={\sigma }_1{|x_1-{\widehat{x}}_1|}^{2/3}sign\left(x_1-{\widehat{x}}_1\right)+{\widehat{x}}_2$$

(19)

$${\dot{\widehat{x}}}_2=f\left(\widehat{x}\right)+g\left(\widehat{x}\right)\tau \left(t\right)+{\sigma }_2{\left|x_1-{\widehat{x}}_1\right|}^{1/3}sign\left(x_1-{\widehat{x}}_1\right)-\widehat{y}$$

(20)

$$\dot{\widehat{y}}=-{\sigma }_3\mathrm{sign}\left(x_1-{\widehat{x}}_1\right)$$

(21)

where $\widehat{x}$ denotes the observation value of the disturbance observer and ${\sigma }_1$, ${\sigma }_2$, ${\sigma }_3$ denotes the suitable observer gain value. Subtracting (17) and (18) to (19)–(21), and defining the system state’s estimation errors $\tilde{x}=x-\widehat{x}$, we have

$${\dot{\tilde{x}}}_1=-{\sigma }_1{\left|{\tilde{x}}_1\right|}^{2/3}sign\left({\tilde{x}}_1\right)+{\tilde{x}}_2$$

(22)

$${\dot{\tilde{x}}}_2=-{\sigma }_2{\left|{\tilde{x}}_1\right|}^{1/3}sign\left({\tilde{x}}_1\right)+\widehat{\Pi }\left(x,\widehat{x},t\right)+\widehat{y}$$

(23)

$$\dot{\widehat{y}}=-{\sigma }_{3}sign\left({\tilde{x}}_1\right)$$

(24)

By introducing variables ${\widehat{y}}_0=\widehat{\Pi }\left(x,\widehat{x},t\right)+\widehat{y}$, the sliding-mode observer becomes

$${\dot{\tilde{x}}}_1=-{\sigma }_1{\left|{\tilde{x}}_1\right|}^{2/3}sign\left({\tilde{x}}_1\right)+{\tilde{x}}_2$$

(25)

$${\dot{\tilde{x}}}_2=-{\sigma }_2{\left|{\tilde{x}}_1\right|}^{1/3}sign\left({\tilde{x}}_1\right)+{\widehat{y}}_0$$

(26)

$${\dot{\widehat{y}}}_0=-{\sigma }_{3}sign\left({\tilde{x}}_1\right)+\dot{\widehat{\Pi }}\left(x, \widehat{x}, t\right)$$

(27)

The estimation errors (25)–(27) is the standard form of the second-order robust accurate differentiator, and its finite-time stability has been successfully proved in [32].

4. MPC Controller Design

In the preceding section, joint velocities and system centralized uncertainty were effectively estimated by designing an SMO. In this section, an MPC-based controller is designed for solving system constraints and model uncertainty, so that image feature points converge to expected values. Figure 3 reflects the overall control scheme. The control instruction is the joint torque of the robot manipulator and the output is the joint angle q. The system uncertainty is estimated using an SMO in the feedforward loop.

The model predictive controller obtains the optimal control input for each control cycle by solving a constrained optimization problem. The design of the model predictive controller is described below.

The visual servo system model is represented by the following discrete-time model:

$$\left(\begin{array}{c}{s}_n\left(k+1\right)\\ \dot{q}\left(k+1\right)\end{array}\right)=\left(\begin{array}{c}{s}_n\left(k\right)\\ \dot{q}\left(k\right)\end{array}\right)+\left(\begin{array}{c}\frac{1}{z^i}C\left(k\right)\dot{q}\left(k\right)T_d\\ (f\left(q\left(t\right), \dot{q}\left(t\right)\right)+\Delta f)T_d\end{array}\right),$$

(28)

where $T_d$ denotes the sampling period. The servoing system control inputs can be counted through dealing with the below constrained finite-time optimization problem:

$$\begin{array}{c}\hfill J\left(k\right)=\sum _{i=1}^{N_p}{∥\overline{s}(k+i)-s_d∥}_{Q_F}^2+\sum _{i=1}^{N_p}{∥\stackrel{\overline{\dot{}}}{q}\left(k+i\right)∥}_{Q_G}^2\\ \hfill +\sum _{i=1}^{N_c}{∥\overline{\tau }\left(k+i-1\right)∥}_R^2\end{array}$$

(29)

where $\overline{s}\left(k+i\right)$, $\stackrel{\overline{\dot{}}}{q}\left(k+i\right)$, and $\overline{\tau }\left(k+i-1\right)$ denote the image which is predicted by the MPC, joint velocity, and output torque, respectively, to be modified at time k. $s_d$ denotes the desired coordinates of image features. $N_p$ denotes the prediction step length, horizon and $N_c$ the control step length, which is usually $N_c\ge N_p$. The cost function $J\left(k\right)$ consists of three parts: image-feature bias, joint velocity, and torque. $Q_F\ge 0$, $Q_G\ge 0$ and R represent the associated weighting matrices.

The MPC-based visual servo system with constraints can be described as

$$\Psi \underset{\dot{q}\left(k\right)}{minJ\left(k\right)}.$$

(30)

To make sure that the feature points are in the camera’s visual field, the visibility constraint is defined as

$$s_n^{min}⩽s_n\left(k\right)⩽s_n^{max},$$

(31)

where $s_n^{min}$ and $s_n^{max}$ denote the range of coordinates, respectively, of the image plane.

The torque constraint is expressed as

$${\tau }^{min}⩽\tau \left(k\right)⩽{\tau }^{max},$$

(32)

where ${\tau }^{min}$ and ${\tau }^{max}$ stand for the boundary value of the force torque. The sequential quadratic programming (SQP) algorithm is an algorithm that converts complex nonlinear constrained optimization problems into simpler quadratic programming (QP) problems. Therefore, this paper adopts the SQP algorithm for the optimal control problem.

The Lyapunov function is defined as as follows:

$$V\left(k\right)=\sum _{i=1}^N{∥\overline{s}\left(k+i\right)-s_d∥}_{Q_F}^2+\sum _{i=1}^N{∥\stackrel{\overline{\dot{}}}{q}\left(k+i\right)∥}_{Q_G}^2+\sum _{i=1}^N{∥\overline{\tau }\left(k+i-1\right)∥}_R^2$$

(33)

Here, for the sake of brevity, the control time domain and the prediction time domain are made equal, that is $N_P=N_C=N.$ Let

$$R\left(i\right)={∥\overline{s}\left(k+i\right)-s_d∥}_{Q_F}^2+{∥\stackrel{\overline{\dot{}}}{q}\left(k+i\right)∥}_{Q_G}^2+{∥\overline{\tau }\left(k+i-1\right)∥}_R^2, i\in I\left[1,N\right]$$

(34)

then

$$\begin{array}{cc}\hfill V(k+1)& \left.=\underset{\tau }{min}\left\{\sum _{i=1}^N{∥\overline{s}(k+i+1)-s_d∥}_{Q_F}^2+\sum _{i=1}^N\parallel \stackrel{\overline{\dot{}}}{q}{(k+i+1)\parallel }_{Q_G}^2+\sum _{i=1}^N{\parallel \overline{\tau }(k+i)\parallel }_R^2\right\}\right]\hfill \\ & =\underset{\tau }{min}\left\{\sum _{i=1}^{N}R\left(i\right)-R\left(1\right)+R\left(N\right)\right\}\hfill \\ & =-R\left(1\right)+\underset{\tau }{min}\left\{\sum _{i=1}^{N}R\left(i\right)+R\left(N\right)\right\}\hfill \\ & \le -R\left(1\right)+V\left(k\right)+\underset{\tau }{min}\left\{\sum _{i=1}^{N}R\left(N\right)\right\}\hfill \end{array}$$

(35)

and $R\left(1\right)\ge 0$, ${min}_{\tau }\left\{{\sum }_{i=1}^{N}R\left(N\right)\right\}=0$. As a result, $V(k+1)\le V\left(k\right)$, namely, $V(k+1)-V\left(k\right)\le 0$.

Therefore, the stability of the closed-loop system is proven.

5. Simulation Results

In order to verify the effectiveness and robustness of the MPC-TOSM method, the comparative simulations are carried out in this section.

5.1. Comparative Simulations with Model Uncertainty

First of all, the comparative simulations of visual servoing with model uncertainty are carried out in this part. Three control strategies used for the comparative simulations are shown below.

• Case 1: The control strategy is the traditional visual servo control method in [30].
• Case 2: The control method is MPC and the observer is a traditional SMO in [31].
• Case 3: The control strategy is the MPC-TOSM method proposed in this paper.

The simulation object adopts the two-degree-of-freedom (2-DOF) manipulator from [27]. In the parameters of the 2-DOF robot manipulator, for the first and second links, their lengths, masses, mass centers and inertia are expressed as $l_1$ and $l_2$, $m_1$ and $m_2$, $l_{c1}$ and $l_{c2}$, $I_1$ and $I_2$, respectively. In the initial rough parameters of the 2-DOF robot manipulator, for the first and second links, their lengths, masses, mass centers and inertia are expressed as ${\widehat{l}}_1$ and ${\widehat{l}}_2$, ${\widehat{m}}_1$ and ${\widehat{m}}_2$ and $l_{c2}$, ${\widehat{l}}_{c_1}$ and ${\widehat{l}}_{c_2}$, ${\widehat{I}}_1$ and ${\widehat{I}}_2$, respectively. The specific values are shown in

Table 1. Parameters of the 2-DOF robot manipulator.

ith Joint	${\mathit{l}}_{\mathit{i}}$ (m)	${\mathit{m}}_{\mathit{i}}$ (kg)	${\mathit{l}}_{{\mathit{c}}_{\mathit{i}}}$ (m)	${\mathit{I}}_{\mathit{i}}$ (kgm${}^2$)
1	0.18	23.9	0.091	1.27
2	0.15	4.44	0.105	0.24

and

Table 2. Initial rough parameters of the 2-DOF robot manipulator.

ith Joint	${\widehat{\mathit{l}}}_{\mathit{i}}$ (m)	${\widehat{\mathit{m}}}_{\mathit{i}}$ (kg)	${\widehat{\mathit{l}}}_{{\mathit{c}}_{\mathit{i}}}$ (m)	${\widehat{\mathit{I}}}_{\mathit{i}}$ (kgm${}^2$)
1	0.05	20	0.05	1.0
2	0.05	4	0.05	0.2

.

Furthermore, we assume that the exact camera parameters are unknown in the comparative simulations. The real camera parameters and initial rough camera parameters are shown in

Table 3. Camera parameters.

	Focal Length (m)	Coordinates of the Camera Stagnation Point in the Image Frame (pixels)	Scaling Factors along u Axis and v Axis (pixels/m)
Real camera parameters	0.0005	(646, 482)	(269,167, 267,778)
Initial rough camera parameters	0.0005	(500, 500)	(250,000, 250,000)

. In this paper, the focal length is denoted by f.The real and initial rough coordinates of the camera stagnation point are $\left(u_0, v_0\right)$ and $\left({\widehat{u}}_0, {\widehat{v}}_0\right)$, which are in the image frame. The real and initial rough scaling factors along the u and v axes are $\left(k_u, k_v\right)$ and $({\widehat{k}}_u, {\widehat{k}}_v)$. The range of the u axis are $u_{min}=0$, $u_{max}=1292$ pixels. The range of the v axis are $v_{min}=0$, $v_{max}=964$ pixels. The minimum and maximum torques of the actuator are −t10 Nm and 10 Nm. Both visibility constraints and actuator constraints manifest in the visual servoing task. The real and initial rough camera parameters are shown in Table 3.

The Cartesian coordinates of Feture Point 1 to Feture Point 4 that relative to the robot base are ${\left(0.0618,0.0516,0.3\right)}^T$ m, ${\left(-0.0159,0.0516,0.3\right)}^T$ m, ${\left(0.0618,0.023,0.3\right)}^T$ m, and ${\left(-0.0159,0.023,0.3\right)}^T$ m, respectively. The initial 2D coordinates of the image feature points are ${\left(500,300\right)}^T$ pixels, ${\left(700,300\right)}^T$ pixels, ${\left(500,425\right)}^T$ pixels, and ${\left(700,425\right)}^T$ pixels. The desired 2D coordinates of the image-feature points are ${\left(425,525\right)}^T$ pixels, ${\left(425,325\right)}^T$ pixels, ${\left(625,525\right)}^T$ pixels, and ${\left(625,325\right)}^T$ pixels. The initial states of robot manipulator are chosen as $q_1\left(0\right)=q_2\left(0\right)=\frac{\pi }{6}$, ${\dot{q}}_1\left(0\right)={\dot{q}}_2\left(0\right)=0$. The coordinate transformation matrix of the end-effector framework relative to the camera framework $H_c^e$ is expressed as

$$H_c^e=\left[\begin{array}{cccc}-1& 0& 0& 0.01\\ 0& -1& 0& -0.02\\ 0& 0& 1& 0.015\\ 0& 0& 0& 1\end{array}\right].$$

(36)

In the experiments, setting the initial rough estimation of camera extrinsic parameter matrix ${\widehat{H}}_c^e$ as

$${\widehat{H}}_c^e=\left[\begin{array}{cccc}-1& 0& -0.25& 0.005\\ 0& -1& 0& 0\\ 0& 0& 0.95& 0.01\\ 0& 0& 0& 1\end{array}\right].$$

(37)

The parameters in the third-order sliding mode are chosen as ${\sigma }_1=2.2L^{1/3}$, ${\sigma }_2=2L^{2/3}$, ${\sigma }_3=L$, with $L=6$. The parameters in the MPC controller are chosen as $Q_F=30I_{8\times 8}$, $Q_G=0I_{2\times 2}$, and $R=5I_{2\times 2}$, $N_p=10$, and $N_c=8$.

With an aim of comparing the control effects of several methods, the same initial positions of the image features are set, which are represented by circles in Figure 4, and the same desired positions of the image features are represented by boxes. Figure 4 displays the trajectories of features in the image plane in Case 1, 2, and 3 control strategies, respectively. The trajectories of Feture Point 1 to Feture Point 4 are shown in red, green, blue and black, respectively. In Case 1, the rough initial parameters make the classical image-based visual servoing invalid, the feature points deviate from the desired positions, and the visual servoing task cannot be completed. In Case 2 and 3, the image features both reach the desired final position. However, the trajectory of image features fluctuates during the convergence process in Case 2. In Case 3, the trajectory of image features converges to the desired position smoothly and quickly. Figure 4c indicates that the proposed method can obtain satisfactory control effects without joint-angle measurement in the company of visibility constraints, actuator constraints, and system model uncertainty. Figure 5 displays the feature response deviations in the image plane in Case 1, 2, and 3 control strategies, respectively. The deviation is defined as the difference between the current feature point coordinates and the desired point coordinates. In Figure 5b,c, it can be seen that the stabilization time of the control system is about 10 s in Case 3, faster than in Case 2 (15 s). In Figure 6, it can be observed that the traditional SMO control strategy leads to the chattering phenomenon of the joint torques, while the proposed MPC-TOSM method can suppress the chattering phenomenon effectively and better complete the visual servo goals. Figure 7 and Figure 8 display the estimation errors of the joint velocities together and the uncertainty estimation errors of the observers in Case 2 and Case 3. We can see that the TOSM observer has higher estimation accuracy than the traditional sliding mode observer. Higher estimation accuracy brings better control performance to the control system.

5.2. Comparative Simulations with Time-Varying External Disturbances

The simulation results in the previous section prove that the visual servo controller proposed in this paper can effectively realize the visual servoing by considering system constraints and model uncertainty without measurements of joint velocity. However time-varying disturbances always exist in practical application scenarios, such as joint torque friction. Therefore, we discuss the control effect of the proposed method in visual servoing tasks with time-varying external disturbances in this subsection. The comparative simulations are divided into two parts, which are conducted on the 2-DOF robot manipulator and 6-DOF robot manipulator, respectively. The comparative simulations are carried out between the control strategies proposed in this paper and in [27], which is called Case 4 in this paper.

5.2.1. 2-DOF Robot Manipulator

The parameters of the robot manipulator and camera are the same as in the previous section. The special condition added is that the time-varying external disturbances were given as ${\tau }_d={\left[4sin\left(\frac{3\pi }{7}t\right), 4sin\left(\frac{3\pi }{7}t\right)\right]}^T$, which is time-varying.The simulation results are as follows. Figure 9 displays the trajectories of the image features with time-varying external disturbances in Case 3 and Case 4, respectively. The trajectories of Feture Point 1 to Feture Point 4 are shown in red, green, blue and black, respectively. Although there exist fluctuations and turn-back in the trajectories, the image features both respond from the initial positions to the desired positions in Case 3 and Case 4 successfully. However as we can see from Figure 9 and Figure 10, the convergence speed of image errors in Case 3 (about 10 s) is faster than in Case 4 (about 15 s). At the same time, the absolute value of steady-state image errors in Case 3 (about 7 pixels) is smaller than errors in Case 4 (about 18 pixels), as shown in Figure 10. Figure 11 shows the response of the torques under the two strategies. Compared with Case 3, Case 4 brings more frequent ripple to the joint torques as shown in Figure 11, which means a better quality of control. It can also be considered that the input torques are more stable in Case 3. Figure 12 displays the estimation errors of joint velocities in Case 3 and Case 4. It can be seen in Figure 12 that the joint velocities are more accurately estimated by the TOSM observer in Case 3. Figure 13 displays the estimation errors of uncertainties in Case 3. The estimation of uncertainties is given accurately by the TOSM observer in Case 3, then feed back to the controller. While the observer in Case 4 does not have the ability to estimate the uncertainties. As shown in Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13, it can be concluded that the control strategy of Case 3 is superior to that of Case 4 in 2-DOF robot manipulator visual servoing with time-varying external disturbances.

5.2.2. 6-DOF Robot Manipulator

The 6-DOF robot manipulators are most widely used in practical applications. Therefore, it is necessary to verify the effectiveness of the MPC-TOSM method on a 6-DOF robot manipulator. The dynamic parameters and model of 6-DOF robot manipulator are given in [33]. The initial rough camera parameters are given the same settings as above experiments. The Cartesian coordinates of Feture Point 1 to Feture Point 4 that relative to the robot base are ${\left(0.6237,0.4651,2\right)}^T$ m, ${\left(-0.16,0.4651,2\right)}^T$ m, ${\left(0.6237,0.25,2\right)}^T$ m, and ${\left(-0.16,0.25,2\right)}^T$ m, respectively. The initial 2D coordinates of the image feature points are ${\left(600,300\right)}^T$ pixels, ${\left(1000,300\right)}^T$ pixels, ${\left(600,425\right)}^T$ pixels, and ${\left(1000,425\right)}^T$ pixels. The desired 2D coordinates of the image-feature points are ${\left(450,525\right)}^T$ pixels, ${\left(450,325\right)}^T$ pixels, ${\left(850,525\right)}^T$ pixels, and ${\left(850,325\right)}^T$ pixels. The initial states of robot manipulator are chosen as $q_1\left(0\right)=q_2\left(0\right)=0$, $q_3\left(0\right)=q_4\left(0\right)=\frac{\pi }{6}$, $q_5\left(0\right)=q_6\left(0\right)=\frac{\pi }{12}$, ${\dot{q}}_1\left(0\right)={\dot{q}}_2\left(0\right)={\dot{q}}_3\left(0\right)={\dot{q}}_4\left(0\right)={\dot{q}}_5\left(0\right)={\dot{q}}_6\left(0\right)=0$. The coordinate transformation matrix of the end-effector framework relative to the camera framework $H_c^e$ is expressed as

$$H_c^e=\left[\begin{array}{cccc}-1& 0& 0& 0.01\\ 0& -1& 0& -0.02\\ 0& 0& 1& 0.015\\ 0& 0& 0& 1\end{array}\right].$$

(38)

In the experiments, setting the initial rough estimation of camera extrinsic parameter matrix ${\widehat{H}}_c^e$ as

$${\widehat{H}}_c^e=\left[\begin{array}{cccc}-0.98& 0& -0.25& 0.005\\ 0& -0.96& 0& 0\\ 0& 0& 0.95& 0.01\\ 0& 0& 0& 1\end{array}\right].$$

(39)

The parameters in the third-order sliding mode are chosen as ${\sigma }_1=3L^{1/3}$, ${\sigma }_2=2L^{2/3}$, ${\sigma }_3=2L$, with $L=6$. The parameters in the MPC are chosen as $Q_F=25I_{8\times 8}$, $Q_G=0.2I_{6\times 6}$, and $R=1I_{6\times 6}$, $N_p=4$, and $N_c=2$. The minimum and maximum torques of the actuator are −80 Nm and 80 Nm. The system uncertainty was given as ${\tau }_d= {\left[5sin\left(\frac{2\pi }{5}t\right), 5sin\left(\frac{2\pi }{5}t\right), 2sin\left(\frac{2\pi }{5}t\right), sin\left(\frac{2\pi }{5}t\right), sin\left(\frac{2\pi }{5}t\right), sin\left(\frac{2\pi }{5}t\right)\right]}^T$, which is time-varying.

The simulation results are given as follows:

Figure 14 displays the trajectories of image features in Case 3 and Case 4 with time-varying external disturbances, respectively. The trajectories of Feture Point 1 to Feture Point 4 are shown in red, green, blue and black, respectively. Although there exist fluctuations and turn-back in the trajectories, the image features both respond from the initial positions to the desired positions successfully in Case 3 and Case 4. However as we can see from Figure 14 and Figure 15, the convergence speed of image errors in Case 3(about 8s)is faster than in Case 4 (about 10s). At the same time, the absolute value of steady-state image errors in Case 3 (about 10 pixels) is smaller than errors in Case 4 (about 20 pixels), as shown in Figure 14 and Figure 15. Figure 16 shows the response of the torques under the two strategies. Compared with Case 3, Case 4 brings more frequent ripple to the joint torques as shown in Figure 16. It can also be considered that the input torques are more stable in Case 3. Figure 17 displays the estimation errors of joint velocities with time-varying external disturbances in Case 3 and Case 4. It can be seen that the joint velocities are estimated more accurately by the TOSM observer in Case 3. Figure 18 displays the estimation errors of uncertainties with time-varying external disturbances in Case 3. The estimation of uncertainties is given accurately by the TOSM observer in Case 3, then fed back to the controller. According to Figure 14, Figure 15, Figure 16, Figure 17 and Figure 18, it can be concluded that the control strategy of Case 3 is superior to that of Case 4 in 6-DOF robot manipulator visual servoing.

To sum up, according to the simulation results and discussions above, the proposed MPC-TOSM strategy is a satisfactory method to deal with IBVS tasks by considering system constraints, model uncertainty and time-varying external disturbances with joint velocity unknown.

6. Conclusions

For dealing with the image-based visual servoing problem of robot manipulators with model uncertainty, system constraints, unmeasurable joint angles, and time-varying disturbances, a new MPC method on the basis of a third-order sliding-mode (TOSM) observer is presented in this current research. A TOSM observer is designed for evaluating the joint velocity and system centralized uncertainty simultaneously, and feed-forward the estimated value to the model predictive controller. Using an MPC strategy, the optimal input torque is calculated by decreasing the cost function on the basis of image deviation as far as possible while fully considering the visibility constraints and actuator constraints. Simulation results verify the effectiveness of the proposed MPC-TOSM visual servoing control strategy. In planned subsequent work, we will study the visual servoing problem of tracking dynamic feature points and fully consider the time-delay sensitivity of visual feature feedback in the dynamic tracking process.