Discrete-Time Visual Servoing Control with Adaptive Image Feature Prediction Based on Manipulator Dynamics

Abstract

In this paper, a practical discrete-time control method with adaptive image feature prediction for the image-based visual servoing (IBVS) scheme is presented. In the discrete-time IBVS inner-loop/outer-loop control architecture, the time delay caused by image capture and computation is noticed. Considering the dynamic characteristics of a 6-DOF manipulator velocity input system, we propose a linear dynamic model to describe the motion of a robot end effector. Furthermore, for better estimation of image features and smoothing of the robot’s velocity input, we propose an adaptive image feature prediction method that employs past image feature data and real robot velocity data to adopt the prediction parameters. The experimental results on a 6-DOF robotic arm demonstrate that the proposed method can ensure system stability and accelerate system convergence.

1. Introduction

Vision perception-based robot control is of great significance in intelligent robots. In the fields of autonomous mobile robots [1,2,3,4], humanoid robots [5,6], aerial vehicles [7,8], etc., visual perception is used to improve robot autonomy, environmental adaptability, and intelligence. In addition, in the field of industrial applications, 6-DOF (six degrees of freedom) manipulators with visual servo control are widely used for tasks such as grasping, inserting, and assembling [9,10,11]. There are three visual servo control schemes reported in the literature, namely the (1) position-based visual servoing scheme (PBVS), (2) the image-based visual servoing scheme (IBVS), and (3) the hybrid visual servoing scheme (HVS) [12,13,14]. In the PBVS scheme, the features of the target object are associated with the geometric model of the object to calculate its pose. On the contrary, IBVS does not estimate the pose of the target object but directly uses image features on its controller to control robot’s velocity. Therefore, it does not require precise camera calibration and geometric object models [13,15].

In classical IBVS, the points, lines, and regions of objects are often employed as image features due to their simple recognition and robustness to image noise [16,17]. Then, the controller algorithm is designed using the image Jacobian matrix to convert the image error into the desired robot motion velocity to calculate the control signal. Generally, the proportional control law is employed in classical IBVS to complete the positioning tasks [18]. Although these methods are applicable to practical visual servo tasks, they rely on the geometric features of the object, and the calculation of their Jacobian matrix requires the depth value of the target object. In addition, these methods may cause singularity problems with the image Jacobian matrix [19]. In order to apply the IBVS method to scenes where the target object is prone to deformation, many researchers have explored new image features. Image moments offer a way to describe image features without relying on the geometric characteristics of the target object. Therefore, visual servo methods based on image moments have been widely studied. F. Chaumet proposed a visual servo method based on the general features of image moments [20]. In this study, the interaction matrices of various analytical forms based on image moments were determined. Furthermore, O. Tahri used a normalization method to process image moments and achieved decoupling control of the system by combining the scale-invariant image moments [21]. Another method called direct visual servoing, which uses features such as image histograms [22], Gaussian mixtures [23,24], and wavelets [25], has also received attention from some scholars. To avoid calculating object depth values, Liu et al. [26] constructed a Jacobian matrix that no longer requires depth information by extracting the depth variable from the interaction matrix. Ali et al. [27] proposed a depth-free IBVS method that uses the area of the power port to represent the depth information. To solve the singularity problems, many studies have been devoted to image moment-based visual servoing methods [28,29].

In addition to designing image features, extensive studies have also been conducted on visual servo controllers to achieve better control performance. Xie et al. [9] proposed a switching controller to keep image features within the view range of the camera. Li et al. [30] presented an enhanced controller based on a hybrid proportional derivative control and sliding mode control (PD-SMC) method to ensure that the robot could quickly track target objects and handle uncertainties in depth. Furthermore, considering limitations in terms of the robot’s workspace and velocity, in addition to visibility constraints and parameter uncertainties, visual servoing methods based on model predictive control (MPC) have been proposed [31]. Allibert et al. [32] formulated the constraints as state constraints, as well as input and output constraints. In Hajiloo’s study [33], a robust model predictive control-based IBVS controller was presented, which reduces computational costs and enables MPC to be implemented in a fast system.

Although the above methods solve the problems of classical IBVS by designing new features and different controllers, most of them are designed in continuous-time cases without considering discrete-time system performance [34]. In addition, since robots employed in IBVS are not ideal rigid systems, their dynamic models should also be considered. This will benefit us for predicting image features in discrete-time visual servoing systems. Conticelli et al. derived an interaction model between a camera and an object in discrete time and implemented positioning tasks by designing a visual controller in discrete time [35]. Bjerkeng et al. [36] studied the stability of robot control based on sensor date feedback in discrete time. The authors derived the stability results of visual servoing considering the input/output time delays and the robot dynamics. The result of small-gain proportional control was obtained for the high-delay problem of sensor feedback. He [37] designed an input-mapping method to achieve stability in a discrete visual servo system. Past image errors and camera motion data were employed to calculate the uncertain part of the model. Although these methods have developed visual servo systems from a discrete perspective and designed controllers to ensure system stability, they have not improved system performance by increasing the frequency of image information feedback.

Unlike continuous visual servo methods, this paper describes a visual servo system in discrete time. Furthermore, by analyzing the inner–outer loop control architecture of a visual servo system, the influence of sampling frequency is presented. In order to improve the performance of discrete visual servo systems, this paper develops a discrete-time visual servo controller with adaptive image feature prediction. In the proposed method, we consider both image processing delay and robot dynamics in the system. Past image feature data and robot end-effector velocity data are used to adaptively adjust predictor parameters. The main contributions of this study are summarized as follows:

(1)

We present a visual servoing method from the perspective of discrete-time analysis. Based on the visual servo inner–outer loop control architecture, the impact of low-frequency visual feedback on system performance is analyzed.

(2)

A linear dynamic model is investigated to describe the movement of the robot under the velocity control mode. By using this model, the actual velocity of the robot end effector can be more accurately estimated.

(3)

An adaptive image feature prediction method is proposed to predict the positions of image features during the image sampling and processing time. Past image feature data and real robot velocity data are employed to adjust the predictor parameters.

The rest of this paper is organized as follows. Section 2 describes the discrete-time visual servo model and the problem in a practical discrete-time visual servoing system. Section 3 presents the adaptive image feature prediction based on the liner dynamics model. Section 4 reports the experimental results. Finally, the conclusion is presented in Section 5.

2. System Description and Control Architecture

2.1. System Description

Consider fixing the camera on the end effector of a collaborative robot, as shown in Figure 1. In positioning tasks, the purpose of the IBVS method is to control the robot’s motion until the image features of the object move to the desired feature position in the image, which is typically defined as

$$\mathbf{e}\left(t\right)=\mathbf{s}\left(t\right)-{\mathbf{s}}^{*},$$

(1)

where $\mathbf{s}\left(t\right)$ is the image feature vector captured by vision sensors at time t, and vector ${\mathbf{s}}^{*}$ is the desired value of the features. $\mathbf{e}\in ϵ$ is the vector of image error variables, with $ϵ$ being a domain of ${\mathbb{R}}^m$. As shown above, in the IBVS scheme, the task error ($\mathbf{e}$) is defined in image space, and the input of the manipulator controller is defined by joint coordinates or Cartesian coordinates. Let $\overline{r}$ be the location of the target object where the features of the object (points, lines, etc.) naturally exist. According to [13], selecting appropriate object features and applying the projection mapping principles, we can obtain

$$\mathbf{s}=\mathit{F}\left(\overline{r}\right),$$

(2)

where $\mathit{F}$ is the perspective projection mapping. Take the derivative of Equation (2) and then the velocities of image features are described as

$$\dot{\mathbf{s}}={\displaystyle \frac{\partial \mathbf{s}}{\partial \overline{r}}}{\displaystyle \frac{\mathrm{d}\overline{r}}{\mathrm{d}t}}=\mathbf{L}{\mathbf{V}}_c,$$

(3)

where $\dot{\mathbf{s}}$ represents the image feature velocities in the image plane and ${\mathbf{V}}_c=[v_x,v_y,v_z,w_x,w_y,w_z{]}^{\top }$ is the camera velocity screw with respect to the target object. $\mathbf{L}\in {\mathbb{R}}^{k\times 6}$ is generally called the image Jacobian or the interaction matrix [4,13,18]. If the system uses image points as features, $\mathbf{L}$ can be described as

$$\mathbf{L}=\left[\begin{array}{cccccc}-{\displaystyle \frac{\lambda }{z}}& 0& {\displaystyle \frac{u}{z}}& {\displaystyle \frac{uv}{\lambda }}& -{\displaystyle \frac{{\lambda }^2+u^2}{\lambda }}& v\\ 0& -{\displaystyle \frac{\lambda }{z}}& {\displaystyle \frac{v}{z}}& {\displaystyle \frac{{\lambda }^2+v^2}{\lambda }}& -{\displaystyle \frac{uv}{\lambda }}& -u\end{array}\right],$$

(4)

where z is the height of image features along the optical axis in the camera coordinates, and $\lambda$ is the focal length of the camera lens. u and v are the positions of image features in the image space [13].

In a discrete-time system, the image feature position at time k is given by $\mathbf{s}\left(k\right)$. Let us set the camera sample time as ${\mathrm{T}}_i$, and the continuous time is given by $t=k{\mathrm{T}}_i$. For simplicity, the following formula uses k instead of $k{\mathrm{T}}_i$ to represent the current time. To obtain the discrete-time model, Equation (1) is expressed as

$$\mathbf{e}\left(k\right)=\mathbf{s}\left(k\right)-{\mathbf{s}}^{*}$$

(5)

Furthermore, with Equations (3) and (5), the discrete-time model can be obtained as

$$\mathbf{e}(k+1)=\mathbf{e}\left(k\right)+\mathrm{T}\mathbf{L}\left(k\right){\mathbf{V}}_c\left(k\right)$$

(6)

To converge image errors to zero, a proportional control scheme is often adopted in classical visual servoing controllers [18], in which

$$\dot{\mathbf{e}}\left(k\right)=-{\mathbf{K}}_{\mathrm{p}}\mathbf{e}\left(k\right)$$

(7)

Combining Equations (6) and (7), the camera velocity can be expressed as

$${\mathbf{V}}_c\left(k\right)=-{\mathbf{K}}_{\mathrm{p}}{\mathbf{L}}^{+}\left(k\right)\mathbf{e}\left(k\right),$$

(8)

in which ${\mathbf{L}}^{+}$ is the pseudo-inverse of $\mathbf{L}$.

2.2. Control Architecture

Defining $\mathbf{q}\in {\mathbb{R}}^n$ as the robot configuration variables, the IBVS scheme can be expressed as follows:

$$\mathbf{f}:\mathbf{q}\subseteq {\mathbb{R}}^n\mapsto \mathbf{e}\subseteq {\mathbb{R}}^m,\mathbf{e}=\mathbf{f}\left(\mathbf{q}\right)$$

(9)

For example, in an IBVS system based on a manipulator, $\mathbf{e}$ is the vector of image feature errors, and $\mathbf{q}$ is the vector of robot joint position or robot end-effector position.

In IBVS robotic systems, the classic control frame is an inner-loop/outer-loop architecture, as shown in Figure 2. The inner loop is typically composed of a robot controller and robot motor servo drivers. The controller controls the robot’s motion by sending position, velocity, or torque commands to the robot drive. Based on the required speed command, the robot controller uses a robot kinematics or dynamics model to calculate the joint velocities and sends these to the robot. Due to low computational complexity and short transmission delay, the control frequency of the inner loop is usually as high as 500–4000 Hz.

The outer loop, based on image information feedback, executes task layer applications such as grasping, placing, and object tracking. Considering the time required for image data acquisition and processing, the outer loop typically has high computational costs and a low control frequency, typically in the range of 30 Hz to 200 Hz.

As shown in Figure 3, since the robot control frequency exceeds the image sampling rate, the robot uses the image features of the previous moment during each image feature sampling period until it obtains a new image. ${\mathbf{T}}_{\mathrm{O}}$ is the outer-loop cycle time, and ${\mathbf{T}}_{\mathrm{I}}$ is the inner-loop cycle time. Therefore, Equation (8) can be rephrased as

$${\mathbf{V}}_c\left(1+(k-1)\mathrm{N}\right)=-{\mathbf{K}}_{\mathrm{p}}{\mathbf{L}}^{+}\mathbf{e}\left(k\right),$$

(10)

where $\mathrm{N}={\mathbf{T}}_{\mathrm{O}}/{\mathbf{T}}_{\mathrm{I}}$ and

$${\mathbf{V}}_c(1+(k-1)\mathrm{N})={\mathbf{V}}_c(m+(k-1)\mathrm{N}),$$

(11)

in which $m=1,2,\cdots ,N-1,$ and $k=1,2,\cdots$.

It is noted that the time delay has a great influence on the visual servoing system. The controller uses the same image feature signal within $\mathrm{N}$ control cycles, which may lead to system stability and oscillation issues in the worst-case scenarios. Although many scholars have studied image error estimation, they have not explicitly considered the difference between the control period and the image feedback period.

3. Adaptive Image Feature Prediction Based on Robot Velocity Linear Dynamic Model

In this section, we introduce the linear dynamic model of the robot end-effector velocity, which benefits us in predicting the image features. Then, the implementation of adaptive image feature prediction is introduced in detail.

3.1. Linear Dynamic Model of Robot and Camera Motion

Let frame ${\mathbf{F}}_{\mathrm{E}}$ link to the robot end effector and ${\mathbf{F}}_{\mathrm{C}}$ link to the camera. The displacement between the robot end-effector coordinate system (${\mathbf{F}}_{\mathrm{E}}$) and the camera coordinate system (${\mathbf{F}}_{\mathrm{C}}$) can be denoted as ${\mathbf{P}}_{\mathrm{C}}^{\mathrm{E}}=(x_c,y_c,z_c)\in {\mathbb{R}}^3$, employing ${\mathbf{R}}_{\mathrm{C}}^{\mathrm{E}}\in {\mathbb{R}}^{3\times 3}$ as the rotation matrix from ${\mathbf{F}}_{\mathrm{C}}$ to ${\mathbf{F}}_{\mathrm{E}}$. The feature point in the camera coordinate system is denoted as ${\mathbf{P}}_{\mathrm{C}}$, and it is represented as ${\mathbf{P}}_{\mathrm{E}}$ in the robot end-effector coordinate system, so we have

$${\mathbf{P}}_{\mathrm{E}}={\mathbf{T}}_{\mathrm{C}}^{\mathrm{E}}\ast {\mathbf{P}}_{\mathrm{C}},$$

(12)

where ${\mathbf{T}}_{\mathrm{C}}^{\mathrm{E}}$ is the homogeneous transformation matrix, and

$${\mathbf{T}}_{\mathrm{C}}^{\mathrm{E}}=\left[\begin{array}{cc}{\mathbf{R}}_{\mathrm{C}}^{\mathrm{E}}& {\mathbf{P}}_{\mathrm{C}}^{\mathrm{E}}\\ 0& 1\end{array}\right]$$

(13)

${\mathbf{V}}_e$ is defined as the robot’s end-effector velocities. Considering the camera is fixedly connected to the robot end effector, the robot end-effector velocities are

$${\mathbf{V}}_e=\mathbf{H}{\mathbf{V}}_c,$$

(14)

where

$$\mathbf{H}=\left[\begin{array}{cc}{\mathbf{R}}_{\mathrm{C}}^{\mathrm{E}}& S\left({\mathbf{P}}_{\mathrm{C}}^{\mathrm{E}}\right){\mathbf{R}}_{\mathrm{C}}^{\mathrm{E}}\\ 0_{3\ast 3}& {\mathbf{R}}_{\mathrm{C}}^{\mathrm{E}}\end{array}\right],$$

(15)

where S is the skew symmetric matrix and $S\left({\mathbf{P}}_{\mathrm{C}}^{\mathrm{E}}\right)$ is given by

$$S\left({\mathbf{P}}_{\mathrm{C}}^{\mathrm{E}}\right)=\left[\begin{array}{ccc}0& -z_c& y_c\\ z_c& 0& -x_c\\ -y_c& x_c& 0\end{array}\right]$$

(16)

Ideally, the output of the robot’s velocity mode is expected to be a step response, which means that

$${\overline{\mathbf{V}}}_e\left(n\right)={\mathbf{V}}_e^{ref}\left(n\right),$$

(17)

where ${\mathbf{V}}_e^{ref}\left(n\right)$ is the expected velocity output by the visual servo controller, and ${\overline{\mathbf{V}}}_e\left(n\right)$ is the real velocity of the robot end effector.

In practice, the body of a robot is flexible, and its velocity control mode is not ideal. Considering the nonlinear part of its response, we propose a linear model taking into account robot dynamics that is suitable for robots that compute torque in the robot controller [36,38].

Assuming the end velocity of the robot is a uniform acceleration model, within the robot control sample time (${\mathrm{T}}_r$), the robot’s motion distance (L) is

$$L={\displaystyle \frac{{\mathbf{V}}_e(n-1)+{\mathbf{V}}_e\left(n\right)}{2}}{t}_1+{\mathbf{V}}_e\left(n\right)t_2,$$

(18)

where $t_1$ is the acceleration time, $t_2$ is the time for uniform motion, and ${\mathrm{T}}_r=t_1+t_2$. ${\mathbf{V}}_e\left(n\right)$ is the actual velocity of the robot. With knowledge of the definition of velocity in physics, the real velocity can be obtained as follows:

$$\begin{array}{cc}\hfill {\overline{\mathbf{V}}}_e\left(n\right)& =L/\mathbf{T}\hfill \\ \hfill & ={\displaystyle \frac{t_1}{2\mathbf{T}}}{\mathbf{V}}_e(n-1)+{\displaystyle \frac{t_1+2t_2}{2\mathbf{T}}}{\mathbf{V}}_e\left(n\right)\hfill \end{array}$$

(19)

Let $\alpha \in (-1,1)$ be the convergence rate, employing $\beta$ as the gain value for the expected velocity. According to Equation (19), the linear dynamic model of the robot velocity is given by

$${\overline{\mathbf{V}}}_e\left(n\right)=\alpha {\mathbf{V}}_e(n-1)+\beta {\mathbf{V}}_e^{ref}\left(n\right)$$

(20)

When $\alpha =0$, the robot velocity controller is perfect and can converge within one step. If the value of $\alpha$ approaches 1, the system is overdamped. For a high-performance robot in velocity control mode, $\beta \approx 1$, indicating that the robot can achieve the required velocity command with a small error. In practice, the robot motion control system is stable. The deviation between the desired velocity and the actual motion velocity is bounded. Thus, we have

$$\left|\right|{\overline{\mathbf{V}}}_e\left(n\right)\left|\right|⩽\eta \left|\right|{\mathbf{V}}_e^{ref}\left(n\right)\left|\right|,$$

(21)

where $\eta \in (0,{\eta }_1),{\eta }_1\in \mathbb{R}$.

Currently, most robots provide velocity operation mode in Cartesian space, which greatly facilitates the operation of the robot. Therefore, we can directly send velocity commands to the robot without considering complex joint velocity calculations. According to Equation (14), we have

$${\overline{\mathbf{V}}}_c\left(n\right)=\alpha {\mathbf{H}}^{+}{\mathbf{V}}_e(n-1)+\beta {\mathbf{H}}^{+}{\mathbf{V}}_e^{ref}\left(n\right)$$

(22)

3.2. Adaptive Image Feature Prediction

In this subsection, we assume that the robot does not have singularity problems. Since the majority of time is spent on image capture and processing in the outer loop, we set ${\mathbf{T}}_{\mathbf{O}}={\mathrm{T}}_i$ and ${\mathbf{T}}_{\mathbf{I}}={\mathrm{T}}_r$. In most IBVS systems, the time (${\mathrm{T}}_i$) is much longer than the robot controller time (${\mathrm{T}}_r$). Therefore, we use the robot velocity information at $n{\mathrm{T}}_r$ to estimate the image error. Considering the model dynamics, according to Equations (14) and (20), we have

$$\mathbf{e}(n+1)=\mathbf{e}\left(n\right)+\mathrm{T}\mathbf{L}\left(n\right){\mathbf{H}}^{+}{\overline{\mathbf{V}}}_e\left(n\right)$$

(23)

When $n=k({\mathrm{T}}_{\mathrm{i}}/{\mathrm{T}}_{\mathrm{r}})$, $n,k=1,2,\cdots$, we can obtain the image error directly from the camera. If $n\ne k({\mathrm{T}}_{\mathrm{i}}/{\mathrm{T}}_{\mathrm{b}})$, we employ Equation (23) to estimate the image error.

It is important to accurately update $\alpha$ and $\beta$ to estimate image feature positions. It should also be noted that due to the uncertainty of robot dynamics, $\alpha$ and $\beta$ vary with robot posture.

Here, an effective method based on the least squares method is proposed for adaptive adjustment of $\alpha$ and $\beta$. The image feature errors and robot end-effector velocities are employed as observational data. In the first control cycle of the robot control inner loop, according to Equation (23), we can obtain

$$\mathbf{e}\left(1\right)-\mathbf{e}\left(0\right)={\mathrm{T}}_r\mathbf{L}\left(1\right){\mathbf{H}}^{+}(\alpha {\mathbf{V}}_e\left(0\right)+\beta {\mathbf{V}}_e^{ref}\left(1\right))$$

(24)

Similarly, the image feature errors are calculated sequentially in other control cycles as

$$\begin{array}{cc}\hfill \mathbf{e}\left(2\right)-\mathbf{e}\left(1\right)& ={\mathrm{T}}_r\mathbf{L}\left(2\right){\mathbf{H}}^{+}(\alpha {\mathbf{V}}_e\left(1\right)+\beta {\mathbf{V}}_e^{ref}\left(2\right))\hfill \\ \hfill \mathbf{e}\left(3\right)-\mathbf{e}\left(2\right)& ={\mathrm{T}}_r\mathbf{L}\left(3\right){\mathbf{H}}^{+}(\alpha {\mathbf{V}}_e\left(2\right)+{\beta }^{+}{\mathbf{V}}_e^{ref}\left(3\right))\hfill \\ \hfill & ⋮\hfill \\ \hfill \mathbf{e}(n-m)-\mathbf{e}(n-m-1)& ={\mathrm{T}}_r\mathbf{L}(n-m){\mathbf{H}}^{+}(\alpha {\mathbf{V}}_e(n-m-1)+\beta {\mathbf{V}}_e^{ref}(n-m))\hfill \end{array}$$

(25)

Let $\mathbf{F}\left(n\right)={\mathrm{T}}_r\mathbf{L}\left(n\right){\mathbf{H}}^{+}$; for the above iterative process, the prediction equation is

$$\mathbf{e}\left(n\right)-\mathbf{e}(n-1)=\mathbf{F}\left(n\right)(\alpha {\mathbf{V}}_e(n-1)+\beta {\mathbf{V}}_e^{ref}\left(n\right)).$$

(26)

In order to better represent the parameters of the linear dynamic model of robot end velocity, we set $Z\left(n\right)={\mathbf{F}}^{-1}\left(n\right)[\mathbf{e}\left(n\right)-\mathbf{e}(n-1)]$ as the system output, and Equation (26) can be rewritten as

$$Z\left(n\right)=\left[\begin{array}{c}{\mathbf{V}}_e(n-1),{\mathbf{V}}_e^{ref}\left(n\right)\end{array}\right]\left[\begin{array}{c}\alpha \\ \beta \end{array}\right]$$

(27)

Assuming that the length of the moving window is K and $\widehat{\mathit{\theta }}={[\alpha ,\beta ]}^{\mathrm{T}}$, we collect Equations (25) and (26) in a compact matrix form as

$$\mathbf{Z}\left(n\right)={\overrightarrow{\mathbf{F}}}^{-1}\mathbf{Y}=\Phi \widehat{\mathit{\theta }},$$

(28)

where

$$\mathbf{Z}\left(n\right)={[Z\left(n\right),Z(n-1),\dots ,Z(n-K+1)]}^{\mathrm{T}}$$

(29)

$$\overrightarrow{\mathbf{F}}={\left[\begin{array}{cccc}\mathbf{F}\left(n\right)& 0& \cdots & 0\\ 0& \mathbf{F}(n-1)& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮\\ 0& 0& 0& \mathbf{F}(n-K+1)\end{array}\right]}_{K\times K}$$

(30)

$$\mathbf{Y}={\left[\begin{array}{c}e(n+1)-e\left(n\right)\\ e\left(n\right)-e(n-1)\\ ⋮\\ e(n-K+2)-e(n-K+1)\end{array}\right]}_{K\times 1}$$

(31)

$$\Phi ={\left[\begin{array}{cc}{\mathbf{V}}_e\left(n\right)& {\mathbf{V}}_e^{ref}(n+1)\\ {\mathbf{V}}_e(n-1)& {\mathbf{V}}_e^{ref}\left(n\right)\\ ⋮& ⋮\\ {\mathbf{V}}_e(n-K+1)& {\mathbf{V}}_e^{ref}(n-K+2)\end{array}\right]}_{K\times 2}$$

(32)

We employ $\widehat{Z}\left(n\right)$ as the measurement value of the system. When $n=k({\mathrm{T}}_{\mathrm{i}}/{\mathrm{T}}_{\mathrm{r}})$, $\widehat{Z}\left(n\right)$ is

$$\widehat{Z}\left(n\right)={\mathbf{F}}^{-1}\left(n\right)[\widehat{\mathbf{e}}\left(n\right)-\mathbf{e}(n-1)]$$

(33)

where $\widehat{\mathbf{e}}\left(n\right)$ is the actual image error at the image sample time. During the non-sampling period of the camera, let $\widehat{Z}\left(n\right)=Z\left(n\right)$. Thus, the measurement value ($\widehat{Z}\left(n\right)$) can be expressed as

$$\widehat{Z}\left(n\right)=\left\{\begin{array}{cc}{\mathbf{F}}^{-1}\left(n\right)[\widehat{\mathbf{e}}\left(n\right)-\mathbf{e}(n-1)]\hfill & n=k({\mathrm{T}}_{\mathrm{i}}/{\mathrm{T}}_{\mathrm{r}})\hfill \\ Z\left(n\right)\hfill & n\ne k({\mathrm{T}}_{\mathrm{i}}/{\mathrm{T}}_{\mathrm{r}})\hfill \end{array}\right.$$

(34)

We set $\widehat{\mathbf{Z}}$ as the vector representation of $\widehat{Z}\left(n\right)$ in K control cycles. We define the cost function as

$$\mathbf{J}=\sum _{n=1}^K{[\widehat{Z}\left(n\right)-Z\left(n\right)]}^2$$

(35)

By using the least squares method, the $\widehat{\mathit{\theta }}$ parameter can be obtained as

$$\widehat{\mathit{\theta }}={({\Phi }^{\mathrm{T}}\Phi )}^{-1}{\Phi }^{\mathrm{T}}\widehat{\mathbf{Z}}$$

(36)

Figure 4 shows our overall visual servo system. In contrast to basic visual servoing methods, the proposed feature prediction method combines past image data and robot end-effector velocity to predict the current image features. The method steps are given in Algorithm 1.

Algorithm 1 Visual Servoing Controller With Adaptive Image Feature Prediction

Input: the robot end-effcetor velocity ${\mathbf{V}}_e\left(n\right)$ and the image features $\mathbf{e}\left(k\right)$ Output: the robot end-effector velocity command ${\mathbf{V}}_e^{ref}\left(n\right)$;  1: Set initial state, such as $\alpha =0,\beta =1$, the window length K ($K>({\mathrm{T}}_{\mathrm{i}}/{\mathrm{T}}_{\mathrm{r}})$), the control gain ${\mathbf{K}}_{\mathrm{p}}$, the final image feature errors $\delta$, the robot control time $n=1$ and the image feature sampling time $k=1$. $\mathbf{H}$ is the constant matrix and $\mathbf{e}\left(1\right)$ can be directly obtained from the camera.  2: Calculate the robot end-effector command ${\mathbf{V}}_e^{ref}$ by (8) and (14).  3: If $k{\mathrm{T}}_i=n{\mathrm{T}}_r$, the image error $\mathbf{e}$ is obtained through the camera and $k\leftarrow k+1$. Otherwise, the image error $\mathbf{e}$ is predicted through (20) and (23).  4: Update the matrix $\mathbf{Y}$ and $\Phi$ by the past data. If $n>K$, then calculate the $\widehat{\mathit{\theta }}$ by (36).  5: If $\mathbf{e}>\delta$, update $n\leftarrow n+1$ and come back to Setp. 2.

3.3. Stability Analysis

The Lyapunov function is designed as

$$\mathbf{V}\left(n\right)={\displaystyle \frac{1}{2}}{\left|\right|e\left(n\right)\left|\right|}^2={\displaystyle \frac{1}{2}}{e}^{\mathrm{T}}\left(n\right)e\left(n\right)$$

(37)

Furthermore, we define $\Delta \mathbf{V}\left(n\right)=\mathbf{V}(n+1)-\mathbf{V}\left(n\right)$. According to Equations (8), (21), and (23), we have

$$\Delta \mathbf{V}\left(n\right)={\displaystyle \frac{1}{2}}\left(\right||(\mathrm{I}-\eta \mathrm{T}{\mathbf{K}}_{\mathrm{p}})e\left(n\right){\left|\right|}^2-\left|\right|e\left(n\right){\left|\right|}^2)$$

(38)

To ensure system stability, the following condition should be satisfied [37]:

$$\left|\right|\mathbf{I}-\eta {\mathrm{T}}_r{\mathbf{K}}_{\mathrm{p}}{\left|\right|}^2<\mathbf{I}$$

(39)

In practice, $\eta$ is close to 1, and $\mathrm{T}$ is very small. We can choose the appropriate ${\mathbf{K}}_{\mathrm{p}}$ to ensure the system’s stability and improve its performance.

4. Experiments

Several comparative experiments were designed to demonstrate the impact of the proposed method on system performance. We used a lightweight collaborative robot and a camera fixed on a 6-DOF collaborative robot end effector. When the robot is in velocity mode, the controller can send speed commands to control the movement of the robot’s end effector. We also selected the corner points of the ArUco marker as the feature points. The experimental equipment is shown in Figure 5.

The robot has a velocity control mode with a communication frequency of 1000 Hz. A Galaxy camera is employed to capture the position of the feature points, with a resolution of 1920 × 1080. Considering the image acquisition time and image processing time, the feedback frequency of image features is about 30 Hz. According to the design and calibration parameters, ${\mathbf{T}}_{\mathrm{C}}^{\mathrm{E}}$ is

$${\mathbf{T}}_{\mathrm{C}}^{\mathrm{E}}=\left[\begin{array}{cccc}0& 1& 0& 0.05\\ -1& 0& 0& 0\\ 0& 0& 1& 0.08\\ 0& 0& 0& 1\end{array}\right]$$

(40)

We conducted comparative experiments of visual servoing using the following three methods:

(1)

Method 1 utilized the classical visual servoing scheme (VSC) [18]. In VSC, the four points are employed as image features. Due to the easy recognition of image features, the simple interaction matrix of point features, and high computational efficiency, this scheme has been widely applied in practice.

(2)

Method 2 (VSC-OB) employed the image prediction method based on the observer, which only uses Equation (6) and does not consider the robot dynamic model.

(3)

Method 3 (VSC-P) utilized the proposed adaptive image prediction based on the input-mapping method in combination with the robot’s dynamic model.

The robot moves from the initial position where the joint angles are ($\mathbf{q}=[-100.5,29.6,97.1,-36.68,-90.37,-10.26]$) and the image points are ($(1214,581),(1214,960),(836,960)$) and $(836,581)$. The length of the moving window is set as $K={\mathrm{T}}_i/{\mathrm{T}}_r=33$. We only considered the translational motion of the robot in the experiments, disregarding the rotational movements.

4.1. Controller with Smaller Proportional Gain

In this subsection, a comparative experiment with the controller selecting a small proportional gain is designed to compare the three different methods. In this comparative experiment, the gain is set as 15.

Figure 6 shows the feature trajectory within the image space of the three methods. It can be observed that all the methods are capable of accomplishing the visual servoing task of moving the robot from the start position to the desired location. At a small proportion of gain, the trajectories in the image space are smooth and close to straight lines.

Figure 7 displays the image feature error, and Figure 8 shows the end-effector velocities and joint velocities of the robot. We can see that with the application of a small proportional gain controller, all three visual servo methods can cause the robot’s end-effector velocity to converge. However, due to the low frequency of image information feedback, the VSC scheme exhibits a stepped error in its image features and experiences significant speed fluctuations during the servo process. On the contrary, owing to the presence of the image feature observer, VSC-OB can predict image features and minimize the velocity changes of the robot end effector. VSC-P integrates an image feature observer and considers the robot’s dynamic characteristics, resulting in smoother velocity control at the robot end effector.

4.2. Controller with Large Proportional Gain

To explore the control performance of the proposed method with larger proportional gains, a controller with a high proportional gain value is used in this subsection. In this experiment, the proportional gain value is set as 50.

Figure 9 illustrates the moving trajectory of the robot end effector using three different schemes, while Figure 10 shows the feature trajectory in the image space. It can be observed that with a larger proportional gain controller, the VSC method induces vibration as the robot end effector approaches the target position. In the VSC-OB method, the trajectory of the robot end effector experiences a slight jitter. Compared to the first two methods, the proposed method achieves a smoother trajectory for the robot’s motion and ensures that there are no oscillations when the robot approaches the target position.

The image errors of the three methods are shown in Figure 11, and the robot end-effector and joint velocities are shown in Figure 12a–c. It is obvious that in the case of large gain values, the VSC scheme causes oscillation of the robot’s end-effector movement during the visual servo process. In contrast, because of the precise prediction of changes in image error during the control cycle, the control signal can decrease as the image error diminishes. Therefore, both the VSC-OB method and the proposed VSC-P method can not only achieve convergence but also improve the convergence speed of the system when employing a larger proportion of controllers. Furthermore, we can see that compared with the VSC-OB method, the proposed method takes into account the dynamic characteristics of the robot’s velocity mode, making image error prediction more accurate. Therefore, the robot’s end-effector motion is smoother than under the VSC-OB method.

5. Conclusions

In this paper, a visual servoing control method is investigated from the perspective of a discrete-time model. To solve the time delay issue of the outer loop, image features are directly estimated in the image plane, which is suitable for scenes with low camera sampling frequency and complex image processing. This work designed an adaptive image feature prediction method by using past image feature data and robot end-effector velocity data. The controller with adaptive feature prediction smooths the velocity input of the IBVS system and still maintains the stability of the system under large gains. Compared with the classical IBVS scheme, the VSC-P method is more applicable for situations with a large outer-loop time delay. For scenarios that require high resolution and high-precision positioning, such as industrial assembly, our method can greatly reduce the problem of slow system convergence caused by long image computation time. It should be noted that the proposed method does not take into account cases of image feature feedback errors or image information loss. Therefore, in future work, we will further investigate the estimation of image features at the sampling time to reduce the impact of image feature information loss or image noise on system performance.