Adaptive Visual Control for Robotic Manipulators with Consideration of Rigid-Body Dynamics and Joint-Motor Dynamics

Abstract

A novel cascade visual control scheme is proposed to tailor for electrically driven robotic manipulators that operate under kinematic and dynamic uncertainties, utilizing an uncalibrated stationary camera. The proposed control approach incorporates adaptive weight radial basis function neural networks (RBFNNs) to learn the behaviors of the uncertain dynamics of the robot and the joint actuators. The controllers are designed to nullify the approximation error and mitigate unknown disturbances through an integrated robust adaptive mechanism. A major advantage of the proposed approach is that prior knowledge of the dynamics of the robotic manipulator and its actuator is no longer required. The controller autonomously assimilates the robot and actuator dynamics online, thereby obviating the need for fussy regression matrix derivation and advance dynamic measurement to establish the adaptive dynamic parameter update algorithm. The proposed scheme ensures closed-loop system stability, bounded system states, and the convergence of tracking errors to zero. Simulation results, employing a PUMA manipulator as a testbed, substantiate the viability of the proposed control policy.

1. Introduction

Visual servoing control, leveraging the capabilities of visual cameras as efficient sensors, has emerged as a pivotal technique in robotic manipulation. Widely applied across various intelligent fields, this method plays a critical role in enhancing the precision and adaptability of robotic systems in numerous industries [1,2,3,4]. Hence, it is routinely deployed to tackle tasks of complexity and elevated difficulty [5,6]. With the development of visual control methods, a variety of control strategies have been explored and established. At present, mainstream visual servoing control methods are primarily categorized into two types, based on the differences in the construct approach of their error functions. The first type, position-based visual servoing (PBVS), calculates control errors based on the variance between the current and desired poses, as relayed by corresponding image features [3,7,8,9]. This method fundamentally relies on the mapping relationship between the pose and the image, which makes it notably sensitive to the precision of visual camera parameters and susceptible to disturbances from image noise. In contrast, image-based visual servoing (IBVS) derives system error directly from image feedback features, rendering it more robust and reliable compared with PBVS [10,11,12,13]. Given these distinct advantages and limitations, some researchers have proposed a combined approach, hybrid visual servoing (HVS) [14,15,16]. This method integrates error types from both PBVS and IBVS into a cohesive control policy [17]. Nevertheless, the effectiveness of this hybrid approach, as with all visual servoing techniques, remains dependent on the measurement accuracy and parameter settings of the visual system. Therefore, uncalibrated visual servoing has emerged as a significant research focus within the field of visual control [18].

To address the problem of the uncalibrated camera, a 2 1/2D visual servoing scheme based on the estimate method of the homography matrix was developed in [19]. In [20], a flexible camera calibrated method for computer vision is proposed. Additionally, a planar robot visual servoing control scheme with the robustness of the radial aberrations in the lens and indeterminate camera orientation is presented in [21]. Ref. [22] proposed to utilize the epipolar geometry to compute the goal location and the translate direction with uncalibrated cameras and robots. However, the control policies previously mentioned are solely predicated upon kinematic controllers, rather than the dynamics of the system. It is widely acknowledged that models of robotic dynamics incorporate an extensive array of the manipulator physical parameters, consisting of nonlinear forces, friction, and inertia.

Therefore, a significant number of visual controllers based on robotic nonlinear dynamics have been presented. As opposed to the kinematics-based methods, the dynamics-based approaches rely on the analytical approach to derive the unknown parameters of the explicit Jacobian matrix and design a matched algorithm to achieve the purpose of system self-calibration. In [23], an adaptive estimate approach for the parameter of the uncalibrated visual system is proposed to design a dynamic controller for planar robots. Informed by this research, a depth-independent image interaction matrix, which can map the image information onto the joint space of the robot without the lacking depth information, is proposed in [24]. The corresponding parameterized method and the adaptive parameter update approach are also developed in the study. Additionally, projective homography-based uncalibrated visual servoing is proposed in [25], which constructs a novel task function from projection theory with prior knowledge of the camera parameters.

Although the aforementioned solution proposes a comprehensive control strategy, the operational dynamics of the robot are subject to variability due to factors such as the interchange and installation of grippers and other attachments. These factors can induce alterations in the physical parameters of the robot, thereby introducing uncertainties in its kinematics and dynamics. To address it, a two-layer control scheme incorporating a model predictive strategy and an extended state observer is presented for multi-manipulator trajectory tracking systems with parametric uncertainties in [26]. To overcome the uncertainties of the dynamics from the highly kinematically redundant robot, an adaptive sliding model with an online dynamic estimator is designed in [27]. In [28], a novel method, merging feedback linearization with a multilayer neural network (MNN)-based observer, is introduced for precise robotic manipulator control, bypassing complex uncertainty modeling. For the uncalibrated visual servoing control system, a hybrid estimation method, which estimates the unknown camera parameters and the robotic manipulator, is proposed to achieve the purpose of eliminating uncertainty in the study [29], as an extension of the study [24].

However, the aforementioned studies are based on the premise that the joint actuators of robotic manipulators function under ideal input–output conditions. In practice, as the robotic manipulator executes movements, the physical parameters of the joint actuator motors undergo alterations due to factors such as thermal drift, thereby introducing uncertainties into the motor model [30,31,32,33,34,35]. In [32], a controller based on the differential equation approximation method is proposed. Although it addresses the uncertainties in dynamics, the approximation parameters of the differential equation need to be selected through experimentation, which is undoubtedly a complex and costly task. In [34], an inverse Jacobian and an actuator adaptation law are proposed to overcome the uncertain kinematic, dynamic, and actuator parameters of the robotic manipulator, which establish the adaptive rule in virtue of the regressor matrix and only consider the DC motor whose torque is proportional to voltage. The research study of [36], which presents a fuzzy logic system to compensate for the uncertainties of the system, neglects the joint torque error, affected by the uncertainties of the mechanical and physical parameters of the robot. But adaptive fuzzy systems necessitate meticulous calibration of membership functions and rule sets, a process that is inherently complex and requires specialized expertise. An adaptive visual servoing control for electronically driven robotic manipulators is investigated in the work [35], but the provided scheme contains the time derivative of the desired armature current, which is quite difficult to compute, since it involves the derivative of the adaptive parameter and the adaptive estimation of the derivative of parameters.

Motivated by the aforementioned considerations, this study investigates the problem of visual control for robotic manipulators, focusing on the uncertain kinematics and dynamics of the robot and its actuator motor, using an uncalibrated camera. By means of the depth-independent composite Jacobian matrix proposed in [29], a kinematic pose controller is constructed and applied to compute the reference joint angle velocity by the image error feedback. The parameters of this virtual controller are estimated by an adaptive update rule. To track the reference joint velocity and compensate for the uncertainties of the robotic dynamics, a torque controller with the adaptive RBFNN is proposed. Further, a voltage controller, which also includes the RBFNN approximator to eliminate stakes of the uncertain motor dynamics, is designed to track the torque control output as the reality system input. The weight of the RBFNNs is updated by the adaptive algorithm through the tracking error feedback. To eliminate the approximator error and the disturbance in the uncalibrated environment, an adaptive robust mechanism is incorporated separately into the torque controller and voltage controller. Ultimately, the system states and the tracking errors of the different closed loops are globally asymptotically stable under the control scheme proposed. In summary, the work of this study has the following contributions and novelties:

• Compared with existing uncalibrated visual control schemes for robot manipulators, e.g., [23,24,25], the uncertainties in kinematics and dynamics and the influences of joint-motor dynamics are further considered in design and analysis and well addressed by our newly proposed neural network-based IBVS strategy.
• Although some attempts have been made in developing uncalibrated visual control schemes capable of accommodating motor dynamics (see [35] for examples), the implementation of the developed control algorithm requires the time derivatives of the regressor matrix and many other signals, resulting in a large computational burden. Such a drawback is removed with our scheme by making full use of the universal approximation property of neural networks.
• A robust adaptive mechanism is newly designed and added in our visual control scheme, with which good robustness of the proposed control scheme to neural network approximation errors and exogenous disturbances can be well established.

The rest of this paper are organized as follows: In Section 2, the vision system model and the dynamic equation of the robotic manipulator are given, based on which a cascade image-based adaptive visual control scheme is established in Section 3. In Section 4, a rigorous stability analysis of the proposed scheme is provided based on the Lyapunov argument. Additionally, the effectiveness of our scheme is also verified by the simulation results, as seen in Section 5. We conclude the whole paper in Section 6.

Notation 1.

To maintain clarity throughout this paper, we establish a set of notation rules and define the key symbols used herein (

Table 1. Nomenclature table.

Symbol	Implication
$\mathit{p}$	The spatial position vector
$\mathit{q}$	The joint anngle vector
$\mathit{T}$	The homogeneous transformation matrix
$\mathit{y}$	The image coordinate vector
$\Xi$	The perspective projection matrix
${\mathit{\xi }}_{\mathit{i}}$	The ith row of the perspective projection matrix
$\mathit{J}$	The Jacobian matrix
$\mathit{A}$	The depth-independent interaction matrix
$\Lambda$	The depth-independent compound matrix
${\mathit{\lambda }}_{\mathit{i}}$	The depth-independent compound vector
$\mathit{M}$	The inertia force matrix of the robotic manipulator
$\mathit{C}$	The Coriolis force matrix of the robotic manipulator
$\mathit{g}$	The gravitational force vector of the robotic manipulator
$\mathit{\tau }$	The joint torque vector
$\mathit{L}$	The motor armature inductance matrix
$\mathit{R}$	The motor resistance matrix
$\mathit{N}$	The motor electromechanical conversion coefficient matrix
$\mathit{H}$	The RBFNN approximate term
$\mathit{W}$	The output weight matrix of the RBFNN
$\mathbf{\Psi }$	The RBF nodes of the neural network
$\mathit{d}$	The unknown disturbance term
$\mathit{\delta }$	The estimation error of the RBFNN
$\mathit{\omega }$	The summary of the estimation error and unknown disturbance
$\mathbf{\Omega }$	The upper bound of $\left|\mathit{\omega }\right|$
$\mathit{p}$	The compound parameter vector of the camera and manipulator
$\mathit{q}$	The energy-like Lyapunov function
z	The depth value with respect to the camera frame

). In this paper, matrices and vectors are represented by bold letters. The notation ${\mathbf{E}}_m$ signifies an identity matrix of dimensions $m\times m$. Furthermore, the caret symbol ˆ placed above a matrix, vector, or scalar designates its adaptive estimate.

2. Problem Statement and Analysis

In this study, a stationary camera with uncertain external and internal parameters, positioned at an undetermined location, serves as a vision feedback sensor to control a robotic manipulator. This manipulator operates under the uncertainty of both robotic kinematics and dynamics, including the dynamic parameters of the unknown servo motors in the robotic manipulator joints. Moreover, the joint actuators and their motors are subject to various non-differentiable nonlinear disturbances, which introduce additional unknown disturbances into the dynamic control system. The primary control objective is to enable the precise positioning of the robotic end-effector at a specific image location, employing uncalibrated visual feedback despite these challenging conditions. The system frame is established as shown in Figure 1. The following subsection addresses these issues through a detailed discussion of each, further exploring the implications and potential solutions.

2.1. Kinematics

In this subsection, the kinematic relationship between the robot and the camera is discussed. Two coordinate frames are established for delineating the visual robot system: the world frame, positioned at the base of the robotic manipulator, and the visual frame, located at the camera center. The vector $\mathit{p}\in {\mathbb{R}}^{4\times 1}$ represents the spatial position of the feature point marked on the end-effector, and the vector $\mathit{q}\in {\mathbb{R}}^{n\times 1}$ signifies the positions of the manipulator joints. The transform relationship between the world frame and the visual frame is delineated as

$$\begin{array}{c}\hfill {}^c\mathit{p}=\mathit{Tp},\end{array}$$

(1)

the matrix $\mathit{T}$ is the homogeneous transformation matrix that represents the camera extrinsic parameters. Incorporating the intrinsic parameters of the stationary camera, the image coordinate $\mathit{y}\in {\mathbb{R}}^{3\times 1}$ under the projection theory is described as

$$\begin{array}{c}\hfill \mathit{y}={\displaystyle \frac{1}{z}}\mathbf{\Xi }\mathit{p},\end{array}$$

(2)

where $\mathbf{\Xi }\in {\mathbb{R}}^{3\times 4}$ represents the unknown parameters of the camera and constitutes the projection matrix, which is the product of the intrinsic and extrinsic parameter matrices. This equation is the perspective projection equation of the stationary camera. The image position vector is denoted in homogeneous form:

$$\begin{array}{c}\hfill \mathit{y}=\left(\begin{array}{c}u\left(t\right)\\ v\left(t\right)\\ 1\end{array}\right).\end{array}$$

(3)

where $\mathit{y}\left(t\right)$ represents the projection coordinates of the feature point on the camera image plane. In addition, z represents the depth of the feature point with respect to the visual frame, which is a time-varying value. By the projection theory, the depth value can be computed by

$$z={\mathit{\xi }}_3\mathit{p},$$

(4)

Further, the Jacobi mapping relationship between the joint velocity space and the end-effector space is established as

$$\begin{array}{c}\hfill \dot{\mathit{p}}=\mathit{J}\dot{\mathit{q}}\end{array}$$

(5)

where $\mathit{J}$ is the robotic kinematic Jacobi matrix. In traditional approaches, the image Jacobian mapping matrix relies on the depth information, $z\left(t\right)$, of the feature point within the visual frame. However, in the context of an uncalibrated visual servoing control system, this depth feature cannot be directly accessed. To circumvent the issue of unknown depth, a depth-independent interaction matrix is introduced. Through the derivative of the projection model (2) and the depth relationship (4), we obtain the following velocity relationship between the image plane and spatial position:

$$\begin{array}{c}\hfill \dot{\mathit{y}}={\displaystyle \frac{1}{z}}[\mathbf{\Xi }\dot{\mathit{p}}-\mathit{y}\dot{z}]={\displaystyle \frac{1}{z}}(\mathbf{\Xi }-\mathit{y}{\mathit{\xi }}_3)\dot{\mathit{p}},\end{array}$$

(6)

where ${\mathit{\xi }}_i$ is the i row of the matrix $\mathbf{\Xi }$. The novel Jacobi matrix is represented as

$$\begin{array}{c}\hfill \mathit{A}\left(\mathit{y}\right)=\mathbf{\Xi }-\mathit{y}{\mathit{\xi }}_3^T,\end{array}$$

(7)

which is called the depth-independent interaction matrix. The depth-independent interaction matrix depends solely on the image coordinates and the camera parameters. It is important to note that due to the uncertainties associated with the camera parameters and the robotic kinematics, both the joint kinematic Jacobian matrix and the depth-independent interaction matrix exhibit uncertainty in an uncalibrated environment. To facilitate the parametrization and estimation of these parameters, an auxiliary matrix is established. This matrix defines the mapping relationship between the image velocity space and the joint velocity space, enabling parameter estimation through a specified estimation rule. This matrix is denoted by

$$\begin{array}{c}\hfill \Lambda (\mathit{q},\mathit{y})=\mathit{A}\left(\mathit{y}\right)\mathit{J}\left(\mathit{q}\right).\end{array}$$

(8)

The matrices $\mathit{A}\left(\mathit{y}\right)$ and $\mathbf{\Lambda }(\mathit{q},t)$ are linearized by the unknown camera parameters. Consequently, through further derivation, the following construction of $\mathbf{\Lambda }(\mathit{q},\mathit{y})$ can be obtained:

$$\begin{array}{c}\hfill \mathbf{\Lambda }(\mathit{q},\mathit{y})=\left(\begin{array}{ccc}{\mathit{\xi }}_1^T{\mathit{j}}_1-y_1{\mathit{\xi }}_3^T{\mathit{j}}_1& {\mathit{\xi }}_1^T{\mathit{j}}_2-y_1{\mathit{\xi }}_3^T{\mathit{j}}_2& {\mathit{\xi }}_1^T{\mathit{j}}_3-y_1{\mathit{\xi }}_3^T{\mathit{j}}_3\\ {\mathit{\xi }}_2^T{\mathit{j}}_1-y_2{\mathit{\xi }}_3^T{\mathit{j}}_1& {\mathit{\xi }}_2^T{\mathit{j}}_2-y_2{\mathit{\xi }}_3^T{\mathit{j}}_2& {\mathit{\xi }}_2^T{\mathit{j}}_3-y_2{\mathit{\xi }}_3^T{\mathit{j}}_3\\ 0& 0& 0\end{array}\right),\end{array}$$

(9)

where ${\mathit{j}}_i$ is the i column of the kinematic Jacobi matrix and $y_1$ is the ith element of the feature point image projection coordinates. $\mathbf{\Lambda }(\mathit{q},\mathit{y})$ is called the depth-independent compound matrix. Hence, the image projection interaction relationship is reformulated as

$$\begin{array}{c}\hfill \dot{\mathit{y}}={\displaystyle \frac{1}{z}}\mathbf{\Lambda }(\mathit{q},\mathit{y})\dot{\mathit{q}},\end{array}$$

(10)

and the depth feature is recast as

$$\begin{array}{c}\hfill z=\mathit{\lambda }\left(\mathit{q}\right)\dot{\mathit{q}},\end{array}$$

(11)

where $\mathit{\lambda }\left(\mathit{q}\right)$ is defined as the following form:

$$\begin{array}{c}\hfill \mathit{\lambda }\left(\mathit{q}\right)=\left(\begin{array}{ccc}{\mathit{\xi }}_3^T{\mathit{j}}_1& {\mathit{\xi }}_3^T{\mathit{j}}_2& {\mathit{\xi }}_3^T{\mathit{j}}_3\end{array}\right).\end{array}$$

(12)

The matrix $\mathbf{\Lambda }(\mathit{q},\mathit{y})$ and the vector $\mathit{\lambda }\left(\mathit{q}\right)$ include all the unknown parameters of both the robotic manipulator and the camera. Since the combination of the unknown kinematic parameters is linear to the other parameters, the following propositions are given as

Proposition 1.

For any product ${\mathit{\xi }}_i^T{\mathit{j}}_k$, the unknown parameter state can be rewritten as

$$\begin{array}{c}\hfill {\mathit{\xi }}_i^T{\mathit{j}}_k={\mathit{\phi }}_{ik}^T\mathit{\theta },\end{array}$$

(13)

where ${\mathit{\phi }}_{ik}^T$ contains no unknown parameters of the manipulator or the camera and all unknown parameters are reconstructed into the vector θ. The unknown parameters of both the manipulator and the camera are compactly consolidated within the vector θ.

Proposition 2.

For any vector $\mathit{o}\in {\mathbb{R}}^{4\times 1}$, the depth-independent compound matrix $\mathbf{\Lambda }(\mathit{q},\mathit{y})\mathit{o}$ and the vector $\mathit{\lambda }\mathit{o}$ can be parameterized as

$$\begin{array}{c}\hfill \mathbf{\Lambda }(\mathit{q},\mathit{y})\mathit{o}={\mathit{\chi }}_1(\mathit{o},\mathit{q},\mathit{y})\mathit{\theta },\end{array}$$

(14)

$$\begin{array}{c}\hfill \mathit{\lambda }(\mathit{q},\mathit{y})\mathit{o}={\mathit{\chi }}_2(\mathit{o},\mathit{q})\mathit{\theta },\end{array}$$

(15)

where ${\mathit{\chi }}_1(\mathit{o},\mathit{q},\mathit{y})$ and ${\mathit{\chi }}_2(\mathit{o},\mathit{q})$ are regression matrices.

Remark 1.

In this paper, the intrinsic and extrinsic parameters of the used camera are assumed to be unknown. To handle the uncertainty in camera parameters, an adaptive control technique shall be employed in this work, under which, instead of the real values of the parameters, their adaptive estimates are involved in the controller, which will be updated online by the corresponding adaptive laws. In this sense, we only need the approximate values of the unknown parameters to perform initialization of adaptive laws, as seen later on.

2.2. Dynamics

The control policy is formulated based on the dynamics rather than solely on the robotic kinematics in this research. The Lagrangian dynamic equations are utilized to model the motion of the robotic system. The robotic rigid dynamic equation is formulated as follows:

$$\begin{array}{c}\hfill \mathit{M}\left(\mathit{q}\right)\ddot{\mathit{q}}+({\displaystyle \frac{1}{2}}\dot{\mathit{M}}\left(\mathit{q}\right)+\mathit{C}(\mathit{q},\dot{\mathit{q}}))\dot{\mathit{q}}+\mathit{g}\left(\mathit{q}\right)=\mathit{\tau },\end{array}$$

(16)

where $\mathit{M}\left(\mathit{q}\right)$ and $\mathit{C}(\mathit{q},\dot{\mathit{q}})$ represent the inertia and the Coriolis force, respectively; $\mathit{g}\left(\mathit{q}\right)$ is the vector representing the gravitational force; and $\mathit{\tau }$ is the torque input for the robotic manipulator. In our control strategy, the joint torque is generated by the joint actuator motor. Furthermore, for any homogeneous vector, the following property holds:

$$\begin{array}{c}\hfill {\mathit{c}}^T\mathit{C}(\mathit{q},\dot{\mathit{q}})\mathit{c}=0.\end{array}$$

(17)

Note that the uncertainties in the robotic dynamics mean that $\mathit{M}\left(\mathit{q}\right)$, $\mathit{C}(\mathit{q},\dot{\mathit{q}})$, and $\mathit{g}\left(\mathit{q}\right)$ are uncertain. Subsequently, the joint actuator motors of the robotic manipulator are introduced, with the specific type of motors discussed in this study being direct current (DC) motors. Based on the voltage–current and torque equations of DC motors [37], it is proposed to utilize the motor dynamic equation with the following form:

$$\begin{array}{c}\hfill \mathit{L}\dot{\mathit{\tau }}+\mathit{R}\mathit{\tau }+\mathit{N}\dot{\mathit{q}}=\mathit{u},\end{array}$$

(18)

where $\mathit{u}$ is the voltage input of the joint motor. $\mathit{L}$ is the motors armature inductance matrix. $\mathit{R}$ represents the resistance of the motors. $\mathit{N}$ is the motor electromechanical conversion coefficient. All three matrices are diagonally positive definite. Like the dynamics of the robotic manipulators, the matrices $\mathit{L},\mathit{R},$ and $\mathit{N}$ all are uncertain. In our control scheme, the actual control inputs are the voltage inputs to the motors that actuate the individual joints.

Remark 2.

Compared with the control scheme in [24,29], which only considers the torque controller design, the uncertain dynamics of the joint actuator motor is taken into consideration in this research. Since the motor dynamics have the first-order time derivative of the output torque, the ideal output model relies on the precise system parameters, which means that the uncertain motor physical parameters directly affect the torque-tracking performance of the controller. Hence, we conduct further research on this aspect in this paper.

3. Control Design

In this section, we propose a control framework that includes two virtual controllers designed to compute the reference joint velocity and the desired motor output torque, respectively, and a motor voltage controller to track the desired torque. In response to the uncertainties in kinematics and dynamics, coupled with unknown camera projection parameters and disturbances, five adaptive parameter update mechanisms are developed to accurately estimate these unknown parameters.

3.1. Controller

The desired image position of the feature point is denoted by ${\mathit{y}}_d$. Then, the image error between the current position and the desired position of the feature point is defined as

$$\begin{array}{c}\hfill \Delta \mathit{y}=\mathit{y}-{\mathit{y}}_d.\end{array}$$

(19)

By invoking the depth-independent image compound Matrices (9) and (12), the image error is effectively mapped into the joint velocity space. Based on the image error, a virtual controller designed for the real-time computation of reference joint angular velocity is designed as

$$\begin{array}{c}\hfill {\dot{\mathit{q}}}_d=-({\widehat{\mathbf{\Lambda }}}^T+{\displaystyle \frac{1}{2}}{\widehat{\mathit{\lambda }}}_3^T\Delta \mathit{y}\left(t\right))\mathit{K}\Delta \mathit{y}\left(t\right),\end{array}$$

(20)

where $\widehat{\mathbf{\Lambda }}$ and ${\widehat{\mathit{\lambda }}}_3$ are the estimations of the depth-independent compound matrices $\mathbf{\Lambda }$ and ${\mathit{\lambda }}_3$, and $\mathit{K}$ is the image error gain positive definite diagonal matrix.

To track the reference joint velocity by the joint torque controller, the joint velocity error needs to be controlled and is defined as

$$\begin{array}{c}\hfill \Delta \dot{\mathit{q}}=\dot{\mathit{q}}-{\dot{\mathit{q}}}_d.\end{array}$$

(21)

Taking into account the uncertainties inherent in the dynamics of the robotic manipulator, a new dynamic term is constructed. This formulation is derived by invoking the established robotic dynamic equation, as referenced in (16), and is presented in the following form:

$$\begin{array}{c}\hfill {\mathit{H}}_1({\dot{\mathit{q}}}_d,{\ddot{\mathit{q}}}_d,\mathit{q})=\mathit{M}\left(\mathit{q}\right){\ddot{\mathit{q}}}_d+{\displaystyle \frac{1}{2}}\dot{\mathit{M}}\left(\mathit{q}\right){\dot{\mathit{q}}}_d+\mathit{C}(\mathit{q},\dot{\mathit{q}}){\dot{\mathit{q}}}_d+\mathit{g}\left(\mathit{q}\right).\end{array}$$

(22)

This term is designed as the rigid dynamic compensation term. The novel dynamic term includes the unknown physical parameters, which cannot linearize parameters. To overcome it, we propose to employ the RBFNN to approximate it.

$$\begin{array}{c}\hfill {\mathit{H}}_1({\dot{\mathit{q}}}_d,{\ddot{\mathit{q}}}_d,\mathit{q})={\mathit{W}}_1^T{\Psi }_1({\dot{\mathit{q}}}_d,{\ddot{\mathit{q}}}_d,\mathit{q})+{\mathit{\delta }}_1,\end{array}$$

(23)

where ${\mathit{W}}_1^T\in {\mathbb{R}}^{3\times N_1}$ is the ideal weight matrix of the RBF; $N_1$ is the node number of the RBF network, which is an adjustive constant; and ${\mathit{\delta }}_1$ is the rigid dynamic approximator estimate error. By invoking it, the second virtual controller designed for dynamic torque control is proposed as

$$\begin{array}{c}\hfill {\mathit{\tau }}_{\mathit{d}}={\widehat{\mathit{W}}}_1^T{\mathbf{\Psi }}_1({\dot{\mathit{q}}}_d,{\ddot{\mathit{q}}}_d,\mathit{q})+{\dot{\mathit{q}}}_d-{\mathit{K}}_1\Delta \dot{\mathit{q}}+{\mathit{\tau }}_{\mathbf{0}}\end{array}$$

(24)

where ${\widehat{\mathit{W}}}_1$ is the estimate weight matrix updating by the adaptive rule and ${\mathit{K}}_1\in {\mathbb{R}}^{n\times n}$ is the velocity diagonal gain matrix. $\mathit{\delta }\in {\mathbb{R}}^{n\times 1}$ is the estimated error of the RBF approximator. ${\mathit{\tau }}_{\mathbf{0}}$ is the adaptive robust compensator, which is given in the follow-up discussion. After the above virtual controller design, the real control output is proposed here. The torque-tracking error of the motor is denoted by

$$\begin{array}{c}\hfill \Delta \mathit{\tau }=\mathit{\tau }-{\mathit{\tau }}_d.\end{array}$$

(25)

Similarly, the dynamic equation for the joint DC motors, as referenced in (18), is also invoked. A new motor dynamic term is formulated as follows:

$$\begin{array}{c}\hfill {\mathit{H}}_2({\dot{\mathit{\tau }}}_d,\mathit{\tau },\dot{\mathit{q}},{\dot{\mathit{q}}}_d)=\mathit{L}{\dot{\mathit{\tau }}}_d+\mathit{R}\mathit{\tau }+\mathit{N}\dot{\mathit{q}}.\end{array}$$

(26)

As in (23), the unknown physical parameters of the motor are estimated by the RBFNN approximator to compensate for the tracking performance. The ideal weight RBF approximator is presented as

$$\begin{array}{c}\hfill {\mathit{H}}_2({\dot{\mathit{\tau }}}_d,\mathit{\tau },\dot{\mathit{q}},{\dot{\mathit{q}}}_d)={\mathit{W}}_2^T{\mathbf{\Psi }}_2({\dot{\mathit{\tau }}}_d,\mathit{\tau },\dot{\mathit{q}},{\dot{\mathit{q}}}_d)+{\mathit{\delta }}_2.\end{array}$$

(27)

where ${\mathit{W}}_1^T\in {\mathbb{R}}^{3\times N_1}$ is the ideal weight of the second RBFNN and ${\mathit{\delta }}_2$ is the motor dynamic approximator estimate error. To track the desired torque, the motor voltage input controller is designed as

$$\begin{array}{c}\hfill \mathit{u}={\widehat{\mathit{W}}}_2^T{\mathbf{\Psi }}_2({\dot{\mathit{\tau }}}_d,\mathit{\tau },\dot{\mathit{q}},{\dot{\mathit{q}}}_d)-{\mathit{K}}_2\Delta \mathit{\tau }-\Delta \dot{\mathit{q}}+{\mathit{u}}_0,\end{array}$$

(28)

where the matrix ${\mathit{K}}_2$ is the torque error control gain. ${\mathit{u}}_0$ is the adaptive robust term in this controller.

Remark 3.

To improve the tracking performance, the control scheme is divided into three closed-loop feedback subsystems, as showcased in Figure 2. The practical controller output signal is the motor input voltage. The kinematic controller and the torque controller are virtual controllers for auxiliary calculation.

Remark 4.

To approximate the unknown nonlinear relationship of the dynamic equations, we propose to employ the RBFNN to learn the behavior of the rigid dynamics and the motor dynamics. The RBF approximator does not require prior knowledge about the system model and the parameter range, which avoids the assumption “unknown parameters are linear” and the tedious derivation of the establishment of regression matrices in the adaptive control policy. The other advantage of the RBF approximator is that it can avoid taking the higher derivative of the unknown dynamic parameters.

The unknown disturbance in the dynamic system is nonnegligible in the process of controller design. Hence, the unknown disturbances ${\mathit{d}}_1$ and ${\mathit{d}}_2$ are taken into consideration at the joint torque controller and the motor voltage controller. In actual physical systems, the impact of disturbances is often limited. Based on this premise, we propose the following assumption.

Assumption 1.

The absolute value of unknown disturbances ${\mathit{d}}_i$ has an upper bound ${\mathit{D}}_i$ in the dynamic system, and the upper bound is unknown.

Due to the fact that the absolute value of the RBF approximate errors also has an upper bound, the summary of the disturbance and the approximate error can be denoted by ${\mathit{\omega }}_i$ and have the unknown upper bound ${\mathbf{\Omega }}_i$. Hence, the controllers are further rewritten as

$$\begin{array}{c}\hfill {\mathit{\tau }}_{\mathit{d}}+{\mathit{\omega }}_1={\widehat{\mathit{W}}}_1^T{\Psi }_1+{\dot{\mathit{q}}}_d-{\mathit{K}}_1\Delta \dot{\mathit{q}}+{\mathit{\omega }}_1+{\mathit{\tau }}_{\mathbf{0}},\end{array}$$

(29)

$$\begin{array}{c}\hfill \mathit{u}+{\mathit{\omega }}_2={\widehat{\mathit{W}}}_2^T{\Psi }_2-{\mathit{K}}_2\Delta \mathit{\tau }-\Delta \dot{\mathit{q}}+{\mathit{\omega }}_2+{\mathit{u}}_0.\end{array}$$

(30)

Here, to eliminate adverse impact terms ${\mathit{\omega }}_i$, the adaptive robust compensators proposed above are designed as

$$\begin{array}{c}\hfill {\mathit{\tau }}_{\mathbf{0}}=-{\widehat{\mathbf{\Omega }}}_{1}sgn(\Delta \dot{\mathit{q}}),\end{array}$$

(31)

$$\begin{array}{c}\hfill {\mathit{u}}_0=-{\widehat{\mathbf{\Omega }}}_{2}sgn(\Delta \dot{\mathit{\tau }}),\end{array}$$

(32)

where ${\widehat{\mathbf{\Omega }}}_i$, which is defined as the adaptive robust parameter, is the estimation of the upper bound, and $sgn(•)$ is the sign function.

Remark 5.

The RBFNN approximation errors and the unknown disturbances directly impact the system stability and the convergence of system states and errors. A robust adaptive mechanism can ensure the system resistance to disturbances and maintain convergence performance.

Subsequently, the closed-loop dynamic system is delineated, incorporating the controller. Upon substituting the controllers into the dynamic equations, the dynamic systems are rewritten as follows:

$$\begin{array}{c}\hfill \mathit{M}\left(\mathit{q}\right)\ddot{\mathit{q}}+({\displaystyle \frac{1}{2}}\dot{\mathit{M}}\left(\mathit{q}\right)+\mathit{C}(\mathit{q},\dot{\mathit{q}}))\dot{\mathit{q}}+\mathit{g}\left(\mathit{q}\right)={\widehat{\mathit{W}}}_1^T{\Psi }_1+{\dot{\mathit{q}}}_d-{\mathit{K}}_1\Delta \dot{\mathit{q}}+{\mathit{\omega }}_1+{\mathit{\tau }}_{\mathbf{0}}+\Delta \mathit{\tau },\end{array}$$

(33)

$$\begin{array}{c}\hfill \mathit{L}\dot{\mathit{\tau }}+\mathit{R}\mathit{\tau }+\mathit{N}\dot{\mathit{q}}={\widehat{\mathit{W}}}_2^T{\Psi }_2-{\mathit{K}}_2\Delta \mathit{\tau }-\Delta \dot{\mathit{q}}+{\mathit{\omega }}_2+{\mathit{u}}_0.\end{array}$$

(34)

The RBFNN weight error is defined as $\tilde{\mathit{W}}=\mathit{W}-\widehat{\mathit{W}}$. Hence, by combing Equations (22), (23), (26), and (27), the above equations are rewritten as

$$\begin{array}{c}\hfill \mathit{M}\left(\mathit{q}\right)\Delta \ddot{\mathit{q}}+({\displaystyle \frac{1}{2}}\dot{\mathit{M}}\left(\Delta \mathit{q}\right)+\mathit{C}(\mathit{q},\dot{\mathit{q}}))\Delta \dot{\mathit{q}}=-{\tilde{\mathit{W}}}_1^T{\Psi }_1-{\dot{\mathit{q}}}_d-{\mathit{K}}_1\Delta \dot{\mathit{q}}+{\mathit{\omega }}_1+{\mathit{\tau }}_{\mathbf{0}}+\Delta \mathit{\tau },\end{array}$$

(35)

$$\begin{array}{c}\hfill \mathit{L}\Delta \dot{\mathit{\tau }}={\tilde{\mathit{W}}}_2^T{\Psi }_2-{\mathit{K}}_2\Delta \mathit{\tau }-\Delta \dot{\mathit{q}}+{\mathit{\omega }}_2+{\mathit{u}}_0.\end{array}$$

(36)

By left-multiplying (35) by ${\Delta \dot{\mathit{q}}}^T$ and (36) by ${\Delta \dot{\mathit{\tau }}}^T$, the closed-loop dynamic equations are constructed as follows:

$$\begin{array}{ccc}\hfill {\Delta \dot{\mathit{q}}}^T\mathit{M}\left(\mathit{q}\right)\Delta \ddot{\mathit{q}}& +& {\displaystyle \frac{1}{2}}{\Delta \dot{\mathit{q}}}^T\dot{\mathit{M}}\left(\Delta \mathit{q}\right)\Delta \dot{\mathit{q}}=\hfill \\ \hfill & -& {\Delta \dot{\mathit{q}}}^T{\tilde{\mathit{W}}}_1^T{\Psi }_1-{\Delta \dot{\mathit{q}}}^T{\dot{\mathit{q}}}_d-{\Delta \dot{\mathit{q}}}^T{\mathit{K}}_1\Delta \dot{\mathit{q}}+{\Delta \dot{\mathit{q}}}^T{\mathit{\omega }}_1+{\Delta \dot{\mathit{q}}}^T{\mathit{\tau }}_{\mathbf{0}}+{\Delta \dot{\mathit{q}}}^T\Delta \mathit{\tau },\hfill \end{array}$$

(37)

$$\begin{array}{c}\hfill \Delta {\mathit{\tau }}^T\mathit{L}\Delta \dot{\mathit{\tau }}=-\Delta {\mathit{\tau }}^T{\tilde{\mathit{W}}}_2^T{\Psi }_2-\Delta {\mathit{\tau }}^T{\mathit{K}}_2\Delta \mathit{\tau }-\Delta {\mathit{\tau }}^T\Delta \dot{\mathit{q}}+\Delta {\mathit{\tau }}^T{\mathit{\omega }}_2+\Delta {\mathit{\tau }}^T{\mathit{u}}_0.\end{array}$$

(38)

3.2. Adaptive Update Method

To update the unknown parameter estimation in real-time while the controller drives the manipulator, various adaptive parameter update methods are introduced in this subsection. To compute the desired joint velocity and guarantee the robustness and the astringency of the image error, the prior knowledge regarding the known range of the unknown parameter $\mathit{\theta }$, which is not hard to obtain by a rough measurement, is defined by convex compact sets $\mathbf{\Theta }$. It is given as

$$\begin{array}{c}\hfill \mathbf{X}=\{{\widehat{\theta }}_i\left(0\right)\in \widehat{\mathit{\theta }}\left(0\right)|{\underset{¯}{\theta }}_i<{\widehat{\theta }}_i\left(0\right)<{\overline{\theta }}_i\},\end{array}$$

(39)

where $\underset{¯}{{\theta }_i}$ is the lower bound of the convex set and ${\overline{\theta }}_i$ is the upper bound. The estimated error of compound parameters θ is defined as

$$\begin{array}{c}\hfill ({\Delta \mathbf{\Lambda }}^T+{\displaystyle \frac{1}{2}}{\Delta \mathit{\lambda }}_3^T\Delta \mathit{y}\left(t\right))\Delta \mathit{y}\left(t\right)=({\widehat{\mathbf{\Lambda }}}^T+{\displaystyle \frac{1}{2}}{\widehat{\mathit{\lambda }}}_3^T\Delta \mathit{y}\left(t\right))\Delta \mathit{y}\left(t\right)-({\mathbf{\Lambda }}^T+{\displaystyle \frac{1}{2}}{\mathit{\lambda }}_3^T\Delta \mathit{y}\left(t\right))\mathit{K}\Delta \mathit{y}\left(t\right).\end{array}$$

(40)

Further, by applying Proposition 2, the linear unknown parameter term is rewritten as

$$\begin{array}{c}\hfill {({\Delta \mathbf{\Lambda }}^T+{\displaystyle \frac{1}{2}}\Delta \mathit{y}\left(t\right){\Delta \mathit{\lambda }}_3)}^T\mathit{K}\Delta \mathit{y}\left(t\right)=\mathit{Z}(\mathit{q},\mathit{y})\Delta \mathit{\theta }.\end{array}$$

(41)

According to the above derivation, the adaptive parameter update rule for the unknown mapping relationship between the camera and the manipulator is designed as

$$\begin{array}{c}\hfill \dot{\widehat{\mathit{\theta }}}=-{\mathbf{{\rm Y}}}^{-1}{\mathit{Z}}^T(\mathit{q},\mathit{y})\dot{\mathit{q}}+\mathit{\eta },\end{array}$$

(42)

where $\mathbf{{\rm Y}}$ is a positive definite and diagonal matrix, and η is constructed as

$${\eta }_i=\left\{\begin{array}{cc}& \mathrm{if}{\underset{¯}{\theta }}_i<{\widehat{\theta }}_i<{\overline{\theta }}_i\hfill \\ 0,& \mathrm{or}{\widehat{\theta }}_i={\underset{¯}{\theta }}_i\mathrm{and}{\phi }_i⩾0\hfill \\ & \mathrm{or}{\widehat{\theta }}_i={\overline{\theta }}_i\mathrm{and}{\phi }_i⩽0\hfill \\ -{\phi }_i,& \mathrm{otherwise}\end{array}\right.$$

(43)

where the term ${\phi }_i$ is defined as

$$\begin{array}{c}\hfill \mathit{\phi }=-{\mathbf{{\rm Y}}}^{-1}{\mathit{Z}}^T(\mathit{q},\mathit{y})\dot{\mathit{q}}.\end{array}$$

(44)

For the torque and the voltage controller, the corresponding RBF approximate weight matrices are updated by following the parameter update law:

$$\begin{array}{c}\hfill {\dot{\widehat{\mathit{W}}}}_1={\mathbf{\Gamma }}_1{\mathbf{\Psi }}_1\Delta {\dot{\mathit{q}}}^T,\end{array}$$

(45)

$$\begin{array}{c}\hfill {\dot{\widehat{\mathit{W}}}}_2={\mathbf{\Gamma }}_2{\mathbf{\Psi }}_2\Delta {\mathit{\tau }}^T,\end{array}$$

(46)

where the matrices ${\mathbf{\Gamma }}_1$ and ${\mathbf{\Gamma }}_2$ are skew symmetric positive definite update gain matrices.

The parameters of the robust compensators are estimated by

$$\begin{array}{c}\hfill {\dot{\widehat{\mathbf{\Omega }}}}_1={{\mathit{B}}_1}^{-1}|\Delta \dot{\mathit{q}}|,\end{array}$$

(47)

$$\begin{array}{c}\hfill {\dot{\widehat{\mathbf{\Omega }}}}_2={\mathit{B}}_2^{-1}|\Delta \mathit{\tau }|,\end{array}$$

(48)

where the matrices ${\mathit{B}}_1$ and ${\mathit{B}}_2$ are skew symmetric positive definite matrices.

4. System Stability

The efficacy of the proposed cascade controller is validated in this section by using the second Lyapunov method. The stability of the control system is given.

Theorem 1.

With controllers (20), (24), and (28) with adaptive rule (42) and (45)–(48) driving, the system stability and the closed-loop control task $\underset{t\to \infty }{lim}\Delta \mathit{y}=0$ of the visual manipulator with uncertainties in the kinematics and the dynamics and the unknown disturbance can be guaranteed, and the system state errors satisfy uniform boundedness (Algorithm 1).

Algorithm 1: Cascade adaptive visual servoing control algorithm.

Proof. 

The Lyapunov function applied to the stability proof is constructed as

$$\begin{array}{ccc}\hfill V\left(t\right)& =& {\displaystyle \frac{1}{2}}\{z\left(t\right)\Delta {\mathit{y}}^T\mathit{K}\Delta \mathit{y}+\Delta {\dot{\mathit{q}}}^{T}H\Delta \dot{\mathit{q}}+\Delta {\mathit{\tau }}^T\mathit{L}\Delta \mathit{\tau }\hfill \end{array}$$

(49)

$$\begin{array}{ccc}& & +{\Delta \mathit{\theta }}^T\mathbf{{\rm Y}}\Delta \mathit{\theta }+\sum \{{\tilde{\mathbf{\Omega }}}_i^T{\mathit{B}}_i{\tilde{\mathbf{\Omega }}}_i+Tr\{{\tilde{\mathit{W}}}_i^T{\Gamma }_i{\tilde{\mathit{W}}}_i\left\}\right\}\}.\hfill \end{array}$$

(50)

The system Lyapunov function is divided into three parts, corresponding to three controllers (20), (24), and (28), which are denoted by

$$\begin{array}{ccc}\hfill V_1\left(t\right)& =& {\displaystyle \frac{1}{2}}\{z\left(t\right)\Delta {\mathit{y}}^T\mathit{K}\Delta \mathit{y}+{\Delta \mathit{\theta }}^T\mathbf{{\rm Y}}\Delta \mathit{\theta }\},\hfill \end{array}$$

(51)

$$\begin{array}{ccc}\hfill V_2\left(t\right)& =& {\displaystyle \frac{1}{2}}\{\Delta {\dot{\mathit{q}}}^{T}H\Delta \dot{\mathit{q}}+{\tilde{\mathbf{\Omega }}}_1^T{\mathit{B}}_i{\tilde{\mathbf{\Omega }}}_1+Tr\left\{{\tilde{\mathit{W}}}_1^T{\Gamma }_1{\tilde{\mathit{W}}}_1\right\}\},\hfill \end{array}$$

(52)

$$\begin{array}{ccc}\hfill V_3\left(t\right)& =& {\displaystyle \frac{1}{2}}\{\Delta {\mathit{\tau }}^T\mathit{L}\Delta \mathit{\tau }+{\tilde{\mathbf{\Omega }}}_2^T{\mathit{B}}_2{\tilde{\mathbf{\Omega }}}_2+Tr\left\{{\tilde{\mathit{W}}}_2^T{\Gamma }_2{\tilde{\mathit{W}}}_2\right\}\}.\hfill \end{array}$$

(53)

Regarding the first Lyapunov subfunction, through the operation of differentiation, we can derive

$$\begin{array}{c}\hfill {\dot{V}}_1\left(t\right)={\displaystyle \frac{1}{2}}\dot{z}\left(t\right)\Delta {\mathit{y}}^T\mathit{K}\Delta \mathit{y}+z\left(t\right)\Delta {\mathit{y}}^T\mathit{K}\Delta \dot{\mathit{y}}+\Delta {\mathit{\theta }}^T\mathbf{{\rm Y}}\dot{\widehat{\mathit{\theta }}}\end{array}$$

(54)

To facilitate further derivation, the auxiliary term ${\dot{\mathit{q}}}^T{\mathit{q}}_d$ is introduced into the derivation process. Then,

$$\begin{array}{ccc}\hfill {\dot{V}}_1\left(t\right)& =& {\displaystyle \frac{1}{2}}\dot{z}\left(t\right)\Delta {\mathit{y}}^T\mathit{K}\Delta \mathit{y}+z\left(t\right)\Delta {\mathit{y}}^T\mathit{K}\Delta \dot{\mathit{y}}+\Delta {\mathit{\theta }}^T\mathbf{{\rm Y}}\dot{\widehat{\mathit{\theta }}}+{\dot{\mathit{q}}}^T{\mathit{q}}_d-{\dot{\mathit{q}}}^T{\mathit{q}}_d\hfill \\ & =& \dot{\mathit{q}}({\mathbf{\Lambda }}^T+{\displaystyle \frac{1}{2}}{\mathit{\lambda }}_3^T\Delta \mathit{y}\left(t\right))\mathit{K}\Delta \mathit{y}\left(t\right)+\Delta {\mathit{\theta }}^T\mathbf{{\rm Y}}\dot{\widehat{\mathit{\theta }}}\hfill \\ & & +{\dot{\mathit{q}}}^T{\mathit{q}}_d-{\dot{\mathit{q}}}^T({\widehat{\mathbf{\Lambda }}}^T+{\displaystyle \frac{1}{2}}{\widehat{\mathit{\lambda }}}_3^T\Delta \mathit{y}\left(t\right))\Delta \mathit{y}\left(t\right).\hfill \end{array}$$

(55)

By combining adaptive rule (42), this subfunction is derived as

$$\begin{array}{c}\hfill {\dot{V}}_1\left(t\right)=-{\dot{\mathit{q}}}^T{\mathit{q}}_d+\Delta {\mathit{\theta }}^T\mathit{Y}\mathit{\eta }.\end{array}$$

(56)

Following [29], we can know that

$$\begin{array}{c}\hfill \Delta {\mathit{\theta }}^T\mathit{Y}\mathit{\eta }⩽0.\end{array}$$

(57)

Hence, we obtain

$$\begin{array}{c}\hfill {\dot{V}}_1\left(t\right)⩽-{\dot{\mathit{q}}}^T{\mathit{q}}_d.\end{array}$$

(58)

Further, the second Lyapunov Function (52) is subjected to processing. It is differentiated, and closed-loop robotic dynamic Equation (37) is invoked.

$$\begin{array}{ccc}\hfill {\dot{V}}_2\left(t\right)& =& \Delta {\dot{\mathit{q}}}^T\mathit{M}\Delta \ddot{\mathit{q}}+{\displaystyle \frac{1}{2}}\Delta {\dot{\mathit{q}}}^T\dot{\mathit{M}}\Delta \dot{\mathit{q}}+{\tilde{\mathbf{\Omega }}}_1^T{\mathit{B}}_1{\tilde{\mathbf{\Omega }}}_1+Tr\left\{{\tilde{\mathit{W}}}_1^T{\Gamma }_1{\tilde{\mathit{W}}}_1\right\}\hfill \\ & =& -\Delta {\dot{\mathit{q}}}^T{\tilde{\mathit{W}}}_1^T{\Psi }_1+\Delta {\dot{\mathit{q}}}^T{\dot{\mathit{q}}}_d-{\Delta \dot{\mathit{q}}}^T{\mathit{K}}_1\Delta \dot{\mathit{q}}+\Delta {\dot{\mathit{q}}}^T{\mathit{\omega }}_1+\Delta {\dot{\mathit{q}}}^T{\mathit{\tau }}_{\mathbf{0}}\hfill \\ & & +\Delta {\dot{\mathit{q}}}^T\Delta \mathit{\tau }+{\tilde{\mathbf{\Omega }}}_1^T{\mathit{B}}_1{\tilde{\mathbf{\Omega }}}_1+Tr\left\{{\tilde{\mathit{W}}}_1^T{\Gamma }_1{\dot{\tilde{\mathit{W}}}}_1\right\}.\hfill \end{array}$$

(59)

We combine RBF update rule (45) and adaptive robust compensator (31) with the corresponding update law (47):

$$\begin{array}{ccc}\hfill {\dot{V}}_2\left(t\right)& =& -\Delta {\dot{\mathit{q}}}^T{\tilde{\mathit{W}}}_1^T{\Psi }_1+\Delta {\dot{\mathit{q}}}^T{\dot{\mathit{q}}}_d-{\Delta \dot{\mathit{q}}}^T{\mathit{K}}_1\Delta \dot{\mathit{q}}+\sum _i^n\{\Delta {\dot{q}}_i^T({\omega }_{1i}-{\widehat{\Omega }}_{1i}\mathrm{sgn}(\Delta {\dot{q}}_i))\}\hfill \\ & & +\Delta {\dot{\mathit{q}}}^T\Delta \mathit{\tau }+{\tilde{\mathbf{\Omega }}}_1^T|\Delta \dot{\mathit{q}}|+Tr\left\{{\tilde{\mathit{W}}}_1^T{\Gamma }_1{\dot{\tilde{\mathit{W}}}}_1\right\}\hfill \\ \hfill & ⩽& \Delta {\dot{\mathit{q}}}^T{\tilde{\mathit{W}}}_1^T{\Psi }_1-\Delta {\dot{\mathit{q}}}^T{\dot{\mathit{q}}}_d-{\Delta \dot{\mathit{q}}}^T{\mathit{K}}_1\Delta \dot{\mathit{q}}+\sum _i^n\{\left|\Delta {\dot{q}}_i^T\right|\Omega -\widehat{\Omega }\left|\Delta {\dot{q}}_i\right|\}\hfill \\ \hfill & & +\Delta {\dot{\mathit{q}}}^T\Delta \tau +{\tilde{\mathbf{\Omega }}}_1^T|\Delta \dot{\mathit{q}}|+Tr\left\{{\tilde{\mathit{W}}}_1^T{\Gamma }_1{\dot{\tilde{\mathit{W}}}}_1\right\}\hfill \\ & ⩽& \Delta {\dot{\mathit{q}}}^T{\dot{\mathit{q}}}_d-{\Delta \dot{\mathit{q}}}^T{\mathit{K}}_1\Delta \dot{\mathit{q}}+\Delta {\dot{\mathit{q}}}^T\Delta \tau .\hfill \end{array}$$

(60)

For the Lyapunov subfunction associated with the motor section in (53), we derive its derivative as

$$\begin{array}{c}\hfill {\dot{V}}_3\left(t\right)=\Delta {\mathit{\tau }}^T\mathit{L}\Delta \dot{\mathit{\tau }}+{\tilde{\mathbf{\Omega }}}_2^T{\mathit{B}}_2{\dot{\tilde{\mathbf{\Omega }}}}_2+Tr\left\{{\tilde{\mathit{W}}}_2^T{\Gamma }_2{\dot{\tilde{\mathit{W}}}}_2\right\}.\end{array}$$

(61)

Analogously, by combining closed-loop dynamics (38) and adaptive rules (46) and (48), we can derive

$$\begin{array}{ccc}\hfill {\dot{V}}_3\left(t\right)& =& -\Delta {\mathit{\tau }}^T{\tilde{\mathit{W}}}_2^T{\Psi }_2-\Delta {\mathit{\tau }}^T{\mathit{K}}_2\Delta \mathit{\tau }-\Delta {\mathit{\tau }}^T\Delta \dot{\mathit{q}}+\Delta {\mathit{\tau }}^T{\mathit{\omega }}_2+\Delta {\mathit{\tau }}^T{\mathit{u}}_0\hfill \\ & & +{\tilde{\mathbf{\Omega }}}_2^T{\mathit{B}}_2{\dot{\tilde{\mathbf{\Omega }}}}_2+Tr\left\{{\tilde{\mathit{W}}}_2^T{\Gamma }_2{\dot{\tilde{\mathit{W}}}}_2\right\}\hfill \\ & ⩽& -\Delta {\mathit{\tau }}^T{\mathit{K}}_2\Delta \mathit{\tau }+\sum _i^n\{\Delta {\mathit{\tau }}_i{\Omega }_{2i}-{\widehat{\Omega }}_{2i}\Delta {\mathit{\tau }}_i\}+{\tilde{\mathbf{\Omega }}}_2^T|\Delta \dot{\mathit{\tau }}|-\Delta {\mathit{\tau }}^T\Delta \dot{\mathit{q}}\hfill \\ & ⩽& -\Delta {\mathit{\tau }}^T{\mathit{K}}_2\Delta \mathit{\tau }-\Delta {\mathit{\tau }}^T\Delta \dot{\mathit{q}}.\hfill \end{array}$$

(62)

Furthermore, we can directly obtain that the derivative of the Lyapunov function for the entire system satisfies the following constraints:

$$\begin{array}{c}\hfill \dot{V}\left(t\right)⩽-\Delta {\mathit{\tau }}^T{\mathit{K}}_2\Delta \mathit{\tau }-{\Delta \dot{\mathit{q}}}^T{\mathit{K}}_1\Delta \dot{\mathit{q}}-{\dot{\mathit{q}}}_d^T{\dot{\mathit{q}}}_d.\end{array}$$

(63)

Hence, $V⩾0$ and $\dot{V}⩽0$ are proven. The Lyapunov function V is bound. Further, by applying Barbalat’s lemma, we obtain

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}{\dot{\mathit{q}}}_d=0,\end{array}$$

(64)

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}\Delta \mathit{q}=0,\end{array}$$

(65)

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}\Delta \mathit{\tau }=0.\end{array}$$

(66)

By combing (64) and (65), it is easy to derive

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}\dot{\mathit{q}}=0.\end{array}$$

(67)

At the equilibrium point, the reference joint velocity (20) is cited again. Based on the inference mentioned above, we can obtain

$$\begin{array}{c}\hfill \underset{\mathbf{\Theta }(\mathit{\theta },\mathit{y})}{\underbrace{({\widehat{\mathbf{\Lambda }}}^T+{\displaystyle \frac{1}{2}}{\widehat{\mathit{\lambda }}}_3^T\Delta \mathit{y}\left(t\right))}}\mathit{K}\Delta \mathit{y}\left(t\right)=0.\end{array}$$

(68)

A new matrix is constructed as

$$\begin{array}{c}\hfill {\mathbf{\Theta }}^T\mathbf{\Theta }=\left(\begin{array}{cc}\mathrm{N}& {\mathbf{0}}_{2\times 1}\\ {\mathbf{0}}_{1\times 2}& 0\end{array}\right).\end{array}$$

(69)

Hence, $Rank({\mathbf{\Theta }}^T\mathbf{\Theta })=Rank\left(\mathrm{N}\right)$ can be known. If the matrix $\mathrm{N}$ is singular, the following constraint must be satisfied, i.e.,

$$\begin{array}{c}\hfill {\Theta }_{11}\times {\Theta }_{22}-{\Theta }_{12}\times {\Theta }_{21}=0,\\ \hfill {\Theta }_{11}\times {\Theta }_{23}-{\Theta }_{13}\times {\Theta }_{21}=0,\\ \hfill {\Theta }_{12}\times {\Theta }_{23}-{\Theta }_{13}\times {\Theta }_{22}=0,\end{array}$$

(70)

which is impossible to satisfied under the complex set of constraints in the adaptive rule. Hence we can obtain the conclusion that the matrix $\mathrm{N}$ is invertible, and the rank of $\mathbf{\Theta }$ is 2. Therefore, we conclude as follows:

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}\Delta \mathit{y}=0.\end{array}$$

(71)

□

Remark 6.

The consideration of the orientation of the end-effector is particularly important for practical applications. This is also a challenging problem for the study. To overcome it, a possible solution is provided below. By selecting multiple feature points, the concrete kinematic joint velocity controller can be designed in the following form:

$$\begin{array}{c}\hfill {\dot{q}}_r=-\left(\begin{array}{c}({\widehat{\mathbf{\Lambda }}}_1^T+{\displaystyle \frac{1}{2}}{\widehat{\mathit{\lambda }}}_1^T\Delta {\mathit{y}}_1)K\Delta {\mathit{y}}_1\\ ({\widehat{\mathbf{\Lambda }}}_2^T+{\displaystyle \frac{1}{2}}{\widehat{\mathit{\lambda }}}_2^T\Delta {\mathit{y}}_2)K\Delta {\mathit{y}}_2\\ ⋮\\ ({\widehat{\mathbf{\Lambda }}}_m^T+{\displaystyle \frac{1}{2}}{\widehat{\mathit{\lambda }}}_m^T\Delta {\mathit{y}}_m)K\Delta {\mathit{y}}_m\end{array}\right)\end{array}$$

(72)

where m is the number of the feature points marked on the end-effector. The adaptive law is designed as

$$\begin{array}{c}\hfill \dot{\widehat{\mathit{\theta }}}=-{\mathit{Y}}^{-1}{\mathit{Z}}^T(\mathit{q},{\mathit{y}}_1,{\mathit{y}}_2,\dots ,{\mathit{y}}_m)\dot{\mathit{q}}+\mathit{\eta }\end{array}$$

(73)

$${\eta }_i=\left\{\begin{array}{cc}& \mathit{if}{\underset{-}{\theta }}_i<{\widehat{\theta }}_i<{\overline{\theta }}_i\hfill \\ 0& \mathit{or}{\widehat{\theta }}_i={\underset{-}{\theta }}_i \mathit{and}{\phi }_{i}0\hfill \\ & \mathit{or}{\widehat{\theta }}_i={\overline{\theta }}_i\mathit{and}{\phi }_{i}0\hfill \\ & -{\phi }_i\mathit{otherwise}\hfill \end{array}\right.$$

(74)

$$\begin{array}{c}\hfill \mathit{\phi }=-{\mathit{Y}}^{-1}{\mathit{Z}}^T(\mathit{q},{\mathit{y}}_1,{\mathit{y}}_2,\dots ,{\mathit{y}}_m)\dot{\mathit{q}}\end{array}$$

(75)

The non-negative function of the image loop is given as

$$\begin{array}{c}\hfill V\left(t\right)={\displaystyle \frac{1}{2}}\{z\Delta {\mathit{y}}^T\mathit{Ky}+\Delta {\mathit{\theta }}^T\mathrm{{\rm Y}}\Delta \mathit{\theta }\}\end{array}$$

(76)

Then, the derivation of this Lyapunov function is

$$\begin{array}{c}\hfill {\dot{V}}_1\left(t\right)={\displaystyle \frac{1}{2}}\dot{z}\left(t\right)\Delta {\mathit{y}}^T\mathit{K}\Delta \mathit{y}+z\left(t\right)\Delta {\mathit{y}}^T\mathit{K}\Delta \dot{\mathit{y}}+\Delta {\mathit{\theta }}^T\mathbf{{\rm Y}}\dot{\widehat{\mathit{\theta }}}\end{array}$$

(77)

Similarly, we can obtain

$$\begin{array}{c}\hfill {\dot{V}}_1\left(t\right)⩽-{\dot{\mathit{q}}}^T{\mathit{q}}_d.\end{array}$$

(78)

When the image errors are convergent to the equilibrium points, the joint position and the kinematic parameters estimates can be considered as the constant vectors. We denote the transformation matrix of the feature points as $\mathit{T}$. We can have

$$\begin{array}{c}\hfill {\mathit{y}}_i={\displaystyle \frac{1}{{z\mathit{Tp}}_i}}\mathbf{\Xi }\mathit{T}{\mathit{p}}_i\end{array}$$

(79)

The transformation matrix can be represented as six unknown variables. When the number of feature points is larger than $2 m$ and $n⩾6$, the kinematic joint velocity controller can be realized as n nonlinear equations with unique solution, which represent only one set of image vectors that can satisfy it. If n nonlinear equations have more that one solution, the image errors may not be convergent to zero.

5. Simulation

The simulation of this study is based on Matlab-simulink (R2021b) and the robotic toolbox. Based on the above analysis, it is evident that only the spatial coordinates of the end-effector of the robotic arm are required for the entire control task, without consideration of its orientation. Therefore, in this simulation, a simplified three-degree-of-freedom (3-DOF) PUMA560 robot is utilized to verify the validity of the proposed scheme. The system configuration is the electrically driven PMUA560 robotic manipulator with an unmovable camera. The system frame is showcased in Figure 3, and the simulation system configuration is as shown in Figure 4.

The projection matrix of the camera is given as

$$\begin{array}{c}\hfill \mathbf{\Xi }=\left[\begin{array}{cccc}\begin{array}{c}750\\ 0\\ 0\end{array}& \begin{array}{c}362\\ -168.3\\ 0.7071\end{array}& \begin{array}{c}-362\\ -892.4\\ -0.7071\end{array}& \begin{array}{c}281.9\\ 506.9\\ 0.9899\end{array}\end{array}\right].\end{array}$$

(80)

The robotic manipulator is described by the Denavit–Hartenberg (D-H) method, which includes the kinematic parameters of the robot. Combining the dynamic parameters, the robotic physical parameters are showcased in

Table 2. Parameters of the manipulator.

Joint	Angle (rad)	Offset (m)	Length (m)	Twist (rad)	Mass (kg)
1	$q_1$	0	0	$\pi /2$	0
2	$q_2$	0	0.4318	0	$17.4$
3	$q_3$	0.15005	0.0203	$-\pi /2$	$4.8$

.

The parameters of the joint actuator DC motor are set to

$$\begin{array}{c}\hfill \mathit{L}=\left[\begin{array}{ccc}0.08& 0& 0\\ 0& 0.0805& 0\\ 0& 0& 0.0788\end{array}\right],\\ \hfill \mathit{R}=\left[\begin{array}{ccc}5& 0& 0\\ 0& 4.758& 0\\ 0& 0& 5.212\end{array}\right],\mathit{N}=\left[\begin{array}{ccc}0.7& 0& 0\\ 0& 0.721& 0\\ 0& 0& 0.711\end{array}\right].\end{array}$$

(81)

The parameter vector $\mathit{\theta }$ = [323.8500, 156.3283, −156.3283, 0, −72.6682, −385.3248, 0, 0.3053, −0.3053, 363.975, 175.6974, −175.6974, 0, −81.6718, −433.0666, 0, 0.3432, −0.3432, 97.1250, 46.884, −46.884, 0, −21.7937, −115.5618, 0, 0.0916, −0.0916]. The image error gain matrix $\mathit{K}=0.0015$ × ${\mathit{I}}_3$, the velocity error gain matrix ${\mathit{K}}_1=diag(40,55,45)$, and the torque error gain matrix ${\mathit{K}}_2=diag(17,1.2,15)$. The adaptive gain matrices ${\mathbf{\Gamma }}_{\mathbf{1}}=114$ × ${\mathit{I}}_n$, ${\mathbf{\Gamma }}_{\mathbf{2}}=1.75$ × ${\mathit{I}}_n$, ${\mathit{B}}_{\mathbf{1}}=diag(0.5,0.3,0.45)$, and ${\mathit{B}}_{\mathbf{2}}=diag(4,7,0.3)$, where n is the node number of the RBFNNs.

In this simulation, the simulation time is 2 s, and the convergence time of the system is less than 0.8 s. As shown in Figure 5, the image tracking converges to zero, demonstrating that the proposed control strategy is capable of accomplishing the primary control task. In Figure 6, the astringency of the joint angles $\mathit{q}$, angle velocity $\dot{\mathit{q}}$, and actuator torque τ of the robotic manipulator is observed. As the conclusion in the stability analysis (67), the joint velocity $\dot{\mathit{q}}$ converges to zero. Further, we can observe that the tracking errors of the joint angular velocity of the robotic manipulator and the torque output of the motor also converge to zero in Figure 7. It can be observed that when the time reaches 0.6 s, all system errors converge to a minimal value, and the various states of the system begin to stabilize.

The track of the feature point projection on the image plane is shown in Figure 8. The trajectory of the feature points in three-dimensional space is depicted in Figure 9. It should be noted that in our control scheme, the reference joint velocity computed by the kinematic controller only guarantees the astringency of the image error, which means that the end-effector cannot reach the desired space position in the uncalibrated environment. To ensure that the end-effector reaches the desired position in three-dimensional space, it is necessary to select multiple non-collinear feature points on the end-effector. In the control system simulation, the variation trend of the adaptive parameters of the robust compensator ${\widehat{\mathbf{\Omega }}}_1,{\widehat{\mathbf{\Omega }}}_2$ are recorded in Figure 10.

Furthermore, we propose the use of the Euclidean norm as a measure of accuracy. Represented by the norm, it directly indicates the convergence of multidimensional data in the experiment. Through computation, the norms of the image error, joint velocity error, and torque tracking are smaller than $1.2$ (Pixel), 0.01 (rad/s), and 20 (N·m) after the convergence of the control system.

Through the above simulation result, the boundness and astringency of the system states and the tracking errors are showcased, which means that Theorem 1, presented in this paper, is verified. Under the uncertainties in the kinematics and dynamics, the proposed control scheme utilizes state and output feedback to regulate the input voltage of the manipulator joint actuators, thereby accomplishing the task of image position tracking.

Furthermore, for the robustness and anti-interference capabilities of the controller, Gaussian noise is introduced into both the torque controller and the voltage controller. Considering the range of the control signal, Gaussian noise with a mean of 0 and variances of 300 and 700 is introduced into the two controllers. Here, we present the simulation results, including the joint torque error $\Delta \mathit{\tau }$, joint velocity error $\Delta \mathit{q}$, and image error $\Delta \mathit{y}$, in Figure 11.

It can be observed that compared with the scenario without random disturbances, the control performance does not significantly degrade even with the introduction of significant disturbances. These results demonstrate that the proposed adaptive RBFNN controller exhibits robustness against system parameter uncertainties and random disturbances.

Remark 7.

The effectiveness of the proposed control scheme is mainly verified by theoretical analysis and simulation results. Presently, how to confirm it further by experimental results based on robotic dynamics is still difficult for us. Despite the difficulties, we have built a hardware system platform for the preparation of experimental tests. In our future study, we will first perform a practical test based on the kinematic model of the manipulator and then generalize the results to the dynamic model, so that the control scheme proposed in this work can be well validated by theoretical analysis, numerical simulation, and experimental tests simultaneously.

6. Conclusions

For the electrically driven robotic manipulator in an uncalibrated environment, we propose an adaptive RBF cascade visual servoing control scheme to solve the problem of the uncertainties from the camera and the manipulator. The proposed control policy does not require any prior knowledge of the robot dynamics; it only necessitates a rough measurement of the camera and robot kinematics. By dividing the system into three subsystems—the image loop, the joint velocity loop, and the torque loop—the uncertainties can be addressed separately within each subsystem. Additionally, an adaptive update algorithm and RBFNNs are employed to estimate and compensate for unknown camera parameters and uncertainties in kinematics and dynamics. Furthermore, corresponding adaptive robust mechanisms are introduced to ensure the stability of the system. Through Lyapunov theory, the global stability of the system with the proposed control scheme and the boundness and astringency of the system states and tracking error are proven. However, the proposed control scheme is primarily evaluated through theoretical analysis, rigorous stability proofs, and simulations as limited practical verification of its effectiveness.

Although the simulation results show that the image tracking errors under different initial values can converge to a residual around zero in finite time, how to prove this finite-time convergence property theoretically, i.e., establishing some Lyapunov-based conditional inequalities for ensuring finite-time or fixed-time stability, is still difficult for us, which will be regarded as a future research topic and the extension of this work.