Geometry-Constrained Learning-Based Visual Servoing with Projective Homography-Derived Error Vector

Abstract

We propose a novel geometry-constrained learning-based method for camera-in-hand visual servoing systems that eliminates the need for camera intrinsic parameters, depth information, and the robot’s kinematic model. Our method uses a cerebellar model articulation controller (CMAC) to execute online Jacobian estimation within the control framework. Specifically, we introduce a fixed-dimension, uniform-magnitude error function based on the projective homography matrix. The fixed-dimension error function ensures a constant Jacobian size regardless of the number of feature points, thereby reducing computational complexity. By not relying on individual feature points, the approach maintains robustness even when some features are occluded. The uniform magnitude of the error vector elements simplifies neural network input normalization, thereby enhancing online training efficiency. Furthermore, we incorporate geometric constraints between feature points (such as collinearity preservation) into the network update process, ensuring that model predictions conform to the fundamental principles of projective geometry and eliminating physically impossible control outputs. Experimental and simulation results demonstrate that our approach achieves superior robustness and faster learning rates compared to other model-free image-based visual servoing methods.

1. Introduction

Visual servoing utilizes visual feedback to refine robot motion strategies, thereby significantly enhancing adaptability and intelligence. The visual system can be mounted on the end effector of the robot (eye-in-hand configuration), or positioned near the robot (hand-to-eye configuration) [1]. This study mainly focuses on the eye-in-hand configuration.

Visual servoing is generally divided into two main approaches: position-based visual servoing (PBVS) and image-based visual servoing (IBVS) [2,3,4]. PBVS operates in Cartesian space, enabling global convergence by directly associating image features with the pose of the target in 3D space. However, the PBVS is sensitive to camera calibration errors, inaccuracies in the kinematic model, and noise, often requiring precise calibration, making it challenging for non-experts. In contrast, IBVS defines its error signal in the image space, making it less sensitive to calibration errors that primarily affect convergence speed rather than accuracy. Nevertheless, IBVS is prone to local minima, and the appearance or disappearance of feature points during the control process can significantly disrupt the continuity of the control law [5]. An alternative approach is homography-based visual servoing (HBVS), which combines the advantages of PBVS and IBVS. HBVS is resilient to the partial occlusion of features and does not require prior information for 3D reconstruction. Studies [6,7,8,9] have decomposed the homography matrix to obtain the translation vector t and rotation matrix R, converting the image information into motion in the Cartesian space. However, decomposing the homography matrix during the control process increases computational demands and affects the real-time performance of the system. To circumvent this issue, one study [10] has proposed constructing a cost function directly from the elements of the homography matrix. However, this method often requires vanishing-point detection. To address these challenges, projective homography-based uncalibrated visual servoing (PHUVS) was introduced [11]. The PHUVS establishes the task function by estimating the projective homography using only image information within a certain range, eliminating the need for vanishing point detection. Moreover, it can produce Jacobian matrices of fixed dimensions, thereby reducing the computational complexity when inverting matrices.

Visual servoing methods typically rely on a deeper understanding of the Jacobian matrix. This matrix describes the mapping between the visual information and manipulator joint angular displacements. It is a local linear approximation of the nonlinear and highly coupled relationship between the visual and motion spaces, making it crucial for control algorithms. Traditional methods usually require accurate system kinematic models or precise estimations of the intrinsic camera parameters. These requirements significantly limit their applicability. To address the challenges of uncalibrated visual servoing, adaptive controllers have been proposed to estimate unknown parameters online [12,13,14]. However, these methods often require linear parameterization of the system modeling, which is complicated for nonlinear systems with multiple degrees of freedom. Consequently, model-free learning methods have been introduced to compensate for these uncertainties without the need for linear parameterization. These methods treat the estimation of a Jacobian matrix as a dynamic parameter identification problem. They employ recursive estimation techniques such as weighted recursive least squares, Broyden’s method, and dynamic Gauss–Newton algorithms [15,16]. Neural networks are renowned for their powerful nonlinear function approximation capabilities and have been widely applied in robotic systems [17,18,19,20,21,22,23]. However, their application in visual servoing systems is relatively limited. In [17], a method based on perceptron neural networks was proposed to learn the inverse mapping of an unknown interaction matrix; however, it required a large amount of data for offline training. Data-driven online mapping estimation methods have been designed [18,19]. In addition, ref. [20] proposed a data-driven IBVS method that addresses both the target tracking and physical constraints of the robot, such as joint velocity limits and FOV constraints. The study in [21] presented a robotic control system that uses a CMAC network with a Takagi–Sugeno fuzzy framework, capable of online learning joint velocities directly from image feature errors. Recent works have also explored deep learning-based visual servoing frameworks with improved robustness and noise tolerance. For instance, ref. [22] employs deep networks for 3D visual servoing without requiring precise calibration, while [23] introduces a noise-tolerant Jacobian estimation method using neural networks under pixel-level disturbances.

Traditional visual servoing systems often rely on handcrafted features such as SIFT or ORB, which can be sensitive to noise, lighting changes, and viewpoint variations. To improve robustness, learning-based methods like SuperPoint [24] and LightGlue [25] have been developed. These methods offer improved feature repeatability and matching accuracy under challenging conditions, making them promising complements to learning-based visual servoing frameworks.

Current visual servoing algorithms face two main challenges: (1) Existing model-free visual servoing approaches are predominantly based on IBVS. The loss of feature points can cause interruptions in the control process. However, due to changes in lighting and partial occlusion of feature points, it is difficult to ensure that every feature point is detected and accurately matched during control. Hence, IBVS is primarily designed for basic visual primitives, such as distinct points or edges. For objects with sparse or low-texture surfaces, feature matching errors become significantly higher, further degrading the reliability of the control process. Furthermore, the size of the image Jacobian matrix is proportional to the number of feature points, resulting in high computational costs when calculating the Jacobian pseudoinverse. When feature points are used as the inputs or outputs of a neural network, an increase in their number significantly expands the network’s dimensionality, thereby affecting training efficiency and computational complexity. This not only increases the number of network parameters but may also lead to optimization issues such as gradient vanishing or gradient explosion, ultimately impacting convergence speed and stability. (2) Current model-free visual servoing methods typically use neural networks or recursive algorithms to learn the Image Jacobian or map visual features to control commands. However, these approaches often ignore the geometric constraints among visual features, resulting in parameter updates that violate physical laws during training. Without explicitly incorporating these constraints, robots may execute unreasonable motions.

We develop a geometry-constrained learning-based control strategy based on PHUVS. The main contributions of this paper include:

• Learning-Based Control Strategy: A novel control method is proposed for PHUVS, where the CMAC neural network is employed to estimate the Jacobian matrix online. This eliminates the need for traditional calibration and kinematic modeling.
• Fixed-Dimension Visual Error Function via Homography: A new visual error function based on the projective homography matrix is designed. It maintains a fixed dimension and uniform magnitude across all components, thereby improving learning efficiency, reducing computational complexity, and offering robustness against feature point occlusion and detection errors.
• Incorporation of Geometric Constraints in Learning: Fundamental geometric relationships among visual features (e.g., collinearity) are embedded into the CMAC network’s learning process, enhancing both accuracy and convergence speed.

The remainder of this paper is organized as follows: Section 2 provides an overview of the camera/robot model and the projective homography matrix, while clearly defining the control problem. Section 3 details the developed controller. Section 4 presents the stability analysis of the proposed control system, providing theoretical guarantees for the controller’s performance. In Section 5, we analyze the simulation and experimental results, to demonstrate the efficacy of proposed methodology. Section 6 presents the conclusions and highlights the contributions of the study.

2. Preliminary Knowledge

2.1. Kinematics Model

As shown as Figure 1, {F_c} and {F_cd} represent the current and desired camera frames, respectively. Consider ${\mathbf{n}}_{\mathbf{d}}$ as the normal vector of the plane $\pi$ expressed in {F_cd} with length $\parallel {\mathbf{n}}_{\mathbf{d}}\parallel ={\displaystyle \frac{1}{d}}$. Here, d is the distance between the plane and center of the projection at the reference pose. $\mathbf{R}$ denotes the orientation of {F_cd} with respect to {F_c}, and $\mathbf{t}$ denotes the translation vector from {F_c} to {F_cd} expressed in the coordinate frame {F_c}.

Let ${\mathbf{p}}_{\mathbf{i}}$ be the point of the target in 3D space. The Euclidean coordinates expressed in {F_c} and {F_cd} are denoted:

$${\mathbf{p}}_{\mathbf{i}}^{\mathbf{c}}=\mathbf{H}{\mathbf{p}}_{\mathbf{i}}^{\mathbf{cd}},$$

(1)

$$\mathbf{H}=\mathbf{R}+\mathbf{t}{\mathbf{n}}_{\mathbf{d}}^{\mathbf{T}},$$

(2)

where $\mathbf{H}$ is the Euclidean homography matrix. This matrix describes how the 3D points lying on a planar surface are mapped between two Euclidean frames.

The projections of the 3D points into two image spaces are: ${\left[\begin{array}{cc}{\mathit{\epsilon }}_{\mathbf{di}}& 1\end{array}\right]}^T=\mathbf{K}{\displaystyle \frac{{\mathbf{p}}_{\mathbf{i}}^{\mathbf{cd}}}{Z_{di}}}$ and ${\left[\begin{array}{cc}{\mathit{\epsilon }}_{\mathbf{i}}& 1\end{array}\right]}^T=\mathbf{K}{\displaystyle \frac{{\mathbf{p}}_{\mathbf{i}}^{\mathbf{c}}}{Z_i}}$, where $\mathbf{K}$ is the camera intrinsic parameter matrix.

Subsequently, the relationship between ${\mathit{\epsilon }}_{\mathbf{i}}$ and ${\mathit{\epsilon }}_{\mathbf{di}}$ will be:

$$\left[\begin{array}{c}{\mathit{\epsilon }}_{\mathbf{i}}\\ 1\end{array}\right]={\displaystyle \frac{Z_{di}}{Z_i}}\mathbf{G}\left[\begin{array}{c}{\mathit{\epsilon }}_{\mathbf{di}}\\ 1\end{array}\right],$$

(3)

$$\mathbf{G}=\mathbf{KH}{\mathbf{K}}^{-\mathbf{1}},$$

(4)

where $\mathbf{G}$ is the projective homography matrix. In the absence of depth information, we can compute the matrix $\mathbf{G}$ up to a scale factor using only four or more non-collinear matched points $\left\{{\mathit{\epsilon }}_{\mathbf{i}},{\mathit{\epsilon }}_{\mathbf{di}}\right\},i\in \left\{1,2,3,4\cdots \right\}$:$\overline{\mathbf{G}}=\beta \mathbf{G}$.

$\overline{\mathbf{G}}$ is an estimation of $\mathbf{G}$ and the parameter $\beta \in R$ represents an arbitrary positive scale factor. It is crucial to note that this random ratio is independent of the depth ratio: ${\displaystyle \frac{Z_{di}}{Z_i}}$. Typically, the determinant of the $\overline{\mathbf{G}}$ is normalized to 1: $det\left(\overline{\mathbf{G}}\right)=1$ to eliminate the ambiguity introduced by the scale factor. In this case, the matrix $\overline{\mathbf{G}}$ can be represented as: T$\overline{\mathbf{G}}={\displaystyle \frac{\mathbf{G}}{det\left(\mathbf{G}\right)}}$. Then, the relationship between the positions of feature points in the two images can be rewritten as follows:

$$\left[\begin{array}{c}{\mathit{\epsilon }}_{\mathbf{i}}\\ 1\end{array}\right]={\gamma }_i\overline{\mathbf{G}}\left[\begin{array}{c}{\mathit{\epsilon }}_{\mathbf{di}}\\ 1\end{array}\right],$$

(5)

$${\gamma }_i={\displaystyle \frac{1}{{\overline{\mathbf{g}}}_{3,:}{\left(\left[\begin{array}{cc}{\mathit{\epsilon }}_{\mathbf{di}}^{\mathrm{T}}& 1\end{array}\right]\right)}^{\mathrm{T}}}},$$

(6)

where ${\overline{\mathbf{g}}}_{3,:}$ is the 3rd row of the matrix $\overline{\mathbf{G}}$. The subscript i denotes the index of the feature point. In the subsequent text, we use subscript c to denote the object center, where ${\mathit{\epsilon }}_{\mathbf{dc}}$ and ${\mathit{\epsilon }}_{\mathbf{c}}$ represent its desired and current positions in image space, respectively. The values of $\overline{\mathbf{G}}$ and ${\gamma }_i$ can be estimated by m (m > 4) pairs of non-collinear matched feature points. When $\overline{\mathbf{G}}$ tends to the identity matrix ${\mathbf{I}}_{3\times 3}$, it indicates that the two images are perfectly aligned.

In a camera-in-hand visual servoing setup, the camera is mounted on the robot’s end-effector. By solving the kinematics, the position vector ${\mathbf{r}}_{\mathbf{c}}^{\mathbf{b}}$ of the camera in the base frame is a function of the joint variables:

$${\mathbf{r}}_{\mathbf{c}}^{\mathbf{b}}=f\left(\mathbf{q}\right),$$

(7)

where $\mathbf{q}\in R^M$ is the joint angle vector of the manipulator, the spatial velocity of the end-effector can be obtained as:

$${\mathbf{v}}_{\mathbf{c}}^{\mathbf{b}}={\mathbf{J}}_{\mathbf{r}}\left(\mathbf{q}\right)\dot{\mathbf{q}},{\mathbf{J}}_{\mathbf{r}}\in R^{6\times n_q},$$

(8)

where $n_q$ is the number of robot joints, ${\mathbf{J}}_{\mathbf{r}}\left(\mathbf{q}\right)$ is the robot Jacobian, $\dot{\mathbf{q}}$ is the joint velocities, and ${\mathbf{v}}_{\mathbf{c}}^{\mathbf{b}}={\left[v_x,v_y,v_z,{\omega }_x,{\omega }_y,{\omega }_z\right]}^T$ is end-effector spatial velocity, which includes the linear velocity v and angular velocity $\omega$.

2.2. Problem Description and Motivation

Equation (8) describes the velocity mapping from the joint space to the camera motion in Cartesian space. To obtain the mapping from the joint space to the image space, we must construct an image Jacobian, which relates the camera’s velocities to the resulting changes in the image space.

In the methods of IBVS [12,13,14,15,16,17,18,19,20,21], the image error ${\tilde{\mathit{\epsilon }}}_{\mathbf{i}}={\mathit{\epsilon }}_{\mathbf{di}}-{\mathit{\epsilon }}_{\mathbf{i}}$ is directly used as the feedback signal. Its derivative and mapping to the joint space can be expressed as follows:

$$\dot{\tilde{\mathit{\epsilon }}}={\mathbf{J}}_{{\mathbf{c}}_{IBVS}}{\mathbf{J}}_{\mathbf{r}}\dot{\mathbf{q}}={\mathbf{J}}_{IBVS}\dot{\mathbf{q}},{\mathbf{J}}_{IBVS}\in R^{2P\times n_q}$$

(9)

where $\dot{\mathit{\epsilon }}={\left[{\dot{\mathit{\epsilon }}}_{\mathbf{1}}^{\mathbf{T}},{\dot{\mathit{\epsilon }}}_{\mathbf{2}}^{\mathbf{T}},\cdots {\dot{\mathit{\epsilon }}}_{\mathbf{P}}^{\mathbf{T}}\right]}^{\mathbf{T}}\in R^{2P\times 1}$. P and $n_q$ represent the numbers of feature points and robot joints, respectively. The image Jacobian matrix contains information on the feature point positions and image depth: ${\mathbf{J}}_{{\mathbf{c}}_{IBVS}}=\left[{\mathbf{J}}_{{\mathbf{c}}_{IBVS}}^{\mathbf{1}}\left({\mathit{\epsilon }}_{\mathbf{1}},Z_1\right);{\mathbf{J}}_{{\mathbf{c}}_{IBVS}}^{\mathbf{2}}\left({\mathit{\epsilon }}_{\mathbf{2}},Z_2\right);\dots {\mathbf{J}}_{{\mathbf{c}}_{IBVS}}^{\mathbf{N}}\left({\mathit{\epsilon }}_{\mathbf{P}},Z_P\right)\right]$.

Although depth-independent Jacobian matrices have been developed in some studies, several common challenges persist. First, the dimension of the Jacobian is proportional to the number of feature points, implying that as the number of feature points increases, the computational complexity also increases. Second, the task function typically minimizes the position errors of all feature points. When some feature points are occluded, they become unavailable. Changes in the number of feature points can lead to discontinuities in the task function, resulting in discontinuities in the control law.

In study [11], a task error based on the PHUVS was designed as follows:

$${\mathbf{e}}_{\overline{\mathbf{g}}}=\mathrm{re}shape\left(\mathbf{I}-\overline{\mathbf{G}},9,1\right)$$

(10)

The expression linking the derivative of ${\mathbf{e}}_{\overline{\mathbf{g}}}$ with the camera velocity is given as:

$${\dot{\mathbf{e}}}_{\overline{\mathbf{g}}}={\mathbf{J}}_{{\mathbf{c}}_{PHUVS}}\left({\mathbf{e}}_{\overline{\mathbf{g}}}\right){\mathbf{v}}_{\mathbf{c}}^{\mathbf{b}},$$

where the image Jacobian ${\mathbf{J}}_{{\mathbf{c}}_{PHUVS}}\left({\mathbf{e}}_{\overline{\mathbf{g}}}\right)\in R^{9\times 6}$. Combining with Equation (8), the relationship between ${\dot{\mathbf{e}}}_{\overline{\mathbf{g}}}$ and $\dot{\mathbf{q}}$ can be determined as:

$${\dot{\mathbf{e}}}_{\overline{\mathbf{g}}}={\mathbf{J}}_{PHUVS}\left({\mathbf{e}}_{\overline{\mathbf{g}}},\mathbf{q}\right)\dot{\mathbf{q}},{\mathbf{J}}_{PHUVS}\left({\mathbf{e}}_{\overline{\mathbf{g}}},\mathbf{q}\right)\in R^{9\times n_q}$$

(11)

where ${\mathbf{J}}_{PHUVS}\left({\mathbf{e}}_{\overline{\mathbf{g}}},\mathbf{q}\right)={\mathbf{J}}_{{\mathbf{c}}_{PHUVS}}\left({\mathbf{e}}_{\overline{\mathbf{g}}}\right)·{\mathbf{J}}_{\mathbf{r}}\left(\mathbf{q}\right)$ denotes the system Jacobian matrix.

The error function ${\mathbf{e}}_{\overline{\mathbf{g}}}$ in Equation (10) consists of nine elements, each representing a different type of geometric transformation, such as rotation, scaling, perspective, and translation. In conventional neural networks, normalization is challenging due to the uncertain range of each element. Similarly, in the CMAC network, the input quantization process becomes difficult as the elements vary significantly in their numerical scales and physical meanings, complicating the discretization into appropriate regions.

To address the above issues, we first reconstructed a fixed-dimension, uniform-magnitude visual error function. It is important to note that the error function we designed differs from the one presented in reference [11], specifically defined in Equation (10). In contrast, our error function consists of only 10 elements, each with consistent physical meanings and uniform magnitude, which simplifies the input quantization process for the neural network.

Despite the improved error function, a key limitation remains: existing model-free learning approaches update network weights without explicitly enforcing geometric constraints. This omission can lead to physically inconsistent mappings, such as collinear points deviating from collinearity after projection. To address this, we integrate geometric constraints into the weight update process. Specifically, we incorporate regularization terms into the loss function to preserve geometric consistency and employ projected gradient descent to ensure physically meaningful updates.

3. Controller Development

3.1. Novel Fixed-Dimension, Uniform-Magnitude Task Error Function

Assume that the camera captures an image at a reference position. The objective is to adjust the pose of the robot to ensure the current image aligns with the reference image, effectively bringing the camera back to its reference position.

This process involves the detecting and matching of image feature points. In Figure 2, the blue points represent the matched feature point pairs: $\left\{{\mathit{\epsilon }}_{\mathbf{j}},{\mathit{\epsilon }}_{\mathbf{dj}}\right\},j=1,2\dots P$, which are used to estimate the projective homography matrix $\overline{\mathbf{G}}$. To ensure robust estimation of this transformation, we use enough feature points across both images. Having sufficient points provides stability in the homography estimation, as it reduces the sensitivity to noise or small mismatches at individual points. Robust estimation methods, such as random sample consensus (RANSAC), help filter out outliers and produce an accurate homography matrix $\overline{\mathbf{G}}$.

To achieve a fixed-dimensional Jacobian, we introduce five fixed, non-collinear virtual reference feature points around the object center. The expected position of each virtual feature point in the reference image is defined as:

$$\left[\begin{array}{c}{\mathit{\epsilon }}_{\mathrm{d}1}^{\left(\mathrm{aux}\right)}\\ {\mathit{\epsilon }}_{\mathrm{d}2}^{\left(\mathrm{aux}\right)}\\ {\mathit{\epsilon }}_{\mathrm{d}3}^{\left(\mathrm{aux}\right)}\\ {\mathit{\epsilon }}_{\mathrm{d}4}^{\left(\mathrm{aux}\right)}\\ {\mathit{\epsilon }}_{\mathrm{d}5}^{\left(\mathrm{aux}\right)}\end{array}\right]=\left[\begin{array}{c}{\mathit{\epsilon }}_{\mathrm{dc}}^{\left(\mathrm{aux}\right)}+{\left[\begin{array}{cc}-\Delta u& \Delta v\end{array}\right]}^T\\ {\mathit{\epsilon }}_{\mathrm{dc}}^{\left(\mathrm{aux}\right)}+{\left[\begin{array}{cc}-\Delta u& -\Delta v\end{array}\right]}^T\\ {\mathit{\epsilon }}_{\mathrm{dc}}^{\left(\mathrm{aux}\right)}+{\left[\begin{array}{cc}\Delta u& \Delta v\end{array}\right]}^T\\ {\mathit{\epsilon }}_{\mathrm{dc}}^{\left(\mathrm{aux}\right)}+{\left[\begin{array}{cc}\Delta u& -\Delta v\end{array}\right]}^T\\ {\mathit{\epsilon }}_{\mathrm{dc}}^{\left(\mathrm{aux}\right)}\end{array}\right]$$

(12)

where ${\mathit{\epsilon }}_{\mathbf{dc}}$ represents the center of the object in the reference image, and $\Delta u$ and $\Delta v$ are the displacements of the feature points in the u and v directions, respectively.

We define an error matrix as follows:

$${\mathbf{E}}^{\mathbf{i}}=\mathbf{I}-{\gamma }_i^{\left(\mathbf{aux}\right)}\overline{\mathbf{G}}$$

(13)

$${\gamma }_i^{\left(\mathbf{aux}\right)}={\displaystyle \frac{1}{{\overline{\mathbf{g}}}_{3,:}{\left[\begin{array}{cc}{\mathit{\epsilon }}_{\mathbf{di}}^{\left(\mathbf{aux}\right)\mathrm{T}}& 1\end{array}\right]}^{\mathrm{T}}}}$$

(14)

The actual position of the i-th virtual feature point in the current image is denoted as ${\mathit{\epsilon }}_{\mathbf{i}}^{\left(\mathbf{aux}\right)}$. Based on Equations (5), (6), (13) and (14), the position error can be rewritten as:

$${\tilde{\mathit{\epsilon }}}_{\mathbf{i}}^{\left(\mathbf{aux}\right)}={\mathit{\epsilon }}_{\mathbf{di}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{i}}^{\left(\mathbf{aux}\right)}={\mathbf{E}}_{\mathbf{1}:\mathbf{2},:}^{\mathbf{i}}{\left[\begin{array}{cc}{\mathit{\epsilon }}_{\mathbf{di}}^{\left(\mathbf{aux}\right)\mathbf{T}}& 1\end{array}\right]}^T={\mathbf{e}}_{\mathbf{i}},$$

(15)

where ${\mathbf{E}}_{\mathbf{1}:\mathbf{2},:}^{\mathbf{i}}$ represents the first two rows of the matrix ${\mathbf{E}}^{\mathbf{i}}$.

Furthermore, ${\mathbf{e}}_{\mathbf{i}}$ is our designed error task function for the i-th feature. If the estimation $\overline{\mathbf{G}}$ is precise, ${\mathbf{e}}_{\mathbf{i}}$ is equivalent to the feature point position error ${\tilde{\mathit{\epsilon }}}_{\mathbf{i}}^{\left(\mathbf{aux}\right)}$.

We combine the error vectors for the four virtual feature points into a single error task vector:

$$\mathbf{e}=\left[\begin{array}{c}{\mathbf{e}}_{\mathbf{1}}\\ {\mathbf{e}}_{\mathbf{2}}\\ ⋮\\ {\mathbf{e}}_{\mathbf{5}}\end{array}\right]=\left[\begin{array}{c}{\mathbf{E}}_{\mathbf{1}:\mathbf{2},:}^{\mathbf{1}}{\mathit{\epsilon }}_{\mathbf{d}\mathbf{1}}^{\left(\mathbf{aux}\right)}\\ {\mathbf{E}}_{\mathbf{1}:\mathbf{2},:}^{\mathbf{2}}{\mathit{\epsilon }}_{\mathbf{d}\mathbf{2}}^{\left(\mathbf{aux}\right)}\\ ⋮\\ {\mathbf{E}}_{\mathbf{1}:\mathbf{2},:}^{\mathbf{5}}{\mathit{\epsilon }}_{\mathbf{d}\mathbf{5}}^{\left(\mathbf{aux}\right)}\end{array}\right],\mathbf{e}\in R^{10\times 1},$$

(16)

Theorem 1.

The task error function $\mathbf{e}=\mathbf{0}$ if and only if $\mathbf{R}=\mathbf{I}$ and $\mathbf{t}=\mathbf{0}$, which is proven in Appendix A.

As mentioned earlier, in the traditional IBVS, it is essential to accurately track each image feature point throughout the entire visual servoing process. This requirement can be stringent and poses challenges, particularly in dynamic environments where feature points may become obscured. In contrast, our PHUVS-based control system can compute the projective homography matrix using a local set of feature points. RANSAC is employed to filter outliers and enhance the robustness of the system to noise. Furthermore, even if the i-th virtual feature point is occluded, we can still determine the scale factor ${\gamma }_i^{\left(\mathbf{aux}\right)}$ and its corresponding error matrix based on Equations (13) and (14). This implies that the loss of certain feature points do not affect the computation of the task function.

If the projective homography matrix is estimated precisely, the vector error ${\mathbf{e}}_{\mathbf{i}}$ is equivalent to the position error ${\tilde{\mathit{\epsilon }}}_{\mathbf{i}}^{\left(\mathbf{aux}\right)}$ for the i-th auxiliary feature point. Each element of ${\mathbf{e}}_{\mathbf{i}}$ is expressed in the same unit (pixels), ensuring uniformity in magnitude across all components. In the subsequent sections, we use ${\mathbf{e}}_{\mathbf{i}}$ as an input signal for the neural network. This consistency simplifies the input quantization process, thus enhancing the effectiveness of the learning process.

We define the system state as:

$$\mathbf{x}={\left[\begin{array}{cc}{\mathbf{e}}^{\mathbf{T}}& {\mathbf{q}}^{\mathbf{T}}\end{array}\right]}^{\mathbf{T}}$$

(17)

The mapping between the derivative of the new task function and the robot joint velocities is defined as:

$$\dot{\mathbf{e}}=\mathbf{J}\left(\mathbf{x}\right)\dot{\mathbf{q}}$$

(18)

where $\mathbf{J}$ is the Jacobian matrix of the entire system.

3.2. Collinearity Constraint

In visual servoing systems, utilizing geometric relationships between feature points enhances control performance. As shown in Figure 3, we consider two key collinearity relationships: points ${\mathit{\epsilon }}_{\mathbf{d}\mathbf{1}}^{\left(\mathbf{aux}\right)}$, ${\mathit{\epsilon }}_{\mathbf{d}\mathbf{4}}^{\left(\mathbf{aux}\right)}$ and ${\mathit{\epsilon }}_{\mathbf{d}\mathbf{5}}^{\left(\mathbf{aux}\right)}$ form one collinear set, while ${\mathit{\epsilon }}_{\mathbf{d}\mathbf{2}}^{\left(\mathbf{aux}\right)}$, ${\mathit{\epsilon }}_{\mathbf{d}\mathbf{3}}^{\left(\mathbf{aux}\right)}$ and ${\mathit{\epsilon }}_{\mathbf{d}\mathbf{5}}^{\left(\mathbf{aux}\right)}$ form another. These collinearity properties remain invariant under projective transformations, ensuring that corresponding points in the current configuration (${\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}$, ${\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}$, ${\mathit{\epsilon }}_{\mathbf{5}}^{\left(\mathbf{aux}\right)}$) and (${\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}$, ${\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}$, ${\mathit{\epsilon }}_{\mathbf{5}}^{\left(\mathbf{aux}\right)}$) maintain the same geometric relationships. The arrows in the figure represent the velocity vectors of the feature points during the servoing process. Based on these collinearity relationships, we derive velocity constraints that preserve the geometric properties throughout the motion.

For collinear points, the following relationships hold:

$${\mathit{\epsilon }}_{\mathbf{5}}^{\left(\mathbf{aux}\right)}={\alpha }_1{\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}+\left(1-{\alpha }_1\right){\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}$$

(19)

$${\mathit{\epsilon }}_{\mathbf{5}}^{\left(\mathbf{aux}\right)}={\alpha }_2{\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}+\left(1-{\alpha }_2\right){\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}$$

(20)

where parameter ${\alpha }_1$ and ${\alpha }_2$ can be calculated as:

$${\alpha }_1={\displaystyle \frac{{\left({\mathit{\epsilon }}_{\mathbf{5}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}\right)}^T\left({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}\right)}{{\left|{\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}\right|}^2}}$$

(21)

$${\alpha }_2={\displaystyle \frac{{\left({\mathit{\epsilon }}_{\mathbf{5}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}\right)}^T\left({\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}\right)}{{\left|{\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}\right|}^2}}$$

(22)

Differentiating Equations (19) and (20) with respect to time, we obtain:

$${\dot{\mathit{\epsilon }}}_{\mathbf{5}}^{\left(\mathbf{aux}\right)}={\alpha }_1{\dot{\mathit{\epsilon }}}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}+\left(1-{\alpha }_1\right){\dot{\mathit{\epsilon }}}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}+{\dot{\alpha }}_1\left({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{c}\mathbf{4}}^{\left(\mathbf{aux}\right)}\right)$$

(23)

$${\dot{\mathit{\epsilon }}}_{\mathbf{5}}^{\left(\mathbf{aux}\right)}={\alpha }_2{\dot{\mathit{\epsilon }}}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}+\left(1-{\alpha }_2\right){\dot{\mathit{\epsilon }}}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}+{\dot{\alpha }}_2\left({\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}\right)$$

(24)

This velocity relationship indicates that the velocity of point 5, ${\dot{\mathit{\epsilon }}}_{\mathbf{5}}^{\left(\mathbf{aux}\right)}$, is a weighted combination of (${\dot{\mathit{\epsilon }}}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}$,${\dot{\mathit{\epsilon }}}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}$) and (${\dot{\mathit{\epsilon }}}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}$, ${\dot{\mathit{\epsilon }}}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}$).

For a static target object, ${\dot{\mathit{\epsilon }}}_{\mathbf{d}\mathbf{1}}^{\left(\mathbf{aux}\right)}={\dot{\mathit{\epsilon }}}_{\mathbf{d}\mathbf{2}}^{\left(\mathbf{aux}\right)}={\dot{\mathit{\epsilon }}}_{\mathbf{d}\mathbf{3}}^{\left(\mathbf{aux}\right)}={\mathit{\epsilon }}_{\mathbf{d}\mathbf{4}}^{\left(\mathbf{aux}\right)}={\dot{\mathit{\epsilon }}}_{\mathbf{d}\mathbf{5}}^{\left(\mathbf{aux}\right)}=0$. In terms of Equations (15), (23) and (24) can be written as:

$$-{\dot{\mathbf{e}}}_{\mathbf{5}}=-{\alpha }_1{\dot{\mathbf{e}}}_{\mathbf{1}}-\left(1-{\alpha }_1\right){\dot{\mathbf{e}}}_{\mathbf{4}}+{\dot{\alpha }}_1\left({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}\right)$$

(25)

$$-{\dot{\mathbf{e}}}_{\mathbf{5}}=-{\alpha }_2{\dot{\mathbf{e}}}_{\mathbf{2}}-\left(1-{\alpha }_2\right){\dot{\mathbf{e}}}_{\mathbf{3}}+{\dot{\alpha }}_2\left({\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}\right)$$

(26)

where ${\dot{\mathbf{e}}}_{\mathbf{i}}={\dot{\mathit{\epsilon }}}_{\mathbf{i}}^{\left(\mathbf{aux}\right)}-{\dot{\mathit{\epsilon }}}_{\mathbf{di}}^{\left(\mathbf{aux}\right)}$, i = 1, 2,…5. To eliminate the influence of ${\alpha }_1$ and ${\alpha }_2$ in the last terms, we multiply both sides of the equation by $\left({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}\right)\in R^{2\times 1}$ and $\left({\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}\right)\in R^{2\times 1}$, yielding:

$$\left({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}\right)\times \left(-{\dot{\mathbf{e}}}_{\mathbf{5}}+{\alpha }_1{\dot{\mathbf{e}}}_{\mathbf{1}}+\left(1-{\alpha }_1\right){\dot{\mathbf{e}}}_{\mathbf{4}}\right)=0$$

(27)

$$\left({\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}\right)\times \left(-{\dot{\mathbf{e}}}_{\mathbf{5}}+{\alpha }_2{\dot{\mathbf{e}}}_{\mathbf{2}}+\left(1-{\alpha }_2\right){\dot{\mathbf{e}}}_{\mathbf{3}}\right)=0$$

(28)

These velocity constraints are incorporated into our learning algorithm, ensuring that the neural network updates respect the underlying geometric principles of projective transformations.

3.3. Model-Free Jacobian Learning

Traditional methods typically rely on precise mathematical models; however, in practical applications, accurate modeling is often difficult to achieve due to system parameter uncertainties and environmental disturbances. This chapter proposes the use of a CMAC neural network to learn and approximate the system’s Jacobian matrix, enabling more robust and adaptive control strategies. The chapter first introduces the CMAC network structure, then explains the weight learning process. Notably, our weight update approach incorporates geometric constraints of visual features to accelerate the learning process.

3.3.1. CMAC Model

The CMAC network is divided into five layers: input layer, association memory space, receptive-field space, weight memory space and output layer, as shown in Figure 4.

• Input space:
  In this research, we use the state vector $\mathbf{x}$ as the input and employ a neural network to learn the unknown Jacobian matrix $\mathbf{J}$.
  The neural network input dimension is $16\times 1$ ($\mathbf{e}\in R^{10\times 1}$ and the robot has six joints: $\mathbf{q}\in R^{6\times 1}$). The input vector needs to be quantized into discrete regions according to its corresponding range:

  $$I_i=⌊{\displaystyle \frac{x_i-x_{imin}}{x_{imax}-x_{imin}}}·I_m⌋$$

  (29)

  where $I_m$ represents the maximum index value after quantization, and $\lfloor ·\rfloor$ denotes the floor function, ensuring that $I_i$ is an integer. And $x_{imin}$ and $x_{imax}$ represent the minimum and maximum values of ith element of state vector $\mathbf{x}$. Figure 5 illustrates a CMAC neural network with two-dimensional input, where each input is discretized into 10 regions: $I_m=10$.
• Association memory space:
  Each input dimension is uniformly partitioned into several discrete regions. Each complete block includes three adjacent regions. In Figure 5, input $x_1$ and $x_2$ have their regions grouped into blocks (a, b, c, d) and (A, B, C, D), respectively.
  The network creates multiple layers by shifting the block boundaries. For example, layer 2 contains blocks D, E, F for $x_1$ and blocks d, e, f for $x_2$, formed by shifting the original blocks by one region. Subsequent layer 3 follows the same pattern with additional shifts. Each block is represented by a Gaussian membership function:

  $${\psi }_{ik}=exp\left({\displaystyle \frac{-{\left(I_i-m_{ik}\right)}^2}{{\sigma }_{ik}^2}}\right),k\in \left[1,n_b\right]$$

  (30)

  where k denotes the k-th block and $n_b$ represents the total number of blocks for input dimension $I_i$. The parameters $m_{ik}$ and ${\sigma }_{ik}$ represent the center and width of the Gaussian function for the k-th block, respectively.
• Receptive-field space:
  The receptive field refers to the region in the input space that can activate specific neurons or memory cells in the network. A key feature of the CMAC network is the use of overlapping receptive fields. This means that a single input can simultaneously activate multiple memory cells, creating a distributed representation. The degree of overlap affects the network’s smoothness and generalization capability; highly overlapping receptive fields produce smoother function approximations, while less overlap produces more localized responses. As shown in Figure 5, point p in space falls within blocks aB, fD, and Ee, activating the corresponding memory cells. The multidimensional receptive-field function is defined as

  $$b_k=\underset{i=1}{\stackrel{n}{\Pi }}{\psi }_{ik}=exp\left(\sum _{i=1}^n{\displaystyle \frac{-{\left(I_i-m_{ik}\right)}^2}{{\sigma }_{ik}^2}}\right),k=1,2\cdots n_l$$

  (31)

  where $n_l$ represents the number of layers in the receptive field space.
• Weight memory space and output layer:
  Each receptive field space is connected to the weighted memory W. The i-th output of the neural network is:

  $$y_i=\sum _{k=1}^{n_l}{w}_{ik}{b}_k={\mathbf{w}}_{\mathbf{i}}^{\mathbf{T}}\mathbf{b}$$

  (32)

  where ${\mathbf{w}}_{\mathbf{i}}={\left[\begin{array}{cccc}{w}_{i1}& w_{i2}& \cdots & w_{i{n}_l}\end{array}\right]}^T$ and $\mathbf{b}={\left[\begin{array}{cccc}{b}_1& b_2& \cdots & b_{n_l}\end{array}\right]}^T$.

3.3.2. Weight Update of Neural Network

The discrete form of the Equation (18) can be written as:

$$\Delta \mathbf{e}\left(t_i\right)=\mathbf{J}\left(t_i\right)\Delta \mathbf{q}\left(t_i\right)$$

(33)

where the subscript i represents the i-th time step. $\Delta \mathbf{q}\in R^{n_q\times 1}$ represents the joint increment over the sampling interval. Next, the state vector $\mathbf{x}$ is used as input to approximate the Jacobian matrix using the CMAC network.

$$\mathbf{J}=\left[\begin{array}{cccc}{\mathbf{w}}_{\mathbf{1},\mathbf{1}}^{\mathbf{T}}\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)& {\mathbf{w}}_{\mathbf{1},\mathbf{2}}^{\mathbf{T}}\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)& \cdots & {\mathbf{w}}_{\mathbf{1},{\mathbf{n}}_{\mathbf{q}}}^{\mathbf{T}}\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)\\ {\mathbf{w}}_{\mathbf{2},\mathbf{1}}^{\mathbf{T}}\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)& {\mathbf{w}}_{\mathbf{2},\mathbf{2}}^{\mathbf{T}}\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)& \cdots & {\mathbf{w}}_{\mathbf{2},{\mathbf{n}}_{\mathbf{q}}}^{\mathbf{T}}\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)\\ ⋮& ⋮& ⋮& ⋮\\ {\mathbf{w}}_{\mathbf{10},\mathbf{1}}^{\mathbf{T}}\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)& {\mathbf{w}}_{\mathbf{10},\mathbf{2}}^{\mathbf{T}}\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)& \cdots & {\mathbf{w}}_{\mathbf{10},{\mathbf{n}}_{\mathbf{q}}}^{\mathbf{T}}\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)\end{array}\right] +{\mathit{\sigma }}_{\mathbf{J}}$$

(34)

where $\mathbf{w}$ are constant matrices of ideal network weights and ${\mathbf{w}}_{\mathbf{k},\mathbf{m}}\in R^{n_l\times 1}$ is used to approximate the i-th row and jth column element of the Jacobian matrix $j_{km}$, and $\mathbf{b}$ is the vector of receptive-field functions. The approximation error of the neural network is denoted as ${\mathit{\sigma }}_{\mathbf{J}}$.

It is important to note that the estimated incremental task error

$$\Delta \mathbf{e}={\left[\begin{array}{cccc}\Delta {\mathbf{e}}_{\mathbf{1}}^{\mathbf{T}}& \Delta {\mathbf{e}}_{\mathbf{2}}^{\mathbf{T}}& \cdots & \Delta {\mathbf{e}}_{\mathbf{5}}^{\mathbf{T}}\end{array}\right]}^T$$

consists of five 2D vectors, where each $\Delta {\mathbf{e}}_{\mathbf{i}}\in R^{2\times 1}$. To distinguish between these components, we use superscript notation to denote the k-th element of $\Delta \mathbf{e}$: $\Delta e^k$. Thus, we have:

$$\Delta e^k\left(t_i\right)=\sum _{m=1}^{n_q}{j}_{km}\left(t_i\right)\Delta q_m\left(t_i\right)+{\sigma }_k={\mathbf{\Phi }}^{\mathbf{T}}{\mathbf{w}}_{\mathbf{k}}+{\sigma }_k$$

(35)

where ${\mathbf{w}}_{\mathbf{k}}={\left[\begin{array}{cccc}{\mathbf{w}}_{\mathbf{k},\mathbf{1}}^{\mathbf{T}}& {\mathbf{w}}_{\mathbf{k},\mathbf{2}}^{\mathbf{T}}& \cdots & {\mathbf{w}}_{\mathbf{k},{\mathbf{n}}_{\mathbf{q}}}^{\mathbf{T}}\end{array}\right]}^T$ and ${\sigma }_k$ is the approximation error of the neural network. ${\mathbf{\Phi }}^{\mathbf{T}}=\left[\begin{array}{cccc}{\mathbf{b}}^{\mathbf{T}}\left(\mathbf{x}\left(t_i\right)\right)\Delta q_1& {\mathbf{b}}^{\mathbf{T}}\left(\mathbf{x}\left(t_i\right)\right)\Delta q_2& \cdots & {\mathbf{b}}^{\mathbf{T}}\left(\mathbf{x}\left(t_i\right)\right)\Delta q_{n_q}\end{array}\right]$, and $\Delta q_m$ represents the m-th element of $\Delta \mathbf{q}$.

$\Delta \mathbf{e}$ is further expressed as shown below:

$$\Delta \mathbf{e}\left(t_i\right)=\sum _{m=1}^{n_q}{\mathbf{j}}_{\mathbf{m}}\left(t_i\right)\Delta q_m\left(t_i\right)={\mathbf{\Omega }}^{\mathbf{T}}\mathbf{w}+\mathit{\sigma }$$

(36)

where ${\mathbf{j}}_{\mathbf{m}}$ represents the m-th column of $\mathbf{J}$. $\mathit{\sigma }={\left[\begin{array}{cccc}{\sigma }_1& {\sigma }_2& \cdots & {\sigma }_{10}\end{array}\right]}^T$ is the approximation error term introduced by the neural network model. ${\mathbf{\Omega }}^{\mathbf{T}}=\left[\begin{array}{cccc}{\mathbf{\Phi }}^{\mathbf{T}}& \mathbf{0}& \cdots & \mathbf{0}\\ \mathbf{0}& {\mathbf{\Phi }}^{\mathbf{T}}& \cdots & \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \ddots & \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \cdots & {\mathbf{\Phi }}^{\mathbf{T}}\end{array}\right]$ and $\mathbf{w}={\left[\begin{array}{cccc}{\mathbf{w}}_{\mathbf{1}}^{\mathbf{T}}& {\mathbf{w}}_{\mathbf{2}}^{\mathbf{T}}& \cdots & {\mathbf{w}}_{\mathbf{10}}^{\mathbf{T}}\end{array}\right]}^T$.

Assumption 1.

The approximation error σ is assumed to be upper bounded:

$$\parallel \mathit{\sigma }\left(t\right)\parallel \le {\sigma }_M$$

where ${\sigma }_M$ is a finite positive constant. In addition, under a properly designed neural network architecture, σ can be made sufficiently small.

However, $\mathbf{w}$ is unknown. Using the estimated weights, the m-th column and the k-th row of the estimated Jacobian matrix can be given by: ${\widehat{j}}_{km}\left(t_i\right)={\widehat{\mathbf{w}}}_{\mathbf{k},\mathbf{m}}\left(t_i\right)\mathbf{b}\left(\mathbf{x}\left(t_i\right)\right)$ where ${\widehat{\mathbf{w}}}_{\mathbf{k},\mathbf{m}}$ is the estimated value of desired ${\mathbf{w}}_{\mathbf{k},\mathbf{m}}$.

The estimated value of $\Delta e^k$ is:

$$\Delta {\widehat{e}}^k\left(t_i\right)={\mathbf{\Phi }}^{\mathbf{T}}{\widehat{\mathbf{w}}}_{\mathbf{k}}$$

(37)

The estimated incremental change in task error is:

$$\Delta \widehat{\mathbf{e}}\left(t_i\right)=\widehat{\mathbf{J}}\left(t_i\right)\Delta \mathbf{q}\left(t_i\right)={\mathbf{\Omega }}^{\mathbf{T}}\widehat{\mathbf{w}}$$

(38)

To incorporate geometric constraints into the learning process, we leverage the velocity relationships between collinear feature points defined in Equations (27) and (28). The Jacobian matrix estimated by our neural network must satisfy the following constraints:

$$h_1=f_1\left(\Delta \widehat{\mathbf{e}}\right)=\left({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)}\right)\times \left(-\Delta {\widehat{\mathbf{e}}}_{\mathbf{5}}+{\alpha }_1\Delta {\widehat{\mathbf{e}}}_{\mathbf{1}}+\left(1-{\alpha }_1\right)\Delta {\widehat{\mathbf{e}}}_{\mathbf{4}}\right)$$

(39)

$$h_2=f_2\left(\Delta \widehat{\mathbf{e}}\right)=\left({\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)}\right)\times \left(-\Delta {\widehat{\mathbf{e}}}_{\mathbf{5}}+{\alpha }_2\Delta {\widehat{\mathbf{e}}}_{\mathbf{2}}+\left(1-{\alpha }_2\right)\Delta {\widehat{\mathbf{e}}}_{\mathbf{3}}\right)$$

(40)

We then define a loss function that incorporates both prediction accuracy and geometric constraints:

$$minL=min\left({\displaystyle \frac{1}{2}}{\left(\Delta \mathbf{e}-\Delta \widehat{\mathbf{e}}\right)}^{\mathbf{T}}\left(\Delta \mathbf{e}-\Delta \widehat{\mathbf{e}}\right)+{\displaystyle \frac{\lambda }{2}}\sum _{j=1}^2{h}_j^2\right)$$

(41)

Then, the weights of neural network are updated by:

$$\widehat{\mathbf{w}}\left(t_{i+1}\right)=\widehat{\mathbf{w}}\left(t_i\right)+\beta {\displaystyle \frac{\partial L}{\partial \widehat{\mathbf{w}}}}=\widehat{\mathbf{w}}\left(t_i\right)-\beta \left(\mathbf{\Omega }\left(t_i\right)\Delta \tilde{\mathbf{e}}\left(t_i\right)-\sum _{j=1}^2\lambda h_j{\displaystyle \frac{\partial h_j}{\Delta \widehat{\mathbf{w}}}}\right)$$

(42)

where $\beta$ represents the learning rate parameter for the neural network weight update process, and $\Delta \tilde{\mathbf{e}}=\Delta \mathbf{e}-\Delta \widehat{\mathbf{e}}$ is the estimation error.

3.4. Controller Design

The control law of a robot is defined as:

$$\mathbf{u}=-\eta {\widehat{\mathbf{J}}}^{+}\left(\left.\mathbf{x}\right|\widehat{\mathbf{w}}\right)\mathbf{e}$$

(43)

where $\eta$ is a positive value.

Figure 6 illustrates the control flow diagram. The process begins with image acquisition and feature point detection. Matched feature pairs $\left\{{\mathit{\epsilon }}_i,{\mathit{\epsilon }}_{di}\right\},i=1,2,\cdots ,P$ are utilized to estimate the projective homography matrix $\overline{G}$ using the RANSAC algorithm. Based on $\overline{G}$, the task error vector $\mathbf{e}$ and a set of virtual feature points ${\mathit{\epsilon }}_i^{\left(\mathrm{aux}\right)}$ are computed according to Equations (12)–(16).

Next, the estimated system state vector, which includes the task error $\mathit{e}$ and the joint position $\mathit{q}$, is input to the CMAC neural network. The network predicts the task error increment $\Delta \widehat{\mathit{e}}$ and estimates the Jacobian matrix. Importantly, the geometry constraint module is activated after this estimation phase. It incorporates $\Delta \widehat{\mathit{e}}$ and the virtual feature points to compute constraint terms (see Equations (39) and (40)), which are then embedded into the neural network’s weight update rule (Equation (42)) to preserve geometric consistency, such as point collinearity under projective transformations.

Finally, the joint controller utilizes the estimated Jacobian and task error to compute the control command $\mathit{u}$, as described in Equation (43), which is then sent to the robot actuators. The entire procedure operates in a closed-loop fashion, with the visual feedback continually guiding subsequent control iterations.

4. Stability Analysis

In this section, we establish the stability of the proposed geometry-constrained learning-based visual servoing controller.

4.1. Geometric Constraint Relationships

Let us define the weight estimation error as $\tilde{\mathbf{w}}=\mathbf{w}-\widehat{\mathbf{w}}$, where $\mathbf{w}$ represents the ideal weights. According to Equations (35) and (37), we have:

$$\Delta {\tilde{e}}^k=\Delta e^k-\Delta {\widehat{e}}^k={\mathbf{\Phi }}^{\mathrm{T}}{\tilde{\mathbf{w}}}_k+{\sigma }_k$$

(44)

Similarly, from Equations (36) and (38), we can derive:

$$\Delta \tilde{\mathbf{e}}=\Delta \mathbf{e}-\Delta \widehat{\mathbf{e}}={\mathbf{\Omega }}^{\mathrm{T}}\tilde{\mathbf{w}}+\mathit{\sigma }$$

(45)

For the actual error increment $\Delta \mathbf{e}$, the geometric constraints must be satisfied, i.e., $h_i(\Delta \mathbf{e})=0$. We establish the relationship between these constraint equations and the estimation error of the task error increment:

$$\begin{array}{cc}\hfill h_1(\Delta \widehat{\mathbf{e}})-h_1(\Delta \mathbf{e})& =h_1\hfill \\ \hfill & =({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)})\times (-\Delta {\mathbf{e}}_{\mathbf{5}}+{\alpha }_1\Delta {\mathbf{e}}_{\mathbf{1}}+(1-{\alpha }_1)\Delta {\mathbf{e}}_{\mathbf{4}})\hfill \\ \hfill & -({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)})\times (-\Delta {\widehat{\mathbf{e}}}_{\mathbf{5}}+{\alpha }_1\Delta {\widehat{\mathbf{e}}}_{\mathbf{1}}+(1-{\alpha }_1)\Delta {\widehat{\mathbf{e}}}_{\mathbf{4}})\hfill \\ \hfill & =({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)})\times (-\Delta {\tilde{\mathbf{e}}}_{\mathbf{5}}+{\alpha }_1\Delta {\tilde{\mathbf{e}}}_{\mathbf{1}}+(1-{\alpha }_1)\Delta {\tilde{\mathbf{e}}}_{\mathbf{4}})\hfill \end{array}$$

(46)

Similarly, for the second constraint:

$$\begin{array}{cc}\hfill h_2& =({\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)})\times (-\Delta {\tilde{\mathbf{e}}}_{\mathbf{5}}+{\alpha }_2\Delta {\tilde{\mathbf{e}}}_{\mathbf{2}}+(1-{\alpha }_2)\Delta {\tilde{\mathbf{e}}}_{\mathbf{3}})\hfill \end{array}$$

(47)

It should be noted that $\Delta {\tilde{\mathbf{e}}}_{\mathbf{i}}={[\Delta {\tilde{e}}^{2i-1}\Delta {\tilde{e}}^{2i}]}^T$ is a two-dimensional vector containing the $(2i-1)$-th and $2i$-th elements of $\Delta \tilde{\mathbf{e}}$. The constraints $h_1$ and $h_2$ can be reformulated as:

$$\begin{array}{cc}\hfill h_1& ={\mu }_u(-\Delta {\tilde{e}}^9+{\alpha }_1\Delta {\tilde{e}}^1+(1-{\alpha }_1)\Delta {\tilde{e}}^7)\hfill \\ \hfill & -{\mu }_v(-\Delta {\tilde{e}}^{10}+{\alpha }_1\Delta {\tilde{e}}^2+(1-{\alpha }_1)\Delta {\tilde{e}}^8)\hfill \\ \hfill & ={\mu }_u(-{\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_9-{\sigma }_9+{\alpha }_1{\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_1+{\alpha }_1{\sigma }_1+(1-{\alpha }_1){\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_7+(1-{\alpha }_1){\sigma }_7)\hfill \\ \hfill & -{\mu }_v(-{\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_{10}-{\sigma }_{10}+{\alpha }_1{\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_2+{\alpha }_1{\sigma }_2+(1-{\alpha }_1){\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_8+(1-{\alpha }_1){\sigma }_8)\hfill \\ \hfill & ={\mathbf{n}}_1^T\tilde{\mathbf{w}}+{\mu }_u(-{\sigma }_9+{\alpha }_1{\sigma }_1+(1-{\alpha }_1){\sigma }_7)-{\mu }_v(-{\sigma }_{10}+{\alpha }_1{\sigma }_2+(1-{\alpha }_1){\sigma }_8)\hfill \\ \hfill & ={\mathbf{n}}_1^T\tilde{\mathbf{w}}+{\sigma }_{h1}\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill h_2& ={\kappa }_u(-\Delta {\tilde{e}}^9+{\alpha }_2\Delta {\tilde{e}}^3+(1-{\alpha }_2)\Delta {\tilde{e}}^5)\hfill \\ \hfill & +{\kappa }_v(-\Delta {\tilde{e}}^{10}+{\alpha }_2\Delta {\tilde{e}}^4+(1-{\alpha }_2)\Delta {\tilde{e}}^6)\hfill \\ \hfill & ={\kappa }_u(-{\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_9-{\sigma }_9+{\alpha }_2{\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_3+{\alpha }_2{\sigma }_3+(1-{\alpha }_2){\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_5+(1-{\alpha }_2){\sigma }_5)\hfill \\ \hfill & -{\kappa }_v(-{\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_{10}-{\sigma }_{10}+{\alpha }_2{\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_4+{\alpha }_2{\sigma }_4+(1-{\alpha }_2){\mathbf{\Phi }}^T{\tilde{\mathbf{w}}}_6+(1-{\alpha }_2){\sigma }_6)\hfill \\ \hfill & ={\mathbf{n}}_2^T\tilde{\mathbf{w}}+{\kappa }_u(-{\sigma }_9+{\alpha }_2{\sigma }_3+(1-{\alpha }_2){\sigma }_5)-{\kappa }_v(-{\sigma }_{10}+{\alpha }_2{\sigma }_4+(1-{\alpha }_2){\sigma }_6)\hfill \\ \hfill & ={\mathbf{n}}_2^T\tilde{\mathbf{w}}+{\sigma }_{h2}\hfill \end{array}$$

(49)

where ${\sigma }_{h1}$ and ${\sigma }_{h2}$ represent the combined approximation errors in the constraint equations. ${\mu }_u$ and ${\mu }_v$ are the first and second elements of $({\mathit{\epsilon }}_{\mathbf{1}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{4}}^{\left(\mathbf{aux}\right)})$, respectively. Similarly, ${\kappa }_u$ and ${\kappa }_v$ are the first and second elements of $({\mathit{\epsilon }}_{\mathbf{2}}^{\left(\mathbf{aux}\right)}-{\mathit{\epsilon }}_{\mathbf{3}}^{\left(\mathbf{aux}\right)})$.

The vectors ${\mathbf{n}}_1$ and ${\mathbf{n}}_2$ are defined as:

$$\begin{array}{cc}\hfill {\mathbf{n}}_1^T& =\hfill \\ \hfill & \left[\begin{array}{cccccccccc}{\mu }_u{\alpha }_1{\mathbf{\Phi }}^T& -{\mu }_v{\alpha }_1{\mathbf{\Phi }}^T& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& {\mu }_u(1-{\alpha }_1){\mathbf{\Phi }}^T& -{\mu }_v(1-{\alpha }_1){\mathbf{\Phi }}^T& -{\mu }_u{\mathbf{\Phi }}^T& {\mu }_v{\mathbf{\Phi }}^T\end{array}\right]\hfill \\ \hfill {\mathbf{n}}_2^T& =\hfill \\ \hfill & \left[\begin{array}{cccccccccc}\mathbf{0}& \mathbf{0}& {\mu }_u{\alpha }_2{\mathbf{\Phi }}^T& -{\mu }_v{\alpha }_2{\mathbf{\Phi }}^T& {\mu }_u(1-{\alpha }_2){\mathbf{\Phi }}^T& -{\mu }_v(1-{\alpha }_2){\mathbf{\Phi }}^T& \mathbf{0}& \mathbf{0}& -{\mu }_u{\mathbf{\Phi }}^T& {\mu }_v{\mathbf{\Phi }}^T\end{array}\right]\hfill \end{array}$$

(50)

The geometric constraints contribute to the neural network weight update rule as shown in Equation (42). The second term in this equation can be reformulated as:

$$\begin{array}{cc}\hfill \sum _{j=1}^2{h}_j{\displaystyle \frac{\partial h_j}{\partial \widehat{\mathbf{w}}}}& =h_1{\displaystyle \frac{\partial h_1}{\partial \tilde{\mathbf{w}}}}{\displaystyle \frac{\partial \tilde{\mathbf{w}}}{\partial \widehat{\mathbf{w}}}}+h_2{\displaystyle \frac{\partial h_2}{\partial \tilde{\mathbf{w}}}}{\displaystyle \frac{\partial \tilde{\mathbf{w}}}{\partial \widehat{\mathbf{w}}}}\hfill \\ \hfill & =({\mathbf{n}}_1^T\tilde{\mathbf{w}}+{\sigma }_{h1})(-{\mathbf{n}}_1)+({\mathbf{n}}_2^T\tilde{\mathbf{w}}+{\sigma }_{h2})(-{\mathbf{n}}_2)\hfill \\ \hfill & =-{\mathbf{n}}_1{\mathbf{n}}_1^T\tilde{\mathbf{w}}-{\sigma }_{h1}{\mathbf{n}}_1-{\mathbf{n}}_2{\mathbf{n}}_2^T\tilde{\mathbf{w}}-{\sigma }_{h2}{\mathbf{n}}_2\hfill \\ \hfill & =-\mathbf{N}\tilde{\mathbf{w}}-{\mathit{\sigma }}_{\mathbf{N}}\hfill \end{array}$$

(51)

where $\mathbf{N}={\mathbf{n}}_1{\mathbf{n}}_1^T+{\mathbf{n}}_2{\mathbf{n}}_2^T$ and ${\mathit{\sigma }}_{\mathbf{N}}={\sigma }_{h1}{\mathbf{n}}_1+{\sigma }_{h2}{\mathbf{n}}_2$ represent the combined approximation error effect.

The weight error dynamics can thus be expressed as:

$$\begin{array}{cc}\hfill \tilde{\mathbf{w}}\left(t_{i+1}\right)& =\tilde{\mathbf{w}}\left(t_i\right)-\beta \left(\mathbf{\Omega }\left(t_i\right)\Delta \tilde{\mathbf{e}}\left(t_i\right)-\lambda \sum _{j=1}^2{h}_j{\displaystyle \frac{\partial h_j}{\partial \widehat{\mathbf{w}}}}\right)\hfill \\ \hfill & =\tilde{\mathbf{w}}\left(t_i\right)-\beta \left(\mathbf{\Omega }\left(t_i\right)({\mathbf{\Omega }}^T\tilde{\mathbf{w}}+\mathit{\sigma })+\lambda \mathbf{N}\tilde{\mathbf{w}}+\lambda {\mathit{\sigma }}_{\mathbf{N}}\right)\hfill \\ \hfill & =\tilde{\mathbf{w}}\left(t_i\right)-\beta \mathbf{\Omega }\left(t_i\right){\mathbf{\Omega }}^T\tilde{\mathbf{w}}-\beta \mathbf{\Omega }\left(t_i\right)\mathit{\sigma }-\beta \lambda \mathbf{N}\tilde{\mathbf{w}}-\beta \lambda {\mathit{\sigma }}_{\mathbf{N}}\hfill \\ \hfill & =(I-\beta \mathbf{M}-\beta \lambda \mathbf{N})\tilde{\mathbf{w}}\left(t_i\right)-\beta \mathbf{\Omega }\left(t_i\right)\mathit{\sigma }-\beta \lambda {\mathit{\sigma }}_{\mathbf{N}}\hfill \end{array}$$

(52)

where $\mathbf{M}=\mathbf{\Omega }\left(t_i\right){\mathbf{\Omega }}^T$.

4.2. Lyapunov Stability Analysis

To analyze the stability of the learning process, we define a positive definite Lyapunov function candidate:

$$V\left(t_i\right)={\displaystyle \frac{1}{2}}{\tilde{\mathbf{w}}}^T\left(t_i\right)\tilde{\mathbf{w}}\left(t_i\right)$$

(53)

The change in this Lyapunov function between consecutive time steps is:

$$\begin{array}{cc}\hfill \Delta V& =V\left(t_{i+1}\right)-V\left(t_i\right)\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}{\tilde{\mathbf{w}}}^T\left(t_{i+1}\right)\tilde{\mathbf{w}}\left(t_{i+1}\right)-{\displaystyle \frac{1}{2}}{\tilde{\mathbf{w}}}^T\left(t_i\right)\tilde{\mathbf{w}}\left(t_i\right)\hfill \end{array}$$

(54)

Substituting the weight error dynamics:

$$\begin{array}{cc}\hfill \Delta V& ={\displaystyle \frac{1}{2}}{[(I-\beta \mathbf{M}-\beta \lambda \mathbf{N})\tilde{\mathbf{w}}\left(t_i\right)-\beta \mathbf{\Omega }\left(t_i\right)\mathit{\sigma }-\beta \lambda {\mathit{\sigma }}_{\mathbf{N}}]}^T\hfill \\ \hfill & ·[(I-\beta \mathbf{M}-\beta \lambda \mathbf{N})\tilde{\mathbf{w}}\left(t_i\right)-\beta \mathbf{\Omega }\left(t_i\right)\mathit{\sigma }-\beta \lambda {\mathit{\sigma }}_{\mathbf{N}}]\hfill \\ \hfill & -{\displaystyle \frac{1}{2}}{\tilde{\mathbf{w}}}^T\left(t_i\right)\tilde{\mathbf{w}}\left(t_i\right)\hfill \end{array}$$

(55)

where $M=\Omega \left(t_i\right){\Omega }^T\left(t_i\right)$ and $N=n_1{n}_1^T+n_2{n}_2^T$ are both positive semi-definite matrices.

Equation (55) can be simplified as:

$$\begin{array}{cc}\hfill \Delta V& =-\beta {\tilde{\mathbf{w}}}^T\left(t_i\right)(\mathbf{M}+\lambda \mathbf{N})\tilde{\mathbf{w}}-\beta {\tilde{\mathbf{w}}}^T\left(t_i\right)(\mathbf{\Omega }\left(t_i\right)\mathit{\sigma }+\lambda {\mathit{\sigma }}_{\mathbf{N}})\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\beta }^2{\tilde{\mathbf{w}}}^T\left(t_i\right){(\mathbf{M}+\lambda \mathbf{N})}^2\tilde{\mathbf{w}}\hfill \\ \hfill & +{\beta }^2{\tilde{\mathbf{w}}}^T\left(t_i\right)(\mathbf{M}+\lambda \mathbf{N})(\mathbf{\Omega }\left(t_i\right)\mathit{\sigma }+\lambda {\mathit{\sigma }}_{\mathbf{N}})\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\beta }^2{\parallel \mathbf{\Omega }\left(t_i\right)\mathit{\sigma }+\lambda {\mathit{\sigma }}_{\mathbf{N}}\parallel }^2\hfill \\ \hfill & =-\beta {\tilde{\mathbf{w}}}^T\left(t_i\right)\left[(\mathbf{M}+\lambda \mathbf{N})-{\displaystyle \frac{\beta }{2}}{(\mathbf{M}+\lambda \mathbf{N})}^2\right]\tilde{\mathbf{w}}\hfill \\ \hfill & -\beta {\tilde{\mathbf{w}}}^T\left(t_i\right)(I-\beta (\mathbf{M}+\lambda \mathbf{N}))(\mathbf{\Omega }\left(t_i\right)\mathit{\sigma }+\lambda {\mathit{\sigma }}_{\mathbf{N}})\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\beta }^2{\parallel \mathbf{\Omega }\left(t_i\right)\mathit{\sigma }+\lambda {\mathit{\sigma }}_{\mathbf{N}}\parallel }^2\hfill \end{array}$$

(56)

Let us define $\mathbf{P}=(\mathbf{M}+\lambda \mathbf{N})-{\displaystyle \frac{\beta }{2}}{(\mathbf{M}+\lambda \mathbf{N})}^2$ and $\mathbf{Q}=I-\beta (\mathbf{M}+\lambda \mathbf{N})$. Then:

$$\begin{array}{cc}\hfill \Delta V& =-\beta {\tilde{\mathbf{w}}}^T\left(t_i\right)\mathbf{P}\tilde{\mathbf{w}}-\beta {\tilde{\mathbf{w}}}^T\left(t_i\right)\mathbf{Q}(\mathbf{\Omega }\left(t_i\right)\mathit{\sigma }+\lambda {\mathit{\sigma }}_{\mathbf{N}})\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\beta }^2{\parallel \mathbf{\Omega }\left(t_i\right)\mathit{\sigma }+\lambda {\mathit{\sigma }}_{\mathbf{N}}\parallel }^2\hfill \end{array}$$

(57)

Based on Assumption 1, and noting that both the joint increment $\Delta q$ and the activation of the receptive field $\mathbf{b}\left(x\right)$ are bounded, it follows that $\Omega \left(t_i\right)$ is also bounded. Consequently, there exists a small positive constant $ϵ>0$ such that:

$$\parallel \mathbf{\Omega }\left(t_i\right)\mathit{\sigma }+\lambda {\mathit{\sigma }}_{\mathbf{N}}\parallel \le ϵ,\forall t_i$$

(58)

To rigorously ensure stability, we assume the following Persistent Excitation (PE) condition:

Assumption 2 (Persistent Excitation).

There exists a positive constant ${\gamma }_{min}>0$ and a finite time interval $T>0$, such that for all t:

$${\displaystyle \frac{1}{T}}{\int }_t^{t+T}\mathbf{\Omega }\left(\tau \right){\mathbf{\Omega }}^T\left(\tau \right)d\tau \ge {\gamma }_{min}I$$

(59)

Under this assumption, define the average matrix as:

$${\mathbf{M}}_{avg}={\displaystyle \frac{1}{T}}{\int }_t^{t+T}\mathbf{M}\left(\tau \right)d\tau \ge {\gamma }_{min}I$$

(60)

Hence, the averaged matrix ${\mathbf{M}}_{avg}+\lambda \mathbf{N}$ is strictly positive definite, ensuring:

$$x^T({\mathbf{M}}_{avg}+\lambda \mathbf{N})x\ge {\gamma }_{min}{\parallel x\parallel }^2,\forall x\ne 0$$

(61)

Selecting the learning rate β to be sufficiently small:

$$0<\beta <{\displaystyle \frac{2}{{\lambda }_{max}({\mathbf{M}}_{avg}+\lambda \mathbf{N})}}$$

(62)

the Lyapunov function increment becomes:

$$\Delta V\le -\beta {\gamma }_{min}\parallel \tilde{\mathbf{w}}\left(t_i\right){\parallel }^2+\beta \parallel \tilde{\mathbf{w}}\left(t_i\right)\parallel ϵ+{\displaystyle \frac{1}{2}}{\beta }^2{ϵ}^2$$

(63)

Therefore, the weight estimation error $\tilde{\mathbf{w}}\left(t_i\right)$ is uniformly ultimately bounded, and the system error converges to a small neighborhood of the origin, ensuring robust convergence and stability under the persistent excitation condition.

4.3. Convergence to Desired Pose

In practice, the estimated Jacobian $\widehat{\mathbf{J}}$ differs from the true Jacobian $\mathbf{J}$ due to learning or modeling errors. We define the Jacobian estimation error as:

$$\tilde{\mathbf{J}}\left(t_k\right)=\mathbf{J}\left(t_k\right)-\widehat{\mathbf{J}}\left(t_k\right)$$

(64)

Assume the control update is given by $\mathbf{u}\left(t_k\right)=-\eta {\widehat{\mathbf{J}}}^{+}\left(t_k\right)\mathbf{e}\left(t_k\right)$. Then, the error propagation becomes:

$$\begin{array}{cc}\hfill \mathbf{e}\left(t_{k+1}\right)& =\mathbf{e}\left(t_k\right)+\mathbf{J}\left(t_k\right)\mathbf{u}\left(t_k\right)\hfill \\ \hfill & =\mathbf{e}\left(t_k\right)-\eta \mathbf{J}\left(t_k\right){\widehat{\mathbf{J}}}^{+}\left(t_k\right)\mathbf{e}\left(t_k\right)\hfill \\ \hfill & =\left(\mathbf{I}-\eta \mathbf{S}\left(t_k\right)\right)\mathbf{e}\left(t_k\right)\hfill \end{array}$$

(65)

where we define the matrix $\mathbf{S}=\mathbf{J}{\widehat{\mathbf{J}}}^{+}=\left(\widehat{\mathbf{J}}+\tilde{\mathbf{J}}\right){\widehat{\mathbf{J}}}^{+}=\mathbf{I}+{\mathit{\delta }}_{\mathbf{s}}$.

Given that the neural network estimation error has been proven to converge, and with an appropriate neural network structure design along with sufficient pre-training data to obtain quality initial weights, it is reasonable to assume that ${\mathit{\delta }}_{\mathbf{s}}$ is bounded. There exists a constant ${\sigma }_{\Delta s}$ such that:

$$\parallel {\mathit{\delta }}_{\mathbf{s}}{\parallel }_2\le {\sigma }_{\Delta s}$$

Define the Lyapunov function:

$$V\left(t_k\right)={\displaystyle \frac{1}{2}}{\parallel \mathbf{e}\left(t_k\right)\parallel }^2$$

(66)

The difference is:

$$\begin{array}{cc}\hfill V\left(t_{k+1}\right)-V\left(t_k\right)& ={\displaystyle \frac{1}{2}}\parallel \mathbf{e}\left(t_{k+1}\right){\parallel }^2-{\displaystyle \frac{1}{2}}{\parallel \mathbf{e}\left(t_k\right)\parallel }^2\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}{∥\left(\mathbf{I}-\eta \mathbf{S}\left(t_k\right)\right)\mathbf{e}\left(t_k\right)∥}^2-{\displaystyle \frac{1}{2}}{\parallel \mathbf{e}\left(t_k\right)\parallel }^2\hfill \\ \hfill & =-\eta {\mathbf{e}}^T\left(t_k\right)\mathbf{S}\left(t_k\right)\mathbf{e}\left(t_k\right)+{\displaystyle \frac{{\eta }^2}{2}}{\parallel \mathbf{S}\left(t_k\right)\mathbf{e}\left(t_k\right)\parallel }^2\hfill \end{array}$$

(67)

To ensure convergence (i.e., $V\left(t_{k+1}\right)-V\left(t_k\right)<0$), we require that $\eta {\mathbf{e}}^T\mathbf{S}\mathbf{e}>{\displaystyle \frac{{\eta }^2}{2}}{\parallel \mathbf{S}\mathbf{e}\parallel }^2$. This inequality holds for all $\mathbf{e}\ne 0$ if the matrix $\mathbf{D}=\mathbf{S}+{\displaystyle \frac{\eta }{2}}{\mathbf{S}}^T\mathbf{S}$ is positive definite. We now derive a conservative bound on $\eta$ to guarantee the positive definiteness of $\mathbf{D}$.

For any unit vector $x\in {\mathbb{R}}^n$, we have:

$$x^T\mathbf{S}x=x^T(\mathbf{I}+{\mathit{\delta }}_{\mathbf{s}})x=1+x^T{\mathit{\delta }}_{\mathbf{s}}x\ge 1-{\sigma }_{\Delta s}$$

(68)

$${\lambda }_{min}\left({\mathbf{S}}^T\mathbf{S}\right)={\lambda }_{min}^2\left(\mathbf{S}\right)\ge {(1-{\sigma }_{\Delta s})}^2$$

(69)

Hence:

$$x^T\mathbf{D}x=x^T\mathbf{S}x+{\displaystyle \frac{\eta }{2}}{x}^T{\mathbf{S}}^T\mathbf{S}x\ge (1-{\sigma }_{\Delta s})+{\displaystyle \frac{\eta }{2}}{(1-{\sigma }_{\Delta s})}^2$$

(70)

To ensure $x^T\mathbf{D}x>0$, we require:

$$(1-{\sigma }_{\Delta s})+{\displaystyle \frac{\eta }{2}}{(1-{\sigma }_{\Delta s})}^2>0\Rightarrow \eta >{\displaystyle \frac{-2(1-{\sigma }_{\Delta s})}{{(1-{\sigma }_{\Delta s})}^2}}$$

(71)

When the Jacobian estimation error is relatively small (${\sigma }_{\Delta s}<1$), the system can maintain stability across a wide range of control gains. When the error becomes too large (${\sigma }_{\Delta s}>1$), a larger control gain is required to maintain stability, but this may lead to other issues (such as excessive oscillation).

Combined with Theorem 1, which shows that $\mathbf{e}=\mathbf{0}$ if and only if $\mathbf{R}=\mathbf{I}$ and $\mathbf{t}=\mathbf{0}$, it follows that the camera pose converges to the desired pose. The proposed controller thus ensures robust convergence despite approximation errors in the neural network.

5. Simulation and Experiment Results

5.1. Pre-Training Process

To ensure efficient initialization and promote global convergence during the visual servoing task, a pre-training phase is conducted for the neural network. In this phase, the robot performs small randomized motions around its initial configuration. During these exploratory movements, the system collects training data consisting of system states $\mathbf{x}$, the corresponding joint displacements, and the incremental task errors.

The collected system state data are denoted as $\mathbf{X}=\left\{\mathbf{x}\left(t_0\right),\mathbf{x}\left(t_1\right),\cdots ,\mathbf{x}\left(t_{n_s}\right)\right\}$, where each sample may activate a distinct subset of local receptive fields within the neural network. The union of all receptive fields activated by the sample set $\mathbf{X}$ is represented by ${\mathbf{b}}_s\left(\mathbf{X}\right)\in {\mathbb{R}}^{n_l\times 1}$, where $n_l$ denotes the total number of receptive fields. Each entry in ${\mathbf{b}}_s\left(\mathbf{X}\right)$ indicates whether the corresponding receptive field is activated by any sample in the dataset.

The activation matrix $\mathbf{A}\left(\mathbf{X}\right)$ is defined as:

$$\mathbf{A}\left(\mathbf{X}\right)=\left[\begin{array}{ccc}{\mathbf{b}}_s^{\top }\left(\mathbf{x}\left(t_0\right)\right)\Delta q_1\left(t_0\right)& \cdots & {\mathbf{b}}_s^{\top }\left(\mathbf{x}\left(t_0\right)\right)\Delta q_{n_q}\left(t_0\right)\\ {\mathbf{b}}_s^{\top }\left(\mathbf{x}\left(t_1\right)\right)\Delta q_1\left(t_1\right)& \cdots & {\mathbf{b}}_s^{\top }\left(\mathbf{x}\left(t_1\right)\right)\Delta q_{n_q}\left(t_1\right)\\ ⋮& \ddots & ⋮\\ {\mathbf{b}}_s^{\top }\left(\mathbf{x}\left(t_{n_s}\right)\right)\Delta q_1\left(t_{n_s}\right)& \cdots & {\mathbf{b}}_s^{\top }\left(\mathbf{x}\left(t_{n_s}\right)\right)\Delta q_{n_q}\left(t_{n_s}\right)\end{array}\right],\mathbf{A}\in {\mathbb{R}}^{n_s\times \left(n_q{n}_l\right)}.$$

where each row of $\mathbf{A}$ corresponds to a training sample, and each column corresponds to the influence of a specific receptive field on a particular joint.

The k-th element of incremental task error $\Delta \mathbf{e}$ is expressed as:

$$\Delta {\mathbf{e}}^{k,\mathrm{sample}}=\mathbf{A}\left(\mathbf{X}\right){\mathbf{w}}_k,$$

(72)

where ${\mathbf{w}}_k\in {\mathbb{R}}^{n_q{n}_l\times 1}$ denotes the weight vector of the neural network associated with the k-th row of the estimated Jacobian matrix. The vector $\Delta {\mathbf{e}}^{k,\mathrm{sample}}\in {\mathbb{R}}^{n_s\times 1}$ contains the incremental task errors for all samples in the dataset, and is defined as:

$$\Delta {\mathbf{e}}^{k,\mathrm{sample}}={\left[\Delta e^k\left(t_0\right),\Delta e^k\left(t_1\right),\cdots ,\Delta e^k\left(t_{n_s}\right)\right]}^{\top }.$$

The weights of neural network are initially estimated by solving the linear regression problem:

$${\mathbf{w}}_k={\left({\mathbf{A}}^T\mathbf{A}\right)}^{-1}{\mathbf{A}}^T\Delta {\mathbf{e}}_{k,\mathrm{sample}},k=1,2,\cdots ,10.$$

(73)

This pre-training strategy effectively initializes the network weights, thus increasing the chances of achieving global convergence.

5.2. Simulations

5.2.1. Performance Analysis

To validate our algorithm, we conducted simulations using a 6-DOF Universal Robot with a camera mounted on its end effector. Recognizing that unsuitable initial weights in the neural network could cause the control error to settle on a local minimum, we implemented pre-training on the neural network.

The initial and final camera positions, as well as the camera’s target position, are shown in Figure 7a. It can be observed that the final camera position coincides with the reference position. Figure 7b displays the motion trajectories of 25 feature points, with “*” denoting their initial positions and hollow circles “O” indicating their target positions. Solid dots “.” represent the trajectories of the feature points, illustrating their gradual convergence towards the reference positions. Furthermore, in Figure 8a, the defined error vector $\mathbf{e}$, as described in Equation (16), also converges to zero. This convergence indicates that as $\mathbf{e}$ converges, the image feature error also converges, and the camera’s pose aligns with the reference pose (R = I, t = 0).

$\lambda$ is a critical parameter controlling the strength of geometric constraints in our model. As shown in Figure 8a,c, selecting an appropriate value of $\lambda$ significantly influences both the convergence rate and stability of the neural network training process. Figure 8b illustrates the trajectory of feature point centers in image space, while Figure 8c demonstrates the evolution of feature point position errors over time.

When $\lambda =0$, the network exhibits oscillatory behavior with slower convergence, as evidenced by the fluctuations in Root Mean Square Error (RMSE) shown in Figure 8c. Introducing moderate constraint penalties ($\lambda =0.001$ to $\lambda =0.003$) effectively dampens these oscillations and accelerates convergence, leading to smoother trajectories in image space as depicted in Figure 8b. However, we observe that when $\lambda$ becomes too large ($\lambda =0.006$), the system exhibits different limitations. While initial error reduction is rapid, when feature point errors become small, the convergence rate significantly decreases, and the system struggles to minimize the residual errors as shown in the latter part of Figure 8c. This trade-off suggests that intermediate values (around $\lambda =0.003$) provide the optimal balance between fast convergence, stability, and final accuracy.

5.2.2. Comparisons with Other Studies

Currently, model-free visual servoing systems can be categorized into two approaches. The first category encompasses neural network-based methods [17,18,19,21], which typically use feature point positions as inputs/outputs, resulting in high-dimensional networks when processing numerous feature points. The second category consists of numerical iteration-based methods, such as Broyden’s method [15,16,20], which avoid explicit modeling by iteratively updating the Jacobian matrix. To validate the effectiveness of our proposed method, we conducted a series of simulations in MATLAB R2024a (MathWorks, Natick, MA, USA), comparing our geometry-constrained learning-based visual servoing framework with two representative model-free approaches: Broyden’s update method [20] and a neural network-based visual servoing method [21]. To ensure a fair comparison, all methods were independently tuned to achieve their best performance under the same experimental conditions. In addition, for the neural network-based method [21], we implemented a structure similar to ours, using a CMAC network with four overlapping associative memory layers. Each input dimension was discretized into nine regions, with every layer internally organizing the inputs into three blocks per dimension.

In the simulation environment, we constructed scenarios with 100 feature points and evaluated performance under both noise-free and noisy image conditions. The evaluation metrics included the RMSE of feature positions and the convergence trajectory of the object center in image space.

As shown in Figure 9b, under noise-free conditions, all three methods eventually achieved convergence, but with significant differences in their convergence characteristics. Both our proposed method and Broyden’s method [20] demonstrated superior dynamic performance, rapidly reducing RMSE to near zero within approximately 6 s. Among these, our method achieved the fastest convergence rate. In contrast, the neural network-based method [21], while ultimately converging, exhibited notable fluctuations and required approximately 35 s to fully stabilize.

When image noise was introduced, the differences in robustness among the methods became more distinct. In this test, we applied random noise with a maximum amplitude of 20 pixels to 10 feature points to simulate measurement uncertainties in real-world environments. Under these conditions, our method maintained its rapid convergence characteristics and minimal steady-state error, while the comparative methods showed significant performance degradation as shown as Figure 9d. Specifically, the neural network-based method [21] eventually converged but required longer convergence time. Broyden’s method [20] demonstrated noticeable performance degradation, with its RMSE stabilizing at approximately 100 pixels, indicating high sensitivity to noise.

Figure 9a,c further visualize the object center trajectories in image space under noisy and noise-free conditions, respectively. The results clearly indicate that, in both scenarios, our method produces smoother trajectories and successfully converges to the target position even in the presence of noise.

Through these two simulation comparisons, we can conclude that our proposed algorithm outperforms existing methods in terms of convergence speed, stability, and noise robustness. This performance improvement can be attributed to two key innovations:

(1)

Using a fixed-dimension task space error function as the neural network input effectively reduces network complexity. This smaller input dimension results in fewer learnable parameters, making the learning process more efficient. In contrast, other neural networks use feature point positions as inputs, where the input dimension increases proportionally with the number of feature points. As a result, when the number of feature points is large, these methods require learning an excessive number of parameters, significantly increasing computational complexity.

(2)

Incorporating geometric constraints of feature points to assist network learning, which not only accelerates the learning process, but also ensures the physical feasibility of control outputs. In comparison, although Broyden’s method [20] avoids the problem of high-dimensional neural networks, its stability and convergence performance when processing image noise remain significantly inadequate, limiting its effectiveness in practical applications.

(3)

Leveraging the Projective Homography matrix to enhance noise robustness. Unlike methods that rely on individual feature points, homography-based estimation effectively filters out the influence of measurement noise and ensures a more stable and reliable Jacobian estimation.

5.2.3. Computational Complexity Analysis

To evaluate the computational efficiency of the proposed method, we compare it with two representative data-driven image-based visual servoing (IBVS) approaches: Broyden’s update method [20] and a fuzzy CMAC-based neural network method [21].

The method in [21] builds a fuzzy CMAC controller for each joint and uses the image feature error vector, which has a dimension of $2P$, as input. As a result, both the complexity of the inference and the training process grow linearly with the number of joints $n_q$ and the number of feature points P, leading to a complexity per step of $\mathcal{O}(n_q·P)$.

The method in [20] updates the Jacobian matrix and a dynamic projection matrix during each iteration. Although it avoids direct matrix inversion, the update process still involves matrix multiplications and control computation. This results in a per-step complexity of $\mathcal{O}(P·n_q+n_q^2)$, which increases significantly when P becomes large.

In contrast, the proposed method uses a compact input vector composed of homography-based task error and joint states, with a fixed dimension that does not depend on the number of feature points. The inference complexity per step is only $\mathcal{O}\left(n_q\right)$. This design reduces computational cost and improves real-time performance in visual tasks with many image features.

5.3. Experiments

As shown in Figure 10, the experimental setup consists of a 6-DOF UR5 collaborative robot with an Intel RealSense D435i RGB-D camera (30 FPS) rigidly mounted on its end-effector. The visual servoing system was implemented using the Robot Operating System (ROS). A visual processing node (20 Hz) performed feature extraction, matching, and control computation. A control interface node (125 Hz) converted the control outputs into joint velocity commands. Communication between nodes was handled via ROS topics over TCP/IP. Computation was performed on a workstation with an Intel i7-9700K CPU, 32 GB RAM, and an NVIDIA RTX 4080Ti GPU.

To validate the effectiveness of our algorithm, we conducted two comparative experiments using different geometric constraint parameters: $\lambda$. Both experiments tested robustness by introducing occlusions of partial feature points during robot motion.

In both experiments, the robot was initially programmed to follow a random trajectory near its starting position to collect pre-training data. Feature matching and homography estimation were performed using the LightGlue framework combined with RANSAC, ensuring robust performance under varying lighting conditions and viewpoint changes while effectively filtering out outliers.

In the first experiment ($\lambda =0$), Figure 11a shows the progress of the images captured during online learning compared to the reference images. The sequence displays images at the initial moment and at 1.18 s, 7.08 s, and 28.05 s, respectively. Figure 11c tracks the evolution of the task error e, revealing an initial error increase during the first 1 s due to insufficient neural network learning, followed by eventual convergence as learning progressed.

In the second experiment ($\lambda =0.001$), the system exhibited enhanced performance characteristics, as illustrated in Figure 11b,d. These figures demonstrate accelerated convergence toward the reference image compared to the unconstrained approach. Figure 12 presents the comparison of task error RMSE between both experimental conditions, revealing the advantages of incorporating geometric constraints. The constrained implementation (red line) achieves error reduction at a substantially higher rate.

Throughout both experiments, we deliberately occluded certain feature points at various times, yet the error consistently converged. This demonstrates our algorithm’s robustness in maintaining accuracy even under challenging conditions with varying numbers of visible feature points.

The experimental results confirm that incorporating geometric constraints through our proposed ($\lambda$) parameter significantly accelerates learning.

6. Conclusions

In this paper, we introduce a geometry-constrained learning-based controller for visual servoing systems based on projective homography. Our approach utilizes a neural network to learn the system’s Jacobian matrix, thereby eliminating the need for precise camera calibration parameters and detailed robot kinematic models. Unlike other neural network approaches, our method utilizes a newly defined error vector related to projective homography as input, ensuring a constant input size irrespective of the number of image feature points. We also demonstrated in Appendix A that the defined error vector e = 0 is a sufficient and necessary condition to achieve R = I and t = 0, which signifies the alignment of the camera with the target. Furthermore, we incorporated geometric constraints between feature points in the network update process. By ensuring that model predictions conform to the fundamental principles of projective geometry, we significantly improved learning efficiency. Through simulations and experiments, we validate that our approach achieves superior performance compared to other model-free visual servoing methods, exhibiting faster convergence rates, enhanced robustness to image noise and partial occlusions.

Looking forward, while the proposed framework is robust and calibration-free, it still assumes sufficient exploration and network convergence during deployment. In practice, especially in unfamiliar or dynamic environments, learning may be incomplete, potentially leading to unsafe control actions. As a future direction, we plan to embed safety constraints into the control law, such as limits on joint velocities, workspace boundaries, and proximity to humans or delicate objects. These constraint-aware mechanisms will help ensure safe robot behavior even under imperfect Jacobian estimation, improving system reliability and enabling deployment in safety-critical applications.