Image Based Visual Servoing for Floating Base Mobile Manipulator Systems with Prescribed Performance under Operational Constraints

Abstract

This paper presents a novel Image-Based Visual Servoing (IBVS) control approach for Floating Base Mobile Manipulator Systems (FBMMSs) that imposes prescribed transient and steady-state response on the image feature coordinate errors while satisfying the visibility constraints that arise owing to the camera’s limited field of view. The proposed control strategy does not incorporate any knowledge on either the FBMMS dynamic model, the exogenous disturbances, or the inevitable camera calibration and depth measurement errors. More specifically, it guarantees: (i) predefined behavior in terms of overshoot, convergence rate, and maximum steady-state error value of the image features and system velocities tracking errors; (ii) satisfaction of camera field of view constraints; (iii) bounded closed-loop control signals, and (iv) reduced design and implementation complexity. Additionally, the performance of the developed scheme is solely determined by certain designer-specified performance functions/parameters, and it is fully decoupled by the control gains selection. The efficiency of the proposed scheme is demonstrated via a realistic simulation study, using an eye-in-hand Underwater Vehicle Manipulator System (UVMS) as a test-bed FBMMS platform.

1. Introduction

During the last few decades, robotic systems have gained a significant amount of attention from both research and industry communities and are already employed in various industrial fields and in today’s production lines [1]. However, the focus of researchers seems to have recently shifted from the development of conventional robots for structured industrial environments to the development of autonomous mobile robots operating in unstructured environments [2]. These autonomous robotic systems can be applied to a variety of demanding practical tasks such as surveillance, maintenance, and rescue of survivors, in uncertain and dynamic environments (e.g., underwater, space, or disaster areas) that may be rather dangerous for humans.

Therefore, the high demand for human replacement in such unsafe environments has already led to new areas of robot employment [3]. Moreover, in addition to mobility and flexibility, in many of these tasks, the mobile robots are also required to be endowed with interaction and manipulation capabilities in order to facilitate, improve and expedite specific operations that were usually carried out by humans exclusively [4].

A challenging and special case of mobile manipulator is the so-called Floating Base Mobile Manipulator System (FBMMS), where the vehicle base is not located at the ground and has more than three Degrees of Freedom (DoFs). In general, based on their operating environments, the FBMMSs are usually categorized into different groups such as: (i) Free-Floating Space Manipulators (FFSMs) [5,6], (ii) Underwater Vehicle Manipulator Systems (UVMSs) as in Figure 1 [7], and (iii) Unmanned Aerial Manipulator Systems (UAMSs) [8,9]. All these systems exhibit specific technical characteristics and limitations, which are defined by the diversity of the operating environments. For example, a UVMS may suffer from bandwidth issues in communication, significant external disturbances and latency in the thrusters’ response [10,11]. Similarly, during the control of a FFSM several issues must be dealt such as communication delays, zero gravity effect, and the absence of damping forces in the space environment.

1.1. Related Literature

Despite the difference in their operational specifications, in most interaction tasks where FBMMSs are employed, the system needs to incorporate visual information into a feedback control scheme to facilitate autonomous grasping and manipulation [2,12]. This results in Visual Servoing schemes, where the camera perceives the environment and the visual feedback is employed to determine the robot control input. Structurally, Visual Servoing can be categorized as: (i) Position-Based Visual Servoing (PBVS) where the extracted visual features are used to calculate the 3D pose of the target and the control error function is defined in the Cartesian space, (ii) Image-Based Visual Servoing (IBVS), where the control error is directly formulated in the image plane, and (iii) 2-1/2 Visual Servoing or hybrid visual servoing [13] where the $3D$ PBVS is combined with $2D$ IBVS [14,15,16].

The approaches mentioned above have their merits and drawbacks depending on the task at hand. In our approach, we will consider an IBVS scheme, as it is more efficient in mobile manipulation tasks due to better local stability properties and robustness against camera model uncertainties and depth estimation errors [17]. A rather important issue during visual servo control is the efficient handling of visibility constraints given that the image features should always be retained within the camera Field of View (FoV) during the motion of the robot [18,19]. Towards this direction, the authors in [20,21] presented path planning techniques for the image features based on the motion of the camera in 3D space. Optimization-based control approaches have also been employed in visual servoing for various robotic applications such as medicine [22], as well as for the navigation of autonomous aerial robots [23], ground mobile robots [24], and autonomous underwater vehicles [25]. Furthermore, a path planning strategy via LMI optimization has been studied in [26], while a novel approach imposing prescribed performance specifications on the image feature error was presented in our previous works [27,28].

Visual Servoing has been already implemented in mobile manipulator systems and significant studies towards this direction can be found in the literature. Direct IBVS approaches combined with optimization techniques were proposed in [7,29,30] in order to perform the guidance of a FBMMS. The coupling dynamics between the vehicle-manipulator system as well as the disturbances in a predicted motion of the end-effector were considered in [31,32]. A two layer IBVS scheme was proposed in [33] to overcome a potential idle condition by employing the system dynamics. Hybrid Visual Servoing techniques for FBMMSs were studied in [34,35,36,37] considering the kinematic limitations of the base motion. It is evident that the presence of conflicting operational limitations i.e., visibility constraints, transient and steady state response specifications, manipulator joint limits, system’s manipulability, etc., as well as various modelling uncertainties in the vehicle-manipulator system dynamics and the camera calibration parameters, increase notably the complexity of the IBVS-FBMMS control problem.

As it is evident from the literature review, none of the above schemes, besides our previous works [27,28], deals directly with the design of a predefined transient and steady state response of the image features errors during an IBVS visual servoing task. The notion of predefined or prescribed performance of a complex robotic system, such as a FBMMS, especially in vision-aided control, is of utmost importance, since it is able to guarantee smooth evolution of the control errors, hence safe motion of the overall robotic system. Moreover, most of the related studies found in literature, handle the visual control problem in pure kinematics (including our previous works [27,28]) ignoring the dynamics of the system, which in many cases (e.g., underwater vehicle manipulator systems, and aerial manipulators) are rather significant due to the external disturbances acting on the system in the form of generalized forces and torques, or the real model is too complex to be successfully captured solely by its kinematics.

On the other hand, there exist some limited studies that consider the complete or partial dynamic model of the system (e.g., [31,32,33]), including the interaction forces with the environment and the interconnected components of the systems, such as the interaction between the vehicle and the on-board manipulator. However, in such cases, the explicit dynamic model is required, which implies an exhaustive and time consuming modelling and system identification procedure, where depending on the nature of the system (e.g., underwater) and lack of appropriate sensors may lead to ambiguous results. Below, we summarize the main contribution of our work.

1.2. Contributions

In general, the prescribed performance control technique guarantees predefined transient and steady state error performance, in terms of overshot, rate of convergence and maximum state state error. It is a model-free method that does not require explicit knowledge of the system dynamic model, however it exhibits robustness against external disturbances. The predefined performance is achieved via specific pre-designed performance functions. The overall control scheme is of low complexity and can be easily implemented to an embedded on-board computer of an autonomous robotic vehicle.

In this work, we propose for the first time, a complete IBVS scheme based on the prescribed performance control notion, which is formulated specifically for mobile manipulator systems with floating base. It is important to highlight that even though the proposed scheme does not explicitly incorporate the dynamics of the complete system, it is able to efficiently calculate the control commands in the form of generalized forces and torques for the vehicle’s base as well as joint torques for the manipulator. The presented method has guarantee local convergence properties and it is accompanied by a rigorous stability analysis. More specifically, via the prescribed performance (PP) architecture, the proposed scheme guarantees predefined transient and steady state performance of the image feature errors, while handling efficiently field of view constraints without requiring any information of either the FBMMS dynamics or of any exogenous disturbances. The specific contributions of the proposed PP-IBVS scheme can be summarized as follows:

• Model-free IBVS for FBMMS calculating low-level commands (i.e., forces and torques), without explicit knowledge of the system model, which in most cases is quite complicated and difficult to identify with standard techniques.
• Predefined overshoot behavior, maximum steady-state error and rate of convergence.
• Compliance with the system’s geometrical and operational limitations such as joint limits and/or system’s manipulability.
• Robust steady-state behaviour against external disturbances.
• Regulation of the system’s performance by pre-designed performance functions, which are decoupled from the selection of control gains.
• IBVS scheme with low complexity which can be easily implemented to an embedded on-board computing system of FBMMS.

The efficiency and applicability of the proposed scheme are verified via a realistic simulation study using an eye-in-hand UVMS as a test FBMMS platform. To the best of the authors knowledge and by the time this manuscript was written, no similar IBVS scheme has been previously reported in the related literature not only for FBMMSs but also for mobile manipulator systems in general.

2. Problem Formulation

In this section, we initially present the mathematical model of the Floating Base Manipulator System. Next, we provide the basic IBVS modelling along with the underlying operational specifications. Finally, we describe analytically the problem at hand.

2.1. Floating Base Mobile Manipulator Modeling

At first, we consider a FBMMS consisting of total $n_r=n_v+n_m$ DoFs, where $n_v$ denotes the number of actuated DoFs of the floating base robotic vehicle and $n_m$ denotes the number of DoFs for the manipulator (see Figure 1). The state variables of the FBMMS are denoted by $\mathit{q}={[{\mathit{q}}_v^{\top },{\mathit{q}}_m^{\top }]}^{\top }\in {\Re }^{n_r}$, where ${\mathit{q}}_v={[{\mathit{\eta }}_{\mathbf{1}}^{\top },{\mathit{\eta }}_{\mathbf{2}}^{\top }]}^{\top }\in {\Re }^6$. More specifically, ${\mathit{\eta }}_1={[x_v,y_v,z_v]}^{\top }$ denotes the position vector and ${\mathit{\eta }}_2={[{\varphi }_v,{\theta }_v,{\psi }_v]}^{\top }$ the Euler-angles orientation of the vehicle, expressed w.r.t to an inertial frame $\left\{I\right\}$. The vector of the manipulator’s joint angles is denoted as ${\mathit{q}}_m\in {\Re }^{n_r-6}$. We also consider frame $\left\{E\right\}$ being attached at the end-effector of the FBMMS with position vector ${\mathit{x}}_e={[x_e,y_e,z_e]}^{\top }\in {\Re }^3$ and rotation matrix ${\mathit{R}}_e=[{\mathit{n}}_e,{\mathit{o}}_e,{\mathit{\alpha }}_e]$, expressed in $\left\{I\right\}$. Moreover, we denote by ${\mathit{\omega }}_e$ the angular velocity of the end-effector that satisfies $\mathit{S}\left({\mathit{\omega }}_e\right)={\dot{\mathit{R}}}_e{\mathit{R}}_e^{\top }$, with $\mathit{S}\left({\mathit{\omega }}_e\right)$ being the skew-symmetric matrix of ${\mathit{\omega }}_e$. Furthermore, we define vector ${\mathit{v}}_e={[{\mathit{t}}_e^{\top },{\mathit{\omega }}_e^{\top }]}^{\top }\in {\Re }^6$ which includes the linear ${\mathit{t}}_e$ and angular ${\mathit{\omega }}_e$ velocities of the end-effector frame. According to [38], it holds:

$${\mathit{v}}_e=\mathit{J}\left(\mathit{q}\right)\mathit{\zeta }$$

(1)

where $\mathit{\zeta }={[{\mathit{v}}^{\top },{\dot{\mathit{q}}}_{m,i}^{\top }]}^{\top }\in {\Re }^{n_r}$ includes the vehicle body and manipulator joint velocities $\mathit{v}$, ${\dot{\mathit{q}}}_{m,i},i\in \{1,\dots ,n_r-6\}$, respectively and $\mathit{J}\left(\mathit{q}\right)\in {\Re }^{6\times n_r}$ is the Jacobian matrix of the augmented vehicle-manipulator system. By invoking again [38], we are able to formulate the FBMMS dynamics, as follows:

$$\begin{array}{c}\mathit{M}\left(\mathit{q}\right)\dot{\mathit{\zeta }}+\mathit{C}(\mathit{q},\mathit{\zeta })\mathit{\zeta }+\mathit{D}(\mathit{q},\mathit{\zeta })\mathit{\zeta }+\mathit{g}\left(\mathit{q}\right)+\mathit{\delta }(\mathit{q},\mathit{\zeta },t)=\mathit{\tau }\end{array}$$

(2)

The term $\mathit{\delta }(\mathit{q},\mathit{\zeta },t)$ is used to encapsulate bounded unmodeled terms as well as external disturbances. The vector $\mathit{\tau }\in {\Re }^{n_r}$ includes the control inputs at the joint (manipulator) and thruster (vehicle) level, $\mathit{M}\left(\mathit{q}\right)$ is the positive definite inertial matrix, $\mathit{C}(\mathit{q},\mathit{\zeta })$ contains the coriolis and centrifugal terms, $\mathit{D}(\mathit{q},\mathit{\zeta })$ describes the damping effects, while $\mathit{g}\left(\mathit{q}\right)$ is the gravity-buoyancy vector.

Remark 1.

The matrix $\mathit{D}(\mathit{q},\mathit{\zeta })$ is used almost exclusively for Underwater Vehicle Manipulator Systems, where the damping effect of water on the motion of the vehicle is quite notable.

2.2. Mathematical Modeling of IBVS

Herein, the mathematical formulation of the image based visual servoing problem is presented. We consider a camera with a frame $\left\{\mathcal{C}\right\}$ attached at its center $O_c$ and ${[X_c,Y_c,Z_c]}^{\top }$ the camera frame axes. The image frame $\left\{{\mathcal{I}}_m\right\}$ coordinates, as shown in Figure 2, are decribed by ${[u,v]}^{\top }$ with $O_{im}$ being the image center [39].

According to the camera geometrical model, a set of n fixed 3D points ${\mathbf{p}}_{\mathbf{i}}={[x_i,y_i,z_i]}^T$, $i=1,\dots ,n$ expressed w.r.t the camera frame, are actually projected to the 2D image plane as ${\mathit{s}}_{\mathit{i}}={[u_i,v_i]}^T$, $i=1,\dots ,n$ in pixels, via the following equation [40]:

$${\mathit{s}}_{\mathit{i}}=\left[\begin{array}{c}{u}_i\\ v_i\end{array}\right]=\frac{\lambda }{z_i}\left[\begin{array}{c}{x}_i\\ y_i\end{array}\right]$$

(3)

with $\lambda$ denoting the focal length of the camera. The time derivative of the image features is related to the camera velocity as follows:

$${\dot{\mathit{s}}}_{\mathit{i}}={\mathit{L}}_i(z_i,s_i){\mathit{v}}_{\mathit{c}},i=1,\dots ,n$$

(4)

where:

$${\mathit{L}}_i\left(z_i,s_i\right)=\left[\begin{array}{cccccc}-\frac{\lambda }{z_i}& 0& \frac{u_i}{z_i}& \frac{u_i{v}_i}{\lambda }& -\frac{{\lambda }^2+u_i^2}{\lambda }& v_i\\ 0& -\frac{\lambda }{z_i}& \frac{v_i}{z_i}& \frac{{\lambda }^2+v_i^2}{\lambda }& -\frac{u_i{v}_i}{\lambda }& -u_i\end{array}\right]$$

(5)

is the interaction matrix [40], and ${\mathit{v}}_c\triangleq {\left[{\mathit{t}}_c^T,{\mathit{\omega }}_c^T\right]}^T$ contains the linear ${\mathit{t}}_c$ and rotational ${\mathit{\omega }}_c$ velocities of the camera frame. Next, we define as $\mathit{s}={\left[s_1^T,\dots ,s_n^T\right]}^T\in {\Re }^{2n}$ the overall image feature vector. Hence, the time-derivative of the image features is given by:

$$\dot{\mathit{s}}=\mathit{L}(z,s){\mathit{v}}_{\mathit{c}}$$

(6)

where $\mathit{L}\left(z,s\right)={\left[{\mathit{L}}_1^T\left(z_1,s_1\right),\dots ,{\mathit{L}}_n^T\left(z_n,s_n\right)\right]}^T\in {\Re }^{2n\times 6}$ is the interaction matrix including all the available features and $\mathit{z}={\left[z_1,\dots ,z_n\right]}^T$ their respective depths. Inevitably, visibility constraints are imposed by the limited camera field of view, via the following inequalities:

$$\begin{array}{cc}\hfill & u_{min}\le u_i\le u_{max},i=1,\dots ,n\hfill \end{array}$$

(7a)

$$\begin{array}{cc}\hfill & v_{min}\le v_i\le v_{max},i=1,\dots ,n\hfill \end{array}$$

(7b)

where $u_{min}$, $v_{min}$ and $u_{max}$, $v_{max}$ describe the lower and upper bounds (in pixels) of the image coordinates as dictated by the camera resolution ($\mathrm{width}\times \mathrm{height}$). The task of retaining the features of interest inside the camera field of view is of utmost importance in IBVS control of mobile manipulators, since otherwise not only the mission is compromised, e.g., object grasping or target tracking, but also unpredicted motions that may harm the system may occur due to the loss of visual feedback in the control loop.

2.3. Problem Statement

Let ${\mathit{s}}^d={\left[{{\mathit{s}}_1^d}^{\top },\dots ,{{\mathit{s}}_n^d}^{\top }\right]}^{\top }\in {\Re }^{2n}$ denote the vector of desired image features. The objective is to control the Floating Base Mobile Manipulator System subject to the visibility and system’s operational constraints in such a way that the actual image features $\mathit{s}\left(t\right)$ converge to a desired compact set that includes the desired image features vector ${\mathit{s}}^d$. Hence, the problem of this paper is defined as follows:

Problem 1.

Given a Floating Base Mobile Manipulator System equipped with a camera at its end effector and a desired image features vector ${\mathit{s}}_d$ that corresponds to a desired configuration of the camera with respect to the object, design a robust feedback control protocol, without considering any knowledge of either the system dynamics, or the exogenous disturbances, so that the following are satisfied:

• The image features do not escape the image plane during the control operation (field of view constraints);
• Predetermined overshoot, rate of convergence and steady state error for the image features and the system velocities;
• Respect system’s operational limitations, e.g., manipulator joint limits, system’s manipulability;
• Robustness against exogenous disturbances, system model uncertainties, camera calibration and depth estimation errors.

3. Control Methodology

In this work, regarding the $2D$ control of image features in the image plane, we adopt the Prescribed Performance Image Based Visual Servoing (PP-IBVS), presented in our previous work [27,28] to achieve prescribed transient as well as steady state response for all image feature errors while respecting the FoV constraints (7a,b). It should be highlighted that the PP-IBVS scheme exhibits proven robustness with respect to camera calibration and depth estimation errors. Moreover, in a similar way, regarding the system control at the dynamic level, we adopt the prescribed performance control technique [41,42,43] in order to attain prescribed performance for the system velocity errors.

The overall control architecture is depicted in Figure 3. As it can be seen, a cascaded control approach is employed, where the PP-IBVS acts as input for the PPC velocity controller, which yields the required actuation forces and torques without the need of a dynamic model. More specifically, a vision algorithm is responsible for detecting the position of features $s\left(t\right)$ in the image plane, while the desired feature positions $s_d$ are defined by the user (e.g., using a reference image). The formulated image error is fed to the PP-IBVS scheme which is responsible for calculating the desired velocities for each DoF of the overall system, in such a manner where the image errors and their transient response are prescribed, while simultaneously the field of view constraints are satisfied (i.e., the target is always visible from the camera). Next, the PPC scheme realizes the aforementioned velocities by calculating the appropriate forces and torques for the actuators of the robotic system, while achieving prescribed performance for the velocity errors and their transient responses and simultaneously satisfying actuation (e.g., joint motors) limits.

In a real system, the joint velocities can be measured by the encoders of the manipulator, while the vehicle velocities can be estimated on-line by fusing data from the appropriate navigation sensors. Since in our work we consider a UVMS as a test-bed, the vehicle’s velocities could be obtained by the fusion of Doppler Velocity Log (DVL) and Inertial Measurement Unit (IMU) measurements, e.g., via an Extended Kalman Filter (EKF). In the presented simulation scenarios, the required system velocities are available by the realistic ROS-based simulation framework UwSim [44].

3.1. PP-IBVS Control Design

At first, we define the image feature errors as:

$$\begin{array}{cc}\hfill e_i^u\left(t\right)& =u_i\left(t\right)-u_i^d,i=1,\dots ,n\hfill \end{array}$$

(8a)

$$\begin{array}{cc}\hfill e_i^v\left(t\right)& =v_i\left(t\right)-v_i^d,i=1,\dots ,n\hfill \end{array}$$

(8b)

where $u_i^d$, $v_i^d$ are the desired image features and $e\triangleq {\left[e_1^u,e_1^v,\dots ,e_n^u,e_n^v\right]}^T$ denotes the overall error. By employing the prescribed performance notion, we aim at imposing a strict evolution of the image feature errors within predefined and bounded regions, inside the image space. These regions are defined by appropriately selected strictly positive and decreasing performance functions of time. According to the prescribed performance control methodology we define the following inequalities for all $t\ge 0$:

$$\begin{array}{cc}\hfill -\underline{M}{}_i^u{\rho }_i^u\left(t\right)& <e_i^u\left(t\right)<\overline{M}{}_i^u{\rho }_i^u\left(t\right),i=1,\dots ,n\hfill \end{array}$$

(9a)

$$\begin{array}{cc}\hfill -\underline{M}{}_i^v{\rho }_i^v\left(t\right)& <e_i^v\left(t\right)<\overline{M}{}_i^v{\rho }_i^v\left(t\right),i=1,\dots ,n\hfill \end{array}$$

(9b)

where

$$\begin{array}{cc}\hfill {\rho }_i^u\left(t\right)& =\left(1-\frac{{\rho }_{\infty }}{max\left\{\underline{M}{}_i^u,\overline{M}{}_i^u\right\}}\right)exp(-lt)+\frac{{\rho }_{\infty }}{max\left\{\underline{M}{}_i^u,\overline{M}{}_i^u\right\}}\hfill \end{array}$$

(10a)

$$\begin{array}{cc}\hfill {\rho }_i^v\left(t\right)& =\left(1-\frac{{\rho }_{\infty }}{max\left\{\underline{M}{}_i^v,\overline{M}{}_i^v\right\}}\right)exp(-lt)+\frac{{\rho }_{\infty }}{max\left\{\underline{M}{}_i^v,\overline{M}{}_i^v\right\}}\hfill \end{array}$$

(10b)

are designer-specified smooth, bounded and decreasing functions of time with l, ${\rho }_{\infty }>0$ incorporating the desired transient and steady state performance specifications, respectively, and $\underline{M}{}_i^u,\overline{M}{}_i^u,\underline{M}{}_i^v,\overline{M}{}_i^v$ are positive parameters appropriately selected to satisfy the visibility constraints (7a,b). In particular, the decreasing rate of ${\rho }_i^u\left(t\right),{\rho }_i^v\left(t\right)$, which is regulated by the parameter l, introduces a lower bound on the speed of convergence of $e_i^u\left(t\right),e_i^v\left(t\right)$, $i=1,\dots ,n$. Depending on the accuracy of the vision detection algorithm, the parameter ${\rho }_{\infty }$ can be set arbitrarily small ${\displaystyle {\rho }_{\infty }<<\underset{i=1,\dots ,n}{min}\{\underline{M}{}_i^u,\overline{M}{}_i^u,\underline{M}{}_i^v,\overline{M}{}_i^v\}}$, in order to achieve convergence of $e_i^u\left(t\right),e_i^v\left(t\right)$, $i=1,\dots ,n$ close to zero. Moreover, we select:

$$\begin{array}{cc}\hfill & \underline{M}{}_i^u=u_i^d-u_{min}\&\overline{M}{}_i^u=u_{max}-u_i^d,i=1,\dots ,n\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill & \underline{M}{}_i^v=v_i^d-v_{min}\&\overline{M}{}_i^v=v_{max}-v_i^d,i=1,\dots ,n\hfill \end{array}$$

(11b)

Under the logical assumption that the features are initially located inside the camera field of view (i.e., $u_{min}<u_i\left(0\right)<u_{max}$ and $v_{min}<v_i\left(0\right)<v_{max}$, $i=1,\dots ,n$) it is guaranteed that:

$$\begin{array}{cc}\hfill -\underline{M}{}_i^u{\rho }_i^u\left(0\right)& <e_i^u\left(0\right)<\overline{M}{}_i^u{\rho }_i^u\left(0\right),i=1,\dots ,n\hfill \end{array}$$

(12a)

$$\begin{array}{cc}\hfill -\underline{M}{}_i^v{\rho }_i^v\left(0\right)& <e_i^v\left(0\right)<\overline{M}{}_i^v{\rho }_i^v\left(0\right),i=1,\dots ,n\hfill \end{array}$$

(12b)

Hence, employing the decreasing property of ${\rho }_i^u\left(t\right),{\rho }_i^v\left(t\right)$, $i=1,\dots ,n$ and invoking (9a,b) for all $t>0$, we achieve:

$$\begin{array}{c}\hfill -\underline{M}{}_i^u<e_i^u\left(t\right)<\overline{M}{}_i^u,i=1,\dots ,n\end{array}$$

(13a)

$$\begin{array}{c}\hfill -\underline{M}{}_i^v<e_i^v\left(t\right)<\overline{M}{}_i^v,i=1,\dots ,n\end{array}$$

(13b)

which owing to (8a,b) and (11a,b), yields:

$$\begin{array}{cc}\hfill u_{min}& <u_i\left(t\right)<u_{max},i=1,\dots ,n\hfill \end{array}$$

(14a)

$$\begin{array}{cc}\hfill v_{min}& <v_i\left(t\right)<v_{max},i=1,\dots ,n\hfill \end{array}$$

(14b)

for all $t>0$, thus respecting the visibility constraints as required by Section 2.3.

Next, we proceed to the design of a state feedback control scheme without incorporating any knowledge regarding the system dynamic model (2). The proposed scheme will be robust against inaccurate depth measurements and camera parameters, and will also guarantee via (9a,b) for all $t\ge 0$, prescribed performance under visibility constraints for the given IBVS problem. The overall control architecture is illustrated in Figure 3. More specifically, initially, we define the normalized image feature errors as:

$$\begin{array}{c}\hfill {\xi }_i^u(u_i,t)=\frac{e_i^u}{{\rho }_i^u\left(t\right)}\&{\xi }_i^v(v_i,t)=\frac{e_i^v}{{\rho }_i^v\left(t\right)},i=1,\dots ,n\end{array}$$

(15)

and the transformed image feature errors as:

$$E_i^u\left({\xi }_i^u(u_i,t)\right)=ln\left({\displaystyle \frac{1+{\displaystyle \frac{{\xi }_i^u(u_i,t)}{\underline{M}{}_i^u}}}{1-\frac{{\xi }_i^u(u_i,t)}{\overline{M}{}_i^u}}}\right)\&E_i^v\left({\xi }_i^v(v_i,t)\right)=ln\left({\displaystyle \frac{1+\frac{{\xi }_i^v(v_i,t)}{\underline{M}{}_i^v}}{1-\frac{{\xi }_i^v(v_i,t)}{\overline{M}{}_i^v}}}\right)$$

(16)

for which $e_i^u\to 0$ ($e_i^v\to 0$) implies $E_i^u\to 0$ ($E_i^v\to 0$), $i=1,\dots ,n$. Finally, the IBVS controller is formulated as:

$${\mathit{v}}_c^r(s,t)=-k{\widehat{\mathit{L}}}^{+}\mathit{E}(\mathit{s},t),k>0$$

(17)

with ${\widehat{\mathit{L}}}^{+}\triangleq {\left({\widehat{\mathit{L}}}^{\top }\widehat{\mathit{L}}\right)}^{-1}{\widehat{\mathit{L}}}^{\top }$ being the Moore-Penrose pseudo-inverse of the estimated interaction matrix [45], expressed in the regular image space, and

$$E(\mathit{s},t)\triangleq {\left[E_1^u,E_1^v,\dots ,E_n^u,E_n^v\right]}^{\top }.$$

(18)

3.2. Handling of Operational Specifications and Limits

The frame $\left\{\mathcal{C}\right\}$ of the camera that is rigidly attached on the End-Effector at $O_C$ does not necessarily coincide with the End-Effector frame $\left\{E\right\}$. Therefore, considering a spatial motion transformation matrix ${}^e{\mathit{V}}_c\in {\Re }^{6\times 6}$, the PP-IBVS given in (17) can be transformed from the camera frame to the End-Effector frame via [15]:

$${\mathit{v}}_e^r(s,t)={}^e{\mathit{V}}_c{\mathit{v}}_c^r(s,t)$$

(19)

Moreover, the End-Effector desired motion profile ${\mathit{v}}_e^r(s,t)$ is transformed to the configuration space as follows:

$$\begin{array}{c}\hfill {\mathit{\zeta }}^r\left(t\right)=\mathit{J}{\left(\mathit{q}\right)}^{\#}{\mathit{v}}_e^r+\left({\mathit{I}}_{n\times n}-\mathit{J}{\left(\mathit{q}\right)}^{\#}\mathit{J}\left(\mathit{q}\right)\right){\mathit{v}}_e^0\in {\Re }^n\end{array}$$

(20)

where $\mathit{J}{\left(\mathit{q}\right)}^{\#}$ denotes the generalized pseudo-inverse [46] of the Jacobian $\mathit{J}\left(\mathit{q}\right)$ of the vehicle-manipulator system and ${\mathit{v}}_e^0$ denotes secondary tasks, in order to satisfy vital system’s operational limitations (e.g., manipulator’s joint limits, manipulability) to be tuned independently since they do not affect the end-effector’s velocity [47] (i.e., they belong to the null space of the Jacobian $\mathit{J}\left(\mathit{q}\right)$). In the proposed work, image feature error minimization and field of view constraints are identified as primary tasks handled by the first part (i.e., $\mathit{J}{\left(\mathit{q}\right)}^{\#}{\mathit{v}}_e^r$) of (20), while the secondary tasks such as manipulator joint limits, avoiding kinematic singularity representations as well as system manipulability are incorporated in ${\mathit{v}}_e^0$ and handled by the second part (i.e., $\left({\mathit{I}}_{n\times n}-\mathit{J}{\left(\mathit{q}\right)}^{\#}\mathit{J}\left(\mathit{q}\right)\right){\mathit{v}}_e^0$) of (20) in a fully decoupled manner. More details on task priority based control for FBMMSs can be found in [47,48].

3.3. Prescribed Performance Velocity Control

Given the desired configuration space motion profile ${\mathit{\zeta }}^r\left(t\right)$ in (20) that satisfies different operational limitations, we proceed with the design of a model-free velocity controller that does not incorporate any information regarding the system’s dynamics, or exogenous disturbances, and achieves certain predefined transient and steady state response. Similarly to the PP-IBVS, the first step is to define the velocity error vector:

$$\begin{array}{c}\hfill {\mathit{e}}_{\zeta }\left(t\right)\triangleq {[e_{{\zeta }_1}\left(t\right),\dots ,e_{{\zeta }_{n_r}}\left(t\right)]}^{\top }=\mathit{\zeta }\left(t\right)-{\mathit{\zeta }}^r\left(t\right)\in {\Re }^{n_r}\end{array}$$

(21)

and then select the corresponding performance functions:

$$\begin{array}{c}\hfill {\rho }_{{\zeta }_i}\left(t\right)=({\rho }_{{\zeta }_i}^0-{\rho }_{{\zeta }_i}^{\infty })exp(-l_{{\zeta }_i}t)+{\rho }_{{\zeta }_i}^{\infty },i=1,\dots ,n_r\end{array}$$

(22)

with ${\rho }_{{\zeta }_i}^0>\left|e_{{\zeta }_i}\left(0\right)\right|$, ${\rho }_{{\zeta }_i}^{\infty }>0$ and $l_{{\zeta }_i}>0,i=1,\dots n_r$. Notice that similarly to the image features’ errors we aim to impose transient and steady state response on the system velocities errors $e_{{\zeta }_i}\left(t\right),i=1,\dots ,n_r$ as well, by satisfying:

$$\begin{array}{c}\hfill -{\rho }_{{\zeta }_i}\left(t\right)<e_{{\zeta }_i}\left(t\right)<{\rho }_{{\zeta }_i}\left(t\right),\forall t\ge 0i=1,\dots ,n_r\end{array}$$

(23)

Next, the transformed velocity error vector is defined as:

$$\begin{array}{c}{\mathit{\epsilon }}_{\zeta }\left({\mathit{\xi }}_{\zeta }\right)\triangleq {[{\epsilon }_{{\zeta }_1}\left({\xi }_{{\zeta }_1}\right),\dots ,{\epsilon }_{{\zeta }_{n_r}}\left({\xi }_{{\zeta }_{n_r}}\right)]}^{\top }\hfill \\ ={[ln\left({\displaystyle \frac{1+{\xi }_{{\zeta }_1}}{1-{\xi }_{{\zeta }_1}}}\right),\dots ,ln({\displaystyle \frac{1+{\xi }_{{\zeta }_{n_r}}}{1-{\xi }_{{\zeta }_{n_r}}}})]}^{\top }\hfill \end{array}$$

(24)

where

$$\begin{array}{c}\hfill {\mathit{\xi }}_{\zeta }\left(t\right)\triangleq {[{\xi }_{{\zeta }_1},\dots ,{\xi }_{{\zeta }_{n_r}}]}^{\top }={\mathbf{P}}_{\zeta }^{-1}\left(t\right){\mathit{e}}_{\zeta }\left(t\right)\end{array}$$

(25)

is the normalized velocity error vector, with ${\mathbf{P}}_{\zeta }\left(t\right)=\underset{i=1,\dots ,n_r}{\mathrm{diag}}\left[{\rho }_{{\zeta }_i}\left(t\right)\right]$ and we design the state feedback control law:

$$\begin{array}{c}\hfill \mathit{\tau }({\mathit{e}}_{\zeta }\left(t\right),t)=-{\mathit{K}}_{\zeta }{\mathbf{P}}_{\zeta }^{-1}\left(t\right){\mathit{R}}_{\zeta }\left({\mathit{\xi }}_{\zeta }\right){\mathit{\epsilon }}_{\zeta }\left({\mathit{\xi }}_{\zeta }\right)\end{array}$$

(26)

where

$$\begin{array}{c}\hfill {\mathit{R}}_{\zeta }\left({\mathit{\xi }}_{\zeta }\right)=\underset{i=1,\dots ,n_r}{\mathrm{diag}}\left[\frac{2}{1-{\xi }_{{\zeta }_i}^2}\right]\end{array}$$

(27)

and ${\mathit{K}}_{\zeta }>0$ is a diagonal gain matrix.

Remark 2.

The error transformation (visual errors for PP-IBVS and velocity errors in PPC velocity control) is realized in order to transform a constrained nominal dynamic system to an equivalent unconstrained one whose stability is sufficient to achieve the tracking control of the original constrained system with a priori prescribed performance.

4. Stability Analysis

In this section, we prove analytically that the proposed control architecture deals successfully with the IBVS problem for floating base mobile manipulator systems with prescribed performance under operational constraints.

Theorem 1.

Consider $n\ge 4$ fixed visual features in the workspace and a pinhole camera mounted on the end-effector of a floating base manipulator system with $n_r\ge 6$ DoFs that aims at achieving the desired positions for the feature coordinates on the image plane, while respecting the field of view constraints as well as operational limitations in the form of manipulator joint limits and manipulability. Assuming that all visual features initially lie sufficiently close to their desired values as well as within the field of view of the camera, the proposed IBVS architecture comprising of (17), (19), (20) and (26) guarantees local practically asymptotic stabilization of the feature errors.

Proof. 

First, let us define the overall normalized error vector $\mathit{\xi }={[{\mathit{\xi }}_s^{\top },{\mathit{\xi }}_{\zeta }^{\top }]}^{\top }$ with ${\mathit{\xi }}_s={[{\xi }_1^u,{\xi }_1^v,\dots ,{\xi }_n^u,{\xi }_n^v]}^{\top }$. Differentiating (15) and (25) with respect to time and substituting the system dynamics (2), (6), (21), (19) and (26), we obtain the closed loop system dynamics:

$$\begin{array}{c}{\dot{\mathit{\xi }}}_s\triangleq h_s({\mathit{\xi }}_s,t)\end{array}$$

(28a)

$$\begin{array}{c}{\dot{\mathit{\xi }}}_{\zeta }({\xi }_{\zeta },t)\triangleq h_{\zeta }({\mathit{\xi }}_{\zeta },t)\end{array}$$

(28b)

which can be written in compact form as:

$$\begin{array}{c}\hfill \dot{\mathit{\xi }}\triangleq h(\mathit{\xi },t)={[h_s^{\top }({\mathit{\xi }}_{\mathit{s}},t),h_{\zeta }^{\top }({\mathit{\xi }}_{\zeta },t)]}^{\top }\end{array}$$

(29)

Let us define the open set ${\mathsf{\Omega }}_{\xi }={\mathsf{\Omega }}_{{\xi }_s}\times {\mathsf{\Omega }}_{{\xi }_{\zeta }}$ with ${\mathsf{\Omega }}_{{\xi }_s}\triangleq (-\underline{M}{}_1^u,{\overline{M}}_1^u)\times (-\underline{M}{}_1^v,{\overline{M}}_1^v)\times \dots \times (-\underline{M}{}_n^u,{\overline{M}}_n^u)\times (-\underline{M}{}_n^v,{\overline{M}}_n^v)$ and ${\mathsf{\Omega }}_{{\xi }_{\zeta }}\triangleq {(-1,1)}^{n_r}$. In what follows, we proceed in two phases. First we ensure the existence of a unique maximal solution $\mathit{\xi }\left(t\right)$ of (29) over the set ${\mathsf{\Omega }}_{\xi }$ for a time interval $[0,t_{\max })$ (i.e., $\mathit{\xi }\left(t\right)\in {\mathsf{\Omega }}_{\xi },\forall t\in [0,t_{\max })$). Then, we prove that the proposed control protocol guarantees, for all $t\in [0,t_{\max })$ the boundedness of all closed loop signals as well as that $\mathit{\xi }\left(t\right)$ remains strictly within the set ${\mathsf{\Omega }}_{\xi }$, which leads by contradiction to $t_{\max }=\infty$ and consequently to the satisfaction of (9a,b) and (23), thus completing the proof.

Phase A: The set ${\mathsf{\Omega }}_{\xi }$ is nonempty and open. Moreover (12a,b) and (23) leads to $-\underline{M}{}_i^u<{\xi }_i^u\left(0\right)<{\overline{M}}_i^u$, $i\in \{u_1,\dots u_n\}$, $-\underline{M}{}_i^v<{\xi }_i^v\left(0\right)<{\overline{M}}_i^v$, $i\in \{v_1,\dots v_n\}$ and $-1<{\xi }_{{\zeta }_i}\left(0\right)<1$, $i=1,\dots ,n_r$. Thus, we guarantee that ${\mathit{\xi }}_s\left(0\right)\in {\mathsf{\Omega }}_{{\xi }_s}$ and ${\mathit{\xi }}_{\zeta }\left(0\right)\in {\mathsf{\Omega }}_{{\xi }_{\zeta }}$. Additionally, $h(\mathit{\xi },t)$, as defined in (29), is continuous on t and locally Lipschitz on $\mathit{\xi }$ over ${\mathsf{\Omega }}_{\xi }$. Therefore, the hypotheses of the Theorem 54 in [49] (p. 476) hold and the existence of a maximal solution $\mathit{\xi }\left(t\right)$ of (29) on a time interval $[0,t_{\max })$ such that $\mathit{\xi }\left(t\right)\in {\mathsf{\Omega }}_{\xi },\forall t\in [0,t_{\max })$ is ensured.

Phase B: In Phase A, we proved that $\mathit{\xi }\left(t\right)\in {\mathsf{\Omega }}_{\xi },\forall t\in [0,t_{\max })$, thus it can be concluded that:

$$\begin{array}{c}{\xi }_i^u\left(t\right)=\frac{e_i^u\left(t\right)}{{\rho }_i^u\left(t\right)}\in (-\underline{M}{}_i^u,{\overline{M}}_i^u),i\in \{1,\dots ,n\}\end{array}$$

(30a)

$$\begin{array}{c}{\xi }_i^v\left(t\right)=\frac{e_i^v\left(t\right)}{{\rho }_i^v\left(t\right)}\in (-\underline{M}{}_i^v,{\overline{M}}_i^v),i\in \{1,\dots ,n\}\end{array}$$

(30b)

$$\begin{array}{c}{\xi }_{{\zeta }_i}\left(t\right)=\frac{e_{{\zeta }_i}\left(t\right)}{{\rho }_{{\zeta }_i}\left(t\right)}\in (-1,1),i\in \{1,\dots ,n_r\}\end{array}$$

(30c)

for all $t\in [0,t_{\max })$, from which we obtain that $e_i^u\left(t\right)$, $e_i^v\left(t\right)$ and $e_{{\zeta }_i}\left(t\right)$ are lower and upper bounded by $-\underline{M}{}_i^u{\rho }_i^u\left(t\right),{\overline{M}}_i^u{\rho }_i^u\left(t\right)$, $-\underline{M}{}_i^v{\rho }_i^v\left(t\right),{\overline{M}}_i^v{\rho }_i^v\left(t\right)$ and $-{\rho }_{{\zeta }_i}\left(t\right),{\rho }_{{\zeta }_i}\left(t\right)$, respectively. Therefore, the transformed error vectors $E_i^u\left({\xi }_i^u(u_i,t)\right)$, $E_i^v\left({\xi }_i^v(v_i,t)\right)$ and ${\epsilon }_{{\zeta }_i}\left({\xi }_{{\zeta }_i}\right)$ designated in (16) and (24), respectively, are well-defined for all $t\in [0,t_{\max })$. Hence, consider the positive definite and radially unbounded function:

$$\begin{array}{c}\hfill V\left(\mathit{E}\right)=\frac{1}{2}{\mathit{E}}^{\top }\left({\mathit{\xi }}_s\right)\mathit{E}\left({\mathit{\xi }}_s\right)\end{array}$$

(31)

Differentiating $V\left(\mathit{E}\right)$ with respect to time, substituting (28a) and considering ${\dot{u_i}}^d\left(t\right)$, ${\dot{v_i}}^d\left(t\right)$, ${\rho }_i^u\left(t\right),{\rho }_i^v\left(t\right)i\in \{1\dots n\}$ and ${\rho }_{{\zeta }_i}\left(t\right),i=1,\dots ,n_r$ are bounded by construction and ${\xi }_i^u,{\xi }_i^v$, ${\xi }_{{\zeta }_i}$ are also bounded within the compact sets ${\mathsf{\Omega }}_{{\xi }_s}$ and ${\mathsf{\Omega }}_{{\xi }_{\zeta }}$ owing to (30a–c), we conclude the existence of a position constant ${\overline{\epsilon }}_s$ such that:

$$\begin{array}{c}\hfill |\mathit{E}\left({\xi }_s\left(t\right)\right)|\le {\overline{\epsilon }}_s,\forall t\in [0,t_{\max })\end{array}$$

(32)

Furthermore, from (18), and invoking the inverse logarithmic function, we obtain:

$$\begin{array}{c}\hfill -\underline{M}{}_i^u<{\underline{\xi }}_i^u\le {\xi }_i^u\left(t\right)\le {\overline{\xi }}_i^u<{\overline{M}}_i^u,\forall t\in [0,t_{\max }),i\in \{1\dots n\}\end{array}$$

(33a)

$$\begin{array}{c}\hfill -\underline{M}{}_i^v<{\underline{\xi }}_i^v\le {\xi }_i^v\left(t\right)\le {\overline{\xi }}_i^v<{\overline{M}}_i^v,\forall t\in [0,t_{\max }),i\in \{1\dots n\}\end{array}$$

(33b)

where:

$$\begin{array}{c}\hfill {\underline{\xi }}_i^u=-\underline{M}{}_i^u\frac{exp\left({\overline{\epsilon }}_s\right)-1}{exp\left({\overline{\epsilon }}_s\right)+\frac{\underline{M}{}_i^u}{{\overline{M}}_i^u}},{\overline{\xi }}_i^u={\overline{M}}_i^u\frac{exp\left({\overline{\epsilon }}_s\right)-1}{exp\left({\overline{\epsilon }}_s\right)+\frac{{\overline{M}}_i^u}{\underline{M}{}_i^u}}\end{array}$$

(34)

$$\begin{array}{c}\hfill {\underline{\xi }}_i^v=-\underline{M}{}_i^v\frac{exp\left({\overline{\epsilon }}_s\right)-1}{exp\left({\overline{\epsilon }}_s\right)+\frac{\underline{M}{}_i^v}{{\overline{M}}_i^v}},{\overline{\xi }}_i^v={\overline{M}}_i^v\frac{exp\left({\overline{\epsilon }}_s\right)-1}{exp\left({\overline{\epsilon }}_s\right)+\frac{{\overline{M}}_i^v}{\underline{M}{}_i^v}}\end{array}$$

(35)

Owing to (33a,b) and (20) it can concluded that the reference velocity vector ${\mathit{\zeta }}^r$ remains bounded for all $t\in [0,t_{\max })$ as well. Moreover, invoking $\mathit{\zeta }={\mathit{\zeta }}^r\left(t\right)+{\mathbf{P}}_{\zeta }\left(t\right){\mathit{\xi }}_{\zeta }$ from (25), we also conclude the boundedness of $\mathit{\zeta }$ for all $t\in [0,t_{\max })$. Finally, differentiating ${\mathit{\zeta }}^r\left(t\right)$ w.r.t time and employing (28a,b), (30a–c) and (33a,b) we conclude the boundedness of ${\dot{\mathit{\zeta }}}^r\left(t\right)$, $\forall t\in [0,t_{\max })$ too.

Now let us consider the positive definite and radially unbounded function $V_{\zeta }\left({\epsilon }_{\zeta }\right)=\frac{1}{2}\left|\right|{\mathit{\epsilon }}_{\zeta }{\left|\right|}^2$. Differentiating $V_{\zeta }$ with respect to time, substituting (28b) and employing the continuity of $\mathit{M}$, $\mathit{C}$, $\mathit{D}$, $\mathit{g}$, $\mathit{\delta }$, ${\mathit{\xi }}_{\zeta }$,${\dot{\mathbf{P}}}_{\zeta }$, ${\dot{\mathit{\zeta }}}^r$, $\forall t\in [0,t_{\max })$, we obtain:

$$\begin{array}{c}\hfill {\dot{V}}_{\zeta }\le \left|\right|{\mathbf{P}}_{\zeta }^{-1}{\mathit{R}}_{\zeta }\left({\mathit{\xi }}_{\zeta }\right){\mathit{\epsilon }}_{\zeta }\left|\right|\left(B_{\zeta }-{\lambda }_M{\mathit{K}}_{\zeta }\left|\right|{\mathbf{P}}_{\zeta }^{-1}{\mathit{R}}_{\zeta }\left({\mathit{\xi }}_{\zeta }\right){\mathit{\epsilon }}_{\zeta }\left|\right|\right)\end{array}$$

$\forall t\in [0,t_{\max })$, where ${\lambda }_M$ is the minimum eigenvalue of the positive definite matrix ${\mathit{M}}^{-1}$ and $B_{\zeta }$ is a positive constant independent of $t_{\max }$, that satisfies:

$$\begin{array}{cc}\hfill B_{\zeta }\ge & \left|\right|{\mathit{M}}^{-1}(\mathit{C}·({\mathbf{P}}_{\zeta }{\mathit{\xi }}_{\zeta }+{\mathit{\zeta }}^r\left(t\right))+\mathit{D}·({\mathbf{P}}_{\zeta }{\mathit{\xi }}_{\zeta }+{\mathit{\zeta }}^r\left(t\right))\hfill \\ \hfill & +\mathit{g}+\mathit{\delta }\left(t\right)+{\dot{\mathbf{P}}}_{\zeta }{\mathit{\xi }}_{\zeta }+{\dot{\mathit{\zeta }}}^r)\left|\right|\hfill \end{array}$$

Thus, we conclude that:

$$\begin{array}{c}\hfill \left|\right|{\mathit{\epsilon }}_{\zeta }\left({\mathit{\xi }}_{\zeta }\right)\left|\right|\le {\overline{\epsilon }}_{\zeta },\forall t\in [0,t_{\max })\end{array}$$

(36)

Furthermore, from (27) and invoking $|{\epsilon }_{{\zeta }_i}|\le {\overline{\epsilon }}_{\zeta }$, we obtain:

$$\begin{array}{c}\hfill -1<\underline{\xi }{}_{{\zeta }_i}\le {\xi }_{{\zeta }_i}\left(t\right)\le {\overline{\xi }}_{{\zeta }_i}<1,\forall t\in [0,t_{\max }),i=1,\dots ,n_r\end{array}$$

(37)

where

$$\begin{array}{c}\hfill \underline{\xi }{}_{{\zeta }_i}=\frac{-exp\left({\overline{\epsilon }}_{\zeta }\right)-1}{-exp\left({\overline{\epsilon }}_{\zeta }\right)+1},{\overline{\xi }}_{{\zeta }_i}=\frac{exp\left({\overline{\epsilon }}_{\zeta }\right)-1}{exp\left({\overline{\epsilon }}_{\zeta }\right)+1}\end{array}$$

(38)

which also leads to the boundedness of the control law (26) for all $t\in [0,t_{\max })$.

Subsequently, we will show that $t_{\max }$ can be extended to infinity. Obviously, notice by (33a,b) and (37) that $\mathit{\xi }\left(t\right)\in {\mathsf{\Omega }}_{\xi }^{{}^{\prime }}\triangleq {\mathsf{\Omega }}_{{\xi }_s}^{{}^{\prime }}\times {\mathsf{\Omega }}_{{\xi }_{\zeta }}^{{}^{\prime }},\forall t\in [0,t_{\max })$, where:

$$\begin{array}{cc}\hfill {\mathsf{\Omega }}_{{\xi }_s}^{{}^{\prime }}=& [\underline{\xi }{}_1^u,{\overline{\xi }}_1^u]\times [\underline{\xi }{}_1^v,{\overline{\xi }}_1^v]\times \dots [\underline{\xi }{}_n^u,{\overline{\xi }}_n^u]\times [\underline{\xi }{}_n^v,{\overline{\xi }}_n^v]\hfill \\ \hfill {\mathsf{\Omega }}_{{\xi }_{\zeta }}^{{}^{\prime }}=& [\underline{\xi }{}_{{\zeta }_1},{\overline{\xi }}_{{\zeta }_1}]\times \dots \times [\underline{\xi }{}_{{\zeta }_{n_r}},{\overline{\xi }}_{{\zeta }_{n_r}}],\hfill \end{array}$$

are nonempty and compact subsets of ${\mathsf{\Omega }}_{{\xi }_s}$ and ${\mathsf{\Omega }}_{{\xi }_{\zeta }}$, respectively. Hence, assuming that $t_{\max }<\infty$ and since ${\mathsf{\Omega }}_{\xi }^{{}^{\prime }}\subset {\mathsf{\Omega }}_{\xi }$, Proposition C.3.6 in [49] (p. 481) dictates the existence of a time instant $t{}^{{}^{\prime }}\in [0,t_{\max })$ such that $\mathit{\xi }\left(t^{{}^{\prime }}\right)\notin {\mathsf{\Omega }}_{\xi }^{{}^{\prime }}$, which is a clear contradiction. Therefore, $t_{\max }=\infty$. Thus, all closed loop signals remain bounded and moreover $\mathit{\xi }\left(t\right)\in {\mathsf{\Omega }}_{\xi }^{{}^{\prime }},\forall t\ge 0$. Finally, from (30a–c) and (33a,b), we conclude that:

$$\begin{array}{c}\hfill -\underline{M}{}^u{}_i{\rho }_i^u\left(t\right)<-\underline{M}{}^u{}_i\frac{exp\left({\overline{\epsilon }}_s\right)-1}{exp\left({\overline{\epsilon }}_s\right)+\frac{\underline{M}{}^u{}_i}{{\overline{M}}^u{}_i}}{\rho }_i^u\left(t\right)\le e_i^u\left(t\right)\le {\overline{M}}^u{}_i\frac{exp\left({\overline{\epsilon }}_s\right)-1}{exp\left({\overline{\epsilon }}_s\right)+\frac{{\overline{M}}^u{}_i}{\underline{M}{}^u{}_i}}{\rho }_i^u\left(t\right)<{\overline{M}}^u{}_i{\rho }_i^u\left(t\right)\end{array}$$

(39)

$$\begin{array}{c}\hfill -\underline{M}{}^v{}_i{\rho }_i^v\left(t\right)<-\underline{M}{}^v{}_i\frac{exp\left({\overline{\epsilon }}_s\right)-1}{exp\left({\overline{\epsilon }}_s\right)+\frac{\underline{M}{}^v{}_i}{{\overline{M}}^v{}_i}}{\rho }_i^v\left(t\right)\le e_i^v\left(t\right)\le {\overline{M}}^v{}_i\frac{exp\left({\overline{\epsilon }}_s\right)-1}{exp\left({\overline{\epsilon }}_s\right)+\frac{{\overline{M}}^v{}_i}{\underline{M}{}^v{}_i}}{\rho }_i^v\left(t\right)<{\overline{M}}^v{}_i{\rho }_i^v\left(t\right)\end{array}$$

(40)

for all $t\ge 0$, which completes the proof. □

5. Simulation Results

5.1. System Components and Parameters

In order to verify the proposed IBVS approach for FBMMSs, two realistic simulation scenarios were performed. An Underwater Vehicle Manipulator System (UVMS) is considered as candidate FBMMS platform. It is worth mentioning that the highly nonlinear dynamics of such systems are heavily affected by the almost unknown undersea water dynamics. Therefore, due to the large dynamic uncertainties, they constitute an ideal test-bed verifying the proposed control strategy. The simulation environment is implemented using the UwSim dynamic simulator [44] running on the Robot Operating System (ROS) [50], and the source code is written in C++ and Python. In particular, we consider two scenarios involving a UVMS consisting of an underwater vehicle with 4 DoFs (Surge, Sway, Heave Yaw) and a 4 rotational DoFs manipulator system attached to the underside of the vehicle. A camera with $640\times 480$ pixels at 30 frames per second (fps) is mounted on the manipulator’s end-effector. In each of the two simulations the UVMS is located at an initial configuration so that it is able to observe a visual target located at the bottom of the sea (see Figure 4 and Figure 5). The target is located at a fixed configuration inside the workspace and consists of four markers. Each marker center denotes an image feature that is detected using the Computer Vision ArToolkit library [51] and its ROS implementation. The loop rate of the PP-IBVS contoller was set at $0.03 \mathrm{s}$, while the loop rate of the PPC velocity controller was set at $0.02 \mathrm{s}$. It is worth mentioning that the considered initial pose for each of the two scenarios (Figure 4 and Figure 5) is rather challenging for IBVS schemes, due to the significant rotation about the x axis of the camera frame as well as the large distance relative to the target configuration. The desired feature coordinates $s^d=\left[\begin{array}{cccc}\hfill 251& \hfill 379& \hfill 251& \hfill 379\\ \hfill 196& \hfill 196& \hfill 318& \hfill 318\end{array}\right]$ were the same for both scenarios and extracted by a still image captured at the desired pose of the camera.

5.2. Results

In the following simulation studies, the objective for the proposed IBVS control scheme is to to autonomously guide the system from the initial configuration to the desired $s^d$. Moreover, the proposed controller should simultaneously satisfy: (i) the visibility constraints i.e., keep the object inside the field of view of the camera (ii) operational limitations i.e., manipulator joint limits, as well as (iii) predefined transient and steady-state performance specifications both on image features and system velocities errors.

5.2.1. Scenario 1

In the first scenario, the UVMS is initialized at the configuration depicted in Figure 4a. More specifically, the vehicle was located at $[x_{\upsilon },y_{\upsilon },z_{\upsilon },{\psi }_{\upsilon }]=[0.63,2.37,1.78,-1.57]$ and the manipulator’s joints were configured at $[q_{m1},q_{m2},q_{m3},q_{m4}]=[0.6,-0.25,-0.25,-0.0]$, which led to a very difficult initial configuration for an IBVS scheme, $s_{s1}^0=$$\left[\begin{array}{cccc}\hfill 205& \hfill 264& \hfill 234& \hfill 290\\ \hfill 207& \hfill 176& \hfill 273& \hfill 238\end{array}\right]$, where the target is very close to the image boundaries and can easily escape from the camera field of view. The parameters of the performance functions for the PP-IBVS scheme were set at ${\rho }_{\infty }=100$ and $\lambda =0.25$, while the values of $\underline{M}{}_i^u,{\overline{M}}_i^u,-\underline{M}{}_i^v,{\overline{M}}_i^v,i=1\dots 4$ were formulated according to the values of $s^d$ and the image boundaries dictated by the camera analysis ($640\times 480$). The evolution of image features errors is depicted in Figure 6. As it can be easily observed, the feature coordinate errors were retained within the corresponding performance envelopes and consequently the visibility constraints were continuously satisfied. The system was successfully guided to the desired configuration as depicted in Figure 4b.

Similarly, Figure 7 depicts the performance of the system velocity errors. The parameters of the performance functions for the vehicle (ROV) DoFs (Surge, Sway, Heave, Yaw) were set at ${\rho }_{{\zeta }_{ROV}}^0=1.5$, ${\rho }_{{\zeta }_{ROV}}^{\infty }=0.2$, ${\lambda }_{ROV}=0.25$. The parameters of the performance functions for the manipulator joints were set at ${\rho }_{{\zeta }_{man}}^0=1.5$, ${\rho }_{{\zeta }_{man}}^{\infty }=0.3$, ${\lambda }_{man}=0.25$. As it can be observed from Figure 7 the system velocity errors were constantly preserved within their corresponding performance envelopes. Therefore, the predefined behavior in terms of overshoot, rate of convergence and steady state was satisfied. The evolution of the underwater vehicle’s control inputs in the form of generalized forces & torques is presented in Figure 8. The control bounds of the vehicle are $[\underline{F}{}_{X_{ROV}},{\overline{F}}_{X_{ROV}}]=[-68.7,68.7]$, $[\underline{F}{}_{Y_{ROV}},{\overline{F}}_{Y_{ROV}}]=[-29.4,29.4]$, $[\underline{F}{}_{Z_{ROV}},{\overline{F}}_{Z_{ROV}}]=[-29.4,29.4]$, $[\underline{N}{}_{ROV},{\overline{N}}_{ROV}]=[-8.4,8.4]$. Therefore, it is clearly shown that the vehicle control inputs are always kept within these bounds. The manipulator’s control inputs in the form of joint torques are presented in Figure 9. The control bounds of the manipulator joints are $[\underline{{\tau }_q}{}_{m_1},{\overline{{\tau }_q}}_{m_1}]=[-3.5,3.5]$, $[\underline{{\tau }_q}{}_{m_2},{\overline{{\tau }_q}}_{m_2}]=[-2.8,2.8]$, $[\underline{{\tau }_q}{}_{m_3},{\overline{{\tau }_q}}_{m_3}]=[-1.5,1.5]$, $[\underline{{\tau }_q}{}_{m_4},{\overline{{\tau }_q}}_{m_4}]=[-1.5,1.5]$. Hence, the manipulator control inputs are always kept within these bounds. Moreover, the evolution of the vehicle as well as the manipulator joints states during the simulation operation are presented in Figure 10 and Figure 11, respectively. The bounds of the manipulator joints are $[\underline{q}{}_{m_1},{\overline{q}}_{m_1}]=[-\pi ,\pi ]$, $[\underline{q}{}_{m_2},{\overline{q}}_{m_2}]=[-\pi /2,0.36\pi ]$, $[\underline{q}{}_{m_3},{\overline{q}}_{m_3}]=[-0.72\pi ,\pi /2]$, $[\underline{q}{}_{m_4},{\overline{q}}_{m_4}]=[-\pi ,\pi ]$. It can be easily seen that the manipulator joint limits were preserved during the complete operation. Finally, the evolution of the camera frame’s velocities is given in Figure 12, where it can be observed that they converge close to zero as the system reaches its desired position.

5.2.2. Scenario 2

In the second scenario, the UVMS is initialized at the configuration depicted in Figure 5a. More specifically, the vehicle was located at $[x_{\upsilon },y_{\upsilon },z_{\upsilon },{\psi }_{\upsilon }]=[1.4,2.7,2.3,-1.57]$ and the manipulator’s joints were configured at $[q_{m1},q_{m2},q_{m3},q_{m4}]=[-0.35,-0.4,-0.4,-0.1]$, which also led to a challenging initial configuration, $s_{s2}^0=\left[\begin{array}{cccc}\hfill 416& \hfill 493& \hfill 390& \hfill 464\\ \hfill 294& \hfill 326& \hfill 369& \hfill 402\end{array}\right]$, very close to the image boundaries. The parameters of the performance functions were set at the same values as in Scenario 1. The evolution of image features errors is depicted in Figure 13. As it can be easily observed, the feature coordinate errors were retained within the corresponding performance envelopes and consequently the visibility constraints were continuously satisfied. The system was again successfully guided to the desired configuration as depicted in Figure 5b.

Likewise, Figure 14 depicts the performance of the system velocity errors in the second scenario. The parameters of the performance functions for the vehicle and the manipulator joints were set at the same values as in Scenario 1. As it can be observed from Figure 14 the system velocity errors were constantly preserved within their corresponding performance envelopes. Therefore, the predefined behavior in terms of overshoot, rate of convergence and steady state was satisfied. The evolution of the underwater vehicle’s control inputs in the form of generalized forces & torques as well as the manipulator’s control inputs in the form of joint torques are indicated in Figure 15 and Figure 16, respectively. The control inputs for the vehicle and the manipulator are always preserved within the operational bounds. Moreover, the evolution of the vehicle as well as the manipulator joints states during the simulation operation are presented in Figure 17 and Figure 18, respectively. The manipulator joint limits were again preserved during the complete operation. The evolution of the camera frame’s velocities is given in Figure 19, where it can also observed that they converge close to zero as the system reaches its desired position.

The performance of the proposed IBVS framework is also demonstrated at a short video that can be found at the following https://youtu.be/dO51LNMMADE accessed date (3 July 2022).

As mentioned previously, the evaluation of the proposed method was realized in a simulation framework. However, in a real system, we expect lower convergence rate and possibly more oscillations during the transient response. This expected behavior will be mainly due to external disturbances (e.g., ocean currents for an underwater vehicle manipulator system), sensor noises (camera, encoders, navigation system) and the difference in latency between the joint motors and the motor thrusters, where the latter usually exhibit faster response. However, the closed loop will retain its stability properties and the errors will converge close to zero with prescribed performance. In any case, the employed simulation framework is quite realistic, since it includes the dynamics of the complete system and also the target detection algorithm runs in real time on raw images.

6. Conclusions

In this study, we presented an IBVS control scheme for Floating Base Mobile Manipulator Systems. We adopted the prescribed performance control technique, which is able to guarantee predefined transient and steady state performance specifications on the image feature errors and satisfy field of view constraints, without incorporating any information regarding the system dynamics, the exogenous disturbances and in the presence of inevitable camera calibration and depth measurement errors. Various performance specifications such as: (i) maintenance of predefined behavior in terms of overshoot, convergence rate and maximum steady-state error, (ii) compliance with the system’s operational limitations (e.g., manipulator joint limits) and (iii) robust steady-state can be simultaneously achieved. The efficiency of the proposed scheme was demonstrated via a realistic simulation study from different initial camera configurations, using an eye-in-hand Underwater Vehicle Manipulator System (UVMS) as a test FBMMS platform.