Eye-in-Hand Robotic Arm Gripping System Based on Machine Learning and State Delay Optimization ^†

Abstract

This research focused on using RGB-D images and modifying an existing machine learning network architecture to generate predictions of the location of successfully grasped objects and to optimize the control system for state delays. A five-finger gripper designed to mimic the human palm was tested to demonstrate that it can perform more delicate missions than many two- or three-finger grippers. Experiments were conducted using the 6-DOF robot arm with the five-finger and two-finger grippers to perform at least 100 actual machine grasps, and compared to the results of other studies. Additionally, we investigated state time delays and proposed a control method for a robot manipulator. Many studies on time-delay systems have been conducted, but most focus on input and output delays. One reason for this emphasis is that input and output delays are the most commonly occurring delays in physical or electronic systems. An additional reason is that state delays increase the complexity of the overall control system. Finally, it was demonstrated that our network can perform as well as a deep network architecture with little training data and omitting steps, such as posture evaluation, and when combined with the hardware advantages of the five-finger gripper, it can produce an automated system with a gripping success rate of over 90%. This paper is an extended study of the conference paper.

1. Introduction

The COVID-19 pandemic means that there is a greater demand for automated production. To avoid a reduction in production capacity due to insufficient manpower, robotic arms with automatic control systems must perform more complex and more varied tasks.

Nowadays, robot arms for pick-and-place and assembly tasks have matured [1,2,3,4,5,6,7,8], and many factory robot arms now use two- or three-finger grippers for tasks, and many research papers have been conducted in this direction or for experiments [9,10,11,12]. Some use algorithms for control [13] and others use visual images and machine learning to allow robotic arms to perform tasks [14,15]. Other studies convert images into point clouds that use depth distances for better task execution [16,17,18,19,20,21,22,23,24]. For example, the study in [25] combines object recognition by Mobile-DasNet and point cloud analysis to generate the coordinates of the arm endpoints for the apple picking task.

However, if the factory is to become more automated, the robot arm needs to be able to perform more complex movements. On this premise, the original two- and three-finger grippers may not be able to complete more delicate or complex movements, so many people began to develop and use other types of grippers to overcome these shortcomings, including the five-finger grippers. Although the five-finger grippers have a larger gripping range, higher flexibility and fault tolerance, because of their complex structure, many control systems that use two- or three-finger grippers, as to derive the gripping attitude may not be compatible with five fingers.

There have been some studies using five-finger graspers, but most of them studied hardware or other aspects [26,27,28,29]. The studies used for application have used a virtual space as a result of the study [30,31] or used algorithms for gripping pose prediction [32,33].

This paper is an extended study of the original conference paper [1]. We extend the original conference paper on the effects of time delay for the robot manipulator. We investigate state time delays and propose a control method for a robot manipulator. Many studies on time-delay systems have been conducted, but most focus on input and output delays. One reason for this emphasis is that input and output delays are the most commonly occurring delays in physical or electronic systems. An additional reason is that state delays increase the complexity of the overall control system. In this paper, we use the qbSoftHand five-finger grasper in combination with an RGB-D visual recognition system and modify the proposed mechanical learning network by omitting the pose evaluation part so that the arm can automatically move to the target location and accomplish the task of grasping the target object to the placement area. The state delay of the control system is also optimized to enable the system to operate efficiently. Experiments were conducted with objects that were considered difficult to grasp with a two- or three-finger grasper to demonstrate the superiority of the grasping method and five fingers.

2. Control System Architecture

The hardware for this study is a UR5 six-degree-of-freedom arm, qbSoftHand five-finger gripper, HIWIN XEG-32 gripper and a RealSense D435 camera. The system architecture is shown in Figure 1. Object recognition is achieved using Yolo [34,35]. The point cloud for the object is generated after sampling and adjustment. The data are used as input data for an AI model that predicts the gripping posture for the arm and sends this information to the arm to perform the gripping action.

The pose that is predicted by the network is sent to the UR5 control system via the socket library of the network cable. The current joint angle, the tool center point (TCP) and the analog and digital signals for the arm are also obtained in this way.

A successful grasping action is achieved if the arm moves to the target location, picks up the object and moves it to the target position without dropping the object.

3. Point Cloud Data and Machine Learning Architecture

3.1. Convolutional Neural Network (CNN) Architecture

The architecture of the CNN is shown in Figure 2. This section describes the hidden layer (convolutional layer).

The convolutional layer is the core of the CNN [36]. When the image is input into the convolution layer, it performs convolutional operations using the convolutional kernel. The formula is:

$$X_j^l=f({\displaystyle {\sum }_{i\in Pj}{X}_i^{l-1}}*k_{ij}^l+b_j^l)$$

(1)

where $f({\displaystyle {\sum }_{i\in Pj}{X}_i^{l-1}}*k_{ij}^l+b_j^l)$ is a tanh function, $P_j$ is a local receptive field, $X_i^{l-1}$ is the value of the l-1 feature on the i window, $(i,j)$ is the position on the first floor, $k_{ij}^l$ and $b_j^l$ are the respective weights of the convolution kernel and the offset of the feature. More details can be found in [37].

The input data for this study are not images, but numerical values, so the convolution is a one-dimensional convolution. Differences between this and a two-dimensional convolution are described in Section 3.3.

The AI network for this study consists of three one-dimensional convolutional layers, a pooling layer and a multi-layer perceptron [25]. The point cloud data are input and the position at which the object is grasped is predicted. This information is transmitted to the UR5 arm control system. The network architecture using the CNN is shown in Figure 3.

3.2. Min-Pnet Architecture

Min-Pnet is a network designed according to the architecture of PointNet [37] (see Figure 4) In PointNet research, it is mentioned that many studies transform the input data into regular 3D voxel grids or collections of multi-angle images, generating large amounts of unnecessary data and destroying the natural invariance of the original data. This study uses point clouds directly to avoid these problems and to make it easier to learn.

In order to protect the network itself from point cloud disorder, PointNet uses its “feature extraction layer” to convert disordered point cloud data into 1024-dimensional features to perform subsequent tasks.

The feature extraction layer is to transform the input points into features by assembling the input data into a canonical space and using T-net to predict an affine transformation matrix. The partial and global feature extraction of the point cloud is performed without affecting the correlation and invariance of the points.

In this study, the Min-Pnet (see Figure 5) uses one feature extraction layer instead of two to reduce the training time and to avoid overfitting. The original 2D conv part was changed to 1D conv. This is because our input data size is different from the original setting of Pointnet, which causes problems in the matrix multiply part of the network.

3.3. Input Point Cloud Data

To ensure that the point clouds are reliable and not easily affected by the external environment, a PCL [38] chopbox and the Yolo Bounding Box are used to remove most of the unwanted point clouds (walls and desktops; see Figure 6).

To increase the amount of training data and to ensure that the model resists manipulation in response to slight errors that are caused by the hardware, Gaussian noise is used (Figure 7.)

To ensure that the input data are the same size, the point clouds for these objects are sampled and processed. The processed input data are a 3000 × 3 (N × 3 in Figure 8) matrix of the XYZ coordinates of the points (Figure 7). These data correspond to grasping point data, including the angle of each joint and a coordinate system based on UR5. These data are the input for the AI network and the prediction.

A one-dimensional convolution layer (1D conv) is used for convolution and feature extraction from 1D data. Most AI frameworks that are used for object recognition or for grasp prediction, which use images, use a two-dimensional convolution layer (2D conv). The differences between these systems are shown in the Figure 9.

In the CNN network, 1D conv directly extracts features from the input point cloud, and after processing with the maximum pooling layer (Max Pooling), it finally inputs the multi-layer perceptron (MLP) composed of several fully connected layers to obtain the output of the final predicted grasp position. This study uses the position of the object in space to predict the grasping position of the gripper so a 1D conv is used to extract the main features. After we sample the point cloud data, it is input to the model as a network, and the model outputs the predicted grasp posture coordinates or joint angles.

A 1D conv is initially used for natural language and data analysis and can also be used to analyze point clouds in data format. It is less computationally intensive and requires a shorter training time than a 2D conv. A 2D conv cannot be used because the point cloud data are not continuous (Figure 9), so the extracted features cannot be applied, which affects the prediction results.

3.4. Pooling Layer

The pooling layer reduces the dimensions of features and filters redundant features to reduce the computational burden and increase the generalization of the network. The pooling layer uses the maximum pooling (Max Pooling; see Figure 10) and mean pooling (Average pooling). The pooling process is expressed as:

$$X_j^i=f(pool(X_i^{l-1})+b_j^l)$$

(2)

where $X_i^{l-1}$ is the value of the ith window in the layer l-1 input feature, $b_j^l$ is the offset for the jth window in layer l and pool represents the sampling function.

3.5. Multi-Layer Perceptron (MLP)

Multi-layer perceptron uses several fully connected layers. Neurons in the full connection layer are connected to each other in the previous layer. The formula is:

$$X^l=f(u^l).u^l=W^l{X}^{l-1}+b^l$$

(3)

where $f(u^l)$ is the activation function, $W^l$ is the weight of layers l-1 to 1, $b^l$ is the offset for Layer 1, and $X^{l-1}$ is the output feature of Layer l-1.

The last fully connected layer outputs six parameters that are used to predict joint angles or TCP coordinates. To avoid overfitting and prediction errors in the AI model, an exit layer is inserted before the final output layer.

4. State Delays Using Digital Redesign

We present the principles of optimal digital redesign in this section. This system removes the following parts of the original transformation of a time-delay system to a delay-free system [39]:

$$\dot{x}(t)= A x(t)+ B u(t)$$

(4)

$$y(t)= C x(t) , x(0)= x_0$$

(5)

The optimal quadratic state feedback control law is used to minimise the following performance cost function:

$$J= {\displaystyle {\int }_{ 0}^{ \infty } \left\{{[ Cx(t)- r(t) ]}^T Q_c [Cx(t)- r(t) ]+ u_{}{}^T(t)R_{c}u(t) \right\} dt}$$

(6)

where $Q_c\ge 0$, $R_c>0$. The following formula is obtained by optimising the controller:

$$u(t)= -K_{c}x(t)+ E_{c}r(t)$$

(7)

The entire closed loop system can then be expressed as

$$\dot{x}(t)= (A- B{K}_c)x(t)+ B{E}_{c}r(t)$$

(8)

For $m=p$, we obtain

$$K_c= R_c{}^{-1}{B}^{T}P$$

(9)

$$E_c= -R_c{}^{-1}{B}^T{[{(A- B{K}_c)}^{-1}]}^T{C}^T{Q}_c$$

(10)

Here, $P$ is the solution to the Riccati equation given below.

$$A^{T}P+ PA- PB{R}_c{}^{-1}{B}^{T}P+ C^T{Q}_{c}C= 0$$

(11)

The linear quadratic regulator (LQR) (Equation (6)) design characteristics make the resulting closed loop system stable. If the system state is unmeasurable, an observer must be designed to measure the system state. The linear observable continuous system (see Figure 11) that is described by the equation is shown below:

$$\dot{\widehat{x}}(t)= A\widehat{x}(t)+ Bu(t)+ L_c[y(t)- C\widehat{x}(t)]$$

(12)

where $\widehat{x}(t)$ is the estimated state, and $L_c$ is the gain of the observer:

$$L_c=P_o{C}^T{R}_o{}^{-1}$$

(13)

We apply digital redesign to the analogue controller (Equation (7)) to obtain a more practical digital controller. The operation of the discrete time state feedback controller is described by the following equation:

$$u(k{T}_s)= -K_{d}x(k{T}_s)+ E_d r^{*}(k{T}_s)$$

(14)

where

$$K_d= {(I+ K_{c}H)}^{-1}{K}_{c}G$$

(15)

$$E_d= {(I+ K_{c}H)}^{-1}{E}_c$$

(16)

$$r^{*}(k{T}_s)= r(k{T}_s+ T_s)$$

(17)

$$G= e^{A{T}_s}$$

(18)

$$L_d=(G-I)A^{-1}{L}_c {(I+C(G-I)A^{-1}{L}_c)}^{-1}$$

(19)

For the linear model of the sampling system, the digital tracker based on the observer and the observer are shown in Figure 12.

5. Results

This study uses bottles, bowls and sports balls for the experiments (see Figure 13). The training parameters for the AI network are a batch size = 32 and an epoch = 10,000 and the loss parameter uses mean squared error (MSE). The selected optimizer is ‘adam’.

$$MSE(y,\widehat{y})=\frac{1}{n}{\displaystyle \sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2}$$

(20)

$y_i-{\widehat{y}}_i$ is error, n is number of data in Dataset.

This study uses Python 3.7.7 and a Windows10 system environment for model training and a crawling test.

In order to verify that the proposed control method [40] can be applied in subsequent real-world tests and to collect data more easily, a simulator was used to design a model of a nonlinear MIMO robot manipulator, as shown in Figure 14.

5.1. The Test of the Robot Manipulator

The dynamic equation of the two-link robot system is given below:

$$M(q)\ddot{q}+C(q,\dot{q})\dot{q}+G(q)=\mathrm{\Gamma }$$

(21)

where

$$\begin{array}{ll}M(q)=\left[\begin{array}{cc}(m_1+m_2)l_1^2& m_2{l}_1{l}_2(s_1{s}_2+c_1{c}_2)\\ m_2{l}_1{l}_2(s_1{s}_2+c_1{c}_2)& m_2{l}_2^2\end{array}\right]& , C(q,\dot{q})=m_2{l}_1{l}_2(c_1{s}_2-s_1{c}_2)\left[\begin{array}{cc}0& -{\dot{q}}_2\\ -{\dot{q}}_1& 0\end{array}\right]\\ G(q)=\left[\begin{array}{c}-(m_1+m_2)l_1{g}_r{s}_1\\ -m_2{l}_2{g}_r{s}_2\end{array}\right]\end{array}$$

and $q={\left[\begin{array}{cc}{q}_1& q_2\end{array}\right]}^T$, where $q_1$ and $q_2$ are angular positions, $M(q)$ is the moment of inertia, $C(q,\dot{q})$ includes the Coriolis and centripetal forces, $G\left(q\right)$ is the gravitational force, and $\mathrm{\Gamma }$ is the applied torque vector. Here, we use the short-hand notations $s_i=\sin (q_i)$ and $c_i=\cos (q_i)$. The nominal parameters of the system are as follows: the link masses are $m_1=5 \mathrm{kg}$ and $m_2=2.5 \mathrm{kg}$, the lengths are $l_1=l_2=0.5 \mathrm{m}$, and the gravitational acceleration is $g_r=9.81{ \mathrm{ms}}^{-2}$. Then, (30) can be rewritten in the following form:

$$\ddot{q}=M^{-1}(q)(\mathrm{\Gamma }-C(q,\dot{q})\dot{q}-G(q))$$

(22)

Let $x$ and $f(x)$ represent the state of the system and a nonlinear function of the state $x$, respectively. The following notation is used:

$$x(t)\equiv {[x_1 x_2 x_3 x_4]}^T={[q_1 {\dot{q}}_1 q_2 {\dot{q}}_2]}^T, f(x(t))\equiv {[f_1 f_2 f_3 f_4]}^T,$$

where

$f_1=x_2$, $f_3=x_4$, and ${[f_2 f_4]}^T=M^{-1}(-C{[x_2 x_4]}^T-G)$. Let $u\equiv \mathrm{\Gamma }$, where $\mathrm{\Gamma }={\left[\begin{array}{cc}{\mathrm{\Gamma }}_1& {\mathrm{\Gamma }}_2\end{array}\right]}^T$.

The inverse of the matrix $M$ is calculated as $M^{-1}=\left[\begin{array}{cc}{p}_{11}& p_{12}\\ p_{21}& p_{22}\end{array}\right]$, such that $g(x(t))={\left[\begin{array}{cccc}0& p_{11}& 0& p_{21}\\ 0& p_{12}& 0& p_{22}\end{array}\right]}^T$.

The dynamic equation of the two-link robot system can, thus, be reformulated as follows:

$$\dot{x}(t)=f(x(t))+g(x(t))u(t-{\tau }_i)$$

(23)

$$y(t)=Cx(t-{\tau }_o)$$

where $C=\left[\begin{array}{c}\begin{array}{cccc}1& 0& 0& 0\end{array}\\ \begin{array}{cccc}0& 0& 1& 0\end{array}\end{array}\right]$ and the initial condition is $x(0)={\left[\begin{array}{cccc}0& 0& 0& 0\end{array}\right]}^T$.

First, OKID is applied to convert the nonlinear system (Equation (23)) to an equivalent linear system. The system (Equation (23)) is injected with white noise $u\left(t\right)=\left[\begin{array}{cc}{u}_1\left(t\right)& u_2\left(t\right)\end{array}\right]$ with a zero mean and covariance $diag\left(\mathrm{cov}\left(u_{1,2}(t)\right)=\left[\begin{array}{cc}0.2& 0.2\end{array}\right]\right)$ at the sampling time $T=0.01$ s. Figure 15 shows that the error between the output of the identified equivalent linear system and the original nonlinear system (Equation (23)) can be controlled to within ${10}^{-6}~{10}^{-5}$.

We then consider two different reference inputs $r\left(t\right)={\left[\begin{array}{cc}{r}_1& r_2\end{array}\right]}^T$ for the identified equivalent linear system at the sampling time $T=0.01 \mathrm{s}$ with an input delay ${\tau }_i=0.5\times T$ and an output delay ${\tau }_o=0.3\times T$ for the optimal DR method presented in Section 4. To test whether the designed control can effectively suppress a state delay, we gradually increase the state delay ${\tau }_s=0\times T$ for two different reference inputs, i.e., Types 1 and 2. Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, Figure 21, Figure 22 and Figure 23 show the ability of the control to suppress the Types 1 and 2 state delays.

Table 1. The ability of the robot manipulator to suppress state delays.

Reference Inputs	Delay Parameters	Max. Output Error (Rad)	Tolerable Delay Time of Manipulator
Type 1	${\tau }_i=0.5\times T$$, {\tau }_o=0.3\times T$$, {\tau }_s=0.13\times T$	$6.48\times {10}^2$	$<2.6 \mathrm{s}$
Type 2	${\tau }_i=0.5\times T$$, {\tau }_o=0.3\times T$$, {\tau }_s=0.13\times T$	$9.886$	$<2.6 \mathrm{s}$

summarises the results in Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, Figure 21, Figure 22 and Figure 23, showing that the robot manipulator suppresses the state delay for approximately 2.6 s.

TYPE 1:

Case 1: ${\tau }_i=0.5\times T$, ${\tau }_o=0.3\times T$, ${\tau }_s=0\times T$

Figure 16. TYPE 1. Case 1: for the ability to suppress state delay.

Figure 16. TYPE 1. Case 1: for the ability to suppress state delay.

Case 2: ${\tau }_i=0.5\times T$, ${\tau }_o=0.3\times T$, ${\tau }_s=0.1\times T$

Figure 17. TYPE 1. Case 2: for the ability to suppress state delay.

Figure 17. TYPE 1. Case 2: for the ability to suppress state delay.

Case 3: ${\tau }_i=0.5\times T$, ${\tau }_o=0.3\times T$, ${\tau }_s=0.12\times T$

Figure 18. TYPE 1. Case 3: for the ability to suppress state delay.

Figure 18. TYPE 1. Case 3: for the ability to suppress state delay.

Case 4: ${\tau }_i=0.5\times T$, ${\tau }_o=0.3\times T$, ${\tau }_s=0.13\times T$

Figure 19. TYPE 1. Case 4: for the ability to suppress state delay.

Figure 19. TYPE 1. Case 4: for the ability to suppress state delay.

TYPE 2:

Case 1: ${\tau }_i=0.5\times T$, ${\tau }_o=0.3\times T$, ${\tau }_s=0\times T$

Figure 20. TYPE 2. Case 1: for the ability to suppress state delay.

Figure 20. TYPE 2. Case 1: for the ability to suppress state delay.

Case 2: ${\tau }_i=0.5\times T$, ${\tau }_o=0.3\times T$, ${\tau }_s=0.1\times T$

Figure 21. TYPE 2. Case 2: for the ability to suppress state delay.

Figure 21. TYPE 2. Case 2: for the ability to suppress state delay.

Case 3: ${\tau }_i=0.5\times T$, ${\tau }_o=0.3\times T$, ${\tau }_s=0.12\times T$

Figure 22. TYPE 2. Case 3: for the ability to suppress state delay.

Figure 22. TYPE 2. Case 3: for the ability to suppress state delay.

Case 4: ${\tau }_i=0.5\times T$, ${\tau }_o=0.3\times T$, ${\tau }_s=0.13\times T$

Figure 23. TYPE 2. Case 4: for the ability to suppress state delay.

Figure 23. TYPE 2. Case 4: for the ability to suppress state delay.

5.2. AI Training Results

The next section shows the training results of the neural network. There are 325 training data points for the three subjects in the training and (bottle: 180; bowl and ball: 50). Noisy point cloud data are used for training to give a total of 595 training samples. There are 45 data points in a test set (bottle: 25; bowl and ball: 10). During the training period, the predicted output (joint angles or TCP) was standardization. The standardization range (

Table 2. Standardization range.

Range	Joint 1	Joint 2	Joint 3	Joint 4	Joint 5	Joint 6
Max (1)	−160°	−80°	140°	−10°	120°	0°
Min (0)	−220°	−120°	80°	−40°	40°	−50°
Range	X	Y	Z	α (Rx)	β (Ry)	γ (Rz)
Max (1)	1.5 m	1.5 m	1.5 m	1.5 m	1.5 m	1.5 m
Min (0)	−1.5 m	−1.5 m	−1.5 m	−1.5 m	−1.5 m	−1.5 m

) is set according to the working range of the arm.

Three models were trained for these three types of objects. The output for these models in

Table 3. AI Model training results.

Model	Bowl Loss	Bowl Val-Loss	Ball Loss	Ball Val-Loss	Time Consumed
CNN	2.2469 × 10−5	0.0011	1.2599 × 10−5	1.7467 × 10−4	17 ms/epoch
Min-Pnet	1.4181 × 10−4	0.0011	5.3420 × 10−5	3.6437 × 10−5	215 ms/epoch
OBB	2.8651 × 10−4	0.0012	1.2412 × 10−4	2.4541 × 10−4	25 ms/epoch

above is the UR5 arm joint angle that qbSoftHand uses to grasp the object. These angles are input into the UR5 control system and the arm is moved using the MoveJ command.

5.3. Real Machine Grasping

For the grasping test, the camera observes objects in the work space at an angle of about 45 degrees to the desktop (Figure 24).

5.3.1. Five-Finger and Two-Finger Claw Gripping Experiment

The experiment first tested the grip comparison between the two-finger and five-finger grippers (Figure 25). A network was trained using oriented bounding box (OBB) framework data [41] and a point cloud model using a two-finger gripper as a control group. Simultaneously, the results were compared with those of a study similar to [25]. The results are displayed in

Table 4. Bowl and ball gripping test results.

Object and Model	Grasping Success Rate
5-Fin CNN Bowl	90.0%
5-Fin CNN Ball	80.0%
5-finger CNN Bottle	80.0%
2-Fin CNN Bowl	58.82%
2-Fin CNN Ball	52.94%
2-finger CNN Bottle	64.15%
Indoor [25]	85%
Outdoor [25]	80%

.

According to the above test results, it can be observed that compared to the two fingers, the five-finger gripper achieved a success rate of more than 80% in the grip of various objects. The performance of the two-finger gripper (Figure 26), although it had a success rate of more than half, was much worse than that of the five-finger gripper. In fact, in most of the failed grips with the two-finger, the predicted position was very close to the object, but the object could not be grasped because the gripping area of the gripper was not wide enough and the force applied at the contact point. This demonstrates that even the five-finger grippers, which are only capable of open grip action, show a significant advantage over the two-finger on the hardware level. Reference [25] used a three-finger gripper and achieved at least an 80% success rate in both indoor and outdoor tests as in this study. Considering the larger number of samples they collected (570 in total), the similar success rate achieved in our study may be attributed to the adaptability of the five-finger gripper, in addition to the different experimental environment (clutter vs. single).

5.3.2. Comparison of CNN and Min-Pnet Networks

The next section compares CNN networks, Min-Pnet networks designed with reference to the PointNet concept, and networks using OBB. Since the advantages of five fingers in hardware have been demonstrated in the previous section, it is shown that they can effectively compensate for the shortcomings of neural networks. In order to compare the differences in network architectures more clearly, the hardware was compared using a two-finger gripper. The results are shown in

Table 5. Test result of each network.

Object and Model	Grasping Success Rate
2-Fin Min-Pnet Bowl	82.60%
2-Fin Min-Pnet Ball	73.07%
2-finger Min-Pnet Bottle	69.01%
2-Fin CNN Bowl	58.82%
2-Fin CNN Ball	52.94%
2-finger CNN Bottle	64.15%
2-Fin OBB Bowl	60.00%
2-Fin OBB Ball	33.34%
2-Fin OBB Bottle	53.33%

.

Compared to the CNN network, the results of Min-Pnet were significantly better in all aspects. Although the difference in the results on the bottles was not large (5%), the bowl and sports ball performed significantly better (24% and 20%), which may be due to the hardware limitation mentioned in the previous section, while for the OBB, except for the bowls with a larger successful area, the results were not very good. In the OBB experiment, it was found that objects that were too close or too far away were not successful, probably due to the angle of view and distance, so that the OBB frame point did not fully represent the size and position of the object.

5.3.3. Out-of-Training Set Object Gripping

Then, we tested whether our network could produce a corresponding grip on an unknown object (see Figure 27) with a similar training set. The objects we tested were: an alcohol spray bottle (considered as bottle), a sock wrapped in a ball, and a small doll (considered as sport ball). The network included CNN, Min-Pnet and OBB, and the hardware was the same as in the previous section. The results are shown in

Table 6. Out-of-training set object grabbing test.

Object and Model	Grasping Success Rate
2-Fin Min-Pnet Ballobj	75.00%
2-Fin CNN Ballobj	50.00%
2-Fin OBB Ballobj	25.00%
2-Fin Min-Pnet Bottleobj	73.33%
2-Fin CNN Bottleobj	60.00%
2-Fin OBB Bottleobj	53.33%

.

Although the number of experiments is small, the success rate of Min-Pnet proves that the feature transformation architecture can indeed analyze point clouds more effectively, demonstrating its adaptability by still being able to predict effective grip positions when faced with objects that are similar but not identical to the training data. This also demonstrates the drawback of OBB’s difficulty in displaying asymmetric object features.

6. Conclusions

Simulation results in Section 5.1 are presented demonstrating effective optimal digital re-design control of a robot manipulator with a nonlinear delay. In particular, the proposed method can effectively control a state delay within the tolerable scope.

In terms of gripper hardware comparison, although the qbSoftHand used in this study cannot grasp objects more flexibly due to hardware limitations, it can still grasp many objects that are difficult to grasp with the two-finger gripper, such as smooth bowls, balls, and bottles. A comparison with the two-finger gripper and the three-finger gripper of [25] demonstrates the advantage of the five-finger gripper on the hard surface.

In the case of objects outside the training set, the Min-Pnet achieved a success rate of more than 70% despite the small number of experiments, and after analysis, it was found that most of the gripping failures were due to hardware limitations. When testing the limits of the network, it is also proved that the features and poses of the objects cannot be fully expressed by using only the OBB framework.

Due to hardware and time constraints, it is not possible to conduct more complex experiments such as identifying and grasping multiple objects in a chaotic environment or testing the grasping of more objects in this paper. However, our research has partially demonstrated the advantages of the five-finger gripper and provided a simple and effective grasping system to automate the task of grasping objects.

There are many future research directions, such as using wireless communication to control the arm. The original data collection part can be accomplished in a virtual environment, or using 5G combined with AR for arm training, etc.