A Fast 6DOF Visual Selective Grasping System Using Point Clouds

Abstract

Visual object grasping can be complex when dealing with different shapes, points of view, and environments since the robotic manipulator must estimate the most feasible place to grasp. This work proposes a new selective grasping system using only point clouds of objects. For the selection of the object of interest, a deep learning network for object classification is proposed, named Point Encoder Convolution (PEC). The network is trained with a dataset obtained in a realistic simulator and uses an autoencoder with 1D convolution. The developed grasping algorithm used in the system uses geometry primitives and lateral curvatures to estimate the best region to grasp without previously knowing the object’s point cloud. Experimental results show a success ratio of 94% for a dataset with five classes, and the proposed visual selective grasping system can be executed in around 0.004 s, suitable for tasks that require a low execution time or use low-cost hardware.

1. Introduction

Today, flexible and scalable robotic systems are a key component, considering the new trend associated with Industry 4.0, and, more specifically, the new mass-customization production concept [1]. The evolution of visual perception with RGB-D sensors and computational power has transformed autonomous and semi-autonomous tasks in the industry. Tasks involving robot navigation [2], manufacturing operations [3], and grasping [4] have become more accessible and efficient over the years. From this perspective, machine learning technologies should be used to improve grasping systems for object recognition, localization, and handling [5].

Object detection networks, such as [6,7], can detect objects in the environment in robotic applications but lack depth, reducing their efficiency in tasks where scale or more precise ambient perception are needed. To improve this, studies on point clouds have been used in semantic segmentation tasks such as [8,9,10].

In robotic manipulators, point cloud has been used to estimate the best region to grasp without prior knowledge of the object [11]. The systems presented by [12,13] use deep learning networks to estimate the best position to grasp, with some limitations, such as planar grasping, where the gripper is always perpendicular to the table or needs a heuristic to select one of the possible grasps returned by the network. Neural networks are commonly applied to grasp objects using partial point clouds observation [14], pose refinements [15,16], and grasping for multiple robotic manipulators [17]. Recent grasping methods are evolving towards 6D pose estimation, such as [18], where a method and dataset use RGB or RGB-D to segment, detect, and estimate an object’s 6D pose. The segmentation and point cloud generation of objects in RGB and RGB-D [19], and an improvement in the 6D estimation of moving objects can be seen in [20,21], respectively.

This paper proposes a new selective grasping system in 6D using only point clouds from RGB-D sensors. This paper has four main contributions: (i) validation of a new grasping algorithm based on lateral curvatures and geometric primitives in a real scenario, which is an improvement of the work proposed by [11,22]; (ii) a new deep learning network for object classification, named Point Encoder Convolution (PEC), using point clouds enabling selective grasping in an environment with several objects; (iii) a new point cloud classification dataset obtained in a simulator, named LARS Classification Dataset, which facilitates the manipulation of object classes in different poses; (iv) extensive validation of the complete grasping system using a UR5 robotic manipulator, a Robotiq 2F140 gripper, and an Intel Realsense D435 RGB-D sensor.

The remainder of this document is organized as follows. Section 2 presents the proposed grasping system overview, showing each step of the grasping process. Section 3 proposes a new deep learning network. In Section 4, the 6D grasping method is presented. Section 5 offers experimental results. Finally, the conclusions are drawn in Section 6.

2. Grasping System

The grasping system overview proposed in this paper can be seen in Figure 1, where the stages of the system are as follows:

• Filtering stage:

  1.

  Input point cloud: the initial point cloud $p_i$ returned by the RGB-D sensor.

  2.

  Gripper and table removal: the point cloud after being filtered by removing the gripper and the table, having only the objects over the table. The filtering process works as follows:

  (a)

  Removing the gripper by limiting the z-axis (depth axis) between $[z_{min},z_{max}]$ meters, where $z_{min}$ and $z_{max}$ are the parameters of the system setup. The gripper and objects far away from the camera are removed, for example, the floor.

  (b)

  Removing the table using plane segmentation and an Outliner filter.

  3.

  Clustering to separate the objects in the point cloud. Returns to the point cloud $p_j$ of each object, with $j=1,\dots ,n$, where n is the number of objects.

• Classification stage:

  4.

  Preprocess the data by normalization, downsampling using farthest point sampling (FPS), and grouping the points. Returns the point cloud $p_f{}_j$, with $j=1,\dots ,n$, where n is the number of objects.

  5.

  The objects’ point cloud is classified using the proposed classification network PEC. Returns objects class $c_j$, with $j=1,\dots ,n$, where n is the number of objects.

  6.

  Select the object with the class c desired to be grasped and return its point cloud p to the grasping algorithm. From this step on, selective grasping is possible, as only the point cloud of the desired object will be considered for the grasping stage.

• Grasping stage:

  7.

  Receive the object’s point cloud p and estimate its 6D pose using principal component analysis (PCA).

  8.

  Estimate the grasping region of the selected object in the camera’s field of view. The proposed algorithm uses lateral curvatures and geometric primitives to estimate the grasping region.

  9.

  Returns the 6D pose to grasp the desired object.

3. Point Encoder Convolution

In this subsection, a new deep learning network called Point Encoder Convolution (PEC) will be described, which was developed to classify objects using their respective point clouds as perceptual information.

The problem definition can be described as follows. Considering a group of point clouds $p_j\subset {\mathbb{R}}^3$ as an input, with $j=1,\dots ,n$, where n is the number of objects, containing unknown segmented objects, the PEC predicts the object classes $c_j$. With those predictions, it will return the point cloud p with the desired class c.

3.1. Data Preprocessing

Before applying the point cloud p to the classification network, the data are preprocessed to improve the learning process and data generalization. First, the object centroid is defined, where $p_c$ is the point cloud centroid, N is the number of points of the point cloud, and $p_i$ is a point from the point cloud p, with $i=1,\dots ,N$:

$$p_c=\frac{{\sum }_{i=1}^N{p}_i}{N},p_i\in p,$$

(1)

and $p_f$ is the translated point cloud to the system origin

$$p_f=p-p_c.$$

(2)

This way, the data are normalized around the origin, avoiding data overfitting during the training step. After that, the point cloud $p_f$ is sub-sampled using FPS [23], as recommended by [8], to ensure that the data will have the same size as the network input size. Finally, a pseudo-organized point cloud is created by grouping the points following these steps:

• Select a number of points k, where this number has to be divisible by the total number of points in the point cloud $p_f$.
• Use k-d tree [24] to find k points near the target point.
• Group those points and remove them from the original point cloud.
• Repeat this process until all points are grouped.
• When all points are grouped, obtain each group’s centroid by using Equation (1), and sort them by their distance to the origin, as proposed by [8].
• Merge all the groups back in a point cloud $p_f$ with its points reorganized.

3.2. Network Architecture

The chosen architecture features an autoencoder using 1D convolutions, 1D transpose convolutions, and multiLayer perceptron. The convolution layers use batch normalization to help stabilize the training and make it more consistent. Figure 2 shows the complete PEC architecture, considering an input size of 64 points and a total of 5 classes for a total of 890 K parameters. Such parameterization is part of the convolution layers (input size, output size, kernel size, stride, padding), linear layers (input size, output size), and the transposed convolution layers (input size, output size, kernel size, stride). For the loss, the cross-entropy loss was used due to better generalization when using objects that are not the same as the ones used in training. A stochastic optimization method, Adam [25], was chosen due to having better and faster training compared to other optimization methods. The network output will be a class c and its confidence score, where the confidence score is obtained by applying the softmax function [26] to the output of the network. The hyperparameters for the network were obtained in an empirical way.

3.3. Lars Classification Dataset

To validate the network PEC, a new dataset is proposed, which is one of the contributions of this paper. The simulator Isaac Sim [27] was used to help with this task since it is easy to manipulate objects through scripting and features good graphical fidelity. The graphical settings used were the default ones as of the 2022 version. The generated dataset used five classes, each class with two objects. Each object received one thousand samples by generating random poses in ${\mathbb{R}}^3$ space, with a range between [−2 m, 2 m] and with a range for orientation between [$-\pi$, $\pi$], where on each pose, a simulated RGB-D sensor recorded the point cloud. Figure 3 represents the simulator interface and the generation process.

Figure 4 shows the objects and classes used on the dataset. One of the challenges of this approach is that the data generated in the simulation are ideal; that is, these data do not show the most defects of a real point cloud. To solve this and help with its generalization in real experiments, each data sample had several random points removed from a region, where the region was chosen randomly. The number of points removed varied between 0% and 40% of the total points. This did not affect the accuracy of the training, which can either be related to the environment or be caused by filters and generalization. Figure 5 shows a classification process example, where the network receives the object already segmented. The segmentation of the objects was performed by removing the table using RANSAC to detect the plane [28]. Some defects on the point cloud are noticeable, such as missing spots, and the segmentation process causes others. In some cases where we have multiple of the same object, they can have different confidence rates. This depends on similarities to the objects used in training; on mice, for example, by trying on different models, the confidence rate could vary between 70% and 95%.

4. Lateral Curvatures and Geometric Primitives

This section proposes a new algorithm for 6D pose estimation using lateral curvatures and geometric primitives of point clouds. The new algorithm is based on the works of [11,22] and can work independently of PEC to grasp unknown objects, with an improvement in two significant points: (i) by breaking the object into smaller and more accessible regions to perform the grasp; this way, we can analyze the entire object in less complex regions, improving the algorithm proposed in [22], where it focuses its analysis near the center of mass of the object. This also gives the ability to know which region does not fit the gripper and discard it to avoid unnecessary computation time on its analysis. (ii) improving the geometric primitive estimation performed in [11], where the author tries to reduce even complex objects to a single primitive. By breaking the object into multiple regions, we can check if the object can be reduced to a geometric primitive.

4.1. Problem Statement

The grasping algorithm will receive a point cloud $p\subset {\mathbb{R}}^3$ as an input, where it only contains the desired object to be grasped. Next, we consider a two-finger gripper with gripper stroke $W_g$ and finger width $H_g$, see Figure 6. The proposed algorithm will return a 6D pose $O\subset {\mathbb{R}}^6$ to grasp the object. This pose will have its position aimed at the center of the selected region.

The proposed grasping algorithm will work as follows:

• Use PCA to establish the object’s orientation in relation to the camera, as suggested in [11,22].
• Use the object orientation to align it to the origin axis.
• Generate grasping regions based on the $H_g$ value.
• Obtain the curvature of each region and check if they have the same geometry, as seen in [11].
• If the object can be reduced to a primitive by checking if each region has the same geometric primitive and width, grasp the object by its centroid.
• If it is not, obtain the lateral curvature of each region.
• Return the pose of the region of the smaller curvature that fits inside the gripper.

4.2. Region Generation

Regions K are defined using the object height $H_o$ and the gripper dimensions:

$$K=\frac{H_o}{H_g}$$

(3)

If $H_o\le H_g$, the object is considered too small and grasping will be performed at its center. Otherwise, the 6D grasping algorithm proposed in [29] is applied. Since some objects have regions that can be separated into multiple regions, for each region found, a clustering algorithm is applied to separate them when applicable. This way, a complex object can be broken into smaller and more accessible regions, such as the joypad seen in Figure 7.

4.3. Curvature Calculation

The point cloud surface curvature is obtained using the method proposed by [11,22]. First, the centroid of each region is calculated by using the equation below, where $N_i$ is the number of points in the region, with $i=1,\dots ,N_i$, $p_{oi}$ is a point of the point cloud $p_i$, and $p_c$ is the centroid of the region:

$$p_c=\frac{{\sum }_{i=1}^{N_i}{p}_{oi}}{N_i},p_{oi}\in p_i.$$

(4)

Then, the covariance matrix is obtained:

$$C=\frac{1}{N_i}\sum _{i=1}^{N_i}(p_{oi}-p_c)·{(p_{oi}-p_c)}^T,$$

(5)

and the eigenvalues $\lambda$ and eigenvector for three dimensions $\overrightarrow{v}$:

$$C·{\overrightarrow{v}}_t={\lambda }_t·{\overrightarrow{v}}_t,t\in \{1,2,3\},$$

(6)

and by using PCA, the curvature ${\sigma }_{j_{*}}$ is

$${\sigma }_{j_{*}}=\frac{{\lambda }_0}{{\lambda }_0+{\lambda }_1+{\lambda }_2}$$

(7)

4.4. Geometric Primitive Reduction

The geometric primitives, proposed by [11], are estimated through the curvature around each point of the region using the Equations (7) and (8):

$$\mathrm{\Delta }=\frac{M^{*}}{M}.$$

(8)

where $M^{*}$ is the number of points $j_{*}$∈$j_i$ where ${\sigma }_{j_{*}}$≤ 0.01 e M is the total number of points from the point cloud. This way, the region geometric primitive can be defined as seen in

Table 1. Geometric primitive classification.

Sphere	0.0 ≤ $\mathrm{\Delta }$ ≤ 0.10
Cylinder	0.10 <$\mathrm{\Delta }$ ≤ 0.40
Box	0.40 <$\mathrm{\Delta }$ ≤ 1.0

:

4.5. Region Selection

The best region to grasp is defined considering whether each region has the same geometric primitive and a similar length, inside a margin of error, due to noises and measurement errors in the point cloud. If they have this, consider the entire object a geometric primitive and grasp it on its centroid. The region length $L_i$ is defined by

$$L_i=x_{ma{x}_{c_i}}-x_{mi{n}_{c_i}},$$

(9)

where $x_{ma{x}_{c_i}}$ and $x_{mi{n}_{c_i}}$ are the maximum and minimum values of each generated region on the x axis. The x-axis represents the horizontal axis, $c_i$ represents the region, with $i=0,1,\dots ,N_k$, and $N_k$ the number of regions.

If this condition fails, the curvature around $x_{max{c}_i}$ and $x_{min{c}_i}$ is calculated, and a score called $P_i$ is defined as

$$P_i=curvma{x}_i+curvmi{n}_i,$$

(10)

where $curvma{x}_i$ is the curvature around $x_{max{c}_i}$ and $curvmi{n}_i$ the curvature around $x_{min{c}_i}$.

The best region to grasp is the one where $P_i$ is minimal, and the condition $L\le W_g$ is valid to avoid regions that do not fit inside the gripper. In the last step, PCA is applied to the region as it may have a different orientation to the object, as seen in Figure 8.

The output of the algorithm will be the 6D pose of the region to be grasped $p_r$.

5. Experimental Results

This section presents experimental results for pick and place tasks using an UR5 robotic manipulator, a Robotiq 2F140 gripper, and an Intel Realsense D435 RGB-D sensor. For the development of the system, Python 3.7 was used as a programming language, Open3D [30] for point cloud processing, and Pytorch Lightning [31] for the development of the classification neural network. The results were divided into two subsections: Object Classification, where it will show the results of the PEC network in classification tasks, and Grasping Results, where the entire grasping system was tested. Two computational systems were used for development and testing: a computer with a Ryzen 5 3600 and a RTX 3060ti graphics card and a laptop i5-4200M with a Nvidia GT850M graphics card, respectively.

5.1. Object Classification

The PEC classification network was tested with two datasets: ModelNet10 [32] and LARS classification Dataset, presented in Section 3.3. The results for the ModelNet10 can be seen in

Table 2. Table comparing neural networks on ModelNet10.

Network	Accuracy
$PE{C}_{group4}$	78.8%
$PE{C}_{group8}$	88.74%
$PE{C}_{group16}$	89.5%
$PE{C}_{group32}$	69.8%
$PE{C}_{nogroup}$	79.4%
$PE{C}_{{128}_{group32}}$	92.24%
$PE{C}_{{256}_{group64}}$	82.9%
3DShapeNets [32]	83.5%
OrthographicNet [33]	88.56%
VSL [34]	91.0%
LonchaNet [35]	94.37%
SPNet [36]	97.25%
RotationN [37]	98.46%
VRN Ensemble [38]	97.14%
Panorama-ENN [39]	96.85%
SO-Net [40]	95.7%
CurveNet [41]	96.3%
Voxelized Point Clouds [42]	93.4%
Shape Self-Correction [43]	95.5%

, where the default size of the entry point cloud is 64 points, except for $PE{C}_{128}$ and $PE{C}_{256}$ that used input sizes of 128 points and 256 points, respectively. In order to test the network with different groupings of the sensor point cloud, groupings of 4, 8, 16, 32, and 64 points were formatted.

In Table 2, the results comparing the PEC network to other classification networks can be seen. It can be verified that the group size influences the accuracy of the network and that more points in the input do not translate to better results. The PEC network was trained for this dataset with a learning rate of $0.001$ for 150 epochs.

The accuracy of the network on the LARS classification dataset can be seen in

Table 3. PEC accuracy on LARS Classification Dataset.

Network	Accuracy
$PE{C}_{group8}$	89.2%
$PE{C}_{group16}$	92.5%
$PE{C}_{group32}$	89.1%
$PE{C}_{nogroup}$	67%
$PE{C}_{{128}_{group32}}$	91%
$PE{C}_{{256}_{group64}}$	91.5%

, where the best result was achieved with an input size of 64 points and a group size of 16 points. The dataset was split into 75% for training and 25% used in validation, using a learning rate of $0.01$ and trained for 150 epochs. Figure 9 shows the validation accuracy and loss in the training stage. The validation loss starts high in the training step due to batch normalization. However, over time, it normalizes and results in a faster and more stable training process.

For comparison purposes, using a RTX 3060ti graphics card, the PEC network with the LARS dataset can achieve prediction in around 0.002 s and takes 152 s in the training step with an input size of 64 points. With a GT850M graphics card, it takes around 0.009 s for prediction and 250 s for training. The network training step is only performed once for a set of object classes, with low training time, and low execution time, and it is suggested that it can be used on low-cost hardware and retrained as needed in the case of changing the objects to be classified.

5.2. Grasping Results

The objects were selected based on their challenges to grasp, such as a joypad (which has a complex geometry), a plug adapter (with a simple geometry that can only stand on the table with a specific orientation), pliers (also with a complex geometry), a wire cutter (hard to grasp due to its low thickness and complex handle), and, finally, a cylinder with a simple geometry. Figure 10 shows an example of the sequence of tasks to perform a grasp. This section was divided into two experiments: Grasping Algorithm, where the proposed grasping algorithm will be validated; and Selective Grasping, where the whole system will be validated.

5.2.1. Grasping Algorithm

Table 4. Table comparing the proposed grasping system vs. a Generic Grasping Algorithm (GGA).

Object	Proposed	GGA
Joypad	90%	40%
Cylinder	95%	85%
Plug Adapter	100%	100%
Pliers	100%	65%
Wire cutter	95%	70%
Wire cutter (Open)	85%	0%

shows the results of grasping an object twenty times in different poses to better show the algorithm’s efficiency. The point cloud of the objects and their respective grasping regions can be seen in Figure 11, showing the performance of the grasping stage with objects of different complexities:

• Joypad is a complex object where the gripper has to grasp it close to its center due to better stability.
• In the case of the pliers, both the clusters and the original region are added as possible grasp since some handles may not be ideal to be grasped individually. The grasping algorithm chose to grasp both handles instead of individually.
• The plug adapter is a rectangular object that uses the wall plug to stay up, causing it to have an orientation about the table. Since it possesses a rectangular geometry, the gripper will grasp it by its center.
• The wire cutter has predominantly planar geometry and lacks depth. Therefore, the plane segmentation will consider only the handles.
• The cylinder is another object that possesses a simple geometry so that it will be grasped by its center.

To benchmark the proposed grasping system, a Generic Grasping Algorithm (GGA) is used for comparison. It grasps the object by its center and keeps the gripper perpendicular to the table. For simple objects, such as the cylinder and the plug adapter, the difference in success rate is minimal. The grasp will depend more on other factors, such as correct pose estimation and object dimensions. A more significant gap between the proposed grasping algorithm and the GGA is noticed on more complex objects, as seen in Table 4. It is essential to emphasize the computational cost of the proposed system that takes an average of $0.0018$ s to generate a grasping for the objects seen in these experiments on a CPU Ryzen 5 3600.

5.2.2. Selective Grasping

Selective grasping allows the user to choose an object of interest in a given environment with other objects. Figure 12 and Figure 13 show the system grasping the joypad and the staples, respectively. It is important to point out that the network was tested with objects similar to those used in the simulation training stage. In the experimental results, it was noted that the quality of the point cloud of the objects is essential for a good performance of the system. The point clouds obtained by the sensors have noises and defective regions, reducing the confidence score of the classification, which does not happen in the simulated point clouds. Even so, the system proved to be robust enough for a diverse set of objects. A video with further experiments can be seen in the Supplementary Materials.

6. Conclusions

This article proposes a new visual selective grasping system using only point clouds of objects, and it can be executed in around 0.004 s (0.002 s for the classification of objects plus 0.0018 s for pose generation for feasible graspings). The proposed system was validated using an UR5 robotic manipulator, a Robotiq 2F-140 gripper, and an Intel Realsense D435. The grasping method analyzes the point cloud of the object and generates a region to grasp based on a single view. For this, the grasping algorithm breaks a complex object into smaller and easier-to-analyze regions, and uses curvatures and geometric primitives to estimate the best region to grasp.

A new deep learning network called Point Encoder Convolution (PEC) that uses 1D convolution on a segmented object’s point cloud was introduced to provide the ability to select objects of interest, as well as a new point cloud classification dataset obtained in a simulator. The experimental results showed an average success rate of 94% in objects with different geometries, with a feasible individual grasping generation time of around $0.0018$ s.

The PEC neural network proved to be simple and effective in selective grasping. Using the LARS dataset, it achieved an accuracy of 92.5% and an object classification time of 0.002 s. It is essential to point out that simulated objects were used in the network training stage similar to the real ones, and, even so, the PEC obtained a correct classification with good precision. Using a public dataset, ModelNet10, it obtained an accuracy of 92.24%, proving itself to be generalist and applicable in several applications.

The system is highly dependent on the point cloud quality since it works with a single view. Because of its simplicity, it does not scale well to more complex classification tasks where many classes are used. Nevertheless, on the other hand, it presents low processing times for a small class of objects, feasible for robotic tasks with high repeatability rates. Improvements using multiple views, noise-related problems, defective points cloud, and occlusion can be addressed for future works.