Fast Location of Table Grapes Picking Point Based on Infrared Tube

Abstract

This study investigates the low mechanization level of grape picking, and the problems associated with the difficult location of grape picking points in three-dimensional space. A method for rapidly locating the picking points of table grapes based on an infrared tube was proposed in this paper. Firstly, the Otsu algorithm and the maximum connected domain were used to obtain the image of the target grape, which realized the fast recognition and segmentation of the target grape in two-dimensional space. Secondly, a location device for grape-picking points based on an infrared tube was designed, which resolved the technical problem related to the difficulty of locating picking points in three-dimensional space, and realized the accurate positioning of picking points for table grapes. Finally, the experimental results show that the proposed method can quickly and accurately locate the picking points of table grapes in three-dimensional space. The average running time of the proposed algorithm is 0.61 s, and the success rate of location identification is 90.0%. It provides a feasible scheme for the mechanized picking of table grapes.

1. Introduction

As the world’s largest fruit and vegetable output and consumption country, the picking of fruits and vegetables is mainly artificial in China. With the development of intelligent and machine vision, machine picking to replace manual picking has become a trend [1,2,3,4]. Additionally, accurately locating picking points plays an important role in mechanical picking. In addition, China is one of the countries with the largest grape outputs in the world. Rapidly and accurately locating grape-picking points is the key to realizing automatic mechanical picking of table grapes, which is also of great significance to ensure picking quality, improve competitiveness and increase farmers’ income [5,6,7,8].

Scarfe et al. used the fixed-threshold method to remove the background and used template matching to identify kiwifruit [9]. However, the fixed threshold could not change with the light. Peng Hongxing et al. successfully isolated citrus fruits by using I2 components in I1, I2, and I3 space [10]. In the above study, only the stone fruit (e.g., peach, apricot) and the kernel fruit (e.g., apple, pear) in two-dimensional space were recognized and segmented. However, the shape of the berry fruits (e.g., grapes, cherry tomatoes) varies. So, the picking location method of the stone fruit and the kernel fruit is not suitable for berries.

In recent years, Liu Ping et al. used a K-means clustering algorithm and Chan-Vese model to study the overlapping grapes in two-dimensional space [11]. Luo Lufeng et al. completed the identification and segmentation of overlapping grapes based on a contour analysis [12]. The above studies only completed the identification and segmentation of grapes but did not study the location of grape-picking points. In addition, Xiong Juntao et al. successfully recognized the stem of grapes by OTSU threshold, and located picking points through Hough straight line fitting [13]. However, the localized accuracy of grapes was low. Furthermore, Lei Wangxiong et al. detected and located grape-picking points based on watershed and angle-constraint methods [14]. The success rate of the method was 89.2%, and the average time of position was 0.65 s. In addition, deep learning has been applied more and more widely in fruit and vegetable recognition and segmentation. Ning Zhengtong et al. achieved the detection of grape picking points by Mask R-CNN [15]. Liang et al. realized fruit-stem detection of litchi at night based on improved YOLOv3 [16]. However, these studies, based on deep learning, required the collection of a large number of images. In addition, we must annotate them one by one. This made for a lot of work. Moreover, the sample training has high requirements for the equipment. In conclusion, both classical methods and deep-learning methods are carried out on one side of berry fruits in two-dimensional space. However, due to the variation in the appearance of berry fruits, it is difficult to replace the whole berry fruits in three-dimensional space using only a side image of berry fruits in two-dimensional space. As a result, the success rate of picking location is reduced.

In view of the above problems, many scholars conducted relevant studies. Luo Lufeng et al. proposed a method of grapes based on binocular stereo vision [17]. To accurately locate tomatoes, Zhejiang University used the stereo-detection system built by two CCD color cameras [18]. Recently, Cao Jingjun built an image acquisition system based on a depth camera, which completed the precise positioning of Agaricus Bisporus in three-dimensions space [19]. However, the depth camera is affected by external interference, which has a great impact on the positioning accuracy. Besides, the price of a depth camera is higher than that of an ordinary camera, which increases the cost. Therefore, it is particularly important to study an efficient and fast location method for grape picking points in three-dimensional space.

However, all of the above papers only studied the picking point positioning system or positioning method. They pay less attention to the end-effector. By the end of the 20th century, Japan designed and developed a tomato-picking robot [20]. It used a color camera to extract and identify tomatoes and absorbed the fruit by suction. Furthermore, it cut the stem by the manipulator. The robot had a harvesting efficiency of 15 s^–1, with a success rate of about 70%. In 2019, Duan Hongyan et al. designed an end-effector that adapted to picking string tomatoes [21]. The next year, Lu Jie et al. also developed a tomato-picking end actuator with the ability to clamp and cut [22]. Recently, Wei Bo et al. designed an underactuated end-effector using the combined control of three-finger grip and deflection, with a success rate of 98.3% [23]. After that, Zhang Xiaowei et al. analyzed the process of the person picking safflower and proposed a three-finger pull-type safflower end-effector with a recovery picking rate of 92.71% [24]. However, all the above studies are designed for the end-effector through physical characteristics or software simulation. However, most of the existing end-effectors are selected after the controller completes the picking point positioning. The process, which uses step-by-step working, leads to low harvesting efficiency. Therefore, it is particularly important to design an end-effector that combines picking point positioning and harvesting.

So, this paper proposed a fast and accurate location method of the table grape picking point based on the infrared tubes. This method provides a feasible scheme for the fast and nondestructive picking of table grapes.

2. Materials and Methods

2.1. Image Collection

The planar image of table grapes was collected in Mountain Jinniu of the Shandong Fruit Research Institute. The time was 09:00–12:00, 15th July 2019. A total of 203 images were taken. The grape variety was Summer Black. The rear camera of the smartphone was positioned parallel to the ground to complete the shooting. Since the captured image was about 12 megapixels (3024 × 4032), the bilinear interpolation method was used to scale the image. Finally, the image was adjusted to 1.08 megapixels (900 × 1200). The partial grape images are shown in Figure 1.

2.2. Recognition and Segmentation of Table Grapes in Two-Dimensional Space

In the natural environment, light is the most important factor that affects the recognition and segmentation of table grapes. To reduce the influence of light on the recognition and segmentation of table grapes, this paper proposes an Otsu method based on the S component (HSI color space, as shown in Figure 2) to identify and segment multiple clusters of grapes [25]. The method, based on the maximum connected domain as the criterion of optimal target grape, can rapidly and accurately identify and segment table grapes.

2.3. Image Preprocessing

By using the nonlinear transformation method, which is expressed as Equation (1), the RGB color space is converted into HSI color space.

$$\{\begin{array}{c}H=\{\begin{array}{c}\theta ,B<G\\ 360-\theta ,B>G\end{array}\\ S=1-\frac{3}{R+G+B}\left[\min (R,G,B)\right]\\ I=\frac{R+G+B}{3}\end{array}$$

(1)

where $\theta$ in Equation (1) is

$$\theta ={\cos }^{-1}\left\{\frac{\left[\left(R-G\right)+\left(R-B\right)\right]/2}{{[{(R-G)}^2+\left(R-B\right)\left(G-B\right)]}^{\frac{1}{2}}}\right\}$$

(2)

If the normalization of the RGB color space is completed, the saturation component (S) and intensity component (I) in the HSI color space will be normalized accordingly. Additionally, following this, the initial component H divided by 360 is the normalized component H.

By comparing the H, S, and I component histograms in all images of table grapes, an obvious boundary becomes apparent between grapes and the background in the S component, as shown in Figure 3c. The S component has a better effect on peaks and troughs. Therefore, the S component was used as the basis of the segmentation of table grapes.

2.4. Recognition and Segmentation of Table Grapes Based on Otsu

Because the grapes have a distinct boundary with the background in the S component, the rapid and effective separation between the grapes and background is realized based on Otsu. Firstly, the threshold ($T\left(k\right)=k, 0<k<1$) between the grapes and the background in the image is determined. Then, the threshold of the grapes image is divided into $C_1$ and $C_2$, where $C_1\in \left[0,k\right],C_2\in \left[k+1,1\right]$. Furthermore, the threshold ${\theta }_B^2\left(K\right)$ that maximizes the variance of the two types of pixels $C_1$ and $C_2$ is obtained by Equation (3). Then, the table grapes can be rapidly segmented. The results of recognition and segmentation are shown in Figure 4a.

$${\theta }_B^2\left(k\right)=P_1\left(k\right)\left[m_1\left(k\right)+m_G{]}^2+P_2\left(k\right)\right[m_2\left(k\right)+m_G{]}^2$$

(3)

$$p_1\left(k\right)={\displaystyle \sum }_{i=0}^k{p}_i,p_2\left(k\right)=1-p_1\left(k\right)$$

(4)

where $m_1\left(k\right)$ is the pixel mean values of $C_1$, $m_2\left(k\right)$ is the pixel mean values of $C_2$, $m_G$ is the mean value of total pixel value $n$, $P_1\left(k\right)$ is the probability of $C_1$ and $P_2\left(k\right)$ is the probability of $C_2$.

It can be seen from Figure 4b that there are many small noises in the image, so the mathematical morphology method was adopted to remove noise quickly and effectively. At the same time, to avoid the influence of interfering grapes (grapes other than the optimal target grape), the area threshold method was used to obtain the maximum connected domain within the visual range. So, the target grape can be rapidly and accurately segmented using the maximum area threshold, as shown in Figure 4c. Additionally, the Sobel operator was used to extract the boundary contour of the target grape. The result of recognition is represented as an external rectangle. Meanwhile, the center point of the external rectangle was labeled to obtain the planar coordinate information of the target grape. The results of recognition and labeling are shown in Figure 4d.

Overall, 100 images of table grapes were selected as samples from 203 images. The success rate of segmentation was 87%. Among them, the thicker branches of grapes were similar in color to grapes, which led to the failure of the experiment on nine occasions. The number of failures caused by illumination was three. Another failure was caused by overlapping grapes. Part of the experimental results is shown in Figure 5.

From the experimental results, it is not difficult to analyze that the identification method proposed in this paper can complete the identification of grapes very quickly. Of course, this paper also compares other methods, as shown in

Table 1. Comparison of multiple methods.

Types of Algorithms	Mean Running Time/s	Recognition Success Rate	The Test Results
The method proposed in this paper	0.59 s	87%
chromatic aberration method (1.1 × G − B)	0.60 s	76%
K-means	3.20 s	90%

. Although the chromatic aberration method (1.1 × G − B) can also achieve segmentation for table grapes, 1.1 × G − B is not suitable for all images because each image has its own color characteristics, which leads to a low success rate of grapes recognition. In addition, the K-means clustering algorithm was adopted for grapes recognition, and the overall recognition success rate was 90%. However, due to the iterative operation of the algorithm, the recognition time was relatively high. Therefore, the recognition method proposed in this paper has high comprehensive performance. In future, deep learning may replace this method with the expansion of data, and we will continue to study this problem.

3. Results

3.1. Design of Detecting Device for Table Grape Picking Point

The outer rectangle of the target grape was obtained according to the method of recognition and segmentation proposed in Section 3. Furthermore, the infrared detection device for the picking points of table grapes was designed based on the infrared tube to locate the picking points.

3.2. Selection of Infrared Tube

To meet the requirements of the method proposed in this paper and to meet the growth characteristics of grapes, an infrared detection device was designed to locate grape-picking points. The shape of the grape cluster is conical. The length of the grape cluster is in the range of 160–230 mm and the width of the grape cluster is in the range of 135–160 mm. In addition, the camera is parallel to the ground in the image taken in this paper, so complete information of the cluster’s width can be obtained. Moreover, due to the limited opening and closing space of the manipulator, the volume of the selected sensor should be as small as possible on the premise of completing the detection task. According to these requirements, the QT30CM infrared tube was selected. Its external size is 21 × 11 × 6 mm. The detection distance is 20–300 mm, with a transmission angle of <5° and receiving angle of <10°. The electro-optical conversion efficiency is high and the power consumption is low. Furthermore, both the transmission and receiving angles are small [26]. Therefore, this sensor meets the requirements of grape picking location.

3.3. Structural Design of Picking Point Detection Device

According to the characteristics of grape clusters, six groups of infrared tubes were arranged in pairs to ensure the infrared tubes covered the whole width of clusters. Additionally, all the infrared tubes were fixed on custom acrylic plates. In addition, the detection device was fixed under the manipulator, which can realize the synchronous movement of the detection device and the pickup device (Figure 6).

At the same time, to avoid the mutual influence between adjacent infrared tubes, a geometric analysis was conducted on the receiving tube of the infrared tube. According to Equation (5), the distance between point E and point F was calculated, which not only ensures the success rate of positioning but also improves the positioning accuracy. The analysis diagram is shown in Figure 7.

$$\{\begin{array}{c}{L}_{DF}=L_{AC}\times \tan \varphi \\ L_{EF}=L_{DF}-L_{DE}\end{array}$$

(5)

where $L_{DF}$ is the distance between point D and point F. Because the ear width of the Summer Black grape is 135–160 mm, and $L$ is the distance between the infrared transmitting tube and infrared receiving tube, $L_{AC}=L/2=80 \mathrm{mm}$ was obtained. In addition, $L_{DE}=5.5 \mathrm{mm}$ can be obtained with the QT30cm technical parameters. In order to ensure the best working state of the receiving tube, a receiving angle of $\varphi =7°$ was used. Finally, $L_{EF}=4.3 \mathrm{mm}$ was obtained. The parameter Settings are shown in

Table 2. Parameter Settings.

Name	Length
$L$	160 mm
$L_{AC}=L/2$	80 mm
$L_{DE}$	5.5 mm
$L_{DF}=L_{BC}=L_{AC}\times \tan \varphi$	9.8 mm
$L_{EF}=L_{DF}-L_{DE}$	4.3 mm

. Therefore, the spacing between the two receiving tubes was $8.6 \mathrm{mm}$, and the infrared transmitting tube corresponds to the receiving tube one by one, so the spacing between the transmitting tubes was also 8.6 mm.

3.4. Location Strategy of Table Grape Picking Point

The location strategy of table-grape picking points proposed in this paper is shown in Figure 8, and the method flow chart of picking-point positioning is shown in Figure 9.

After the end of recognition and segmentation, the controller takes the length of the outer rectangle’s short edge as the opening stroke of the detection device. Then, according to the planar coordinate information, the device used for detection and picking moves to the vertical position of the target grape. The device is then moved vertically while the infrared tube begins detection. Only one set of infrared tubes is triggered when they are near the base of the grape. When the device detection and picking rises, the infrared tube is triggered. If the detection and picking device is moved to the maximum horizontal length of the grape, the number of infrared tubes triggered reaches the maximum. Furthermore, if the detection and picking device continues to rise, the number of infrared tube triggers decreases. Once the number of infrared tube triggers is reduced to the minimum level and the state does not change for 30 mm, it indicates that the picking equipment has reached the position of the grape stem, namely the position of the picking point. The controller sends out the picking letter instruction to pick the target grape.

4. Discussion

In order to verify the feasibility of the method proposed in this paper, a simulation test platform (Figure 10) was built to carry out the test. The available image of the Summer Black grape was collected by RealSense R-200 and processed by MATLAB 2020(b) software. The computer processing platform used in this study is a notebook computer, the processor is Intel (R) Core (TM) i7, the main frequency is 2.60 GHz, with 16 GB running memory, and a 500 GB hard disk.

In order to verify the effectiveness of the proposed method, 100 location experiments were carried out on multiple clusters of table grapes with different shapes using a simulation test platform. The average running time of the algorithm is 0.61 s, and the success rate of localization is 90.0%. Locating failed 10 times. Due to the occlusion of leaves, the identification of the grape boundary contour was not accurate for a total of six times, which led to a calibration error of the enclosing rectangle. Secondly, because of the influence of light on the test site, the table grapes were not identified and segmented successfully three times. In addition, because of the sparse berry of grapes, the area without berry was considered as the fruit stem. So, the picking point location was incorrect in this instance.

The data of 10 randomly selected experimental are shown in

Table 3. Partial test data.

Num.	Running Time	Locating Results
1	0.61 s	Succeed
2	0.53 s	Succeed
3	0.63 s	Succeed
4	0.65 s	Succeed
5	0.51 s	Succeed
6	0.62 s	Succeed
7	0.56 s	Succeed
8	0.59 s	Succeed
9	0.35 s	Failed
10	0.61 s	Succeed

. Illumination is the main reason for the failure of recognition for sample 9, which led to the failure of outside rectangle labeling. Finally, locating the grape picking points failed. Compared with reference [14], the method proposed in this paper not only reduces the positioning time of the table-grape picking point, but also led to a positioning success rate of 90.0%, marking an increase of 7.1%.

5. Conclusions

The method proposed in this paper realizes the fast and accurate locating of grape picking points in three-dimensional space. By analyzing each component diagram in HSI color space, the S component is finally taken as the basis. Additionally, Otsu was selected as the segmentation method. The maximum connected domain was used as the segmentation basis of the optimal target grape. Fast and effective recognition and segmentation of table grapes in the natural environment was achieved. Compared with other methods, although the method proposed in this paper is greatly affected by the environment (e.g., lighting, fruit overlapping), it is simpler and faster for target detection and recognition. More efficient and accurate methods may be used to replace this method in the future.

Furthermore, the detection device of grape picking points was designed based on six groups of infrared tubes, and the picking points of table grapes in three-dimensional space were successfully located by the planar images. Finally, the results show that the average running time is 0.61 s and the positioning success rate is 90.0%. Even if the end-effector does not prevent mechanical damage to the fruit, the bottom-up approach is still a good choice. In addition, the detection device can realize the synchronization of detection and picking, which greatly improves the picking efficiency. The device is suitable for picking a variety of fruits. Therefore, the method proposed has broad application prospects. It also provides a reliable theoretical basis for picking table grapes quickly and without damage.