Development of Apple Detection System and Reinforcement Learning for Apple Manipulator

Abstract

Modern deep learning systems make it possible to develop increasingly intelligent solutions in various fields of science and technology. The electronics of single board computers facilitate the control of various robotic solutions. At the same time, the implementation of such tasks does not require a large amount of resources. However, deep learning models still require a high level of computing power. Thus, the effective control of an intelligent robot manipulator is possible when a computationally complex deep learning model on GPU graphics devices and a mechanics control unit on a single-board computer work together. In this regard, the study is devoted to the development of a computer vision model for estimation of the coordinates of objects of interest, as well as the subsequent recalculation of coordinates relative to the control of the manipulator to form a control action. In addition, in the simulation environment, a reinforcement learning model was developed to determine the optimal path for picking apples from 2D images. The detection efficiency on the test images was 92%, and in the laboratory it was possible to achieve 100% detection of apples. In addition, an algorithm has been trained that provides adequate guidance to apples located at a distance of 1 m along the Z axis. Thus, the original neural network used to recognize apples was trained using a big image dataset, algorithms for estimating the coordinates of apples were developed and investigated, and the use of reinforcement learning was suggested to optimize the picking policy.

1. Introduction

Recently, multimodal deep learning models, which combine several areas in the field of artificial intelligence, have become increasingly popular [1,2,3,4]. Of particular interest in the field of robotics is the combination of computer vision algorithms and reinforcement learning [5]. At the same time, agriculture is one of the most popular areas for introducing intelligent robotics technologies [6,7,8,9]. This is due to the observed increase in the global population, which has led to a demand for increased efficiency in all areas related to food production. For example, according to the United Nations Organization [10], in the coming decades it is expected that the population of the Earth will exceed the threshold of 10 billion people. Obviously, providing food for such a huge number of people is impossible without the optimization of modern agriculture. In addition, the economies of many countries are still facing the consequences of the coronavirus infection, which affected, among other things, the digitalization of agri-food systems [11].

Moreover, problems associated with the complexity of efficient harvesting already exist in horticulture. For example, many apples fail to reach the consumer due to reasons such as a low payback of labor, inaccessibility of picking in some places, etc. As a result, such apples remain in agricultural holdings and rot.

Given the factors listed above, on the one hand, there is the problem of increasing the amount of food produced in agriculture, and, on the other hand, the need to ensure fast and efficient harvesting with the help of intelligent robotic solutions is clear. Our article aims to solve the second problem. In addition to the harvesting robot itself, solving this issue requires the use of a control system. The main decisions in this case can be made using video information. It is clear that the detection of objects of interest in the video is required, followed by the localization of these objects. Also of interest is the method of movement of the manipulator during harvesting. Thus, the following three related problems are considered: detection of objects in the image [12], determination of the coordinates of objects relative to the manipulator [13], and the formation of a route for moving behind objects [14]. In this article, the objects of interest will be apples. The first and second tasks can be successfully solved using convolutional neural networks, as well as specialized deep cameras.

The recognition of apples in images was achieved using the YOLO neural network architecture [15,16]. An important feature of such a pretrained model is the basic ability to detect apples among 80 object classes. To estimate the depth, we use the methods of computer optics [17], and also perform a comparison for the Intel Real Sense [18] and ZED [19] cameras. The first camera registers optical images and builds a depth map through IR sensors, and the second one provides access to the point cloud as a result of scanning. Since the movement for apples will actually be carried out in a simulator in two-dimensional space, a reinforcement learning model based on Q-learning will be trained [20].

The purpose of this work is to study the efficiency of detecting apples and estimating their coordinates using various stereo cameras. The contribution that this article makes to the scientific community is the development of fruit coordinate estimation algorithms for robot guidance using various pre-trained YOLOv5 network configurations and transfer learning, as well as a comparative analysis in terms of coordinate estimation error for various stereo cameras. Moreover, the article proposes a hardware–software solution for the implementation of such a robot with a GPU computer and control using Raspberry Pi. Finally, for the first time, an attempt was made to generate apple picking trajectories, specifically in two-dimensional space, but with the possibility of generalization in the future to three-dimensional space. Transfer learning made it possible to achieve complete and accurate recognition with improved performance compared to earlier works on this topic.

The solution to a complex problem with a combination of algorithms for computer vision, stereo vision and robotic-arm control based on our developments will be described for the first time.

The structure of the article is as follows. The second section will describe stereo vision technologies, as well as computer vision systems in object detection tasks and examples of the application of reinforcement learning in robotics. The third section is devoted to the development of a software and hardware solution for the control system of a robotic arm for picking apples, and also describes an approach to the formation of a harvesting policy. The fourth section describes the evaluation of the quality and performance of the developed algorithms. The main results and conclusions are reflected in the Conclusion, followed by a list of references.

2. Related Works

Today, as noted earlier, it is difficult to imagine intelligent robotic systems without machine vision. Machine or computer vision includes the tasks of recognizing faces, gender, age, and general human characteristics [21,22]. Animal recognition is also an important task in the field of machine vision [23,24]. A number of datasets have been collected for this task, such as those used in machine learning tasks associated with the recognition of flowers, vegetables and fruits [25,26].

Among the tasks of machine vision, one can single out the acquisition, processing and (pre-processing) of images, and sometimes the prediction of future frames of a sequence of images, associated, among other things, with the simulation of new scenes [27,28]. At the same time, computer vision systems can be considered as systems used to describe the real world using images or their sequences. However, an accurate description of the world is impossible without stereovision systems. The task of stereovision systems is the reconstruction of three-dimensional coordinates of points for depth estimation. The stereo vision system usually consists of two cameras installed in the appropriate way, as shown in Figure 1 [29].

Similarly to the vision of a person who has two eyes, the cameras on the left and right provide two shifted images of the same object. Further, based on these offsets, you can determine the distance to the object. The lot method approximates the differences between two frames to perform depth mapping. The purpose for the development of such systems is to increase the accuracy of building depth maps and the speed of work.

The study in [29] proposes a taxonomy of computer vision algorithms, including methods for evaluating algorithms for the construction of depth maps. In [30], Brown and his co-authors provide approaches to the construction of stereo vision maps, as well as inconsistencies between the systems for obtaining them and the methods of processing occlusion. In addition, the authors paid attention to the implementation of such algorithms in real time. A review paper [31], presented in 2008, compares stereo vision methods in terms of computational costs, as well as the accuracy of matching predicted and actual distances. The authors of [32] consider not only global algorithms, but also more efficient local depth determination algorithms that can be implemented on various software and hardware platforms. Tombari et al. [33] made a great contribution to the development and evaluation of object recognition algorithms in three-dimensional space. In [34], the issue of assessing the suitability and applicability of stereo vision algorithms is explored. The authors collected data on the errors of various algorithms, as well as on the performance of devices that generate depth maps. Emphasis was placed on real-time systems. Although systems for matching and combining several images make it possible to obtain a high accuracy in determining the range, they process occlusions; therefore, stereo matching systems remain the most preferable in a number of tasks. Such problems, as a rule, are associated with the need to have a fixed geometry, the number of images being limited, and the speed of decision making, which also plays an important role [35]. Undoubtedly, robotic solutions satisfy all the above conditions; therefore, the use of stereo vision in robotics is appropriate.

It is noteworthy that the solution of the problem of estimating the coordinates of objects in the case of using stereo cameras can be separately divided into the tasks of estimating local coordinates on a color image frame, and recalculating coordinates relative to the camera using a depth map. Modern models consider the problem of detecting deterministic objects in close connection with pattern recognition [36]. The networks of the Region-Based CNN (R-CNN) family [37] can be attributed to classical neural network detectors of previously known classes of objects. In fact, such a network cut out a set of smaller rectangular images, the so-called regions, from the original image and process them using a classification neural network, such as AlexNet [38]. The non-optimality of this approach, firstly, consisted of the slow speed of work, and secondly, the inefficient choice of regions. The problem of increasing performance was solved to some extent by a modification called Fast R-CNN [39]. The acceleration occurred due to the fact that the entire image was first processed using the convolutional network, regions were then formed for verification using the obtained feature map, and these regions were then passed through subsequent layers of the neural network for classification. In addition, to reduce the error when determining the coordinates in the Fast R-CNN network, the regression of bounding boxes was proposed. This was also required in order to transfer the frame from the feature map for which the regions were formed to the original image. However, the performance of the Fast R-CNN model was still quite poor. In fact, the selective selection method led the neural network to perform many “idle” recognitions in regions without objects of interest. The modified Faster R-CNN method [40] used another specialized neural network which suggested regions for testing based on the feature map obtained in the first step.

Despite the optimization of the selection of test regions, real-time operation required either very powerful computing devices or processing optimization tools [41,42]. However, such works consider hardware acceleration options. At the same time, more high-performance detection algorithms were implemented in single-pass detectors such as Single Shot-Multiple Detections. Localization occurs in the same pass as detection. Examples of the architecture of convolutional single-pass detectors include the neural networks YOLO [43], SSD [44], and RetinaNet [45].

Let us take a closer look at the You Only Look Once (YOLO) architecture. At present, version 7 of this architecture has already appeared. However, no less important was the appearance of the third and fifth versions. Firstly, YOLOv3 provided an acceptable detection quality in many tasks, and the YOLOv5 network implemented on the PyTorch framework with different sizes made it possible to obtain fairly fast solutions for object detection. The basic recognition network in YOLO is the DarkNet network [46]. The selective selection algorithm has been replaced by an algorithm for dividing the original image into square areas of different sizes, for which objects are recognized in the single-pass mode. Each square is checked using three bounding boxes, for which the probability of the presence of objects of certain classes is determined. These probabilities can be used to make a final decision about the presence or absence of an object in an area. This approach made it possible to use YOLO neural networks for processing video sequences [47,48].

In recent years, there has been a trend towards the use of transformer architectures in the field of object detection in images [49,50]. However, their main disadvantage compared to convolutional models is their low performance, despite their often high accuracy. Therefore, convolutional detection architectures are more often used for video analysis. For example, YOLO has already proven itself in the problems of recognizing fruits on trees [51]. It is true, however, that [51] does not take into account data from video sequences, and the processing concerns individual frames. Moreover, many works do not estimate the real distance to apples in any way, since there are practically no datasets with such information. An increase in the quality of processing is achieved through the use of a variety of image preprocessing algorithms, as was shown in [52]. The authors of [53] studied the impact of customized datasets on the quality of detection. In addition, augmentations can improve the metrics of models [54]. Indicators of the F1-score metric at the level of 93% in the apple recognition problem were obtained in [55]. At the same time, the used convolutional networks lag behind in accuracy when detecting green apples. However, the computer vision system proposed in [56] recognizes apples and pears, and also tracks objects in the wind using the Kalman filter [57].

Finally, the problems of applying depth maps are considered in [58,59,60]. In particular, the work in [58] is devoted to studying the efficiency of estimating the distances between the robot and landmarks based on data from the Intel Real Sense Depth Camera D435. These distances are used to determine the trajectory of movement. A comparative analysis of the line of Intel Real Sense cameras is provided by the authors in [59]. However, the disadvantages of using only depth cameras when constructing motion trajectories are shown in [60]. Problems can arise when there is noise in the image or when exposed to visual attacks [61].

However, the optimal trajectories for the movement of robots are not always obvious, and learning to form them is difficult to imagine as a learning task with a teacher. In this case, reinforcement learning algorithms are used. In [62], the indoor navigation of robots is achieved using reinforcement learning. Combining algorithms for computer vision, simultaneous localization and mapping (SLAM) estimation, and reinforcement learning for robotics and positioning is proposed by the authors of [63]. The work in [64] is devoted to the application of reinforcement learning methods in a simulation environment to control a robotic arm.

Thus, a review of the related works shows that in order to solve the problem of picking apples using a robotic arm, it is necessary to organize a computer vision system based on convolutional neural networks, develop algorithms for converting coordinates relative to the camera into coordinates relative to the manipulator, and to also consider options for applying reinforcement learning for picking apples.

The scientific novelty of this article is as follows. Firstly, we performed transfer learning on a large volume of images, and secondly, a comparative analysis of two types of cameras is performed. Thirdly, this article also explored the simulation of a method for constructing a path for picking apples based on reinforcement learning.

3. Materials and Methods

To develop an algorithm for detecting apples in images, we used our own dataset consisting of 7218 images in which about 75,000 apples were marked and 3400 images without apples. The data were collected in specialized partner gardens. Red and green apples were considered. It is important to note that all images were taken in good and clear weather with sunlight. There was also no binding to different varieties in order to simplify the task of recognizing apples against the background of everything else.

An example of a training sample image is shown in Figure 2.

The test sample included 1000 images with 9620 apples and 215 images without apples. YOLOv5 architectures with various weights were considered. Since such models are able to detect apples without training, models with and without weight tuning are compared.

YOLOv5 was chosen after testing different architectures using NVIDIA GeForce GTX 1070.

Table 1. Comparative analysis of apple detection algorithms.

Model	Precision	Performance (fps)
YOLOv5	86.32%	6.54
YOLOv3	74.85%	4.75
R-CNN	69.45%	1.06
Fast R-CNN	66.78%	2.78

shows the results of a comparative analysis for apple detection on 100 images with 854 apples and 20 images without apples.

It can be seen that YOLOv5 shows best precision and performance.

The Computer Vision Annotation Toolbox (CVAT) was used for markup. It allows you to make annotations in various formats. The YOLOv5 network itself was trained using RoboFlow. Figure 3 shows an example of data markup in CVAT [65].

Thus, the color image processing model was first trained. Further, the use of specialized depth cameras, such as the Intel Real Sense Depth Camera D415 and Stereo Labs ZED 2, was considered. If the first one supplies color and depth images for processing separately, for which no alignment is required, then the second one provides researchers with a three-dimensional point cloud.

In the case of Real Sense, the pixel coordinates of detected objects need to be recalculated using depth map data. Such a recalculation is carried out using the following expressions [66]:

$$\begin{array}{l}X[mm]=d_{x0,y0}\frac{(C_x-x_0[px])}{f_x},\\ Y[mm]=d_{x0,y0}\frac{(C_y-y_0[px])}{f_y},\end{array}$$

(1)

where $X[mm]$ is the projection of the distance relative to the center of the image on the axis X, which we calculate in millimeters; $Y[mm]$ is the projection of the distance relative to the center of the image on the axis Y, which we calculate in millimeters; $d_{x0,y0}$ is the depth map value at the point with coordinate $(x_0,y_0)$, which the camera calculates in millimeters; $C_x$ is the X-axis coordinate of the image center, which is machine calculated in pixels by definition; $C_y$ is the Y-axis coordinate of the image center, which is machine calculated in pixels by definition; $f_x$ and $f_y$ are internal parameters of the optical system of the camera used to obtain an image, i.e., the focal lengths along the axes X and Y; $(x_0,y_0)$ is the coordinate of the center of the detected object on the image, which the neural network algorithm calculate in pixels.

When using the D415 model, it is necessary to make a correction for the shift of the camera center for Expression (1):

$$X’[mm]=X[mm]-35$$

(2)

where $X’[mm]$ is the unshifted projection of the distance from the center of the camera to the object along the X axis (in mm), and 35 (mm) is the standard offset of the D415 model.

Having calculated the projection of the distance on the Y axis in accordance with expression (1), and the projection of the distance on the X axis in accordance with expression (2), it is easy, knowing the total distance from the depth map, to determine the projection of the distance along the Z axis as follows:

$$Z[mm]=\sqrt{d^2-X^{\prime }{[mm]}^2-Y{[mm]}^2}$$

(3)

Expressions (1)–(3) provide coordinates relative to the camera center. However, the position of the camera on the manipulator is known in its local coordinate system, so the distance from the manipulator to the object can be recalculated by introducing appropriate corrections for each of the coordinate axes. Since stepper motors are used to control the manipulator along the X, Y, and Z axes, the decision regarding the number of pins supplied by the control system is easy to adjust by a given number of millimeters, and can be easily calculated using regression models along the axes. Figure 4 shows a manipulator with a camera and an apple-recognizing computing device.

Figure 4 shows the prototype of a manipulator. There is laptop with a GPU on the front. It is used for object detection. We can also see some drivers and a connection with the depth camera. They are controlled using Raspberry Pi 3 connected to a laptop. The experiment also includes two apples for picking. We can see that robotic arm is aimed at the first (left) apple. The next steps are the movement to the apple and grabbing.

Control is carried out using a single-board microcomputer Raspberry Pi 4, which in the local network receives the number of necessary steps from the computer vision system so as to aim the manipulator at the object. A block diagram for this type of intelligent manipulator is shown in Figure 5.

According to the above diagram, the manipulator control procedure is as follows. First, the stereo vision system evaluates the coordinates of the object relative to the camera, recalculates them relative to the manipulator and transmits the results to the control device. The control device determines the required number of pins to apply to the drive and creates a control action to move to the desired point. When the manipulator reaches the required position, it picks the apple and returns to the starting point.

The system works in a similar way using the ZED 2 stereo camera, which is shown in Figure 6.

As shown in Figure 6 [19] (camera device), the ZED 2 stereo camera has left and right optical cameras, the image difference between which allows you to build a three-dimensional point cloud. Therefore, it can also be used for coordinates estimation.

Let the coordinates of the rectangle bounding the apple be determined using YOLO. Then, we determine the coordinates relative to the center of the ZED 2 camera. Note that in our case, processing was performed for the frame from the left camera of the device. Object detection after YOLO includes the following parameters: coordinates of the upper left corner of the rectangle along the X and Y axes labeled as $(x_l,y_l)$, width $w$ and height $h$.

Accordingly, the point corresponding to the center of the object has coordinates $(x_l+w/2,y_l+h/2)$. The ZED 2 camera interface provides the ability to access the point cloud by the coordinate of the color image frame. Let us estimate the distance to the object using a cloud of points according to the Euclid formula:

$$r=\sqrt{p{c}_x^2(x_l+w/2,y_l+h/2)+p{c}_y^2(x_l+w/2,y_l+h/2)+p{c}_z^2(x_l+w/2,y_l+h/2)}$$

(4)

where $p{c}_{x(y,z)}$ is the distance projection value along the X(Y,Z) axis.

Knowing all projections of distances relative to the left camera, we recalculate them for the manipulator and determine the required number of steps for movement.

All image processing in the laboratory was performed using an ASUS FX 504 computer with an NVIDIA GeForce 1060 video card and CUDA version 11.0.

The formation of the route was solved using Python with Q-learning methods. It was believed that the apples were at an equal distance. A 10 × 10 map was modeled, where 3 apples could be randomly located. Figure 7 shows one of the states of the simulation environment.

In this case, it is assumed that the manipulator agent always starts from the lower right corner. Additionally, an optimal route needs to be built, with shifts occurring only along the X and Y axes; however, an unpicked apple cannot be knocked down when moving towards the apple that the manipulator agent wants to pick.

The action space was described by movements along the X and Y axes and the capture of an apple. A discount factor of 0.9 and a learning rate of 0.005 were used. The q-values were updated according to the following formula [67,68]:

$$Q_{new}(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha (r_t+\gamma \max Q(s_{t+1},a)-Q(s_t,a_t))$$

(5)

where $(s_t,a_t)$ is the state and action at a point in time t, $\alpha$ is the learning rate, $\gamma$ is the discount coefficient, and $r_t$ is the reward at a point in time t.

4. Results

The results of the work of various detection algorithms according to the image frame, estimates of the accuracy of coordinates by different cameras, and the effectiveness of reinforcement learning algorithms are further presented in the tables.

Figure 8 shows an example of detecting apples on a test image.

Table 2. Comparative analysis of apple detection algorithms.

Model	mAP	mAP after Fine-Tuning
Yolov5n (640 px)	46.42%	46.47%
Yolov5s (640 px)	53.22%	53.44%
Yolov5m (640 px)	84.75%	86.22%
Yolov5l (640 px)	88.92%	90.13%
Yolov5x (640 px)	90.31%	94.27%
Yolov5x6 (1280 px)	91.14%	96.22%
Yolov5x6+ (1280 px)	93.72%	97.08%

shows the comparative mAP metrics for different neural network models.

The analysis shows that the largest model provides the best quality. At the same time, transfer learning leads to an increase of up to 5%. The speed of work will be inversely proportional to the size of the models.

Although there were red and green apples in the training data, we did not treat them separately as two classes in training. However, interesting results were observed during testing. It was found that the neural network recognizes red apples somewhat better than green ones. This is probably due to the presence of green leaves in the background.

Table 3. Comparative analysis of red and green apple detection.

Model	mAP Red	mAP Green
Yolov5n (640 px)	47.54 %	46.22%
Yolov5s (640 px)	54.86%	52.91%
Yolov5m (640 px)	86.78%	85.42%
Yolov5l (640 px)	91.07%	88.95%
Yolov5x (640 px)	95.64%	93.12%
Yolov5x6 (1280 px)	96.92%	95.05%
Yolov5x6+ (1280 px)	97.96%	95.38%

presents the results of the detection of red and green apples.

Let us review how the system works in the laboratory. We use red apples on the stand, as the task of estimating coordinates can then be solved.

Figure 9 shows an example of detecting apples and coordinates in the laboratory.

Figure 9 shows the direct processing of data from the video camera. However, since we do not use specialized algorithms when working with video data, such as optical flow, motion vector estimation, and others [42], the neural network is used for detection and processes individual frames of the video stream. This approach is applicable, since the apples on the trees are quite static, which imitates our laboratory stand. No field studies have yet been performed to evaluate effectiveness; thus, results are provided for a laboratory stand.

Table 4. Comparative characteristics of coordinate estimation (RMSE are measured in mm).

Camera	RMSE on X (mm)	RMSE on Y (mm)	RMSE on Z (mm)
Intel Real Sense D415	7.22	6.54	16.59
ZED 2	6.24	4.68	13.22

presents the comparative characteristics of the RMSE metrics for the error of estimating coordinates based on different cameras.

The analysis shows that the accuracy of position estimation using the ZED 2 camera is 10–15% higher than that of the Real Sense camera.

It should also be noted that error data is important for different sizes of blocks and manipulators. In a laboratory study, a three-fingered manipulator with a girth of about 12 cm in a compressed state and apples with a diameter of about 10 cm were investigated. In this regard, it is important that the pointing errors at the center of the apple do not exceed 1 cm in content and width. This allows for the capture of the grip. Figure 10 shows the manipulator grabbing the apple.

Table 5. Movement politics comparison.

Politics	Time (ms)	Penalties
Q-learning	36.85	0
Naive (snake)	124.75	5.6

shows the results of reinforcement learning for picking apples. Note that the time to collect all the apples was considered on average and the number of errors during movement on average.

Clearly, the metrics for the trained model are much better, because the Naive algorithm performs all actions randomly, but our model learned the shortest ways in different states in a simulator.

Undoubtedly, the obtained developments can be used in the future to find rotten and infected fruits [69,70].

5. Conclusions

Thus, several algorithms were implemented in this article. First, transfer learning was performed for the YOLOv5 neural network, which provides 97% apple detection accuracy on real images of horticulture. Moreover, Real Sense and ZED 2 cameras were compared to estimate the coordinates of apples. The analysis showed that the errors in determining the coordinates are smaller for the ZED 2 camera. Along the X and Y axes, this error does not exceed 6.5 mm according to the RMSE metric, and along the Z axis, such an error is slightly more than 1 cm. However, such errors are not critical for collecting apples. Finally, a reinforcement learning model is explored in the simulation, which is several times faster and more accurate than models with naive apple-picking policies. In the future, we plan to optimize the processing of video data, as well as to teach the model to distinguish between red, green and yellow apples. Furthermore, the investigation will be devoted to detection in different weather conditions and during day time. Another important area of research will be the adaptation of algorithms for estimating coordinates in sunlight. Therefore, it will be important to consider solar light interference.