Next Article in Journal
Plant Disease Identification Based on Encoder–Decoder Model
Previous Article in Journal
Responses of Soil Macro-Porosity, Nutrient Concentrations and Stoichiometry Following Conversion of Rice–Wheat Rotation to Organic Greenhouse Vegetable System
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Agronomy
Volume 14
Issue 10
10.3390/agronomy14102209
agronomy-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Balancing Accuracy and Efficiency: The Status and Challenges of Agricultural Multi-Arm Harvesting Robot Research

by
Jiawei Chen
1,
Wei Ma
2,*,
Hongsen Liao
1,
Junhua Lu
1,
Yuxin Yang
1,
Jianping Qian
3 and
Lijia Xu
4
1
School of Mechanical Engineering, Xihua University, Chengdu 610039, China
2
Institute of Urban Agriculture, Chinese Academy of Agricultural Sciences, Chengdu 610213, China
3
Institute of Agricultural Resources and Regional Planning, Chinese Academy of Agricultural Sciences, Beijing 100081, China
4
College of Mechanical and Electrical Engineering, Sichuan Agriculture University, Ya’an 625014, China
*
Author to whom correspondence should be addressed.
Agronomy 2024, 14(10), 2209; https://doi.org/10.3390/agronomy14102209
Submission received: 4 September 2024 / Revised: 24 September 2024 / Accepted: 24 September 2024 / Published: 25 September 2024
(This article belongs to the Section Precision and Digital Agriculture)
Browse Figures
Versions Notes

Abstract

:
As the global fruit growing area continues to increase and the population aging problem continues to intensify, fruit and vegetable production is constrained by the difficulties of labor shortages and high costs. Single-arm harvesting robots are inefficient, and in order to balance harvesting accuracy and efficiency, research on multi-arm harvesting robots has become a hot topic. This paper summarizes the performance of multi-arm harvesting robots in indoor and outdoor environments from the aspects of automatic navigation technology, fruit and vegetable identification and localization, multi-arm workspace optimization, and multi-arm harvesting task planning and analyzes their advantages and challenges in practical applications. The results show that the lack of application of automatic field navigation for multi-arm harvesting robots, the low harvesting rate in non-structured environments, and the complexity of algorithms for multi-arm harvesting robots’ task planning are the main challenges hindering their wide-scale application. Future studies need to focus on building a standardized growing environment to control the amount of information acquired by the robots and optimize the multi-arm control strategy of these challenges, which is an important direction for research on multi-arm harvesting robots.

1. Introduction

According to the statistics of the Food and Agriculture Organization of the United Nations, from 1960 to 2022, the global fruit planting area continued to grow, reaching 67.49 million hectares in 2022. While the cost of fruit and vegetable manual harvesting has risen rigidly, the labor cost of the vegetable and fresh fruit industry has reached more than 60% [1]. In addition, according to World Population Prospects 2022, by 2050, the proportion of the global population aged 65 and over is expected to rise to 16% from 10% in 2022, and labor shortages will further increase the cost of fruit and vegetable harvesting.
Since 1984 when Kawamura et al. [2] developed the first tomato harvesting robot, harvesting robots have made rapid progress. Ji et al. [3] achieved an 85% success rate in cucumber harvesting with a single harvest cycle of 28.6 s. Arad et al. [4] reduced the single harvest time to 15 s, but the optimal harvest success rate was only 61%. In order to improve both harvesting success and rate, researchers developed a multi-arm harvesting robot. Feng et al. [5] developed a four-arm harvesting robot that achieved a harvesting rate of 4.87 s per fruit, which is 1.96 times higher than that of a single-arm robot, and a successful harvesting rate of 82%. Williams et al. [6] invented a four-arm kiwifruit harvesting robot. The robot utilized deep neural networks and stereo-matching techniques and proposed a dynamic fruit scheduling system to coordinate the movements of the robotic arms during harvesting, with a harvesting time of 5.5 s/fruit. The dual-arm harvesting robot designed by Jiang et al. [7] had an 83% harvesting success rate at 9 s/plant. Sepulveda et al. [8] proposed a dual-arm harvesting robot with a 91.67% harvesting success rate.
In view of this, in order to achieve efficient and precise harvesting of fruits and vegetables, researchers have carried out extensive studies on automatic navigation, fruit identification and localization techniques, harvesting space optimization, and multi-arm task planning techniques. In this paper, a comparison of different multi-arm harvesting robots is demonstrated, as shown in Table 1. This paper reviews the research progress of multi-arm harvesting robots, discusses how to effectively improve the efficiency of multi-arm harvesting, and analyzes the limitations and challenges of existing technologies and the development trend of harvesting robots. It also points out that automatic navigation in non-structured environments, improving the accuracy of fruit and vegetable recognition, optimizing the multi-arm control strategy, and enhancing the robot’s adaptive ability are important directions for future research.

2. Key Technologies for Multi-Arm Harvesting Robots

Multi-arm robots have been categorized into two types based on their operation method nationally and internationally. One is the dual-arm collaborative type, which grasps the fruit with one hand and cuts with the other hand, as shown in Figure 1a and as shown in Figure 1b [8,13]. The harvesting success rate of dual-arm collaborative robots performing these two tasks can reach 87.5% and 91.67%, respectively, but the harvesting time is greater than 20 s/fruit in both cases. Therefore, this type has a high success rate but low efficiency, so it is used for harvesting fruits and vegetables with long stems and high economic value. The second is the parallel collaboration mode, as shown in Figure 1c,d [12,15], where the robot harvesting success rate and the average cycle time are 71.28% to 80.45% at 5.8 to 6.7 s/fruit and 86.8% at 8.85 s/fruit, respectively. This type of robot, which focuses on breaking accuracy, success rate, and efficiency, is the main direction of research scholars.

2.1. Navigation Technology

Autonomous navigation is one of the most important technologies in automated agriculture and is widely used in production processes such as agricultural harvesting and farming [23]. For example, Xiong et al. [11] realized autonomous navigation through fixed-point navigation, while Yu et al. [14] utilized QR codes affixed to trees for navigation. However, the operating environments of most harvesting robots are complex outdoor scenarios, and the current multi-arm harvesting robots are still deficient in the application and development of auto-navigation technology. Auto-navigation technology can help multi-arm harvesting robots autonomously plan their paths and accurately perform their tasks, thus improving harvesting efficiency and accuracy. In intelligent agriculture, automatic navigation technology reduces the dependence on human labor and enables robots to make intelligent decisions and optimization in complex environments, thus promoting the intelligent development of agricultural production.
Navigation technologies are essential for agricultural robots to operate efficiently in complex field environments, particularly in non-structured or dynamic settings. These systems are crucial for guiding harvesting robots through their work areas. However, inter-row navigation and boundary following, where the robot moves along crop rows or field edges, often perform poorly in non-structured environments, as shown in Figure 2a,b [24,25]. Simultaneous Localization and Mapping (SLAM) is a key technique used for environment detection in unknown terrains. Sensors capture information about the surroundings, enabling both map construction and robot localization, which is especially important in dynamic and unpredictable field conditions, as shown in Figure 2c [26,27]. Additionally, deep reinforcement learning is applied to optimize path planning. Through continuous interaction with the environment, Deep Q-learning (DQN) enables robots to adaptively adjust their path planning strategies to suit the complex conditions of various orchards, as shown in Figure 2d [28].
Sensing technologies in automated navigation can be categorized into Global Navigation Satellite Systems (GNSSs), Light Detection and Ranging (LiDAR), and vision-based methods. GNSSs have been widely used in precision agriculture, and improvements in systems such as GPS, GLONASS, Galileo, and BeiDou [29] have enabled agricultural robots to achieve higher accuracy in navigation and localization [30]. However, GNSS signals are susceptible to interference in heavily vegetated or canopy-covered areas. LiDAR generates high-resolution 3D point clouds by emitting lasers and measuring the reflection time [31], which provides accurate 3D environment sensing. LiDAR technology is suitable for scenarios where GNSS signals are unreliable, such as densely vegetated areas or indoor environments [32,33]. Vision-based navigation techniques utilize convolutional neural networks (CNNs) to process visual information [34] and realize robot navigation through target detection [35], which improves the robot’s navigation ability in outdoor environments. In addition, systems based on LiDAR and inertial measurement unit (IMU) data fusion [36], and LiDAR combined with GNSSs and cameras [37,38], provide more comprehensive environmental data and navigation information and perform well in complex agricultural environments.

2.2. Target Recognition Technology

The performance of the target detection model is directly related to the efficiency of fruit and vegetable harvesting robots [39]. As shown in Table 1, more than half of the multi-arm harvesting robots operate outdoors and are therefore negatively affected by noise [40], light [12,41], and occlusion [8,11,42], which make it difficult for the robots to accurately recognize fruits, resulting in a low harvesting success rate.

2.2.1. Target Recognition in Adverse Light

With the continuous development of image preprocessing techniques and feature extraction methods, researchers have made significant progress in improving the accuracy and robustness of fruit detection in complex environments, as shown in Table 2.
Image preprocessing effectively improves image quality to optimize detection [58]. Wang et al. [43] utilized the improved FastICA method to effectively deal with Gaussian noise and pretzel noise. Ji et al. [44] utilized the Retinex algorithm with bootstrap filtering so that the edge detail information was enhanced, the contours became smooth, and apples became more prominent in an image. Liu et al. [45] utilized the Gamma transform image enhancement technique, which resulted in significant improvement in brightness, grayscale, contrast and sharpness. Qi et al. [46] utilized the deep learning model Generative Adversarial Networks to generate high-resolution images, thus improving the robustness of the model.
The use of features improves the extraction of objects detected under poor lighting conditions [59]. Naseeb Singh et al. [47] used the color difference method to detect cotton and Liu et al. [48] used two BPNNs to identify image color information and pixel color and position information to improve the recognition accuracy. However, color-based recognition techniques are highly sensitive to light variations. Therefore, Fu et al. [49] extracted kiwi boundaries using Canny operator, and Lin et al. [50] used Partial Shape Matching (PSM) and Probabilistic Hough Transform (PHT) to match fruit features.
The feature extraction and matching techniques have poor adaptability to complex scenes and limitations in handling irregular shapes. Xiong et al. [51] utilized the Otsu algorithm to segment an image to remove the background and used the fuzzy C-mean clustering algorithm to segment a fruit and fruit stalk image to obtain a lychee fruit image. Zhang et al. [52] extracted the external contours of target apples using a distance-transform-based watershed algorithm with 89.79% recognition accuracy under a backlight. Wang et al. [53] segmented fruit images using the K-means clustering method and achieved a recognition rate of 93.6% for cloudy grapes. Sun et al. [54] combined the GrabCut algorithm and the Ncut algorithm to segment green apples and achieved an average F1 of 94.08% under varying illumination conditions.
Although image segmentation techniques are still effective in complex scenes, they have significant shortcomings in feature representation, data processing capability, accuracy performance, and adaptivity compared to deep learning models. Chu et al. [57] added a suppression branch to the standard Mask-R-CNN to suppress non-apple features generated by the original network, with a detection accuracy of 88% under adverse lighting and a 0.25 s per frame detection time. Liang et al. [55] used the K-means clustering algorithm based on YOLOv3 to cluster lychee fruit Bounding Boxes (Boundary Boxes) labeled in the training set to obtain the optimal anchor box size in low brightness and used the pre-trained weights on the ImageNet classification task to initialize the parameters of Darknet-53, with an accuracy in low brightness of 89.30% and a detection time of 0.026 s. Zhang et al. [56] used a squeezed excitation network (SE net) to amplify relevant features while suppressing less relevant features based on YOLOv5, using the EIoU (enhanced concatenated intersection) loss instead of CIoU (complete concatenated intersection) loss function, with an accuracy of 0.954 in unfavorable lighting and 102 ms per frame.

2.2.2. Target Recognition under Occlusion

Occlusion is an unavoidable problem in fruit and vegetable detection. As early as 2012, Li et al. [60] conducted a study on occlusion recognition. At present, multi-sensor detection, shape reconstruction, and target detection algorithms are able to detect fruits and vegetables effectively.
(1) Multi-sensor detection. Madeleine Stein [61] used an unmanned ground vehicle for multi-point-of-view data collection, and the error rate of detecting a single tree was only 1.36%, but it did not have real-time detection capability. Yoshida et al. [9] set up a multi-point camera to construct a good visual image for multi-arm harvesting robots, and it could achieve more than 95% of accuracy even in outdoor settings. But it is affected by light variations and is costly.
(2) Shape reconstruction. Yang et al. [62] proposed an amodal segmentation network based on Swin Transformer and boundary estimation, and the detection accuracy of all tomatoes exceeded 90%. Gong et al. [10] used the Mask-R-CNN model to segment target tomatoes from the original image. They then applied a shape and position reconstruction (SPR) model to recover the depth information of the intact tomatoes. From the reconstructed depth image, they extracted the center of mass and diameter of the intact tomatoes in camera coordinates.
(3) Deep learn. Deep learning has a strong ability to extract high-dimensional features from fruit images and is widely used in fruit detection [63]. Chu et al. [57] proposed an apple detection model based on Mask-R-CNN, with a recognition accuracy of 0.880 under complex occlusion. Hou et al. [64] proposed a cherry tomato detection model based on YOLOv7, achieving a 92.2% success rate for detecting tomatoes with 30–70% occlusion. Chu et al. [65] proposed the occluder–occludee relational network (O2RNet) to detect clusters of apples under different occlusion conditions and achieved 94% accuracy.
Direct grasping of occluded fruits after successful detection can lead to damage caused by collisions between robotic arms and plants, and thus occlusion classification recognition is induced to develop appropriate harvesting strategies based on different types of occlusion to improve harvesting safety. The occlusion effect can be categorized into four classes, non-occlusion, fruit occlusion, leaf occlusion, and branch occlusion, as shown in Figure 3, which helps to guide the robot for harvesting. Rathore et al. [66] classified the detected apples into the abovementioned four classes using the EfficientNet-B0 model, with a classification accuracy of about 91.38%. Such methods are good for harvesting task planning, but the robot will drop the occluded fruits, resulting in low harvesting rates. Therefore, Sun et al. [67] proposed a detection method based on deep learning and active sensing, which detects and grasps the occluded fruits and vegetables by continuously adjusting the robotic arm position through active sensing technology.

2.2.3. Target Recognition at Different Ripeness Levels

During the harvesting process, harvesting robots must accurately assess the ripeness of target crops; otherwise, the quality of the harvest may decline, and immature crops may be mistakenly harvested, leading to reduced economic benefits. To address this issue, ripeness can be determined by analyzing external features such as the size, color, and texture of crops, thereby improving the precision and efficiency of harvesting. Meng et al. [68] used an improved YOLOv7-tiny model to detect cherry tomatoes at different ripeness stages, primarily based on the color characteristics of tomato ripeness. QI et al. [69] determined the ripeness level by calculating the ratio of green to red fruits, achieving an accuracy of 90.84% in ripeness estimation.
Wang et al. [70] improved the YOLOv5s model by enhancing the texture feature fusion of apples at the same ripeness level, significantly increasing the model’s detection accuracy for apples at different ripeness stages. The average detection precision for apples at high, medium, and low ripeness levels was 94.1%, 93.1%, and 93.7%, respectively.
Similarly, Liu et al. [71] improved the YOLOv5s model by introducing a dual attention mechanism (channel and spatial attention, GAM) to enhance the extraction of color and texture features of apples. For diameter detection, the method combined dual features from color and depth images to capture apple contours. Based on the conversion relationship between pixel area and actual area, the apple diameter was calculated. The model achieved a 93.3% success rate in diameter detection, with an average diameter deviation of 0.878 mm, and reached an average ripeness detection accuracy of 98.7%.

2.3. Localization Technology

Fruit detection and localization in agricultural harvesting is a challenging task, requiring accurate object information for effective analysis [72].

2.3.1. Vision and Point Clouds

(1) Based on visual localization technology. This approach provides accurate perception of the environment and target objects by analyzing image data, enabling the robot to achieve accurate localization in diverse scenes.
Monocular cameras, which use a CMOS color camera for apple localization, use the relationship between camera focal length, pixel size, and the center of the apple in the image plane to calculate the distance between the camera and the fruit [73]. Using binocular cameras, by comparing the parallax between these two images, the depth information of the object can be deduced to achieve 3D localization of the object [74]. Gao et al. [75] detected kiwifruit and calyx by YOLOv5x and calculated the depth values using the pairing and matching technique to convert the feature points to 3D coordinates, thus realizing the high accuracy localization of kiwifruit with average deviations of 7.9 mm, 6.4 mm, and 4.8 mm on the X, Y, and Z axes, respectively. An RGB-D camera was used to obtain color images and depth information, and the position of the target object was deduced from the depth map. Hu et al. [76] used an RGB-D camera to obtain aligned depth images and calculate the depth values of the desired points. The two-dimensional coordinates in the RGB image were combined with the depth values to obtain the three-dimensional coordinates of the apple harvesting point, and the localization errors in the X, Y, and Z directions were less than 7 mm, 7 mm, and 5 mm.
(2) Based on point cloud localization technology. By analyzing 3D point cloud data, this approach provides precise spatial information, enabling the robot to accurately locate the target.
Light Detection and Ranging (LIDAR) provides more accurate and extensive point cloud data, can work stably under various lighting conditions, and is especially suitable for the accurate localization of fruits and vegetables in large-scale orchards or farmland. Liu et al. [77] used the ORB-SLAM3 algorithm to generate the current pose and registered the LiDAR scan data to the global coordinate system. A sliding window of the most recent 20 scans was used to generate a dense local point cloud, which was combined with YOLOV5’s apple detection results. The average positioning error under dynamic detection was 21.1 mm. The RGB-D camera generates point cloud data by acquiring the color image and depth information of the scene, which is suitable for object recognition and localization in close range and complex environments. With the rapid development of depth cameras, depth cameras are widely used in machine vision to determine the position of an object and the distance between the object and the camera [78]. Seventeen multi-arm harvesting robots with depth cameras account for 76.5% of the total number of robots. Li et al. [79] used RGBD cameras to acquire the initial point cloud data, localized the apples on the RGB images by a deep learning segmentation network, and constructed a 3D view cone to estimate the depth information of the fruit, thus generating more accurate point cloud data for precise positioning of occlusion fruits with a Center Med error of 5.69 to 14.94 mm.
(3) Based on multi-sensor fusion localization technique. The integration of data from different sensors enables more accurate and robust spatial localization and ensures reliable operation of robots in variable environments. Kang et al. [80] proposed a visual perception method using high-resolution LiDAR–camera fusion. The standard deviations for fruit localization at distances of 0.5 m, 1.2 m, and 1.8 m were 0.253 cm, 0.230 cm, and 0.285 cm, respectively.

2.3.2. Position and Pose

Given the differences in various fruit harvesting methods, the positional information that the robot needs to acquire varies. For nearly spherical fruits such as apples, kiwis, and citrus, it is usually only necessary to obtain the spatial coordinates of the fruit [81]. Li et al. [82] directly computed the 3D coordinates of the apple feature points by selecting the feature points of the apples, with an average standard deviation of 5.1 mm.
However, for soft fruits such as tomatoes, strawberries, and grapes, it is usually necessary to obtain the spatial pose information of the fruits and stems. Jang et al. [83] detected and segmented the tomato body and sepals using YOLOv8. They then determined the correlation between the tomato body and sepals by combining IoU matching with the Hungarian algorithm. For pose estimation, they utilized the point cloud generated by RGB-D data. The angular error of the tomato pose estimation was 6.79 ± 3.18 degrees, as shown in Figure 4a. Yu et al. [84] predicted the rotational bounding box of the fruit target in the camera field of view in real time using R-YOLO, and the hand–eye vision system computed the target fruit inclination slope in real time. The localization accuracy of the pickup point was significantly improved, as shown in Figure 4b. Coll-Ribes et al. [85] performed example segmentation of grapes and peduncles based on the Mask-R-CNN model, and the segmentation of grape bunches and peduncles became more accurate by adding the depth information gained from monocular depth estimation to the RGB images, as shown in Figure 4c.

2.4. Multi-Arm Workspace and Task Planning

Li et al. [12] designed a robot that was unable to grasp fruits lower than 1.5 m, resulting in an abandonment rate of up to 62.04%. The goal of determining the number of robotic arms, spacing, and designing the layout and type of robotic arms is to improve the area of the robot’s workspace [86,87]. Xiong et al. [88] proposed an optimal configuration method based on critical-TOPSIS to ensure complete coverage of the canopy area. Cui et al. [18] combined an equivalent model of the robotic arm workspace and canopy space to determine the relative mounting positions of the dual arms, resulting in a traversal success rate of 92.09% for the average fruit position of the robot. Au et al. [22] analyzed kiwifruit trellises and found that an articulated robot has limitations in irregularly shaped workspaces, whereas a Cartesian harvesting robot can be effective in complete coverage of the standard square space and can effectively reduce the number of harvesting steps, as shown in Figure 5, thereby reducing harvesting time and enhancing robot accessibility.
The existence of common areas in the workspace of multi-arm harvesting robots is unavoidable and adds to the difficulty of using harvesting robots. The common workspace of the articulated robot is small, as shown in Figure 6a [7], so the common space is less difficult to work in, whereas the Cartesian system robot moves the robotic arm through the guide rail, as shown in Figure 6b [12]. Increasing the workspace increases the common workspace, and the multi-arm task planning increases linearly.

2.4.1. Harvesting Sequence

Harvesting robots are often affected by non-structured environments in agricultural production, and the harvesting efficiency and success rate can be significantly improved by optimizing the harvesting sequence [89]. Multi-arm harvesting robots can not only avoid conflicts between robotic arms but also improve the overall efficiency through rational sequence planning.
Xiong et al. [11] used a control algorithm to plan the harvesting sequence of a dual-arm harvesting robot and avoid collisions, as shown in Figure 7a. By optimizing the harvesting sequence, this method can improve the simultaneous working time of both arms, monitor the distance between the arms in real time, and when they are close, the secondary arm (Arm 2) will retreat to a safe position to avoid collisions. The harvesting efficiency of this method is limited by several factors: the fixed harvesting sequence (bottom–up) reduces flexibility, and the secondary arm has to wait when a collision may occur, leading to a reduction in the harvesting efficiency. Rong et al. [15] partitioned the harvesting space as shown in Figure 7b so that the robotic arm prioritizes the harvesting of the nearest target and allows robotic arm A to walk from A1 to A2 first, ensuring that the distance between the two arms is always greater than the safety distance. Limiting the working range of the robotic arm leads to uneven distribution of the work of the robotic arm, which makes the harvesting efficiency decrease. The above two methods, with situation-specific constraints to make the distance between the robotic arms always greater than the safety distance, sacrificed the efficiency of the public harvesting space despite the improved safety. Jiang et al. [7] used a “symmetric space partitioning” method based on depth values to plan the harvesting sequence, as shown in Figure 7c. Through spatial segmentation, the dual arms can work independently, which reduces the coordination time, improves the operation efficiency, and reduces the interference and collision risk between the two arms.

2.4.2. Task Scheduling

Task scheduling of multi-arm harvesting robots is a key issue in this field to optimize task allocation, avoid interference between robotic arms, and balance the task load to improve the efficiency of multi-arm harvesting robots [90]. Compared to single-arm systems, multi-arm systems are more advantageous in terms of shortening the harvesting cycle time because the multiple arms can share the tasks and reduce the workload of the single arm. The extent to which this advantage is realized depends on the effectiveness of task scheduling and the work efficiency of each robotic arm.
Barnett et al. [91] proposed a task division methodology, and the study introduced a work distribution parameter to measure the efficiency of task allocation to minimize the total harvesting time. The task division method takes into account the indivisibility of fruits and the distribution of fruits in the canopy to ensure that a better task assignment can be achieved even in the case of the uneven distribution of fruits. Due to the uneven distribution of fruits in the canopy, the task division sometimes has difficulty achieving the desired uniform distribution, and it is also deficient in terms of the coordination and collision avoidance among the robotic arms. Yang et al. [17] proposed a multi-objective optimization algorithm, IGAACMO, to optimize the harvest trajectories for multi-arm collaboration to achieve the shortest harvest paths and avoid the waste of robotic arm resources due to the fruit clustering problem. However, the study mainly focuses on trajectory optimization and task allocation and lacks the treatment of robotic arm collision and dynamic task adjustment. Li et al. [92] proposed a Multi-intelligent Reinforcement Learning (MARL)-based approach to solve the coordination and optimization problem of multiple robotic arms working together to efficiently harvest fruits. This approach effectively solved the problem of conflict and dynamic task allocation among robotic arms and the problem of multi-arm harvesting robots not being able to adapt to different tasks and working conditions. However, the training process of the multi-intelligent body reinforcement learning algorithm is complex and may require a large amount of computational resources and time, and the theoretical model may face the challenge of environmental complexity and uncertainty in practical applications, which requires further optimization and adjustment.

3. Challenges and Trends

With the unremitting efforts of researchers and scholars at home and abroad, some progress has been made in the field of multi-arm harvesting robots, but the technology still faces many challenges in its application. These challenges mainly include how to perform efficient and accurate fruit identification and localization in non-structured environments, as well as task planning and execution in complex environments. The existence of these problems makes the current multi-arm harvesting robot systems still have much room for improvement in terms of efficiency and accuracy. Carrying out research on multi-arm harvesting robots to gradually realize the efficient and accurate intelligent harvesting of agricultural products is of great significance to improve the current shortage of agricultural labor faced by the world and to realize the development of precision agriculture. Therefore, future research needs to work on the following aspects:
(1) Orchard structuring. Non-structured environments are usually characterized by irregular terrain, dynamic obstacles, and complex information gathering. This leads to unfavorable effects on automatic navigation, fruit recognition, and localization of harvesting robots in unstructured environments. Therefore, the first challenge in the application of multi-arm harvesting robots stems from the complexity of non-structured environments. The operation of multi-arm harvesting robots in complex environments is still the focus of research. Therefore, the future construction of structured environments is orchards. Greenhouse development is more conducive to harvesting robots carrying out harvesting operations, which is also a major direction for future development.
(2) Automated navigation. In order to realize automated harvesting, multi-arm harvesting robots are the key to development, but multi-arm harvesting robots need to flexibly cope with crops with different morphology and growth conditions, and the insufficiency of navigation technology restricts the robots’ flexibility, which may lead to a lower adaptability to different crops. Future development trends include integrating high-precision localization technology, introducing environment sensing and recognition technology, and deeply applying artificial intelligence and machine learning to improve the robots’ ability to navigate in complex agricultural environments and the level of intelligence, so as to achieve smarter automated harvesting.
(3) Recognition and localization. Multi-arm harvesting robots face challenges in identification and localization that make it difficult to simultaneously meet the challenges of lightweight, efficiency, and accuracy, and the use of equipment such as RGB-D cameras and LiDAR to improve localization accuracy increases costs and hinders promotion. Future trends should include optimizing lightweight deep learning models, exploring low-cost and high-precision sensor combination solutions, and controlling costs while improving accuracy and efficiency through multi-sensor fusion and adaptive technologies to promote the widespread use of robots.
(4) Robotic arm task planning. Although multi-arm harvesting robots avoid collisions between robotic arms through spatial division and harvesting sequence planning, this also limits the advantages of multi-arm collaboration, while the clustered distribution of fruits and vegetables leads to uneven task distribution and affects harvesting efficiency. Future trends include intelligent collaborative planning to reduce spatial conflicts and dynamic task scheduling systems to achieve uniform and efficient operation of the robotic arms, thus improving the overall performance.
(5) Robotic arms and end-effectors. The design of end-effectors will evolve to become more flexible and precise, allowing them to adapt to various crop types and shapes. They will need to operate at higher speeds and accuracy levels while minimizing damage to the crops. Optimizing the mechanics of robotic arms will also be crucial, particularly for operations in complex agricultural environments. Future robotic arms will require improved dynamic models to enhance mobility and responsiveness, enabling them to handle dynamic and irregular working conditions. Additionally, accurately assessing crop quality and position will be a key focus in gripper design. By incorporating sensor technologies, such as visual or force sensors, grippers will be able to better evaluate factors like ripeness, size, and location, thereby improving the precision of the harvesting process.

4. Conclusions

To address the issues related to multi-arm harvesting robots, this paper reviews the current state of research on multi-arm harvesting robots. Currently, harvesting robots are under-researched in terms of automatic navigation, and current navigation technology is developing rapidly, providing a wide range of algorithms and sensor technologies that can be used to enhance robotic auto-navigation capabilities, which is a major contribution to the realization of fully automated harvesting in agriculture.
Secondly, recognition and localization are always the core technology of robots, and complex natural environments always affect their efficiency and real-time performance. Uncertain light changes and complex occlusions can cause accuracy degradation. Target detection models based on deep learning effectively improve recognition accuracy, but they require huge data support and long training times. With the rapid development of depth cameras, they have become widely used in harvesting robots due to their ability to capture detailed image and point cloud data, which significantly enhances localization accuracy. Among all of the multi-arm harvesting robots, 76.5% utilize depth cameras. However, while this technology improves harvesting performance, it also increases costs, especially when combined with LiDAR.
Additionally, challenges such as low workspace traversal rates, difficulties in harvesting all fruits from a plant, and coordinating multiple robotic arms in shared spaces affect the overall performance of harvesting robots. The complexity of multi-arm task planning further complicates system design and implementation, limiting practical applications and commercialization potential.
Despite these obstacles, the future of multi-arm harvesting robots holds great promise. These robots have the potential to resolve key issues such as balancing precision and efficiency, addressing labor shortages, and significantly reducing agricultural harvesting costs. By integrating advanced technologies like sensor fusion and intelligent task planning, they ensure precise crop handling without compromising speed. Moreover, their ability to function autonomously in complex environments alleviates the growing reliance on manual labor, providing a sustainable solution for the agriculture industry. As they continue to evolve, multi-arm harvesting robots will play a pivotal role in transforming agriculture into a more efficient, cost-effective, and labor-independent industry.

Author Contributions

Conceptualization: W.M.; Writing—Original Draft Preparation: J.C. and H.L.; Article Ideas: W.M.; Methodology: J.L. and Y.Y.; Article Search: J.C. and H.L.; Article Collation: J.Q. and L.X.; Visualization: J.C.; Supervision: W.M.; Project Administration: W.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Chengdu Agricultural Science and Technology Center, grant numbers NASC2022KR08, NASC2021KR07, and NASC2023ST03.

Data Availability Statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Zhao, C.-J.; Fan, B.-B.; Li, J.; Feng, Q.-C. Agricultural robots: Technology progress, challenges and trends. Smart Agric. 2023, 5, 1–15. [Google Scholar] [CrossRef]
  2. Kawamura, N.; Namikawa, K.; Fujiura, T.; Ura, M. Study on agricultural robot. Mem. Coll. Agric. Kyoto Univ. 1986, 129, 29–46. [Google Scholar]
  3. Ji, C.; Feng, Q.; Yuan, T.; Tan, Y.; Li, W. Development and Performance Analysis on Cucumber Harvesting Robot System in Greenhouse. Robot 2011, 33, 726–730. [Google Scholar]
  4. Arad, B.; Balendonck, J.; Barth, R.; Ben-Shahar, O.; Edan, Y.; Hellstrom, T.; Hemming, J.; Kurtser, P.; Ringdahl, O.; Tielen, T.; et al. Development of a sweet pepper harvesting robot. J. Field Robot. 2020, 37, 1027–1039. [Google Scholar] [CrossRef]
  5. Feng, Q.; Zhao, C.; Li, T.; Chen, L.; Guo, X.; Xie, F.; Xiong, Z.; Chen, K.; Liu, C.; Yan, T. Design and test of a four-arm apple harvesting robot. Trans. Chin. Soc. Agric. Eng. 2023, 39, 25–33. [Google Scholar]
  6. Williams, H.A.M.; Jones, M.H.; Nejati, M.; Seabright, M.J.; Bell, J.; Penhall, N.D.; Barnett, J.J.; Duke, M.D.; Scarfe, A.J.; Ahn, H.S.; et al. Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms. Biosyst. Eng. 2019, 181, 140–156. [Google Scholar] [CrossRef]
  7. Jiang, Y.; Liu, J.; Wang, J.; Li, W.; Peng, Y.; Shan, H. Development of a dual-arm rapid grape-harvesting robot for horizontal trellis cultivation. Front. Plant Sci. 2022, 13, 881904. [Google Scholar] [CrossRef]
  8. Sepulveda, D.; Fernandez, R.; Navas, E.; Armada, M.; Gonzalez-De-Santos, P. Robotic Aubergine Harvesting Using Dual-Arm Manipulation. IEEE Access 2020, 8, 121889–121904. [Google Scholar] [CrossRef]
  9. Yoshida, T.; Onishi, Y.; Kawahara, T.; Fukao, T. Automated harvesting by a dual-arm fruit harvesting robot. ROBOMECH J. 2022, 9, 19. [Google Scholar] [CrossRef]
  10. Gong, L.; Wang, W.; Wang, T.; Liu, C. Robotic harvesting of the occluded fruits with a precise shape and position reconstruction approach. J. Field Robot. 2022, 39, 69–84. [Google Scholar] [CrossRef]
  11. Xiong, Y.; Ge, Y.; Grimstad, L.; From, P.J. An autonomous strawberry-harvesting robot: Design, development, integration, and field evaluation. J. Field Robot. 2020, 37, 202–224. [Google Scholar] [CrossRef]
  12. Li, T.; Xie, F.; Zhao, Z.; Zhao, H.; Guo, X.; Feng, Q. A multi-arm robot system for efficient apple harvesting: Perception, task plan and control. Comput. Electron. Agric. 2023, 211, 107979. [Google Scholar] [CrossRef]
  13. Ling, X.; Zhao, Y.; Gong, L.; Liu, C.; Wang, T. Dual-arm cooperation and implementing for robotic harvesting tomato using binocular vision. Robot. Auton. Syst. 2019, 114, 134–143. [Google Scholar] [CrossRef]
  14. Yu, X.; Fan, Z.; Wang, X.; Wan, H.; Wang, P.; Zeng, X.; Jia, F. A lab-customized autonomous humanoid apple harvesting robot. Comput. Electr. Eng. 2021, 96, 107459. [Google Scholar] [CrossRef]
  15. Rong, J.; Wang, P.; Yang, Q.; Huang, F. A Field-Tested Harvesting Robot for Oyster Mushroom in Greenhouse. Agronomy 2021, 11, 1210. [Google Scholar] [CrossRef]
  16. He, Z.; Ma, L.; Wang, Y.; Wei, Y.; Ding, X.; Li, K.; Cui, Y. Double-Arm Cooperation and Implementing for Harvesting Kiwifruit. Agriculture 2022, 12, 1763. [Google Scholar] [CrossRef]
  17. Yang, S.; Jia, B.; Yu, T.; Yuan, J. Research on Multiobjective Optimization Algorithm for Cooperative Harvesting Trajectory Optimization of an Intelligent Multiarm Straw-Rotting Fungus Harvesting Robot. Agriculture 2022, 12, 986. [Google Scholar] [CrossRef]
  18. Cui, Y.; Ma, L.; He, Z.; Zhu, Y.; Wang, Y.; Li, K. Design and Experiment of Dual Manipulators Parallel Harvesting Platform for Kiwifruit Based on Optimal Space. Trans. Chin. Soc. Agric. Mach. 2022, 53, 132–143. [Google Scholar]
  19. Wang, F.; Urquizo, R.C.; Roberts, P.; Mohan, V.; Newenham, C.; Ivanov, A.; Dowling, R. Biologically inspired robotic perception-action for soft fruit harvesting in vertical growing environments. Precis. Agric. 2023, 24, 1072–1096. [Google Scholar] [CrossRef]
  20. Navas, E.; Fernandez, R.; Sepulveda, D.; Armada, M.; Gonzalez-de-Santos, P. Modular Dual-Arm Robot for Precision Harvesting. In Proceedings of the 4th Iberian Robotics Conference (Robot)—Advances in Robotics, Porto, Portugal, 20–22 November 2019; pp. 148–158. [Google Scholar]
  21. Li, Y.; Feng, Q.; Zhang, Y.; Peng, C.; Zhao, C. Intermittent Stop-Move Motion Planning for Dual-Arm Tomato Harvesting Robot in Greenhouse Based on Deep Reinforcement Learning. Biomimetics 2024, 9, 105. [Google Scholar] [CrossRef]
  22. Au, C.; Barnett, J.; Lim, S.H.; Duke, M. Workspace analysis of Cartesian robot system for kiwifruit harvesting. Ind. Robot-Int. J. Robot. Res. Appl. 2020, 47, 503–510. [Google Scholar] [CrossRef]
  23. He, C.; Chen, Q.; Miao, Z.; Li, N.; Sun, T. Extracting the navigation path of an agricultural plant protection robot based on machine vision. In Proceedings of the 40th Chinese Control Conference (CCC), Shanghai, China, 26–28 July 2021; pp. 3576–3581. [Google Scholar]
  24. Feng, Q.; Wang, X.; Zheng, W.; Qiu, Q.; Jiang, K. A new strawberry harvesting robot for elevated-trough culture. Int. J. Agric. Biol. Eng. 2012, 5, 1–8. [Google Scholar]
  25. Chen, J.; Song, J.; Guan, Z.; Lian, Y. Measurement of the distance from grain divider to harvesting boundary based on dynamic regions of interest. Int. J. Agric. Biol. Eng. 2021, 14, 226–232. [Google Scholar] [CrossRef]
  26. Chen, L.-B.; Huang, X.-R.; Chen, W.-H. Design and Implementation of an Artificial Intelligence of Things-Based Autonomous Mobile Robot System for Pitaya Harvesting. IEEE Sens. J. 2023, 23, 13220–13235. [Google Scholar] [CrossRef]
  27. Aguiar, A.S.; dos Santos, F.N.; Cunha, J.B.; Sobreira, H.; Sousa, A.J. Localization and Mapping for Robots in Agriculture and Forestry: A Survey. Robotics 2020, 9, 97. [Google Scholar] [CrossRef]
  28. Wang, Y.; He, Z.; Cao, D.; Ma, L.; Li, K.; Jia, L.; Cui, Y. Coverage path planning for kiwifruit picking robots based on deep reinforcement learning. Comput. Electron. Agric. 2023, 205, 107593. [Google Scholar] [CrossRef]
  29. Radocaj, D.; Plascak, I.; Jurisic, M. Global Navigation Satellite Systems as State-of-the-Art Solutions in Precision Agriculture: A Review of Studies Indexed in the Web of Science. Agriculture 2023, 13, 1417. [Google Scholar] [CrossRef]
  30. Bai, Y.; Zhang, B.; Xu, N.; Zhou, J.; Shi, J.; Diao, Z. Vision-based navigation and guidance for agricultural autonomous vehicles and robots: A review. Comput. Electron. Agric. 2023, 205, 107584. [Google Scholar] [CrossRef]
  31. Vazquez-Arellano, M.; Griepentrog, H.W.; Reiser, D.; Paraforos, D.S. 3-D Imaging Systems for Agricultural Applications—A Review. Sensors 2016, 16, 618. [Google Scholar] [CrossRef]
  32. Wen, W.; Zhang, G.; Hsu, L.-T. Object-Detection-Aided GNSS and Its Integration with Lidar in Highly Urbanized Areas. IEEE Intell. Transp. Syst. Mag. 2020, 12, 53–69. [Google Scholar] [CrossRef]
  33. Jiang, A.; Ahamed, T. Navigation of an Autonomous Spraying Robot for Orchard Operations Using LiDAR for Tree Trunk Detection. Sensors 2023, 23, 4808. [Google Scholar] [CrossRef] [PubMed]
  34. Zheng, Z.; Hu, Y.; Li, X.; Huang, Y. Autonomous navigation method of jujube catch-and-shake harvesting robot based on convolutional neural networks. Comput. Electron. Agric. 2023, 215, 108469. [Google Scholar] [CrossRef]
  35. Halim, M.H.A.; Subrata, D.M.; Widodo, S.; Solahudin, M. Development of deep learning models on the navigation system for assistant harvesting robot. IOP Conf. Ser. Earth Environ. Sci. 2022, 1038, 012046. [Google Scholar] [CrossRef]
  36. Yang, Z.; Jiang, Q.; Duan, J.; Jin, M.; Fu, H.; Xu, X. A method for measuring banana pseudo-stem phenotypic parameters based on handheld mobile LiDAR and IMU fusion. Front. Plant Sci. 2024, 15, 1369501. [Google Scholar] [CrossRef] [PubMed]
  37. Rovira-Mas, F.; Saiz-Rubio, V.; Cuenca-Cuenca, A. Augmented Perception for Agricultural Robots Navigation. IEEE Sens. J. 2021, 21, 11712–11727. [Google Scholar] [CrossRef]
  38. Xiao, F.; Wang, H.; Li, Y.; Cao, Y.; Lv, X.; Xu, G. Object Detection and Recognition Techniques Based on Digital Image Processing and Traditional Machine Learning for Fruit and Vegetable Harvesting Robots: An Overview and Review. Agronomy 2023, 13, 639. [Google Scholar] [CrossRef]
  39. Xiao, F.; Wang, H.; Xu, Y.; Zhang, R. Fruit Detection and Recognition Based on Deep Learning for Automatic Harvesting: An Overview and Review. Agronomy 2023, 13, 1625. [Google Scholar] [CrossRef]
  40. Jia WeiKuan, J.W.; Zhao Dean, Z.D.; Ruan ChengZhi, R.C.; Shen Tian, S.T.; Chen Yu, C.Y.; Ji Wei, J.W. De-noising algorithm of night vision image for apple harvesting robot. Trans. Chin. Soc. Agric. Eng. 2015, 31, 219–226. [Google Scholar]
  41. Hou, G.; Chen, H.; Jiang, M.; Niu, R. An Overview of the Application of Machine Vision in Recognition and Localization of Fruit and Vegetable Harvesting Robots. Agriculture 2023, 13, 1814. [Google Scholar] [CrossRef]
  42. Tang, Y.; Qiu, J.; Zhang, Y.; Wu, D.; Cao, Y.; Zhao, K.; Zhu, L. Optimization strategies of fruit detection to overcome the challenge of unstructured background in field orchard environment: A review. Precis. Agric. 2023, 24, 1183–1219. [Google Scholar] [CrossRef]
  43. Wang, H.; Duan, W.; Zhao, H.; Yue, Y. Research of Night Vision Image Denoising Method Based on the Improved FastICA. In Proceedings of the IEEE International Conference on Mechatronics and Automation (ICMA), Takamatsu, Japan, 6–9 August 2017; pp. 332–336. [Google Scholar]
  44. Ji, W.; Qian, Z.; Xu, B.; Zhao, D. A nighttime image enhancement method based on Retinex and guided filter for object recognition of apple harvesting robot. Int. J. Adv. Robot. Syst. 2018, 15, 1729881417753871. [Google Scholar] [CrossRef]
  45. Liu, X.; Fan, C.; Li, J.; Gao, Y.; Zhang, Y.; Yang, Q. Identification Method of Strawberry Based on Convolutional Neural Network. Trans. Chin. Soc. Agric. Mach. 2020, 51, 237–244. [Google Scholar]
  46. Qi, C.; Gao, J.; Chen, K.; Shu, L.; Pearson, S. Tea Chrysanthemum Detection by Leveraging Generative Adversarial Networks and Edge Computing. Front. Plant Sci. 2022, 13, 850606. [Google Scholar] [CrossRef] [PubMed]
  47. Singh, N.; Tewari, V.K.; Biswas, P.K.; Pareek, C.M.; Dhruw, L.K. Image processing algorithms for in-field cotton boll detection in natural lighting conditions. Artif. Intell. Agric. 2021, 5, 142–156. [Google Scholar] [CrossRef]
  48. Liu, X.; Zhao, D.; Jia, W.; Ruan, C.; Tang, S.; Shen, T. A method of segmenting apples at night based on color and position information. Comput. Electron. Agric. 2016, 122, 118–123. [Google Scholar] [CrossRef]
  49. Fu, L.; Wang, B.; Cui, Y.; Su, S.; Gejima, Y.; Kobayashi, T. Kiwifruit recognition at nighttime using artificial lighting based on machine vision. Int. J. Agric. Biol. Eng. 2015, 8, 52–59. [Google Scholar] [CrossRef]
  50. Lin, G.; Tang, Y.; Zou, X.; Cheng, J.; Xiong, J. Fruit detection in natural environment using partial shape matching and probabilistic Hough transform. Precis. Agric. 2020, 21, 160–177. [Google Scholar] [CrossRef]
  51. Xiong, J.; Lin, R.; Liu, Z.; He, Z.; Yang, Z.G.; Bu, R. Visual Technology of Picking Robot to Detect Litchi at Nighttime under Natural Environment. Trans. Chin. Soc. Agric. Mach. 2017, 48, 28–34. [Google Scholar]
  52. Zhang, Y.; Liu, M.; Gong, J.; Lan, Y. Apple recognition based on two-level segmentation and region-marked gradient Hough circle transform. Trans. Chin. Soc. Agric. Eng. 2022, 38, 110–121. [Google Scholar]
  53. Wang, C.; Tang, Y.; Zou, X.; SiTu, W.; Feng, W. A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot. Optik 2017, 131, 626–631. [Google Scholar] [CrossRef]
  54. Sun, S.; Jiang, M.; He, D.; Long, Y.; Song, H. Recognition of green apples in an orchard environment by combining the GrabCut model and Ncut algorithm. Biosyst. Eng. 2019, 187, 201–213. [Google Scholar] [CrossRef]
  55. Liang, C.; Xiong, J.; Zheng, Z.; Zhong, Z.; Li, Z.; Chen, S.; Yang, Z. A visual detection method for nighttime litchi fruits and fruiting stems. Comput. Electron. Agric. 2020, 169, 105192. [Google Scholar] [CrossRef]
  56. Zhang, G.; Tian, Y.; Yin, W.; Zheng, C. An Apple Detection and Localization Method for Automated Harvesting under Adverse Light Conditions. Agriculture 2024, 14, 485. [Google Scholar] [CrossRef]
  57. Chu, P.; Li, Z.; Lammers, K.; Lu, R.; Liu, X. Deep learning-based apple detection using a suppression mask R-CNN. Pattern Recognit. Lett. 2021, 147, 206–211. [Google Scholar] [CrossRef]
  58. Feng, Y.; Ma, W.; Tan, Y.; Yan, H.; Qian, J.; Tian, Z.; Gao, A. Approach of Dynamic Tracking and Counting for Obscured Citrus in Smart Orchard Based on Machine Vision. Appl. Sci. 2024, 14, 1136. [Google Scholar] [CrossRef]
  59. Jana, S.; Basak, S.; Parekh, R. Automatic Fruit Recognition from Natural Images using Color and Texture Features. In Proceedings of the 2nd International Conference on Devices for Integrated Circuit (DevIC), Kalyani, India, 23–24 March 2017; pp. 620–624. [Google Scholar]
  60. Li, H.; Wang, K.; Cao, Q.; Yin, J. Tomato targets extraction and matching based on computer vision. Trans. Chin. Soc. Agric. Eng. 2012, 28, 168–172. [Google Scholar]
  61. Stein, M.; Bargoti, S.; Underwood, J. Image Based Mango Fruit Detection, Localisation and Yield Estimation Using Multiple View Geometry. Sensors 2016, 16, 1915. [Google Scholar] [CrossRef]
  62. Yang, J.; Deng, H.; Zhang, Y.; Zhou, Y.; Miao, T. Application of amodal segmentation for shape reconstruction and occlusion recovery in occluded tomatoes. Front. Plant Sci. 2024, 15, 1376138. [Google Scholar] [CrossRef]
  63. Tang, Y.; Chen, M.; Wang, C.; Luo, L.; Li, J.; Lian, G.; Zou, X. Recognition and Localization Methods for Vision-Based Fruit Picking Robots: A Review. Front. Plant Sci. 2020, 11, 510. [Google Scholar] [CrossRef]
  64. Hou, G.; Chen, H.; Ma, Y.; Jiang, M.; Hua, C.; Jiang, C.; Niu, R. An occluded cherry tomato recognition model based on improved YOLOv7. Front. Plant Sci. 2023, 14, 1260808. [Google Scholar] [CrossRef]
  65. Chu, P.; Li, Z.; Zhang, K.; Chen, D.; Lammers, K.; Lu, R. O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environments. Smart Agric. Technol. 2023, 5, 100284. [Google Scholar] [CrossRef]
  66. Rathore, D.; Divyanth, L.G.; Reddy, K.L.S.; Chawla, Y.; Buragohain, M.; Soni, P.; Machavaram, R.; Hussain, S.Z.; Ray, H.; Ghosh, A. A Two-Stage Deep-Learning Model for Detection and Occlusion-Based Classification of Kashmiri Orchard Apples for Robotic Harvesting. J. Biosyst. Eng. 2023, 48, 242–256. [Google Scholar] [CrossRef]
  67. Sun, T.; Zhang, W.; Miao, Z.; Zhang, Z.; Li, N. Object localization methodology in occluded agricultural environments through deep learning and active sensing. Comput. Electron. Agric. 2023, 212, 108141. [Google Scholar] [CrossRef]
  68. Meng, Z.; Du, X.; Xia, J.; Ma, Z.; Zhang, T. Real-time statistical algorithm for cherry tomatoes with different ripeness based on depth information mapping. Comput. Electron. Agric. 2024, 220, 108900. [Google Scholar] [CrossRef]
  69. Qi, Z.; Zhang, W.; Yuan, T.; Rong, J.; Hua, W.; Zhang, Z.; Deng, X.; Zhang, J.; Li, W. An improved framework based on tracking-by-detection for simultaneous estimation of yield and maturity level in cherry tomatoes. Measurement 2024, 226, 114117. [Google Scholar] [CrossRef]
  70. Wang, Y.; Tao, Z.-S.; Shi, X.-Y.; Wu, Y.; Wu, H. Apple target detection method with different ripeness based on improved YOLOv5s. J. Nanjing Agric. Univ. 2024, 47, 602–611. [Google Scholar] [CrossRef]
  71. Liu, J.; Zhao, G.; Liu, S.; Liu, Y.; Yang, H.; Sun, J.; Yan, Y.; Fan, G.; Wang, J.; Zhang, H. New Progress in Intelligent Picking: Online Detection of Apple Maturity and Fruit Diameter Based on Machine Vision. Agronomy 2024, 14, 721. [Google Scholar] [CrossRef]
  72. Fu, L.; Gao, F.; Wu, J.; Li, R.; Karkee, M.; Zhang, Q. Application of consumer RGB-D cameras for fruit detection and localization in field: A critical review. Comput. Electron. Agric. 2020, 177, 105687. [Google Scholar] [CrossRef]
  73. Baeten, J.; Donné, K.; Boedrij, S.; Beckers, W.; Claesen, E. Autonomous Fruit Picking Machine: A Robotic Apple Harvester. In Field and Service Robotics: Results of the 6th International Conference, Chamonix, France, 9–12 July 2007; Laugier, C., Siegwart, R., Eds.; Springer: Berlin/Heidelberg, Germany, 2008; pp. 531–539. [Google Scholar]
  74. Xiang, R.; Ying, Y.; Jiang, H.; Peng, Y. Localization of tomatoes based on binocular stereo vision. Trans. Chin. Soc. Agric. Eng. 2012, 28, 161–167. [Google Scholar]
  75. Gao, C.; Jiang, H.; Liu, X.; Li, H.; Wu, Z.; Sun, X.; He, L.; Mao, W.; Majeed, Y.; Li, R.; et al. Improved binocular localization of kiwifruit in orchard based on fruit and calyx detection using YOLOv5x for robotic picking. Comput. Electron. Agric. 2024, 217, 108621. [Google Scholar] [CrossRef]
  76. Hu, T.; Wang, W.; Gu, J.; Xia, Z.; Zhang, J.; Wang, B. Research on Apple Object Detection and Localization Method Based on Improved YOLOX and RGB-D Images. Agronomy 2023, 13, 1816. [Google Scholar] [CrossRef]
  77. Liu, T.; Kang, H.; Chen, C. ORB-Livox: A real-time dynamic system for fruit detection and localization. Comput. Electron. Agric. 2023, 209, 107834. [Google Scholar] [CrossRef]
  78. Neupane, C.; Koirala, A.; Wang, Z.; Walsh, K.B. Evaluation of Depth Cameras for Use in Fruit Localization and Sizing: Finding a Successor to Kinect v2. Agronomy 2021, 11, 1780. [Google Scholar] [CrossRef]
  79. Li, T.; Feng, Q.; Qiu, Q.; Xie, F.; Zhao, C. Occluded Apple Fruit Detection and Localization with a Frustum-Based Point-Cloud-Processing Approach for Robotic Harvesting. Remote Sens. 2022, 14, 482. [Google Scholar] [CrossRef]
  80. Kang, H.; Wang, X.; Chen, C. Accurate fruit localisation using high resolution LiDAR-camera fusion and instance segmentation. Comput. Electron. Agric. 2022, 203, 107450. [Google Scholar] [CrossRef]
  81. Li, Y.; Feng, Q.; Li, T.; Xie, F.; Liu, C.; Xiong, Z. Advance of Target Visual Information Acquisition Technology for Fresh Fruit Robotic Harvesting: A Review. Agronomy 2022, 12, 1336. [Google Scholar] [CrossRef]
  82. Li, T.; Fang, W.; Zhao, G.; Gao, F.; Wu, Z.; Li, R.; Fu, L.; Dhupia, J. An improved binocular localization method for apple based on fruit detection using deep learning. Inf. Process. Agric. 2023, 10, 276–287. [Google Scholar] [CrossRef]
  83. Jang, M.; Hwang, Y. Tomato pose estimation using the association of tomato body and sepal. Comput. Electron. Agric. 2024, 221, 108961. [Google Scholar] [CrossRef]
  84. Yu, Y.; Zhang, K.; Liu, H.; Yang, L.; Zhang, D. Real-Time Visual Localization of the Picking Points for a Ridge-Planting Strawberry Harvesting Robot. IEEE Access 2020, 8, 116556–116568. [Google Scholar] [CrossRef]
  85. Coll-Ribes, G.; Torres-Rodriguez, I.J.; Grau, A.; Guerra, E.; Sanfeliu, A. Accurate detection and depth estimation of table grapes and peduncles for robot harvesting, combining monocular depth estimation and CNN methods. Comput. Electron. Agric. 2023, 215, 108362. [Google Scholar] [CrossRef]
  86. Wessen, J.; Carlsson, M.; Schulte, C.; Flener, P.; Pecora, F.; Matskin, M. A constraint programming model for the scheduling and workspace layout design of a dual-arm multi-tool assembly robot. Constraints 2023, 28, 71–104. [Google Scholar] [CrossRef]
  87. Mann, M.P.; Zion, B.; Shmulevich, I.; Rubinstein, D.; Linker, R. Combinatorial Optimization and Performance Analysis of a Multi-arm Cartesian Robotic Fruit Harvester-Extensions of Graph Coloring. J. Intell. Robot. Syst. 2016, 82, 399–411. [Google Scholar] [CrossRef]
  88. Xiong, Z.; Feng, Q.; Li, T.; Xie, F.; Liu, C.; Liu, L.; Guo, X.; Zhao, C. Dual-Manipulator Optimal Design for Apple Robotic Harvesting. Agronomy 2022, 12, 3128. [Google Scholar] [CrossRef]
  89. Ji, W.; Zhang, T.; Xu, B.; He, G. Apple recognition and picking sequence planning for harvesting robot in a complex environment. J. Agric. Eng. 2024, 55, 1549. [Google Scholar] [CrossRef]
  90. Zhang, H.; Li, X.; Wang, L.; Liu, D.; Wang, S. Construction and Optimization of a Collaborative Harvesting System for Multiple Robotic Arms and an End-Picker in a Trellised Pear Orchard Environment. Agronomy 2024, 14, 80. [Google Scholar] [CrossRef]
  91. Barnett, J.; Duke, M.; Au, C.K.; Lim, S.H. Work distribution of multiple Cartesian robot arms for kiwifruit harvesting. Comput. Electron. Agric. 2020, 169, 105202. [Google Scholar] [CrossRef]
  92. Li, T.; Xie, F.; Qiu, Q.; Feng, Q. Multi-Arm Robot Task Planning for Fruit Harvesting Using Multi-Agent Reinforcement Learning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA, 1–5 October 2023; pp. 4176–4183. [Google Scholar]
Agronomy 14 02209 g001
Figure 1. Two types of harvesting robots: first type (a,b); second type (c,d). (a) Tomato harvesting robot [13]; (b) aubergine harvesting robot [8]; (c) apple harvesting robot [12]; (d) mushroom harvesting robot [15].
Figure 1. Two types of harvesting robots: first type (a,b); second type (c,d). (a) Tomato harvesting robot [13]; (b) aubergine harvesting robot [8]; (c) apple harvesting robot [12]; (d) mushroom harvesting robot [15].
Agronomy 14 02209 g001
Agronomy 14 02209 g002
Figure 2. Different navigation techniques: (a) interline navigation technique [24]; (b) boundary-following technique [25]; (c) SLAM technique [26]; (d) deep learning navigation technique [28].
Figure 2. Different navigation techniques: (a) interline navigation technique [24]; (b) boundary-following technique [25]; (c) SLAM technique [26]; (d) deep learning navigation technique [28].
Agronomy 14 02209 g002
Agronomy 14 02209 g003
Figure 3. Different types of occlusion: (a) non-occlusion; (b) fruit occlusion; (c) leaf occlusion; (d) branch occlusion.
Figure 3. Different types of occlusion: (a) non-occlusion; (b) fruit occlusion; (c) leaf occlusion; (d) branch occlusion.
Agronomy 14 02209 g003
Agronomy 14 02209 g004
Figure 4. Soft fruit stem localization: (a) tomato fruit stem position and pose information [83]. The arrows depict the tomato pose.; (b) strawberry fruit stem harvesting point [84]; (c) grape fruit stem segmentation [85].
Figure 4. Soft fruit stem localization: (a) tomato fruit stem position and pose information [83]. The arrows depict the tomato pose.; (b) strawberry fruit stem harvesting point [84]; (c) grape fruit stem segmentation [85].
Agronomy 14 02209 g004
Agronomy 14 02209 g005
Figure 5. Harvesting steps for kiwifruit trellis [22]: (a) Cartesian robotic system; (b) articulated robotic system.
Figure 5. Harvesting steps for kiwifruit trellis [22]: (a) Cartesian robotic system; (b) articulated robotic system.
Agronomy 14 02209 g005
Agronomy 14 02209 g006
Figure 6. Multi-arm robot workspace analysis: (a) articulated system robot [7]; (b) Cartesian system robot [12].
Figure 6. Multi-arm robot workspace analysis: (a) articulated system robot [7]; (b) Cartesian system robot [12].
Agronomy 14 02209 g006
Agronomy 14 02209 g007
Figure 7. Three multi-arm harvesting robot harvesting sequences: (a) strawberry harvesting sequence [11]; (b) mushroom harvesting sequence [15]; (c) grape harvesting sequence [7].
Figure 7. Three multi-arm harvesting robot harvesting sequences: (a) strawberry harvesting sequence [11]; (b) mushroom harvesting sequence [15]; (c) grape harvesting sequence [7].
Agronomy 14 02209 g007
Table 1. Comparison of different multi-arm harvesting robots.
Table 1. Comparison of different multi-arm harvesting robots.
Equipment PictureSceneTargetDetectionHarvest Success RateHarvest
Efficiency		FeatureChallengeReference
Agronomy 14 02209 i001OrchardKiwifruit89.6%51.0%5.5 s/fruitMulti-arm robot, deep network-based vision systemAccurate recognition in complex environments[6]
Agronomy 14 02209 i002OrchardApple95.8%~10 s/fruitDual-arm robot design, automation of fruit harvestingComplex non-structured environments, crash risk, and scalability and adaptability[9]
Agronomy 14 02209 i003GreenhouseTomato~73.04%11 s/fruitReconstruction of spatial features of occluded targetsAccuracy of detection and reconstruction of masked fruits[10]
Agronomy 14 02209 i004FarmStrawberry~Clustering below 83.8%4.6 s/fruitAutonomous continuous harvestingFlexibility of automated navigation, harvesting capability for non-structured environments, and flexibility of grippers[11]
Agronomy 14 02209 i005OrchardAppleAP below 70%71.28%~80.45%5.8~6.7 s/fruitMulti-arm harvesting robots combining sensing and task planning and control techniquesPositioning accuracy, operational efficiency, and collision risk[12]
Agronomy 14 02209 i006InteriorTomato96%87.5%29 s/fruitDual-arm collaboration, binocular vision sensorsReal-time control and precision of arm movements in dual-arm collaboration[13]
Agronomy 14 02209 i007InteriorApple82.5%72%14.6 s/fruitHumanoid robotRecognition and harvesting success, harvesting speed, and environmental adaptation[14]
Agronomy 14 02209 i008OrchardGrape88%83%9 s/plantAddressing high grape breakage rates and workspace analysisThe contradiction between high-speed harvesting and breakage rate and the effect of identification in complex environments[7]
Agronomy 14 02209 i009InteriorAubergine88.35%91.67%26 s/fruitDual-arm operation to handle occlusionRobotic arm coordination and harvest time[8]
Agronomy 14 02209 i010GreenhouseMushroomRecognition rate 95%86.8%8.85 s/unitFully automated multi-arm mushroom harvesting robotEfficient operation and system scalability in complex environments[15]
Agronomy 14 02209 i011OrchardsKiwifruitRecognition rate 92.16%86.67%~Area planning and collision detectionCollision detection efficiency and accuracy and real-time system and efficiency[16]
Agronomy 14 02209 i012factoryStraw-Rotting Fungus~97%3 s/unitMulti-arm collaboration to optimize harvest trajectoriesComplexity of multi-objective optimization algorithms and accuracy in practice[17]
Agronomy 14 02209 i013InteriorTomato 82.10%5.86 s/fruitImproved workspace utilizationAccurate spatial optimization and efficiency in operation[18]
Agronomy 14 02209 i014FarmStrawberry~~~Stem harvestingAdaptability and efficient operation in complex environments[19]
Agronomy 14 02209 i015~~~~~Modular design for different cropsComplex detection algorithms and different harvesting operations[20]
Agronomy 14 02209 i016GreenhouseTomato~~~Deep reinforcement learning, intermittent motion planningEnvironmental complexity and real-time requirements[21]
Agronomy 14 02209 i017OrchardKiwifruit~~~Differential analysis of workspace geometry between articulated and Cartesian armsPrecise workspace analysis and optimization[22]
~ Represents data not queried in the paper.
Table 2. Different methods to improve detection accuracy.
Table 2. Different methods to improve detection accuracy.
MethodAdvantageDisadvantageReference
Image preprocessingEnhances image quality and details for the overall improvement of image qualityPoor real-time performance[43,44,45,46]
Color basedCan effectively distinguish the target color and improve detection accuracyReduced effectiveness when colors are inconsistent or lighting is complex[47,48]
Shape and texture basedBetter extraction of shape and texture features and suitable for regular shape object detectionLess effective when dealing with irregular shapes and complex backgrounds[49,50]
Traditional segmentation methodsEffective target segmentation in complex scenes, removing background interference and improving the accuracy of target contour detectionSegmentation is dependent on initial parameter selection and is sensitive to noise and shadows[51,52,53,54]
Deep learning basedStrong feature expression capability and adaptable to complex environments
High accuracy detection and adaptable		Requires large amount of training data, a complex model, and long training time[55,56,57]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Chen, J.; Ma, W.; Liao, H.; Lu, J.; Yang, Y.; Qian, J.; Xu, L. Balancing Accuracy and Efficiency: The Status and Challenges of Agricultural Multi-Arm Harvesting Robot Research. Agronomy 2024, 14, 2209. https://doi.org/10.3390/agronomy14102209

AMA Style

Chen J, Ma W, Liao H, Lu J, Yang Y, Qian J, Xu L. Balancing Accuracy and Efficiency: The Status and Challenges of Agricultural Multi-Arm Harvesting Robot Research. Agronomy. 2024; 14(10):2209. https://doi.org/10.3390/agronomy14102209

Chicago/Turabian Style

Chen, Jiawei, Wei Ma, Hongsen Liao, Junhua Lu, Yuxin Yang, Jianping Qian, and Lijia Xu. 2024. "Balancing Accuracy and Efficiency: The Status and Challenges of Agricultural Multi-Arm Harvesting Robot Research" Agronomy 14, no. 10: 2209. https://doi.org/10.3390/agronomy14102209

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop