Applications of Multi-Robotic Arms to Assist Agricultural Production: A Review

Abstract

With the modernization of agricultural production, single-arm machine systems in agriculture are unable to meet the needs of future agricultural development. In order to further improve agricultural operation efficiency, the collaborative operation of multi-robotic arms has become a hot topic in current research. This paper focuses on the task allocation problem in the collaborative operation of agricultural multi-robotic arms and summarizes the main algorithms currently used, including the genetic algorithm, particle swarm algorithm, etc., in terms of the aspects of work area division and task planning order. On this basis, further analysis is conducted on the path planning problem of agricultural multi-robotic arms. This paper summarizes the key technologies used in current research, including heuristic algorithms, fast search rapidly exploring random trees, reinforcement learning algorithms, etc., and focuses on reviewing the present applications of cutting-edge reinforcement learning algorithms in agricultural robotic arms. In summary, the agricultural multi-robot arms system can help with agricultural mechanization and intelligent production.

1. Introduction

With the promotion of modernization and automation in agricultural production, robotic arms have been widely used in various fields of agricultural production due to their precise operational capabilities and stable operation advantages [1,2,3,4,5]. Driven by technologies such as sensors, computers, and artificial intelligence, agricultural robotic arms have made positive progress in various aspects, such as information perception, precise control, and job execution [6,7,8]. In order to further break through the efficiency bottleneck of single robotic arms, more researchers are focusing on combining multi-robotic arms to jointly complete operational tasks. And, in related fields such as agricultural picking and planting, it has been proven that the collaborative operation of multi-robotic arms in agriculture can improve the efficiency of agricultural operations. The 24-arm robotic arm designed by Agrobot and the multi-robot arms of FFR have increased efficiency by up to 8 times compared to manual labor, and the single-fruit picking time has been compressed to less than 1.5 s [9,10,11]. Although agricultural multi-robotic arms have significant advantages in improving operational efficiency, they face multiple technological challenges as the number of robotic arms increases, including task allocation and path planning between multiple robots [12,13].

In order to ensure complete coverage of the operating range by multi-robotic arms, it is inevitable that the working space of the robotic arms will overlap, which increases the possibility of collisions between the robotic arms. A study by Luo et al. [14] showed that in the overlapping area of multiple arms, robot arms are prone to collision. In addition, the scheduling problem between robotic arms is also a key factor affecting the efficiency of multi-arm collaborative operations. In actual agricultural work environments, task distribution is often uneven. For example, in the field of fruit harvesting, the distribution of fruits is often limited by the canopy structure of fruit trees [15,16,17]. Meanwhile, the different traversal orders of multi-robotic arms on the target points of the operation could also lead to significant differences in operation time. Kurtzer et al. [18] planned the sequence of tasks for harvesting robots and were able to reduce time costs by 12%. Therefore, in order to improve the overall efficiency of agricultural multi-robotic arm systems, it is particularly important to construct a scientific and efficient task allocation strategy for agricultural multi-robotic arms.

The task allocation problem of multi-robotic arms is a complex non-deterministic polynomial (NP) that can be solved using operations research methods [19]. But it is difficult to solve this problem using this method within a limited time. Many scholars believe that heuristic algorithms are effective collaborative evolutionary algorithms for multi-robot task allocation and scheduling in smart farms [20]. And with the development of machine vision, the application of reinforcement learning algorithms has provided new options for solving task allocation problems [21,22]. However, the effective allocation and scheduling of agricultural multi-robotic arms remains a major challenge at present.

In order to achieve reasonable task allocation for multi-robotic arms, researchers have further conducted in-depth research on the path planning problem of agricultural multi-robotic arms. The path planning technology of agricultural robotic arms, as one of the core technologies in the field of agricultural automation, is the underlying support for achieving precision agriculture and smart agriculture [23,24,25]. A reasonable path planning algorithm requires the robotic arm to effectively avoid obstacles and find the shortest feasible path under task constraints. The traditional A * algorithm and Dijkstra algorithm perform well in structured tasks but are difficult to adapt to the complex dynamic environment of agriculture. Many scholars have used intelligent heuristic algorithms such as the artificial potential field method, genetic algorithm, and ant colony algorithm to solve the path planning problem of multi-robotic arms in agricultural scenarios [26,27,28,29]. The sampling-based path planning method has become a commonly used method for solving complex multi-robotic arm path planning due to its adaptability to high-dimensional space [30,31]. Specifically, path planning algorithms based on reinforcement learning have become a cutting-edge direction in the field of path planning due to their excellent environmental perception and self-determination abilities, providing new technical support for the application of agricultural multi-robotic arms [32].

In response, this paper extensively collects and analyzes the relevant literature to explore the question of how to improve the task allocation efficiency of agricultural multi-robotic arms based on different types of robotic arms, as well as how to avoid resource waste and task conflicts among multi-robotic arms. A comprehensive review of path planning algorithms for multi-robotic arms is conducted, with a deep analysis of the advantages and disadvantages of current algorithms. Especially, the application of reinforcement learning algorithms in multi-robotic arm path planning provides a theoretical reference for promoting the application of agricultural multi-robotic arms in agricultural production.

This paper is structured as follows: Section 2 describes the task allocation problem of agricultural multi-robotic arms collaborative operation, including work area division, operation sequence, and planning sequence issues. Section 3 summarizes the application of intelligent heuristic algorithms in agricultural multi-robotic arms path planning, including obstacle avoidance strategies and trajectory optimization. Section 4 summarizes the prospects and applications of reinforcement learning algorithms in multi-arm path planning. Finally, a summary of this paper is provided in Section 5.

2. Task Allocation for Agricultural Multi-Robotic Arms

Realizing scientific task allocation is a prerequisite for ensuring efficient collaborative operation of agricultural multi-robotic arms systems. Therefore, it has attracted extensive attention from researchers. The current task allocation of multi-robotic arms in agriculture mainly faces the following difficulties. Firstly, due to the differences in motion modes and structures of different types of robotic arms, how to ensure complete coverage of the agricultural operation range by the multi-robotic arms system is the primary issue to ensure the comprehensive completion of tasks [33,34,35]. Secondly, the absence of efficient harvesting strategies inherently leads to a heightened risk of collisions, which can severely disrupt the normal operation of collaborative workflows. [36]. In addition, how to reasonably schedule multi-robotic arms systems and achieve efficient traversal of task target points are the core issues that restrict the efficient development of multi-robotic arms collaboration [21].

This paper provides a systematic overview of the structural characteristics of various types of robotic arms. It summarizes the approaches used to categorize different workspaces into indoor and outdoor tasks, as well as collaborative work strategies. Additionally, the paper reviews and analyzes current mainstream task allocation algorithms in detail. These algorithms mainly include reinforcement learning algorithms and heuristic algorithms, such as genetic algorithms and ant colony algorithms.

2.1. Division of Working Areas for Multi-Robotic Arms

In the action of agricultural multi-robotic arms collaborative operation, the multi-robotic arms system should scientifically divide the working area of the robotic arms based on the terrain characteristics and crop distribution of the agricultural environment. By dividing the area reasonably, it is possible to effectively reduce blind spots in the operation and avoid repetitive work of the robotic arms [37]. Meanwhile, due to significant differences in structure, degree of freedom of motion, and working radius among different types of robotic arms. Therefore, it is necessary to select the appropriate type of robotic arm for operation based on specific usage scenarios.

2.1.1. Robotic Arm Classification

As the core component of agricultural intelligent equipment, agricultural robotic arms can be divided into Cartesian robotic arms, articulated robotic arms, and cylindrical robotic arms according to the different crops harvested and planting methods, as shown in Table 1 [38].

The Cartesian robot arm is composed of three mutually perpendicular linear motion axes, and the workspace is composed of three motion axes, forming a spatial right-angle prism. On the one hand, the robot arm moves along three linear motion axes, which have high linear accuracy. Zahid et al. [39] used a three-degree-of-freedom Cartesian robot arm to cut agricultural branches. Thanks to the advantages of a Cartesian robot arm with high-precision positioning, it effectively meets the requirements of cutting points and angles when cutting branches. On the other hand, Cartesian robot arms rely on fixed tracks and frames and cannot adapt to high-density or unstructured environments. Zhang et al. [40] developed a four-degree-of-freedom robot arm with an infrared laser. In the older and lush orchard, the success rate is only 65.2%. The cylindrical agricultural robotic arm is designed based on a cylindrical coordinate system, with a cylindrical workspace that can perform rotational and linear telescopic movements, making it suitable for vertical or radial motion scenarios. Yoshida et al. [41] designed a double cylindrical robotic arm for picking operations in V-shaped trellis orchards, which covers fruits of different heights through telescopic motion. However, in more complex orchard environments, the linear extension and contraction of cylindrical robotic arms are susceptible to obstacles and lack obstacle avoidance capabilities.

Unlike the two types of robotic arms mentioned above, articulated robotic arms are more flexible and widely used in agriculture. Articulated robotic arms are typically composed of multiple transmission joints, as well as the workspace is typically composed of multiple spherical regions. The functions performed by the robotic arm vary depending on the number of rotating joints. Most low-degree-of-freedom robotic arms have simple control and high stability and are commonly used in crop planting, field management, and other applications. Wei et al. [42] designed a three-degree-of-freedom articulated robot arm to replace artificial wheat fertilization and used an angle sensor to control the joint angle, with a conveying efficiency of 0.833 kg/s. Due to its high flexibility and adaptability to complex environments, the high degree of freedom robot arm is widely used in fruit and vegetable picking, flower planting, and other fields. Li et al. [43] used the z-arm four-axis cooperative robot arm to pick cherry. The average fruit picking time is 10.4 s. Xiong et al. [44] designed a strawberry-picking robot, which uses an industrial 5-DOF serial robot arm and binocular vision to pick. The average single-fruit picking time is 7.5 s. For a more complex orchard environment, Zhu et al. [45] used a six-degree-of-freedom robot arm for apple picking, with a success rate of 96.0%.

In fact, three different types of robot arms are widely used not only in outdoor farms but also in indoor farms. However, due to the limited indoor space, most of the indoor operating robot arms are smaller and more compact. It is widely used in standardized crops such as strawberries and tomatoes. Most outdoor robot arms are large in size and suitable for large-scale farmland scenes.

Table 1. Advantages and disadvantages of different types of robot arms.

Applied Crops	Sensor Model	Producing Area	Structure	Equipment Features	Advantage	Disadvantage	Ref.
Cotton	-	Clemson University team	Cartesian robotic arm		Simple structure, high control accuracy, and good rigidity	Only linear operations can be performed, and the space utilization rate is low.	[46]
Apple	SN04-N	America	Cartesian robotic arm				[39]
Apple	Intel RealSense D435	Universal robots	Multi-cylindrical robotic arm		Compact structure, easier to solve spatial trajectories	Complex structure; arm end error increases with the increase in arm length	[41]
Eggplant	Prosilica GC2450C, Mesa SwissRanger SR4000	Kinova Robotics	Dual joint robotic arm		Flexible movements, good obstacle avoidance performance, and the ability to perform complex and precise tasks	Low structural stiffness and more complex driving control	[47]

Table 1. Advantages and disadvantages of different types of robot arms.

Applied Crops	Sensor Model	Producing Area	Structure	Equipment Features	Advantage	Disadvantage	Ref.
Cotton	-	Clemson University team	Cartesian robotic arm		Simple structure, high control accuracy, and good rigidity	Only linear operations can be performed, and the space utilization rate is low.	[46]
Apple	SN04-N	America	Cartesian robotic arm				[39]
Apple	Intel RealSense D435	Universal robots	Multi-cylindrical robotic arm		Compact structure, easier to solve spatial trajectories	Complex structure; arm end error increases with the increase in arm length	[41]
Eggplant	Prosilica GC2450C, Mesa SwissRanger SR4000	Kinova Robotics	Dual joint robotic arm		Flexible movements, good obstacle avoidance performance, and the ability to perform complex and precise tasks	Low structural stiffness and more complex driving control	[47]

2.1.2. Regional Division

According to the different types of robot arms, how to realize the full coverage of the operating range of multi-robot arms is the premise of the efficient operation of a multi-robotic arms system.

Xiong et al. [48] used the double Cartesian robot arms for apple picking and regarded the apple picking area as a rectangle to adapt to the workspace of the Cartesian robot arm cuboid, as shown in Figure 1a. Feng et al. [25] also used Cartesian robot arms for picking operations. According to the apple tree crown, nine rectangular areas are equally divided, including four single-arm operation areas, four double-arm operation areas, and one four-arm operation area, to ensure the complete coverage of the operation range of the robot arm, as shown in Figure 1b. Compared with the Cartesian robot arm, the articulated robot arm is more flexible and widely used in agricultural dobby operations. Cui et al. [49] designed a two-arm picking robot for Kiwifruit picking in a trellis, using two UR5 articulated robot arms for operation. In order to adapt to the matrix area of the scaffolding space, an elliptic workspace is designed. Jiang et al. [50] also conducted oval modeling for the grape picking area and divided it evenly according to the grape growth area, as shown in Figure 1d. Reduce the scope of the work area as much as possible while ensuring no fruit leakage. Zhang et al. [51] equalized the operating range of the six robot arms according to the reachable area of the articulated robot arm, as shown in Figure 1c.

Figure 1. Equal division of multi-robot arms based on working area. (a) Division of dual Cartesian robot arms [48]; (b) division of four Cartesian robot arms [25]; (c) division of six joint robot arms [51]; (d) division of double joint robot arms [50].

Figure 1. Equal division of multi-robot arms based on working area. (a) Division of dual Cartesian robot arms [48]; (b) division of four Cartesian robot arms [25]; (c) division of six joint robot arms [51]; (d) division of double joint robot arms [50].

In fact, this equalization of work areas according to the working environment is often not the optimal solution. Due to the uneven distribution of tasks of the agricultural robot arms, the robot arms with intensive tasks are in high load operation for a long time, and other parts will be idle due to insufficient work, resulting in significant differences between the workload of the arms.

Arikapudi et al. [52] divided the region according to the number of fruits in the region, as shown in Figure 2a. According to the frequency histogram and cumulative distribution function of the fruit, the working boundaries of multi-robot arms are determined to reduce the impact of uneven task distribution. In addition, increasing the overlapping area of the arm is also a feasible scheme. K Lammers et al. [53] set the gap between multi-robot arms to 7 mm, so as to maximize the overlapping area and reduce the difference in workload of each robot arm. Barnett et al. [37] further expanded the common workspace of multiple Cartesian robot arms by sharing the x-axis of multiple robot arms, as shown in Figure 2b.

2.2. Operation Sequence of Agricultural Multi-Robot Arms

Scientifically planning the operation sequence of agricultural multi-robot arms can significantly shorten the waiting time caused by the poor task connection, and effectively prevent the collision between the robot arm.

Cui et al. [49] started from the distance between the robot arms, and, at the same time, the mechanical arm on one side picked and the mechanical arm on the other side was as close to the next picking target as possible to shorten the waiting time. Yu team [54] designed a humanoid apple-picking robot to harvest apples in turn according to the coordinates of the end effector in the workspace. Xiong et al. [55] used the primary–secondary control algorithm to ensure that the distance between the two arms was greater than the safe distance in real time to prevent collision. The harvesting sequence of strawberries is shown in Figure 3a below. Henry et al. [56] controlled the safe distance between each arm to ensure the safe operation of the arm. Although these methods ensure the safety of operation from the perspective of distance, they also lose the efficiency of collaborative operation. Jiang et al. [50] implemented a left-right segmentation strategy through the central axis of the camera’s field of view, dividing the workspace into a left-arm work area and a right-arm work area, ensuring that the two robot arms operate independently while preventing the collision of the robot arms. The harvesting sequence of the double robot arms is shown in Figure 3b below. Feng et al. [25] judged whether the picking tasks of the two robots collided according to the time interval of different robots entering the public area, which effectively reduced the interference and collision of the robots and improved the operation efficiency, as shown in Figure 3c.

2.3. Task Allocation Algorithm

In the process of the development of modern agricultural intelligence, the dynamic planning of agricultural multi-robot arms cooperative operation has become a key problem restricting the large-scale application and efficiency improvement of multi-robot arms. The cooperative operation of multiple robot arms constitutes the multi-traveling salesman problem. Firstly, how to traverse the target points efficiently according to the distribution of the target points is the key to solving this problem. Many researchers use heuristic algorithms to solve such large-scale and complex problems. Secondly, the application of a reinforcement learning algorithm also provides a new strategy to solve this problem. The specific task planning algorithm, division method and average operation time are shown in Table 2.

2.3.1. Intelligent Heuristic Algorithm

A heuristic algorithm is an algorithm based on intuitive or indirect experience, which searches for the optimal solution by constructing a heuristic function. At present, the algorithms commonly used to solve the task allocation of multi-robot arms mainly include the simulated annealing algorithm, ant colony algorithm, genetic algorithm, and so on.

The simulated degradation algorithm is derived from the solid-state annealing process and seeks the optimal solution by finding the termination temperature, which is easily affected by the initial temperature and cooling coefficient [57]. Zhang et al. [51] conducted in-depth research on the initial temperature, internal circulation times, and cooling coefficient of the simulated annealing algorithm. When the initial temperature is 3000 °C and the cooling coefficient is 0.99, the convergence speed is the fastest. The search efficiency of the optimized algorithm is improved by about 20%. Feng et al. [20] proposed AOMTSP-GA based on a genetic algorithm to solve the picking sequence problem of multi-robot arms. When selecting the selection operator, it fused the best individual and roulette strategy to further improve the convergence speed of the genetic algorithm. Compared with the random traversal algorithm, the traversal time is shortened by 40.97%. However, in the case of intensive operation, it is not effective to plan the fruit covered by leaves. Regarding the issue of crop obstruction, Barnet et al. [37] prioritized harvesting fruits that are obstructed to prevent them from falling off during the picking process. Yang et al. [58] proposed a genetic ant colony algorithm (IGAACMO) to solve the problem of mushroom-picking sequences in dense environments. The pheromone concentration of the ant colony algorithm is used to choose the picking order. While ensuring the success rate of the algorithm, it also improves the efficiency of multi-robot cooperation. When there are many targets, the average success rate is as high as 97%.

2.3.2. Reinforcement Learning Algorithm

Although the use of heuristic algorithms has significantly improved the efficiency of task allocation for agricultural multi-robotic arms, it still faces problems such as high computational difficulty and convergence speed is slow. The reinforcement learning algorithm proposed by Minsky provides a new research direction for solving multi-robotic arms task allocation [59]. Thanks to the powerful online learning and dynamic decision-making capabilities of reinforcement learning, it provides a revolutionary solution for the efficient collaborative operation of multiple agricultural robotic arms.

Xie et al. [22] proposed a multi-robotic arms task allocation algorithm based on proximal policy optimization (PPO). The reward function is integrated with time reward, action reward, and conflict penalty to guide model training. The access sequence of the robotic arm is determined based on selecting the appropriate picking action in the action space. Guo et al. [32] integrated a long short-term memory (LSTM) network during the data collection phase to store the past action sequences of the robotic arm and the historical picking records of apples, improving the agent’s ability to capture long-term dependencies. Simultaneously combining the PPO to control the task planning of multi-robotic arms. Li et al. [60] adopted the training strategy of multi-agent reinforcement learning, and the centralized controller integrated the state information of each robot arm to improve the training efficiency of multi-robot arms. In the real orchard environment, the average single-fruit picking time is 5.8 s. Li et al. [36] set the discount factor and learning rate for training, which improved the convergence speed of the near-end strategy optimization model. In addition, the incremental training method is adopted to carry out multi-robot arms operation training from less target environment, and gradually migrate to more target environment, which greatly improves the generalization ability of the model. Gong et al. [61] used the soft actor–critic (SAC) algorithm for task allocation of multi-robot arms. Through the pre-allocation module, entity units are allocated to each task point to effectively avoid the problem of local optimal solutions.

Table 2. Multi-robotic arms task allocation method based on heuristic algorithm and reinforcement learning.

Target	Number	Division of Picking Areas	Task Allocation Method	Average Time	Efficiency Improvement	Planning Road Map	Ref.
Peach	2		Simulated annealing algorithm	-	1.20 times		[51]
Apple	4		Genetic algorithm	7.12 s	1.96 times		[25]
Peach	6		Allocation of work units for waiting fruit load	4 s	1.5 times		[52]
Fruit	4		AOMTSP-GA	3.15 s	4.28 times		[20]
Fungus			IGAACMO	1183 pcs/h	2 times		[58]
Apple	4		PPO	5.8 s	1.33 times		[36]
Apple	2		LSTM-PPO	6.26 s	1.17 times		[32]
Strawberry	2		Active obstacle separation strategy	6.1 s	1.23 times		[55]

Table 2. Multi-robotic arms task allocation method based on heuristic algorithm and reinforcement learning.

Target	Number	Division of Picking Areas	Task Allocation Method	Average Time	Efficiency Improvement	Planning Road Map	Ref.
Peach	2		Simulated annealing algorithm	-	1.20 times		[51]
Apple	4		Genetic algorithm	7.12 s	1.96 times		[25]
Peach	6		Allocation of work units for waiting fruit load	4 s	1.5 times		[52]
Fruit	4		AOMTSP-GA	3.15 s	4.28 times		[20]
Fungus			IGAACMO	1183 pcs/h	2 times		[58]
Apple	4		PPO	5.8 s	1.33 times		[36]
Apple	2		LSTM-PPO	6.26 s	1.17 times		[32]
Strawberry	2		Active obstacle separation strategy	6.1 s	1.23 times		[55]

3. Path Planning of Agricultural Multi-Robot Arms Based on Intelligent Algorithm

In the complex dynamic scene of agriculture, the multi-robot arms system is vulnerable to the dual constraints of natural terrain and crop species in the process of operation. Therefore, on the basis of task allocation, planning an appropriate path for the multi-robot arms system is the premise for the robot arm to reach the target position accurately and complete the task successfully. Unlike the path planning of a single arm, the path planning problem of multi-robot arms not only needs to plan the collision-free optimal path to the target point, but also needs to solve the problem of mutual interference between each robot arm in the overlapping area of multi-robot arms, which is a major test for the real-time, robustness, and collaborative optimization ability of agricultural multi-robot arms path planning algorithm [62].

For the special challenges faced by agricultural multi-robot arms path planning, scholars around the world have conducted a lot of research. At present, the main algorithms for path planning of multi-robot arms can be roughly divided into two categories. One is a heuristic algorithm using empirical laws. The other is a path planning algorithm based on sampling, which mainly includes a probabilistic road map and a fast search random tree algorithm.

3.1. Heuristic Algorithm

Heuristic algorithms often use empirical rules or direct observation and summary algorithms. When facing some complex NP problems, they can skillfully use heuristic information to reduce the search space, so they are suitable for real-time path planning of multi-robot arms. The heuristic information of heuristic algorithms mainly comes from the following parts: the heuristic based on natural laws, the heuristic based on terrestrial organisms, the heuristic based on aquatic organisms, and the heuristic based on aerial organisms. The common algorithms are shown in Figure 4 below. The trajectory and planning time of the arm path planning algorithm are shown in Table 3.

Figure 4. Sources of intelligent heuristic algorithms [63,64,65,66].

Figure 4. Sources of intelligent heuristic algorithms [63,64,65,66].

In the single-arm independent operation area, only one collision-free path needs to be planned. As a traditional heuristic algorithm, the artificial potential field method is inspired by natural physical phenomena [67]. A local path search algorithm is proposed by using the repulsion force field to guide the robot arm to explore the best path. The traditional artificial potential field method uses two-dimensional potential energy for path search, which is difficult to adapt to the complex agricultural operation environment. Zhang et al. [68] transferred the two-dimensional potential energy of the artificial potential field method to three-dimensional space for the path planning of the apple-picking robot arm, and the arm obtained a smoother obstacle avoidance curve. Chen et al. [69] combined the simulated annealing algorithm to search the optimal parameters of the artificial potential field and set up virtual obstacles to solve the local minimum problem, which significantly improved the obstacle avoidance speed of the robot arm. However, in the agricultural environment with more dense obstacles, the planning efficiency is significantly reduced.

Ant colony algorithm, with its distributed search characteristics and global search capability, provides a new solution to improve the efficiency of path planning for multi-robotic arms. Guo et al. [70] improved the initial parameters of the ant colony algorithm using a genetic algorithm to enhance the convergence speed of the algorithm in order to improve the efficiency of the robotic arm in picking red flowers. At the same time, in order to alleviate the errors caused by the positive feedback of the ant colony algorithm. Establish a minimum threshold for the pheromone level to increase its chances of discovering the global optimum. In the simulated picking experiment, compared with the traditional ant colony algorithm, the path length is shortened by 1.33–7.85%. Meng et al. [71] proposed an elite smooth ant colony algorithm to reduce the dependence of harvesting robotic arms on local paths. By introducing attenuation coefficients in the pheromone update strategy, the robotic arm is effectively prevented from becoming stuck in local optimal solutions. Yan et al. [72] also improved the pheromone distribution of the ant colony algorithm by dynamically adjusting the decay rate of pheromones with path length and height and using the beetle antenna algorithm for secondary optimization of the initial path. After experimental verification, the success rate of path planning is as high as 96%.

In the overlapping area of multiple arms, the cooperative operation of multi-robot arms should not only consider the static obstacles in the workspace, but also avoid the dynamic path of other robot arms in real time [12]. Ling et al. [73] defined the potential collision relationship between various parts of the double robot arm, such as allowing the left forearm and right forearm to operate together, implementing strict obstacle avoidance on the waist and shoulders, and effectively avoiding the occurrence of special tracks. Bao et al. [74] Proposed a citrus-picking system based on double robots, with one robot arm responsible for target recognition and the other for picking. The optimal path is solved by a genetic algorithm. Gao et al. [75] improved the fitness function of the genetic algorithm by using the weighted summation method to consider multiple optimization objectives at the same time to meet the path planning requirements of multi-robot arms. Xiong et al. [55] designed a monorail dobby system to pick strawberries. Inspired by the active separation of surrounding obstacles by human hands, they proposed a path planning strategy for active separation of obstacles. The robot arm is used to actively separate obstacles to reduce the picking failure caused by strawberry leaves. In the double-arm operation mode, the single fruit takes about 4.6 s.

During the operation of multi-robotic arms, it is necessary to autonomously adjust their motion trajectory to meet motion constraints. Cao et al. [76] used B-spline curves to interpolate path points, achieving flexible configuration of robotic arm velocity and acceleration, and ensuring the stability of robotic arm motion. In order to overcome the dynamic singular value problem of robotic arms, Wang et al. [77] used constrained particle swarm optimization (PSO) for coordinated trajectory planning of dual-arm space robots. Huang et al. [78] addressed the issue of motion trajectory during double-arm apple picking by generating motion trajectories using the five-degree spline difference method based on the starting and ending joint positions, effectively improving the accuracy and reliability of double-arm picking.

Table 3. Comparison of robot path planning based on heuristic algorithm.

Feature	Algorithm	Working Hours	Success Rate	Path Trajectory	Ref.
	Ant Colony Algorithm	72.245 s	-		[70]
	Elite Ant Colony Algorithm	3.83 s	73.3%		[71]
	-	30 s	87.5%	-	[73]
	Genetic Algorithm	82%	158.9 s		[74]
	Active Separation Path Planning	75%	4.6 s		[55]

Table 3. Comparison of robot path planning based on heuristic algorithm.

Feature	Algorithm	Working Hours	Success Rate	Path Trajectory	Ref.
	Ant Colony Algorithm	72.245 s	-		[70]
	Elite Ant Colony Algorithm	3.83 s	73.3%		[71]
	-	30 s	87.5%	-	[73]
	Genetic Algorithm	82%	158.9 s		[74]
	Active Separation Path Planning	75%	4.6 s		[55]

3.2. Path Planning Algorithm Based on Probability Sampling

In the scenario of multi-robotic arms collaborative operation in agriculture, the computational complexity of heuristic algorithms will increase exponentially due to the increase in the number of robotic arms, making it difficult to develop suitable heuristic functions in a short period of time. The path planning algorithm based on probability sampling provides a novel approach for solving multi-robotic arms path planning problems due to its efficiency and flexibility in complex environments. The path planning algorithm based on probability sampling mainly uses random sampling in the workspace to search for the optimal path using sampling points. The algorithms commonly used for multi-robotic arms path planning include the probabilistic roadmap algorithm and the fast extended random tree algorithm (RRT).

3.2.1. Probability Roadmap

The probability roadmap involves random sampling in the workspace and planning the path from the starting point to the endpoint based on the distance of the sampling points. Therefore, probabilistic roadmap algorithms are highly dependent on accurate modeling of the environment.

Cai et al. [79] used binocular vision to obtain the three-dimensional coordinates of fruits and obstacles and constructed a virtual picking environment to ensure the accuracy of robotic arm path planning. Chen et al. [80] conducted in-depth research on the path of robotic arms in narrow channel environments and proposed an improved probabilistic road map (PRM). By applying virtual force to the sample points, the distribution of sample points has been optimized. Through verification of an 8-degree-of-freedom robotic arm, the time required to search for feasible paths is shorter than that of traditional PRM algorithms. Cheng et al. [81] used Gaussian mixture models to model the work environment and further optimized the path trajectory by considering both Euclidean distance and probability distance. But it only verifies the feasibility of the path, without fully considering whether the path is safe. SepLveda et al. [47] fully considered the influence of a dynamic environment on dual-arm harvesting and used the simple text-oriented messaging protocol (STOMP) to optimize the path of the dual robotic arms. By generating random trajectories and optimizing their gradients, the path quality of the dual robotic arm is improved. Therefore, compared to traditional roadmap methods, dynamic path planning is more suitable. Through experimental verification, the success rate of double-arm picking is as high as 91.67%.

In fact, modeling the working environment is difficult in complex agricultural environments. Therefore, the probability roadmap method is difficult to apply in more complex orchard environments.

3.2.2. Rapidly Exploring Random Trees

Unlike the PRM, the method of rapidly exploring random trees generates a tree structure and finds the optimal path based on the tree structure. It has been widely applied in the path planning problem of high-dimensional space robot arms in the past few decades.

In response to the problems of low planning efficiency and long iteration time in traditional RRT algorithms, Cao et al. [82] used the idea of target gravity to guide the exploration tree to extend towards the target point and selected the optimal search step size to improve the path search capability of the robotic arm. Ye et al. [83] adjusted the adaptive coefficient for obstacle avoidance while introducing target gravity to address the issue of path search failure caused by too many obstacles. And the experimental results verified the effectiveness of the collision-free motion planning method for robotic arms. Liu et al. [84] proposed an attractive step size and step-size binary method for path planning of robotic arms, which is used to solve path searches in repulsive fields. In real environments, the path length has been reduced by 17.88%, which gives it an advantage when facing obstacles with larger occlusion areas. Li et al. [85] proposed an adaptive step-size RRT * algorithm that utilizes a dynamic step-size adjustment mechanism. Increase the step size when there are fewer obstacles, and decrease the step size when there are more obstacles. Ensure the reliability and accuracy of the path planning process for the robotic arm.

The core to improve the convergence speed of the RRT algorithm is to select the appropriate exploration node search problem. Wang et al. [86] used the third point between the starting point of the principle and the target point as the guide point for a four-way search and used P probability sampling instead of random sampling to increase the search efficiency. Wang et al. [87] proposed a method to gradually improve the sampling area to constrain the expansion nodes of the spanning tree. The boundary node is used as a new extension node to prevent the random tree from falling into the local optimal solution.

Tahir et al. [88] have performed a lot of research on the bidirectional RRT* algorithm and added the guided bidirectional search tree in the path search process to accelerate the convergence speed of the algorithm. The characteristics, trajectory and planning time of the robot arms are shown in Table 4 below.

In the field of multi-arm path planning, Zhu et al. [89] proposed an attractive RRT algorithm with adaptive step size. Establish the step-size norm between the configuration space and the workspace, and adjust the step size according to the target position. Kim et al. [90] reduced the dimension of the path planning space according to the task requirements to speed up the path search efficiency of the dual robot arm. Cui et al. [49] deployed a multi-machine distributed system for the integrated control of the two robot arms and used the RRT algorithm and the collision detection library for collision-free path planning. The average picking success rate is 82.10%, but there is still a risk of collision in the shared area of both arms. Shi et al. [91] combined with A * heuristic algorithm to calculate the path cost of each random point in order to prevent the collision problem of double robot arms in the public area. The two arms will be divided into main and auxiliary arms to avoid dynamic and static obstacles, which greatly improves the path’s searching ability of the robot arms.

In addition, the Batch Notification Tree (BIT *) algorithm proposed by Gammell et al. [92] strikes a balance between graph search algorithms and sampling algorithms, achieving optimal sampling path planning through heuristic search of implicit random geometric graphs, providing new ideas for path planning of agricultural multi-robotic arms. Ma et al. [93] established both forward and backward trees during the initialization phase and conducted alternating searches. And use fifth-degree polynomial interpolation on the joint path to further improve the safety of the robotic arm’s motion trajectory. The average execution time is 7.32 s, with an average success rate of 90%, both higher than traditional path search algorithms.

Table 4. Comparison of robot path planning based on RRT algorithm.

Feature	Degree of Freedom	Crop	Improved Algorithm	Picking Success Rate	Path Time	Trajectory Diagram	Ref.
	-	Litchi	tRRT	100%	0.447 s		[82]
	6	Litchi	AtBi-RRT	100%	0.53 s		[83]
	6	Fruit	PIB-RRT	85%	7.84 s		[88]
	7	Tangerine	TO-RRT	-	0.07 s		[84]

Table 4. Comparison of robot path planning based on RRT algorithm.

Feature	Degree of Freedom	Crop	Improved Algorithm	Picking Success Rate	Path Time	Trajectory Diagram	Ref.
	-	Litchi	tRRT	100%	0.447 s		[82]
	6	Litchi	AtBi-RRT	100%	0.53 s		[83]
	6	Fruit	PIB-RRT	85%	7.84 s		[88]
	7	Tangerine	TO-RRT	-	0.07 s		[84]

4. Path Planning of Agricultural Multi-Robot Arms Based on Reinforcement Learning Algorithm

The reinforcement learning algorithm is outstanding in the face of complex environments and tasks and has strong learning ability and self-decision-making ability, which provides a new idea for solving the path planning problem of multi-robot arms. At this stage, the research on path planning of arms based on reinforcement learning mostly focuses on the path planning of agricultural single arms, and the research on agricultural multi-robot arms is relatively less.

The path planning training of the robot arm is mainly determined by the state space, action space, and reward function. According to different training methods, reinforcement learning path planning algorithms can be classified as follows: the reinforcement learning algorithm based on strategy, the algorithm based on Q value, and the reinforcement learning algorithm based on actor–critic (A–C) [94]. The development of the reinforcement learning algorithm is shown in Figure 5.

4.1. Strategy Based Reinforcement Learning Algorithm

The strategy-based reinforcement learning algorithm can learn the optimal control strategy of the robot arm directly through the continuous interaction between the robot arm and the environment. The whole training process mainly collects the state action reward sequence of the robot arm in the whole process and constructs the strategy network. According to this control strategy, the path planning of the agricultural robot arm is guided. At present, PPO and diffusion policy optimization (DPPO) are most widely used in the path planning of agricultural robot arms [95,96].

The PPO algorithm was proposed by OpenAI in 2017. It transforms the on-policy method into the off-policy method, which effectively solves the problems of low sample efficiency and unstable training of the traditional policy gradient algorithm when updating the policy. However, when there are large samples of rewards in the experience replay area, the model is easy to fall into a local optimal solution. Qi et al. [97] decomposed the rewards into multi-dimensional frequency rewards and the sum of exponentially weighted components to reduce the impact of large reward samples. The heuristic idea is introduced into the reward function, which greatly speeds up the convergence speed at the initial stage of model training. Based on the PPO algorithm, the introduction of distributed parallel sampling in diffusion policy optimization (DPPO) has further improved the training efficiency.

Yang et al. [98] combined this with distributed sampling to search the environment in parallel by multiple agents, significantly reducing the time correlation of samples. The generalized dominance estimation is used to optimize the near-end strategy optimization algorithm to further reduce the sample demand. It effectively solves the problem of low sampling efficiency of robot arms in path planning. In addition, Yang et al. [99] combined multi-agent technology with the PPO algorithm to set up nine identical and independent agents to share reward and punishment signals to further improve sample utilization efficiency. Lin et al. [100] conducted in-depth research on the path planning of the picking robot arm. Aiming at the sparse reward problem during the training of the robot arm, they used the idea of an artificial potential field to design the reward and punishment mechanism. When the robot arm gradually approaches the target point or the angle between the robot arm and the target point decreases, they will be rewarded; otherwise, they will be punished. The success rate of obstacle avoidance is as high as 97.5%.

Actually, in the dynamic agricultural scenario, the construction of the strategy network is very complex and difficult to converge. And the training cost of robot arms is too high to be widely used in path planning of agricultural robot arms.

4.2. Reinforcement Learning Based on Q Value

In reinforcement learning, besides updating through policy networks, updates can also be made through Q functions. The Q function, also known as the action value function, is the expected cumulative reward that an intelligent agent can receive in the future after taking a certain action in a certain state. Q-learning (Q-L) and Deep Q-learning (DQN) algorithms are often used in agriculture for the path planning of robotic arms.

4.2.1. Q-Learning Algorithm

The Q-L algorithm will obtain the maximum expected value of future cumulative rewards based on the reward and punishment for selecting the next action in a certain state.

In order to improve the convergence efficiency of traditional Q-L algorithms and reduce training time. Li et al. [101] aimed to study greenhouse environments and adaptively adjust the step size of harvesting robots based on the position of obstacles. Compared to traditional Q-L algorithms, the average training time has been reduced by 34.46%. Xie et al. [102] combined an ant colony algorithm to dynamically update the pheromone matrix and iteratively adapted the path search strategy in the Q-table, effectively reducing the problem of becoming stuck in local optima. Li et al. [103] added a distance metric to Q-L to guide the robotic arm towards the target in order to avoid the problem of dimensionality disaster in path planning. Liu et al. [104] proposed a reinforcement learning strategy that combines expert experience guidance to improve the learning efficiency of the robotic arm in the early stages of training. The training path in an unstructured environment is shown in Figure 6. The selection of action selection strategies for robotic arms is equally important. Wang et al. [105] drew on the short-term convergence of the SARSA algorithm and combined it with the Q-L algorithm to indirectly influence action strategies based on the Q-value. Maoudj et al. [106] explored the Q-table initialization process by utilizing prior knowledge of the environment. At the same time, in order to reduce the search space, a new selection strategy is proposed to further accelerate the convergence of the algorithm.

The Q-L algorithm requires discretization of the state space and action space during training, which to some extent limits its ability to represent complex continuous spaces. In actual agricultural robotic arm path planning, the motion of the robotic arm is often continuous, and the state information in the environment may also be continuously changing. Discretizing it may lose some important information, affecting the accuracy and effectiveness of path planning. Therefore, it is not widely used in agricultural robotic arms.

4.2.2. Deep Q-Learning

In order to handle higher dimensional and continuous state spaces, a new algorithm DQN is formed by combining deep learning algorithms with Q-L. The DQN algorithm approximates the Q function in Q-learning through deep neural networks, enabling it to handle complex environments that traditional Q-learning cannot handle.

In response to the problems of slow convergence speed and poor path planning performance faced by traditional DQN algorithm networks. Li et al. [107] proposed an improved ERDQN algorithm, which recalculates the Q value by recording the frequency of repeated states. The more times a state is repeated, the lower the probability of its next occurrence. To a certain extent, it reduces the risk of the network converging to a local optimum and reduces the number of training rounds required for network convergence. Wang et al. [108] reconstructed the experience backtracking pool to speed up the early training of the model. At the same time, the path evaluation function is constructed and the optimal path is selected.

4.3. Actor–Critic Algorithm

The actor–critic algorithm is most widely used in practical agriculture, combining the advantages of value function-based methods and strategy-based methods. It can effectively handle the continuous motion problem of agricultural robotic arms during operation and efficiently extract multi-dimensional information such as obstacle and target point positions. Common actor–critic (AC) algorithms include soft actor–critic (SAC), deep deterministic policy gradient (DDPG), twin delayed deep deterministic policy gradient algorithm (TD3), etc. The specific algorithm features are shown in Table 5.

4.3.1. Soft Actor–Critic

The SAC algorithm is based on the AC algorithm and introduces a maximum entropy objective function to encourage strategies to maintain diversity. In order to reduce the training time of neural networks, Yang et al. [109] modified the network structure of SAC by adding a class residual structure, which to some extent enhanced the network’s expressive power. Simultaneously using multi-step TD error to optimize the reward mechanism and reduce the possibility of gradient vanishing problems.

Prianto et al. [110] proposed the SAC algorithm based on ex-post-experience replay for periodic motion multi-robot arms system. The deep neural network is used to store the moving position information of obstacles in the past period of time, and the ex-post-experience replay technology is used to effectively use the training data. Xiong et al. [111] improved the a-c network by combining long-term and LSTM and used LSTM to extract the characteristics of the current state. The selection of temperature parameters plays a decisive role in the training process. Xiong et al. [111] designed the neural network to automatically adjust the value of the parameter according to the change in the reward. The parameters converge when the optimal strategy is reached.

On the other hand, the design of the reward function is the key to the path quality of the robot arm. Tao et al. [112] designed a non-sparse reward function to guide the early training sparsity of the agent and guided the agent to approach according to the distance of the obstacle and the target guidance angle. Xiong et al. [113] introduced the concept of target attraction and target repulsion in the artificial potential field method into the design of reward and punishment mechanism and converted the obstacle range punishment into a single-direction punishment. By building a simulation environment in unity in 200 random picking experiments, the success rate of the directional penalty obstacle avoidance function method is 97.5%, which is 11% higher than that of the ordinary reward function method.

4.3.2. Twin Delayed Deep Deterministic Policy Gradient Algorithm

The Twin Delayed Deep Deterministic policy gradient algorithm also utilizes the AC architecture and updates the value function estimation through time differences. The TD3 algorithm proposed by Fujimoto in 2018 provides a new direction for the path planning problem of robotic arms. Yang et al. [114] utilized the double-delay deep deterministic policy gradient algorithm to conduct path planning for the green walnut harvesting robotic arm. By using the post-experience replay mechanism, the exploration ability of the agent is significantly improved, and the problem of sparse rewards in the early stage of model training is effectively alleviated.

4.3.3. Deep Deterministic Policy Gradient

The deep deterministic policy gradient algorithm proposed by Timothy et al. provides an effective method for solving complex reinforcement learning problems in continuous action spaces and is another option for the path planning of agricultural robotic arms.

To solve problems such as low training efficiency and slow convergence caused by the high proportion of illegal strategies in the traditional DDPG algorithm, Dong et al. [115] adopted an adaptive exploration method based on the epsilon–greedy algorithm. Dynamically adjusting the exploration factors and rationally allocating the probabilities of exploration and mining provide algorithmic support for its application in agricultural robotic arms. Lin et al. [116] adopted the recurrent neural network LSTM to store and remember the past states of the environment and retrieve implicit information, using some three-dimensional line segments to replace obstacles. Meanwhile, we set an appropriate reward threshold to ensure that the robotic arm does not collide. Zheng et al. [117] conducted an in-depth study on the trajectory problem of the picking robotic arm, limiting the trajectory planning to the picking plane to reduce the environmental complexity. Meanwhile, the strategies learned in the single-obstacle scenario are transferred to the mixed-obstacle scenario, and the obtained prior knowledge is utilized to guide the path planning task of the robotic arm in the complex unstructured environment.

Table 5. Path comparison of robot arm based on reinforcement learning.

Crop	Algorithm	Feature	Success Rate	Planning Time	Planning Path	Ref.
Guava	Recurrent deep reinforcement learning		90.90%	29 ms		[116]
Apple	Trajectory planning		-	260 s		[117]
Litchi	DPPG		96.7%	124 s		[115]
Walnut	HER-TD3		80.0%	-		[114]

Table 5. Path comparison of robot arm based on reinforcement learning.

Crop	Algorithm	Feature	Success Rate	Planning Time	Planning Path	Ref.
Guava	Recurrent deep reinforcement learning		90.90%	29 ms		[116]
Apple	Trajectory planning		-	260 s		[117]
Litchi	DPPG		96.7%	124 s		[115]
Walnut	HER-TD3		80.0%	-		[114]

In summary, there are three main types of path planning algorithms for agricultural robot arms: path planning algorithms based on intelligent heuristics, path planning algorithms based on probability sampling, and path planning algorithms based on reinforcement learning. The algorithms based on intelligence elicitation, such as the artificial potential field method and genetic algorithm, have high computational complexity. It is difficult to guarantee the optimal solution, and it is difficult to directly apply to the path planning problem of agricultural robot arm, which can assist other algorithms to improve. Path planning algorithms based on probabilistic sampling perform outstandingly in high-dimensional environments. They usually do not need to build an environment model and have good real-time performance and scalability. However, the path quality is relatively low and often requires further optimization. Benefiting from the self-learning ability and decision-making ability of reinforcement learning, the path planning algorithm of reinforcement learning provides an innovative method for solving the complex agricultural environment, which has become an important topic of current research. However, most of the current studies are still at the stage of single-robotic arm research, and there is still great potential in the path planning of multi-robotic arms in agriculture in the future.

5. Conclusions

Multi-robot systems have a wide range of applications in agricultural production practices. This paper systematically reviews recent research advancements in agricultural multi-robot systems, addressing issues such as low efficiency in cooperative tasks, difficulties in task allocation, and poor path planning quality. It provides an overview of algorithms for cooperative task allocation and dynamic environment path planning strategies to promote the future large-scale application of multi-robot systems in smart agriculture. Section 2 focuses on the task allocation problem for agricultural multi-robot systems, systematically discussing the division of work areas and task planning issues for Cartesian and joint robotic arms in different agricultural scenarios. Based on the existing literature, current multi-robot task allocation algorithms mainly include genetic algorithms, ant colony algorithms, and reinforcement learning algorithms. Among these, genetic algorithms, with their excellent search capabilities and adaptive characteristics, have become the most widely used intelligent heuristic algorithms, playing a key role in the orderly operation of agricultural multi-robot systems, while ant colony algorithms and reinforcement learning algorithms have promising research prospects for the future. Section 3 summarizes the path planning strategies for multi-robot systems in complex agricultural scenarios based on the implementation of multi-robot task allocation, reviewing various heuristic algorithms such as genetic algorithms and particle swarm algorithms, as well as sampling-based algorithms like RRT. Heuristic algorithms like genetic and particle swarm algorithms are widely applied in the harvesting operations of strawberries, kiwis, and others. The rapid random tree algorithm, with its adaptability to high-dimensional spaces, provides solutions for fruit harvesting in complex orchard environments. Section 4 reviews cutting-edge reinforcement learning algorithms, including PPO, DQN, and AC algorithms, and their application in the path planning of agricultural robots. Thanks to the strong self-learning and decision-making capabilities of reinforcement learning, path planning efficiency is significantly improved compared to traditional heuristic algorithms. However, they generally face challenges such as high training costs and difficulties in convergence. Additionally, current research is mainly focused on single-robot path planning, with relatively little research on multi-robot path planning. Nevertheless, reinforcement learning algorithms have broad prospects in the path planning of multi-robot systems.

The use of multiple robot systems in agriculture not only demonstrates the progress in agricultural machinery technology but also signifies a crucial shift toward advanced intelligent automation in the field. As technology continues to evolve, these robot systems will play a key role in modernizing agriculture and promoting sustainable development. However, the suitability of such systems for farmers’ investments will ultimately depend on whether the prices of agricultural products justify the costs associated with these investments.