Smarter Robotic Sprayer System for Precision Agriculture

Abstract

The automation of agricultural processes is expected to positively impact the environment by reducing waste and increasing food security, maximising resource use. Precision spraying is a method used to reduce the losses during pesticides application, reducing chemical residues in the soil. In this work, we developed a smart and novel electric sprayer that can be assembled on a robot. The sprayer has a crop perception system that calculates the leaf density based on a support vector machine (SVM) classifier using image histograms (local binary pattern (LBP), vegetation index, average, and hue). This density can then be used as a reference value to feed a controller that determines the air flow, the water rate, and the water density of the sprayer. This perception system was developed and tested with a created dataset available to the scientific community and represents a significant contribution. The results of the leaf density classifier show an accuracy score that varies between 80% and 85%. The conducted tests prove that the solution has the potential to increase the spraying accuracy and precision.

1. Introduction

Agricultural land is limited and can only increase marginally, so we need to produce more with the same resources through higher precision, intelligent agriculture. Precision agriculture benefits the environment by reducing the quantities of pesticides applied and the resources spent on machinery.

Steep slope vineyards represent only 7–12% of the total European vineyards. However, they produce wines with recognized quality (e.g., Port Wine), and they have an undeniable identity, heritage, and historical value, which are being seriously threatened due to climate change.

The cultivation of mountainous and steep-slope vineyards requires a high number of working hours (higher than 1500 h/ha/year), with human labor being increasingly rare and costly. Therefore, grape-growers are considering innovative mechanization solutions to reduce operating costs, execute timely cultural practices, and increase flexibility within their operations.

Spraying is a critical operation for these vineyards since they account for about 20–30% of the total annual work time of the vineyard. The main bottlenecks for spraying these vineyards are the lateral and transversal slope, terrain with a stony surface, narrow row sizes (90–150 cm), bards’ curvature, and canopy heterogeneity.

Currently, spray operation is based on sprayers carried on the person’s back (manual) or, where possible, using a small tractor-based system equipped with air blast sprayers. Both current solutions have significant shortcomings. Manual spraying struggles with the lack of human resources in the region available to perform a heavy and non-ergonomic operation. The use of tractors has low spraying efficiency due to off-target and soil compaction problems.

Inspired by this problem, in this work, we developed a modular and precision terrestrial sprayer robot, the Precision Robotic Sprayer (PRySM), which is capable of operating autonomously on rugged terrain with steep slopes and under the most diverse ground conditions. A robotic platform was adapted to work on complex terrain conditions and whose dimensions and locomotion mechanism allow tight manoeuvring in mountain vineyards with very narrow rows. This robot is equipped with advanced algorithms for self-localisation and navigating using light detection and ranging (LiDAR) sensors and Global Navigation Satellite System (GNSS) receiver data to support precision spraying tasks. The robotic navigation stack considered for this platform is deeply explained in [1,2,3,4,5]. In addition, we developed and integrated a novel precision autonomous spray tool into the developed robotic platform, constructed a dataset for the scientific community, and tested an SVM classifier that calculates the grapevine leaf density.

This paper is organized into six sections: Firstly, Section 2 introduces spraying-related works carried out over the past few years. Section 3 presents the proposed hardware organization and the system design. Section 4 describes the structure of the sprayer: the 3D models and the electrical components used. An electrical schematic of the entire system is also presented. Section 5 is divided into two subsections where first presents the dataset created for the crop perception system and then the fourteen visual descriptors that were designed and tested are described. Finally, Section 6 presents the SVM results using different descriptors and the sprayer performance for different reference values of air flow, water rate, and water density. This is followed by some conclusions and suggestions for future work in Section 7.

2. Background

Spraying is a common task in agriculture that relies on chemical product use. Although these products are efficient, they leave chemical residues in the soil, decreasing soil fertility and the diversity of plants [6].

Precision spraying is a method that reduces pesticides losses. This method controls the amount of pesticide distributed across the field according to specific characteristics [7].

For more than two decades, studies have been carried out to contribute to precision spraying today. The first of these works began with investigating the effects of applying spatially variable herbicide doses [8]. Later, the concept of precision spraying evolved to protection treatments directly applied on crop plants, where the amount of insecticide applied depends on factors such as foliage shape and volume [9].

To improve the perception of these systems, grape clusters and foliage detection algorithms were constructed to guide the selected application of hormones to fruit and pesticide application to foliage [10].

In 2015, an automatically controlled sprayer was developed. The system uses ultrasonic sensors to determine variations in the canopy structure and adjust valve opening to implement variable-rate application. This application was achieved through three nozzles mounted at different heights on a vertical mast for orchard tree-spraying [11].

The first study using an automatic selective system for spraying diseases in speciality crops was conducted in 2016. A robot capable of detecting and spraying from 85% to 100% of the diseased area and reducing the pesticide use from 65% to 85% was developed [12,13].

A new technique for a close-range precision spraying process in vineyards was evaluated. An air-assisted precision spraying and effector was presented, and the percentage of the spray coverage and the number of droplet impacts were evaluated. The results showed that the leaves’ front side’s spraying was good, but on the backside, the spraying was limited [14].

Recently, a research group developed a solution based on a robotic vehicle for distributing plant protection products in vineyards and greenhouses. The system was equipped with a standard spraying machine that was transformed to optimise spraying operations [15].

Human–robot collaboration has also been applied in agricultural sprayer robots. For example, the human and robot can work collaboratively to detect spraying targets, to increase the true positive (TP) rate, and to reduce false positives (FPs) [16]. In this work, the authors proved the feasibility of this collaboration, which reduced the quantity of sprayed material by 50%.

Related to this human–robot collaboration, a semi-autonomous agricultural robot sprayer was developed. In this case, the human helps the robot identify the spraying targets and, if needed, can also control the robot movement to avoid obstacles [17].

A technical-economic analysis was conducted on spraying techniques with different control levels: an air-blast sprayer, an on-off nozzle switching sprayer, and a canopy-optimised distribution sprayer. The results showed that the larger the area to be treated, the higher the adopted technology’s level of precision should be. This analysis also showed that using a robotic platform produces pesticide and labour savings [7].

To make these tasks more efficient and intelligent, recognising crops and weeds is an important task that can be achieved through image processing techniques. Some techniques differentiate crops and weeds from the soil first and then try to classify plants as crops or weeds based on their shape, texture, and colour properties [18].

Recently, a machine learning-based vision system was developed to detect weeds and crops. Based on this, the plant canopy size is calculated and sent to a microcontroller that controls the flow rate of the agrochemical [19].

Vegetation indices are often used in these perception systems to improve their performance. These are important indicators of the health and yield of agricultural crops [20].

The system developed in this work has significant advantages: the robustness to work on rugged terrain and being completely electric, which is advantageous for the environment and allows more efficient control of the system compared with sprayers based on other techniques (

Table 1. Sprayer design approaches comparison.

Solution	Advantages	Disadvantages
Airblast with hydraulic actuators [21,22,23]	- High wind velocity and long reachability - Simple design	- Requires a tractor power take-off - Hydraulic leaks - Hydraulic problems under high temperatures - Low efficiency - Losses higher than 40% - Not adequate for low-volume spraying
Airblast with recovering panel and hydraulic actuators [23,24,25]	- High wind velocity - Reduced losses - Does not require variable-rate technologies to reach high efficiency - Compatible with low-volume spraying	- Requires a tractor power take-off - Low efficiency - Complex design - Heavy system - Complex maneuvering operation - Requires hydraulic circuit - Hydraulic leaks - Hydraulic problems under high temperatures - Potential diseases dissemination
Pneumatic sprayer and hydraulic actuators [21,23]	- High wind velocity - Reduced losses - Compatible with low-volume spraying	- Requires a tractor power take-off - Low efficiency - Complex design - Heavy system - Complex manoeuvring operation - Requires hydraulic circuit - Hydraulic leaks - Hydraulic problems under high temperatures - Potential diseases dissemination
Fully electrical sprayer [15,26]	- Simplest design - More manoeuvrability - More efficient - Simplified integration with variable-rate technologies - Lighter - Compatible with ultra-low-lvolume spraying - Does not require PTO and hydraulic circuit - Less maintenance and leaks	- Requires batteries (potentially more expensive) - Requires more electronics compatible with high temperatures and with high ingress protection (IP)

).

In contrast to sprayers based on robotic manipulators [27], a fully electrical approach with an array of electrical fan sprayers allows higher velocities during the spraying procedure and less complex movements. Additionally, in this paper, we propose a three-component independent sprayer set (fan, atomiser, and nozzle (pump)), which are independently controlled and adjusted in height. Since they are controlled independently, the system can reach high precision, reduce losses, and be more efficient.

The literature focuses on variable-rate technologies and not on the sprayer hardware. Our paper proposes a different perspective on the innovation of the sprayers concept by redesigning the entire system from scratch.

3. Hardware Organisation and System Design

To accomplish PRYSM robot features, we had to organise the hardware to avoid interference between sensors and actuators. The planned general organisation for the PRYSM robot is presented in Figure 1. The sensing, communication, and processing units are stored on a tower in front of the robot. For localisation, mapping, perception, and safety, a LiDAR sensor is placed on the top of this tower to ensure that they have an observability of 180° on the robot’s front. The GNSS antenna improves the localisation precision and is placed on the top of the tower. The sprayer is placed in a mast installed in the robot’s back, far away from the sensors to avoid water projection into the sensors. The water pumps and sprayer power are stored below the sprayer tank. The sprayer controller and sensing sensors are placed on the robot front (AgIoT 2.0 module [28], developed by INESC TEC) to avoid exposure to water and increase the vine canopy visibility.

4. Electric Sprayer Description: PRYSM

We propose a fully electrical PRYSM sprayer using a centrifugal principle. The structure of the sprayer, the PRYSM sprayer, consists of three spray drums and a fertiliser tank. Each drum comprises 3D models and an aluminium plate on the side (Figure 2). These models are used to provide structure and fix the motors and the fertiliser pipe. Both the drums and these models used to fix the motors are fixed on an aluminium profile. This fixing can be performed at different heights according to the specific application. The fertiliser tank has a capacity of 100 L and is fixed in a stainless steel tube structure.

The spraying system (Figure 2) consists of a brushless motor that controls the propellers, a DC motor that controls the centrifugal disk, and a water pump that controls the amount of fertiliser passing through the pipe. In this way, the brushless motor controls the air flow, the DC motor controls the water density, and the water pump controls the water rate.

AgIoT is also part of this system, where it is responsible for all the sensors and actuators.

Figure 3 presents an electrical schematic of the entire system.

There is also a 3D model of a box and a tube to enable the AgIoT to be fixed to the existing structure (Figure 4). The PRYSM sprayer and the AgIoT module were assembled on the PRYSM robot (Figure 5).

5. Crop Perception System

The crop perception system allows, according to the number of leaves observed in the image acquired by the stereo camera, the management of the amount of fertiliser applied. As the sprayer is composed of three independent spray drums, a measurement is performed of the number of leaves in the image area equivalent to each drum’s spray area. The developed software’s objective is to obtain a perception of the environment by processing images captured by the Raspberry Pi Camera Module.

Therefore, it is necessary to consider the distance between the sensor and the sprayer and the robot’s speed to know the delay between the area viewed by the sensor and the spraying area. A dataset was created with several images captured in a vineyard, and then we developed a leaf density classifier.

5.1. Dataset

The canopy classification (leaf area index class) is needed for the sprayer control to adjust the chemical product quantity to the real leaf area index: the larger the leaf area, the more product should be applied on the leaves. Most sprayers are controlled manually by the machinery operator by observing the leaf area index (without variable-rate technologies). To automatise the procedure, we collected 475 images from a real vineyard using a robot and manually annotated them with four leaf area index classes in three regions of interest (ROIs) using our experience.

The dataset consists of 475 images, size 640 × 480, collected from Aveleda’s vineyards using a stereo camera [29] assembled on a static robotic manipulator over a mobile platform, called AgRob v16. The data were collected in video format and saved in a ROSBAG file. After, images were extracted from one of the lenses from the recorded file, sampled every five frames to reduce the correlation between images and avoid annotating similar images.

We selected three ROIs to feed the independent controllers of the three sprayer drums. These controllers needed to be fed the leaf area index, so it was selected using a classifier with four classes to reduce the system complexity. Typically, most vines are fully healthy (with all leaves) or simply dead, and the middle term is residual. Nevertheless, we added two more classes to produce a more precise system to consider this middle term. So, every image was manually annotated using an application that saved 1425 annotations in a text file. The application defines three ROIs (Figure 6, and then asks for the leaf density on each of the ROIs. The value is inserted in the four classes: 0% leaves, 33% leaves, 66% leaves, or 100% leaves. When there is no leaf in the region, it is considered 0%. When there are a few leaves, but in small quantities never exceeding half of the total area, it is 33%. If a large part of the area is covered with leaves, but not totally covered, it is considered 66%, and when the whole area is filled with leaves, it is considered a 100% class (Figure 7). This value is then saved in a text file together with the name of the image. The dataset is publicly accessible at [30].

5.2. Image Sensing System for Leaf Area Index (ISSLA)

We propose an approach called the image sensing system for leaf area index (ISSLA) to quantify the leaf area that needs to be sprayed. In ISSLA, we define three ROIs. For each ROI, with a size of 200 × 140 pixels, a visual descriptor that feeds a support vector machine is extracted. For ISSLA, fourteen descriptors (

Table 2. Different descriptors identification according to the combination of the six concepts used.

Descriptor	LBP	RGBVI	GLI	NGRDI	Average	Hue	Bins
1	X						59
2	X					X	69
3	X	X					69
4	X	X			X		71
5	X	X				X	79
6	X	X			X	X	81
7	X		X				69
8	X		X		X		71
9	X		X			X	79
10	X		X		X	X	81
11	X			X			69
12	X			X	X		71
13	X			X		X	79
14	X			X	X	X	81

) were designed and tested. These descriptors were reached by the concatenation of other histograms, which were built considering six fundamental visual concepts:

• Local binary pattern, with 59 bins, considering the uniform local binary pattern [31];
• Histogram of hue values, with 10 bins (10 bins were selected to have a minimal colour description and to avoid a long descriptor, which necessitates a bigger dataset), equally spaced from 0 to 256;
• Vegetation index, with 10 bins (10 bins were selected to have a minimal vegetation index (leaves health) description and to avoid a long descriptor, which necessitates a bigger dataset), where three were selected:

  –

  Red-green-blue vegetation index (RGBVI) [32],

  –

  Green leaf Index (GLI) [33] and

  –

  Normalized green red difference Iidex (NGRDI) [34].

• Average, with 2 bins, is the average of the vegetation index component of all pixels and the average of the green component of all pixels.

The vegetation index was considered to increase the robustness of the visual descriptor to differentiate leaves from other objects. The vegetation index is one parameter used to measure plant photosynthesis (leaves tend to have a higher index vegetation value than trunks and soil).

The formulation of the RGB-based vegetation indices is presented in

Table 3. Vegetation indexes equations. Rr = red, Rg = green, and Rb = blue.

Vegetation Index	Equation
RGBVI	(Rg × Rg) − (Rr × Rb)/(Rg × Rg) + (Rr × Rb)
GLI	(2 × Rg − Rr − Rb)/(2 × Rg + Rr + Rb)
NGRDI	(Rg − Rr)/(Rg + Rr)

.

The four histograms were concatenated into a single one, from 59 to 81 bins (Figure 8). An equalization was then applied to this histogram, forming the descriptor. This descriptor was used as the input for the SVM classifier (version 0.24.2 from scikit-learn [35]), which outputs the density of leaves in a given ROI of an image.

The result was then used to feed a previously trained support vector machine (SVM) that classifies the quantity of leaves existing in the initially selected descriptor.

5.3. Sprayer Control System

The controller system (Figure 9) is based on the prescription value, the robot velocity, and the quantity of leaves observed. The prescription value is obtained from the prescription map, which indicates the amount of product to apply in the different zones of the map from the location of the robot. In this way, with a prescription value, the robot velocity, the quantity of leaves, and the prescription gain, it is possible to obtain a corrected prescription value. The reference provided to the controller is obtained from the difference between this value and the water flow measured by the flowmeter.

The controller sets, for each spray drum, the air flow, the water rate, and the water density.

6. Tests and Results

6.1. ISSLA Evaluation

The performance of the ISSLA approach in detecting leaf density was evaluated. In this evaluation, we considered all fourteen descriptors (Table 2). Different tests were carried out to verify the accuracy of the ISSLA approach considering different combinations of four components: LBP (59 bins), vegetation index (10 bins), average (2 bins), and hue (10 bins).

In these tests, 475 images were considered, since three ROIs are defined in each image, so the dataset used has 1425 samples. For each sample, the necessary features were extracted for each histogram. The resulting histogram has all its components normalised to ensure that both have the same influence on the image characterisation. To train the SVM, 80% of the data was used, and the remaining 20% was used to validate and test its performance. This procedure was repeated in all fourteen tests and the results obtained from the testing dataset were recorded (

Table 4. SVM results using different combinations of the four features: local binary pattern (LBP), vegetation index (RGBVI, GLI, or NGRDI), average, and hue.

Descriptor	Accuracy
1	0.799621
2	0.832782
3	0.826870
4	0.826870
5	0.836303
6	0.836303
7	0.802721
8	0.806453
9	0.852779
10	0.845316
11	0.801811
12	0.822009
13	0.845316
14	0.851018

).

The results obtained show that using the LBP method (1) produces satisfactory results but can be improved using other components such as the vegetation indices or the hue component. Just by including the vegetation indexes (3, 7, and 11), there is some improvement, but only in descriptor 3, which uses RGBVI, but this improvement is not significant. However, when the average or hue components are introduced, they do not affect the results. The same does not occur in the remaining cases (9, 12, and 13), where these vegetation indexes and these components produce better results. The best combination is obtained on descriptor 9, which uses only three components, but descriptor 14 also has a similar result, where four components are used.

The accuracy (Table 4) was obtained from summing all diagonal values from the confusion matrix and dividing the result by four (average accuracy value). For example, the accuracy for descriptor 9 is the result of summing the diagonal values in

Table 5. Confusion matrix for descriptor 9 with the following features: local binary pattern (LBP), green leaf index (GLI), and hue.

		Predicted Value
		0%	33%	66%	100%
True Value	0 %	0.86440678	0.13559322	0	0
	33 %	0.11764706	0.76470588	0.11764706	0
	66 %	0	0	0.90140845	0.09859155
	100 %	0	0	0.11940299	0.88059701

and dividing by four.

All ISSLA configurations have an accuracy lower than 90%, which is justified by the proximity between classes. For example, in Table 5, we can observe that the ISSLA approach confused class 33% with class 0%. This can be found by the intersection region between these two classes. For example, the 0% class is never identified as 66% or 100%. The 33% class is only confused with the 0% and 66% classes.

The ISSLA performance is shown in a demonstration video (evaluated using real images, which can be accessed at https://youtu.be/911RWCx8WXc). This video presents the ISSLA approach considering a combination of LBP, GLI, and hue—the best descriptor obtained from our benchmark presented in Table 4. In this video, the robot is moving through a region without a vine and another tree in the background and still detecting the region as 0% leaves, proving the reliability of the ISSLA approach. So, we concluded that the ISSLA approach performs well even when another kind of vegetation emerges in the background.

6.2. PRYSM Sprayer Performance Evaluation

The PRYSM sprayer performance was validated according to the size and dispersion of the water particles obtained. Therefore, five tests were performed with water-sensitive paper (Figure 10). The maximum air flow was used in all tests, and the only varied references were both the centrifugal disk and the water pump (

Table 6. Qualitative results for the PRYSM sprayer obtained for five tests.

	Test 1	Test 2	Test 3	Test 4	Test 5
Air flow	100%	100%	100%	100%
Water rate	10%	100%	10%	10%	100%
Water density ${}^1$	10%	10%	100%	50%	50%
Water droplet size average	4 mm	5 mm	<500 $\mathsf{\mu }$m	2 mm	4 mm
Spraying quality	- -	- -	++	-	-

${}^1$ Water density had a direct relation to the centrifugal velocity.

).

Table 6 presents the qualitative results for the PRYSM sprayer. These qualitative results were obtained considering the water-sensitive paper marks made by the PRYSM sprayer tests. Figure 10 shows the water-sensitive papers obtained from the five tests (blue represents the water sprayed on the paper). We verified that for the same water pump reference, increasing the centrifugal disk reference causes the droplet size to decrease and its dispersion to increase. In contrast, for the same centrifugal disk reference, increasing the water pump reference causes the droplet size to increase and its dispersion to decrease. The best value was obtained under test 3, having the centrifugal velocity (water density) at its maximum value with a very low water rate and air flow at the maximum speed.

The final result of this work is presented in a video [36]. This video presents the sprayer design and shows how the robot works in a real scenario.

7. Conclusions and Future Work

In this work, we developed a novel precision spray tool. This tool was then integrated into a robotic platform capable of operating autonomously on rugged terrain with steep slopes and under the most diverse ground conditions. The system was tested in a real scenario in steep-slope vineyards. We also constructed an SVM classifier that measures the leaf density in three different regions according to the equivalent area to each drum’s spray area. The dataset used to test the performance of this system was made public.

The tests conducted with the designed sprayer proved that the solution has considerable potential to increase spraying accuracy and precision and is feasible for application in small robots.

Nevertheless, several improvements for future work were identified. These improvements result from contact with a company specialising in sprayers and certain aspects of which we became aware during this work.

The centrifugal disk design needs to be changed to create grooves from the centre to the edge to better direct the water projected onto the disk. The design used in this work has a flat surface that makes the water’s dispersion on the disk more random and difficult to control.

The system aerodynamics should also be improved to increase the air flow efficiency created by the propellers. The distance between the propellers and the aluminium plate, which forms the circular structure of each drum, should be reduced. The dimensions of the part supporting the water pipe can also be reduced to improve aerodynamics. Related to the air circulation, a change in the propeller motors to those with a higher power to cope with high leaf density can also be considered.

The image processing algorithms can be extended to consider other relevant features for treatments, such as grapes and trunks. It would also be interesting to consider and evaluate the need for a more continuous leaf area classifier, such as a linear regression formulation for the leaf area index.

This work was conducted and validated two different systems: an electric-based sprayer and a crop perception system. In the future, it will be essential to develop a control system capable of using the output of the crop perception system, the leaf density, and decide which values to assign as a reference for air flow, water rate, and water density.