Multi-Agent Vision System for Supporting Autonomous Orchard Spraying

Abstract

In this article, the authors propose a multi-agent vision system supporting the autonomous spraying of orchards and analyze the condition of trees and occurrence of pests and diseases. The vision system consists of several agents: first, for the detection of pests and diseases of fruit crops; second, for the estimation of the height of trees to be covered with spraying; third, for the classification of the developmental status of trees; and fourth, for the classification of tree infections by orchard diseases. For the classification, modified deep convolutional neural networks were used: Xception and NasNetLarge. They were trained using transfer learning and several additional techniques to avoid overfitting. Efficiency tests performed on the datasets with real orchard photos, showing accuracies ranging from 96.88% to 100%. The presented solutions will be used as part of an intelligent autonomous vehicle for orchard works, in order to minimize harm to the environment and reduce the consumption of water and plant protection products.

1. Introduction

Horticulture is a fruit-producing industry in which it is necessary to use plant protection products. Even in ecological orchards, protective treatments are carried out with the use of substances of a natural or biological origin. In the course of spraying, fruit trees are covered with a water-based solution by sprayers. Unfortunately, the chemical sprays expose operators with leaky protective clothing or an insufficiently ventilated tractor cabin to harmful dust. In recent years, the control systems of orchard tractors based on video signals have been improved, which extends vehicle navigation and allows for autonomous driving [1,2,3]. Sprayers and agricultural machines are also equipped with vision systems that ensure precise spraying and targeted cultivation [4,5]. Thus, it can be noticed that the control technology of autonomous vehicles [6,7,8] is beginning to be used for practical applications on arable plantations [9,10,11]. The level of robotic autonomy is raised by introducing control systems using intelligent data processing [12,13].

To systematize the level of advancement of solutions, the Horticulture 4.0 classification was introduced [14]. Horticulture 4.0 classifies the levels of advancement of the digitalization technology used to support production. There are three levels:

• Level 1—use of crop-monitoring sensors.
• Level 2—processing of monitoring data supporting decision-making.
• Level 3—production automation using autonomous systems.

Fruit tree pests and diseases usually infect tree leaves [15]. The leaf is a good indicator of plant morphological variability. A single leaf has a unique pattern. These patterns can be input data that the artificial neural network algorithms can use to recognize leaf types [16]. In laboratory conditions, the recognizability of leaf types was obtained at a level from 94.69% to 97.2%. Achieving high accuracy in recognition was possible by using a seven-layer ConvNet network with data augmentation for leaf recognition [17]. In other papers, a pulse-coupled neural network and a support vector machine (SVM) [18], or a probabilistic neural network (PNN) with image and data processing techniques [19,20], were used. In tomato cultivation under cover, deep artificial neural networks were used to detect leaf diseases and pests [21,22]. Recently, some attempts have also been made to automatically recognize leaf diseases using vision systems of artificial intelligence (AI), which offer high accuracy results in laboratory conditions only [23,24,25,26]. The work in [27] presents methods for distinguishing basic diseases of apple leaves—rust, scab, and black rot—based on AI. There are also first implementations of vision algorithms based on artificial neural networks that work in a real environment [28,29,30,31]. These applications, whether prepared for laboratories or tested at orchard plantations, have some imperfections and recognize only a few types of leaf diseases. Most of the presented solutions, used in practical systems of autonomous spraying, do not allow us to recognize irregularities, which, if they occur, can destroy plantations.

Automating the visual assessment of the condition of leaves and other tree features requires the recording of proper images and their analysis. A process of automatic image analysis consists, generally, of image acquisition, pre-processing, and then appropriate analysis. Marked objects are classified in order to determine their characteristics and/or cluster. The classification of objects is a task of digital image processing and machine learning, which typically consists of two stages: feature extraction and image validation with a selected classifier. To extract the essential features of an object and to reduce the amount of data, which must be further processed, many well-known architectures such as histograms of oriented gradients (HOG) [32], local binary patterns [33], and 1D/2D Haar descriptors [34], or their combinations [35], can be used. Then, the classification decision must be made. Different categories of classifiers are used in the validation stage: support vector machine (SVM), decision trees, AdaBoost, self-organizing maps, deep convolutional neural networks (DCNNs), and their combinations. Some of the most important CNN architectures are AlexNet/CaffeNet [36,37] (as the historically first significant CNN network), VGG [38], ResNet [39], Xception [40], NasNet (in classic or mobile version) [41], EfficientNet and EfficientNetV2 [42,43,44].

The presented literature review has shown individual solutions concerning topics similar to those discussed in this work, but to the best of the authors’ knowledge, no comprehensive fruit tree cultivation system has been proposed so far that would cover so many different aspects. This article presents the results of research conducted to develop vision algorithms for identifying diseases of fruit trees and classifying the developmental states of leaves based on CNN models. Additionally, a concept of using digital image processing algorithms to detect fruit tree pests and determine heights of trees for spraying in real time is presented. These studies constitute an important part of an integrated artificial intelligence system supporting autonomous orchard spraying.

Data for vision algorithms (vision agents) are obtained from the cameras placed on the orchard plantation and on the body of an autonomous orchard tractor with a sprayer. The vision agents supply information about the plantation to the database regardless of the spraying process or during spraying. The most advanced agents operate with artificial neural networks that recognize infected trees in an orchard plantation as a result of extracting leaf disease features and the developmental states of trees. The observation of pheromone traps and the control of leaf wetting time after atmospheric precipitation at an early stage allows for the detection of pests and disease infection, which triggers the implementation of preventive treatments. In this way, the number of treatments, used when symptoms of infection begin to appear on leaves and fruits, is reduced.

This paper is organized as follows: after the Introduction, in Section 2, the characteristics of the occurrence of selected diseases and pests of fruit trees are presented. Next, in Section 3, an intelligent solution is proposed. Section 4 presents the details of a multi-agent vision system. In Section 5, a system for autonomous protective spraying in horticulture, with an integrated AI multi-agent vision system application of convolutional neural networks for supporting orchard spraying, is presented. Section 6 summarizes the obtained results.

2. Characteristics of the Occurrence of Selected Diseases and Pests of Fruit Trees

Before preparing an AI system supporting the cultivation of fruit trees, it is necessary to study the strategies used to protect against the most common pests. Each species and even type of fruit tree requires a separate protective strategy. Obviously, there is the area of common or approximate actions that are taken during the vegetation season of fruit trees in order to prevent infections and pest appearance. In each case, it is extremely important that the right decisions are made regarding how to carry out the protective treatments. It should be noted that most pests that attack orchard plantations appear in particular time gaps, which is why their next appearance in the orchard season is called generation. Experienced growers strive to limit the number of pests as soon as they appear. A similar relationship occurs in the case of fungal diseases. In order to reduce the seeding of primary fungal spores, it is recommended to remove leaves remaining under the trees from the previous season, but the most important action is the appropriate mitigation response or removal of the appearance of the primary fungal infection, which reduces or eliminates the recurrence of the secondary fungal infection.

During the growing season, there may be a few or even a dozen infections that require preventive or intervention treatments. The most important strategies of protection, which have been included in this work, are:

• Strategy of defense against apple scab (lat. Venturia inaequalis): the development and sowing of spores of the fungus occurs in specific weather conditions; therefore, monitoring these conditions is of particular importance in the effective protection of apple trees against scab (Figure 1). Weather stations provide up-to-date information on changes in temperature, precipitation, air humidity and air pressure; these data are fed into the experimentally established model of the development and course of mycosis and are a source of knowledge used to make decision for a protective treatment.
• Strategy of defense against rhagoletis cherry (lat. Rhagoletis cerasi): protection against this pest consists of hanging traps in fruit plantations with a decoy attracting the rhagoletis cherry insect. In the Eastern-Europe geographical rank, outlets are already observed in mid-May; the observation of trap catches in an experimentally determined amount, as the economic harmfulness threshold for cherries, is a determinant for deciding to apply a protective treatment. During the ripening period of cherry fruits, there are several generations of pests that require spraying with a tractor connected to an orchard sprayer. The female pest lays eggs inside the developing fruit by cutting the skin. The white larvae hatch from the eggs and are about 4 mm long (Figure 1b); they cause the worming of fruit, making it unsuitable for consumption and industrial processing.

Effective protection against diseases and orchard pests consists mainly of prophylaxis to prevent the development of fungal pathogens or larvae that cause the degradation of green tissues of trees or fruit infestation. There is also irregular seasonality in the occurrence of certain types of diseases or pests, as well as irregularities in the severity. A strong dependence of the effectiveness of treatments on the weather conditions should also be emphasized. All these factors, together with current information from the visual inspection, should be taken into account when deciding to perform a specific protective or intervention treatment.

3. A Proposal for an Intelligent System for Autonomous Protective Spraying in Horticulture

Using the latest data processing methods based on AI [11,26], as well as the experience of one of the authors on the work in growing fruit trees and known strategies for their protection against pests [45], we propose an intelligent system for protective and autonomous spraying in horticulture based on a multi-agent vision system. The system we propose is located in the Horticulture 4.0 classification at levels 1 and 2 and also contains elements of level 3. The block diagram of this system is presented in Figure 2.

The system consists of ten blocks that perform the following tasks:

• Block 1: Disease monitoring.
• Monitoring the occurrence of diseases of fungal origin is carried out by measuring the wetting time of leaves after rainfall at a specific temperature. The leaf wetting time can be determined using video analysis. The measured values are then fed into the disease development model, on the basis of which the RIM (Relative Infection Measure) is determined [46]. Exceeding the experimentally determined RIM threshold is a signal of the risk of mycosis.
• Block 2: Video pest monitoring.
• Monitoring the occurrence of pests in orchard crops using a vision system provides information on the number of pests caught in special attracting traps [47,48]. Exceeding the experimentally determined number of pests per trap is a signal to perform spraying to remove pests.
• Block 3: Disease and pest data processing.
• Data from Blocks 1 and 2, obtained from weather stations and/or cameras located on the plantation, are processed in Block 3. If a disease or pest is detected in an orchard crop, output information is generated for the decision-making Block 6.
• Block 4: Protective strategy.
• This block contains a database with strategies for the protection of fruit plantations against particular threats. It also contains the assignment of protective measures allowed by relevant permits for fruit production. Each signal from Block 3 requires finding a suitable protection measure in Block 4.
• Block 5: Weather forecasts.
• Based on the data obtained from the forecasted weather conditions, the date of the procedure is calculated. A day is selected when the requirements for specific weather components are met, such as: temperature, humidity, wind strength, etc. This date is critical to the effectiveness of the spraying. For example, too strong a wind makes it difficult to properly cover the entire green tissue of the tree and causes a high consumption of plant protection products, as well as environmental pollution. Further, too low a humidity level causes the evaporation of part of the protection products into the atmosphere. At too low or too high a temperature, the effectiveness of the treatment is significantly lower.
• Block 6: Decision-making.
• After collecting and processing data about diseases and pests (Block 3) and additional data determining the protection measure (Block 4) along with the date of the treatment (Block 5), an automatic decision is generated. At the current stage of testing the proposed system, before further stages (i.e., before performing the tree protection treatment), an additional, manual approval by the experienced fruit grower is required to verify actions and avoid unnecessary treatments in the case of wrong decisions (Figure 3).

• The decision of the grower to approve or cancel the automatic suggestion is additional information that teaches the decision-making AI system. After several seasons of testing and learning the system, it will probably be possible to skip the manual acceptance step.
• Block 7: Orchard messages.
• In this block, messages with up-to-date news generated by the commercial consulting companies from the fruit industry, for an area where the plantation is located, are obtained. These messages indicate, to some extent, the type and date of necessary orchard spraying. The messages can support the decision-making Block 6, but also, the local decision to perform the procedure made by Block 6 can be made available to the consulting companies for further processing or publication (see the two-way green arrow between Blocks 6 and 7 in Figure 2).
• Block 8: Plantation map.
• To perform autonomous protective measures, a numerical map of the orchard should be prepared. This map enables the movement of an autonomous tractor connected to an automated sprayer in a given area of an orchard plantation. The map includes the distribution of trees on the plantation, as well as the possible routes.
• Block 9: Realization of autonomous spraying.
• When the grower accepts the decision made by the integrated AI system, autonomous spraying is carried out. It is performed by an automated orchard tractor with a sprayer. The tractor is equipped with the cameras. The video streams from the cameras are transmitted to the specially designed video processing modules which perform three tasks: They support the vehicle control system, control the automatic sprayer, and collect data about the plantation, e.g., size of trees, condition of leaves, damages, possible tree diseases (especially new symptoms that were not detected before spraying), etc.
• Block 10: Inspection data processing.
• The data acquired during the spraying (Block 9) are then processed and used to make further decisions about the next necessary spraying if infection symptoms are detected. This block provides feedback in the system, making decisions more precise. Additionally, it updates the plantation map (connection to Block 8, see Figure 2).

As presented in the detailed task list above, the system is a multi-agent system, and most of the agents process images. Because the thorough testing of the system requires several annual cycles, currently, the proposed system is in development and still in the preliminary testing phase.

The following sections will primarily describe agents related to video processing and artificial intelligence. Their tasks are mainly located in Blocks 2 and 9 (cf. Figure 2).

4. Multi-Agent Vision System

The presented multi-agent vision system supports three main tasks: pest monitoring, tractor control, and sprayer control. The system uses IP color Gemini 612-23W cameras (Delta-Opti Poznań, Poznań, Poland) with a resolution of 1.4 Mpix (1280 × 720 px) at 25 fps with a 1/4 inch OmniVision OV9712 CMOS sensor, a H.264 video codec, and a 10/100 Base-T Ethernet interface.

4.1. Visual Pest-Monitoring Agent

A scheme of the visual pest-monitoring agent is presented in Figure 4. As was mentioned in previous section, this agent monitors the occurrence of pests in orchard crops and provides information on the number of pests caught in special attracting traps. The pheromone trap is continuously observed by the inspection camera (Figure 5a). The image from the camera is denoised and binarized with a threshold in the preprocessing step. This step allows us to remove noise and unwanted objects (e.g., dust or flies that are not pests) from images. Then, insect pests of a certain size are detected and counted by matching to known object pattern definitions (Figure 5b). If the number of pests in the trap exceeds the given number, the agent produces a signal to perform spraying to remove pests. Optionally, the report is supplemented by the resulting image for additional verification.

4.2. Autonomous Tractor Control Agent

The basic control of a moving vehicle in a closed area inaccessible to outsiders (as in the presented case—in the orchard) can be performed using an autonomous driving agent. The diagram of the control algorithm is shown in Figure 6. The tractor is equipped with a front camera located on the roof of the tractor (Figure 7a). The image from the camera provides information to the autonomous driving agent and can also be used for the remote, manual control of the vehicle via a wireless network (Figure 7b) [7]. Autonomous driving or remote, manual control are much better than classic tractor driving during spraying: the operator is not exposed to harmful conditions. However, it requires the introduction of additional security procedures.

Remote, manual vehicle control may be necessary in emergency situations, e.g., when there is a need to guide the manual vehicle to the correct trajectory or the pre-set trajectory needs to be modified. The tractor’s autonomous control system allows for the detection and recognition of objects appearing in the field of view of the front camera. It supports the function of avoiding obstacles and emergency stopping when it is impossible to avoid an obstacle. Since the orchard is an environment with little variability in time, it is possible to program the spraying trajectory that will be performed by the autonomous driving agent in advance, using the orchard map. Information about the tractor’s location is obtained from the location system. The location system may use a global GPS module or a local vehicle-positioning module based on changes in the radio signal strength provided by local Wi-Fi wireless network stations [7]. In the future, vehicle location may be supported or entirely carried out by a vision system.

4.3. Automatic Sprayer Control Agent

The key function of the vision agent controlling the sprayer is the appropriate dosage of the tree protection product. This agent recognizes the developmental state of trees, detects tree height, and additionally, detects infected trees for further analysis.

A diagram of the sprayer control agent is shown in Figure 8. The agent receives the signal from a camera placed on the tractor hood, which is directed sideways, towards the row of trees being sprayed (Figure 9a,b).

The image from the camera is processed by a vision algorithm [49] that detects the outline of the tree by detecting leaves and branches in four zones (Figure 10). A special function has been prepared for this task. First, edge detection is performed using a Sobel mask, and then one of the morphological operations is performed—dilation. Image processing is performed sequentially to find the outline of a branch with leaves. Then, a structural element is created—a rectangle of a specific size that covers the detected contour. The trees are then assigned to one of four tree height categories. Depending on the assigned tree height category, a smaller or larger number of solenoid valves controlling the sprayer nozzles is opened (Figure 9c). This allows for spraying only the required area of the tree, and at the same time, it saves water and plant protection products.

In addition to detecting tree height, the vision agent detects the tree’s developmental state and tree infections, which modify the local spraying decision. For example, trees without leaves will not be sprayed. Since automatically detecting the developmental state of a tree, as well as tree diseases, based only on image analysis is a difficult task, the authors decided to solve the problem using convolutional neural networks and carefully examine their operation.

5. Application of Convolutional Neural Networks for Supporting the Orchard Spraying

During spraying, the video agent that controls the orchard sprayer, apart from determining the height of the spraying tree, also recognizes the developmental state of trees and inspects the plantation by a classification of infected trees. The last two functions are realized in the presented system using convolutional neural networks (CNNs). In this section, we deeply analyze these tasks by performing experimental analysis.

5.1. Classification of Stages of Fruit Tree Development

Recognizing the developmental stages of fruit trees involves classifying images obtained from a fruit plantation into five separate sets related to the development of trees during the fruit season. For a proper spraying performance, we are interested in distinguishing the first five stages of the fruit season: leafless, before flowering, blooming, after flowering, and bud growth (cf. Figure 11). During the last stage, the leaves develop the most intensively on trees, and it is necessary to regulate the spray pressure depending on the given developmental state in order to obtain a complete coverage of the green tissue.

For training the neural networks to work properly with the specific task and then testing their operation, the proper database of photos should be used. Since no appropriate database of photos of the considered growth stages of fruit trees was found, the authors prepared the photo database themselves. During the growing season, 7751 photos were taken in the fruit orchard at different times of the day.

Table 1. Number of photos for testing the developmental stages of trees.

Tree States	Number of Samples
leafless period	1440
before flowering	1634
flowering period	1408
after flowering	1766
bud growth	1503

and Figure 11 present some details about this database.

5.2. Classification of Trees Infections

The proposed vision agent can also detect tree diseases and classify disease symptoms identified during spraying. This is helpful for identifying rare and non-cyclical diseases, which, if detected early, prevent the destruction of the plantation.

Similarly, as in the previous classifier, we prepared a database of 3354 photos for training and testing the neural networks.

Table 2. Number of photos for testing infected trees.

Trees State	Number of Samples
uninfected	1609
infected	1745

and Figure 12 present some details about the database of photos dedicated to the fruit tree infections.

5.3. Used CNNs: Architectures, Supporting Techniques

5.3.1. Architectures

For performing the assumed tasks, we adopted two convolutional neural network models, namely: Xception [40] and NasNetLarge [41].

The Xception model is an extended version of the Inception V3 model, where Inception blocks have been replaced with convolutional layers used separately for individual image channels. Xception is divided into three main blocks: entry flow, middle flow and exit flow. As a result, it is a linear stack of convolutional layers, with additional connections (residual connections). Compared to Inception V3, Xception shows a small increase in classification accuracy on the ImageNet dataset and a much larger increase in accuracy on the JFT dataset.

The second CNN architecture we used was the NasNet (Neural Architecture Search Network) model. It was originally developed semi-automatically using the reinforcement learning method. Due to the time-consuming process of searching for the optimal architecture, the analysis was performed on a small dataset, and then the block configuration was transferred for verification on a larger dataset. During the experiments, the optimal architecture of a normal cell and a reduction cell was searched for. Although the search was performed on the CIFAR-10 basis, the use of the same cell architecture based on ImageNet allowed us to achieve a very high accuracy of 82.7% (top-1) and 96.2% (top-5).

5.3.2. Transfer Learning

In the case of an insufficient training set, it is possible to use an auxiliary training method—transfer learning. This method uses two stages of training. In the first stage, the model is trained on a very large database (typically, an image net for the object classification task). In the next step, the model is trained on the target learning database. For this purpose, a previously trained model with frozen weights is used. Only the last layers are trained (usually three layers of fully connected neurons), which act as a classifier in the network. In this approach, training is faster and does not require such a large number of training examples.

In the experiments carried out, the Xception network with weights trained on the ImageNet basis was used. Then, the network architecture was adapted to training with the transfer learning method by adding a layer of one-way perceptrons. A summary of the modified Xception network architecture used to classify tree development stages is presented in

Table 3. Summary of the modified Xception network architecture: additional layers: sequential is a linear stack of layers, each with one input and one output tensor, rescaling is a preprocessing step that normalizes or standardizes input, global average pooling summarizes the values of all neurons for each patch of the input data into a feature map, dense is a regular deeply connected neural network layer, dropout is a regularization method that randomly shuts down some fraction of a layer’s neurons during training to reduce overfitting.

Layer (Type)		Output
Shape	Param
input_4 (InputLayer)	[(None, 331, 331, 3)]	0
input_4 (InputLayer)	[(None, 299, 299, 3)]	0
sequential_1 (Sequential)	(None, None, None, 3)	0
rescaling_1 (Rescaling)	(None, 299, 299, 3)	0
NASNet (functional)	(None, 10, 10, 2048)	20,861,480
global_average_pooling2d_1 (GlobalAveragePooling2D)	(None, 2048)	0
dropout_1 (Dropout)	(None, 2048)	0
dense_1	(None, 5)	10,245
Total params: 20,871,725
Trainable params: 10,245
Non-trainable params: 20,861,480

. For tree infection classification, only the number of perceptrons in the last layer was changed. Network training was performed only for the new network layer (10,245 parameters trained); the remaining weights were frozen (i.e., approximately 21 million parameters of the Xception architecture).

The NasNet network in the NasNetLarge version was used with weights trained on the ImageNet basis. Then, the network architecture was adapted to training with the transfer learning method by adding a layer of one-way perceptrons. A summary of the modified NasNetLarge network architecture used to classify tree development stages is presented in

Table 4. Summary of the modified NasNetLarge network architecture: additional layers: sequential is a linear stack of layers, each with one input and one output tensor, rescaling is a preprocessing step that normalizes or standardizes input, global average pooling summarizes the values of all neurons for each patch of the input data into a feature map, dense is a regular deeply connected neural network layer, dropout is a regularization method that randomly shuts down some fraction of a layer’s neurons during training to reduce overfitting.

Layer (Type)		Output
Shape	Param
input_4 (InputLayer)	[(None, 331, 331, 3)]	0
sequential_1 (Sequential)	(None, 331, 331, 3)	0
rescaling_1 (Rescaling)	(None, 331, 331, 3)	0
NASNet (functional)	(None, 11, 11, 4032)	84,916,818
global_average_pooling2d_1 (GlobalAveragePooling2D)	(None, 4032)	0
dropout_1 (Dropout)	(None, 4032)	0
dense_1	(None, 5)	20,165
Total params: 84,936,983
Trainable params: 20,165
Non-trainable params: 84,916,818

. In the case of the classification of tree infections, only the number of perceptrons in the last layer was changed. The network was trained only for the new network layer (20,165 parameters were trained), and the remaining weights were frozen (about 85 million parameters of the NasNet architecture).

5.3.3. Preventing Network Overfitting

Model overfitting is a significant risk when training a neural network. It manifests itself mainly with the high value of accuracy obtained at the model training stage, which does not result in high accuracy on the testing set. In order to ensure the stability and reliability of the artificial network training process, a number of methods were used. The following training techniques were additionally used at different stages of learning [50,51]:

• Data augmentation techniques—adding random manipulations into training images:
• Image mirroring in vertical axis (none, mirroring).
• Image rotation (with angles from −10 to 10°).
• Image zooming (up to 40%).
• Image translation (up to 10%).
• Contrast adaptation (with contrast factor = 0.3).
• Training, validation and testing datasets—the whole database was divided into appropriate parts. The author’s database of photos was divided into training, validation, and testing sets in the proportion of 60%, 20%, and 20%, respectively.
• K-fold cross-validation—K-fold training of the model: each time, a different part of the dataset becomes a testing set.
• Dropout—randomly forgetting certain neurons and not transmitting information.
• Early stopping—saving of model weights for highest accuracy on the validation set during the training process.

5.4. Experimental Results

Results of experiments on the training, validation and testing of the prepared CNN architectures are presented below. The results were evaluated based on the following expression for accuracy:

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$$

where: TP—True Positives, TN—True Negatives, FP—False Positives, FN—False Negatives.

5.4.1. Classification of the Tree Development Stages

The results of experiments for the classification of tree development stages according to K-fold cross-validation are presented in

Table 5. Classification accuracy of tree development stages for the Xception model.

K-fold	Accuracy [%]
-----------	Training	Validation	Tests
1	97.95	99.87	99.16
2	97.98	99.14	97.44
3	96.55	99.42	97.24
4	97.43	97.89	98.57
5	97.21	98.14	98.17
Average	97.42	98.89	98.12

(for the Xception model) and

Table 6. Classification accuracy of tree development stages for the NasNetLarge model.

K-fold	Accuracy [%]
-----------	Training	Validation	Tests
1	98.40	100.00	99.12
2	97.86	98.45	97.23
3	98.23	99.13	96.88
4	98.07	97.45	97.13
5	97.67	98.51	98.21
Average	98.05	98.71	97.21

(for NasNetLarge). The accuracy plots in the training validation sets for both models are presented in Figure 13 and Figure 14, respectively, for the Xception and NasNetLarge models.

5.4.2. Classification of Trees Infections

The results of experiments for the classification of healthy and infected trees according to K-fold cross-validation are presented in

Table 7. Classification accuracy of trees infections for the Xception model.

K-fold		Accuracy [%]
-----------	Training	Validation	Tests
1	97.06	100.00	100.00
2	100.00	100.00	97.06
3	100.00	100.00	97.06
4	97.06	100.00	97.06
5	97.06	100.00	97.06
Average	98.24	100.00	97.65

(for the Xception model) and

Table 8. Classification accuracy of trees infections for the NasNetLarge model.

K-fold	Accuracy [%]
-----------	Training	Validation	Tests
1	99.81	99.50	100.00
2	99.91	99.25	100.00
3	100.00	100.00	100.00
4	100.00	100.00	98.43
5	100.00	98.70	99.20
Average	99.94	99.49	99.78

(for NasNetLarge). The accuracy plots in the training validation sets for both models are presented in Figure 15 and Figure 16, respectively, for the Xception and NasNetLarge models.

5.5. Discussion of Results

For both tasks, i.e., for the classification of tree development stages and for the classification of tree infections, very high accuracies of classifications were achieved (at least 97% on the testing datasets). Regardless of the network model used, the obtained results were quite similar. This was also confirmed by the K-fold cross-validation approach (see Table 5, Table 6, Table 7 and Table 8).

In general, larger fluctuations in accuracy (in fact in loss and accuracy) during the training process (see Figure 14 and Figure 15) were observed for the validation set than in the results obtained on the training set. This fact is related to the smaller size of the validation set. During the testing process, no significant model over-fitting was detected. This proved that strong-enough methods of data augmentation were used.

In the case of the classification of healthy and infected trees, the CNNs achieved even higher accuracy, i.e., up to 99.78% with the NasNetLarge model on the testing dataset. This is due to the fact that the analyzed images with the plant classes of healthy and diseased trees were easier to visually differentiate from each other, and just two classes were used (healthy and diseased trees). In the future, we also intend to expand this database, but this is a difficult task due to the periodic nature of the observed diseases.

6. Conclusions

In this article, a multi-agent vision system supporting the autonomous spraying of orchards and analyzing the condition of trees and the occurrence of pests and diseases is proposed. Local data collected by agents with information about tree pests, diseases and tree development stages, combined with data from external sources, e.g., weather stations or companies presenting information for fruit growers, are input to the AI decision-making module. Complex tasks such as the classification of the developmental status of trees and the classification of tree infections by orchard diseases are carried out using two CNN types: Xception and NasNetLarge. Efficiency tests performed on datasets with real orchard photos have shown accuracies ranging from 96.88% to 100%.

The proposed architecture of the artificial intelligence system is a part of the area of precision horticulture (HA), which optimizes food production areas, minimizes costs and effectively increases production results. The models, based on which the presented research results were carried out, are part of an artificial system prepared to perform integrated autonomous orchard spraying.

The multi-agent vision system saves the time needed for classic plantation inspection by fruit growers and provides precise data on the areas of the orchard that require spraying. The precise control of the sprayer, which detects heights of trees and their infections, makes it possible to save sprayed water and plant protection products.

The presented solution can be widely implemented on new or even used tractors and orchard sprayers after their rather inexpensive modernization.

In the coming growing seasons, the authors plan to implement the presented system as a whole and examine the achieved overall effectiveness of the proposed solution.