A System for the Direct Monitoring of Biological Objects in an Ecologically Balanced Zone

Abstract

This article discusses a model of a robotic platform that can be used for the proximal probing of biological objects in an ecologically balanced zone. The proximal probing is for scanning deciduous and fertile parts of biological objects with a hyperspectral camera at a distance of no more than a few meters. It allows for the obtention of information about the presence of phyto-diseases of tissues and also about the degree of ripeness and other parameters of the internal quality of the fruit. In this article, we report the methods and approaches used to detect fruits in the crown of a tree and also to identify their diseases such as scab and decay with an accuracy of at least 87%. For the autonomous movement of the platform in an ecologically balanced area, visual and inertial navigation is based on a Zed 2i stereo camera. This allows for the moving of biological objects in accordance with a given route indicated on the 2D map. The analysis of the information received from this platform allows for the building of maps of the presence of phyto-deseases in an ecologically balanced zone, and decisions are promptly made regarding the implementation of technical and protective measures that ensure high-quality products.

1. Introduction

The production of fruits and berries is extremely laborious and difficult to automate. There is a need to reduce labor intensity, minimize operating and production costs and the environmental impact and optimize the entire production cycle. The foregoing causes an increase in interest, among both researchers and practitioners, in the development of autonomous tractors [1] and autonomous and semi-autonomous robotic systems [2,3], which automate the main complex of mechanized works in fruit growing. The most typical tasks include soil sampling for agrochemical analyses, tree planting, foliar fertilizing, chemical treatments and fruit harvesting. Recently, routine planting monitoring, providing information for automated decision support in the precision gardening system, has been added.

Robotic platforms for mechanical and chemical weeding are represented by the following types: Hortibot (Aarhus University, Denmark) [2], Vitirover (NaïoTechnologies, France) [3], EcoRobotix (Switzerland), BoniRob (DeepfieldRobotics, Germany) [4], FarmWise (USA), AgBotII, Digital Farmhand (Australia) and FarmDroid (Germany). The efficiency of robotic weeders approaches 90%, whereas herbicide treatment using Drop on Demand (DoD) robotic systems can be 100% efficient [5]. Still, the information about the actual performance of the robotic platforms and the adaptability to different crops is scarce. An increase in their velocity, performance and surface area, as well as improving the accuracy of weed detection, are required to expand the scope of commercial applications of the robotic platforms.

Robotic technologies for the highly precise detection and identification of diseases, as well as insect pests [6], have been developed. Researchers proposed a convolutional neural network-based model for the multiple classification of insects and methods for rough detection and counting YOLO (You Only Look Once) and also classification and exact counting based on Support Vector Machines (SVM) using global functions [6,7]. These algorithms allow for the detection of the symptoms of powdery mildew, tomato spotted wilt virus, Xylella fastidiosa bacterial pathogen, etc.

An automated robotic platform (Bernard, France) has been developed for sugar beet phenotyping. It includes a wheeled mobile robot and a six-degrees-of-freedom manipulator for the colorimetric and geometric measurements of plants. T. Mueller-Sim, M. Jenkins, J. Abel and G. Kantor are with The Robotics Institute of Carnegie Mellon University and devised a robotic ground platform for the high-performance phenotyping of sorghum and corn crops. The fully automated platform for phenotyping from Rothamsted Research, UK includes high-resolution RGB cameras, as well as cameras detecting the brightness of chlorophyll fluorescence and thermal infrared cameras, two hyperspectrometers and two lidars. A number of commercially available autonomous and semi-autonomous robotic platforms for the harvesting of diverse fruits have been created (

Table 1. Robotic solutions for the automated monitoring and picking of fruits.

Fruit Crops	Manufacturer	Navigation	Country
Apples	Abundant Robotics	Lidar	USA
	FF Robotics	-	Israel
Strawberries	Dogtooth Technologies	GPS	UK
	Rubion Octinio	IR Tags	Belgium
	Thorvald II	Lidar	Norway
	Agrobot SW 6010	-	Spain
Bell pepper	Sweeper	Visual	Netherlands
Asparagus	Cerescon	-	Netherlands
Tomatoes	Metomotion	Visual	Israel
	Root-AI	Visual	USA
Oranges	Energid	-	USA
Cherries	Cherry-harvesting robot	Visual	Japan
	Agribot	GPS	Spain
Cucumbers	VanHenten	-	Netherlands
Eggplant	Hayashi	Visual	Japan
	Asparagus-harvesting robot	Visual	Japan
Watermelons	Umeda	Visual	Japan
Mushrooms	Agaricusbisporus	Visual	UK

).

Of particular concern is the development of mechanically reinforced frame elements and active manipulators of robotic systems [8]. Interfaces (for example, grips) should be small, minimal in weight and processed taking into account operational strength and plastic characteristics [9,10].

All of the robotic platforms mentioned above employ technical vision based on RGB and hyperspectral cameras operating in the IR and visible spectrum ranges. Acoustic distance sensors, gyroscopes, lidars, inertial measurement units (IMU) and GPS navigation (GPS RTK) are used for the positioning system. The robotic platforms include either 4- or 6-wheel chassis with 2-, 4- or all-wheel drive by servos and/or stepper motors.

Common tasks for robotic platforms include the route planning, detection and classification of the objects of interest (crops, pests, etc.), finding fruits in three-dimensional space using the input from stereo cameras and lidars, the dosing of chemicals, coping with the navigation system inaccuracy while driving through “unstructured” agricultural areas and ensuring safety by detecting and circumventing people and other obstacles. Roboplatform control problems also arise due to a technical vision failure arising from the uneven lighting of the scene, a nonlinear real-time response, position displacement due to drift and the accumulation of errors by the sensors. In addition, constant communication is necessary between works and objects, which requires the additional development of new methods of deep learning for their orientation in space.

The hyperspectral analysis research review:

Recently, the analysis of hyperspectral reflectance has been increasingly used for the remote and proximal probing of plant organs. This is due to the development of the technique and technology of hyperspectral cameras, increasing their availability and reducing the cost. Hyperspectral reflectance imaging in the visible and near-infrared spectral regions allowed for the detection and classification of apples infected with fruit worm (Laspeyresia pomonella) [11]. The most accurate classification was obtained using five channels (434.0 nm, 437.5 nm, 538.3 nm, 582.8 nm and 914.5 nm): 82% for the calibration dataset; 81% and 86% for healthy and damaged apples.

Reflectance in the visible and near-infrared range (400–900 nm) revealed the rot, bruises, wounds and soil contamination of apple and pear fruits of several varieties [12] using the second derivative of the reflectance spectrum in the absorption band of chlorophyll (centered at 685 nm) and two bands in the NIR region (950–1650 nm), regardless of the fruit color and variety [13]. Optimal results were obtained by using a spectral index in the form of a ratio of reflection coefficients at 1074 nm and 1016 nm. The accuracy of damage detection was 92%. Hyperspectral images were also used to predict the content of soluble solids in the tissues of apples of various colors [14].

In mathematical models describing the reflection spectra in the 450–1000 nm range to predict the dynamics of fruit hardness and the content of soluble solids in Golden Delicious apples, the accuracy of prediction peaked at 675 nm [15]. Hyperspectral reflectance images taken in the range of 400–1000 nm were used to determine the content of glucose, fructose and sucrose in whole kiwi fruits and their slices [16].

The optical properties of the skin and pulp of Breaburn, Kanji and Green Star apples varieties in the range of 500–1850 nm revealed spectral contributions to light absorption by carotenoids (500 nm), anthocyanins (550 nm), chlorophyll (678 nm) and water (970, 1200 and 1450 nm) [17]. The processing of hyperspectral images (400–1000 nm) with an artificial neural network (ANN) detected cold damage to apple fruits [18]. The ANN selected the optimal wavelengths (717, 751, 875, 960 and 980 nm) for classifying apples according to the degree of their damage. Spectral and spatial criteria made it possible to achieve an average accuracy of 98.4% in detecting damaged fruits.

The possibility of determining the stage of maturity of blueberries by hyperspectral imaging was studied [12]. Three methods for selecting informative spectral bands based on information theory and using the Kullback–Leibler divergence were used. This made it possible to achieve a classification accuracy of more than 88%.

Hyperspectral images were also used to monitor the fungal infection of dates by employing four spectral channels in the NIR (1120, 1300, 1610 and 1650 nm), with an accuracy of at least 91% [19]. A similar approach, implemented in the range of 400–780 nm, was used to classify tomato leaves into healthy leaves and those affected by gray mold [20]. Hyperspectral imaging also made it possible to analyze the spatial variation in the diffuse reflection of fruits (apples, peaches, pears, kiwi and plums) and vegetables (cucumbers, zucchini and tomatoes) in the spectral range of 500–1000 nm [21]. The NIR reflectance images made it possible to visualize quality parameters and predict the sugar content and hardness of melons using the Partial Least Squares Regression (PLSR) method, the Principal Component Analysis (PCA) and the Support Vector Machine (SVM) method, as well as an ANN; PLSR was evaluated as the best method [22].

Reflection in the ranges of 750–850 nm and 900–1900 nm has been used to detect bruises in apples of five varieties (Champion, Gloucester, Golden Delicious, Idared and Topaz) [23]. The spectra were pre-processed by baseline displacement/linear correction, multiple scattering correction (MSC), standard normal variation with trend elimination (SNV&DT) and the second derivative calculated by the Savitsky–Golay algorithm. The analysis of hyperspectral images obtained in the range of 500–1000 nm revealed the dependence of the apple storability decline on the energy of the damaging impact [10,11,12,13,24].

Hyperspectral images in the visible and NIR ranges (450–1040 nm) have been successfully used to monitor the ripeness of nectarines of the Big Top and Magic varieties (Prunus persica L. Batsch var. Nucipersica) using RPI and IQI indices calculated on the base of 650–730 nm and 940–1040 nm channels [25]. Hyperspectral images (the bands 650–850 nm and 900–1000 nm) were used to detect bruises and dents in peaches 12, 24, 36 and 48 h after their mechanical damage, as well as to detect various skin defects (damage by insects, phytopathogenic fungi, etc.) of two-colored Pinggu peaches [26].

Navigation systems enable the positioning of autonomous mobile robots and guide them along a preset route. Currently, satellite, inertial and visual navigation systems are widespread in agricultural robots, with a predominance of satellite navigation. However, the latter is limited by the instability of the satellite signal [26]. Therefore, full autonomy of a robot can be achieved only when using a combined navigation system that includes all of the above-mentioned components [27]. The most promising method of robot guiding is based on the imaging of the robot’s surroundings, mimicking the vision of living organisms—a basic and evolutionarily ancient way of perceiving information. It is also intuitively attractive since people and animals often use visual information to navigate in space.

Cameras used for visual navigation are divided into three types: monocular, binocular (stereo cameras) and depth (RGB-D) cameras [28]. Unlike monocular cameras, stereo cameras and RGB-D cameras obtain a depth map representing distances to the surfaces of objects in the camera’s field of view. The advantage of using such cameras for mobile robot navigation is that they provide ample capabilities for pattern recognition during mapping, allowing for navigation by landmarks [29]. Video records of geo-referenced landmarks allow for the determination of true coordinates and plan movement using a topological image of the surroundings.

Visual landmarks can be classified into artificial and natural ones. Detecting and tracking artificial landmarks make the task easier because of the optimal contrast, and their exact geometric and physical properties are known in advance.

The first artificial landmarks used in robotics were contrasting lines and crosses drawn on the floor. The approach when the robot control system constantly compares images of the target and current surroundings to adjust its trajectory is called visual servo control [30].

Another method of the visual navigation of agricultural robots is the use of natural differences in the color, e.g., treated vs. untreated areas of the field. This allows, using simple binarization, for the determination of the boundaries of the land plots. In combination with the least square method and morphological analysis, this approach is efficient in guiding ground robots [31]. The combination of Hough transformation with different navigation methods is more precise (its average error is below 0.5°, which is much lower than the average error of the least square method) and faster (by 35.20 ms, as compared with the calculation time of traditional conversion) [30,31,32].

A more advanced computer vision-based navigation system suggested for intellectual agricultural robots implementing Internet of Things (IoT) technology is navigation with binocular vision. It uses both land plot boundaries and the height of the plants extracted as the navigation parameters. The ANN- and deep learning-based methods of visual navigation are vigorously developing since they are suitable for navigating smart agricultural robots in complicated surroundings [33]. Since precise global positioning systems can be unavailable in orchards, a combination of 3D vision, lidar and ultrasound proximity sensors has been suggested to achieve the reliable navigation of agricultural robots [33,34].

Collectively, hyperspectral control is currently a ubiquitous approach to the monitoring of plant and fruit conditions, although a higher precision is desirable for the application of this method in precision horticulture. The common sources of error are the fluctuations of natural light conditions, shadows, the natural heterogeneity of fruit color, etc. Possible solutions include the use of a refined mathematical model for objects classification and an informed choice of images in spectral channels with minimal interference. Here, we propose such an approach in combination with the visual and inertial navigation on a single robotic platform.

2. Materials and Methods

2.1. A System for the Proximal Sensing of Apple Orchards

We developed a system for the proximal monitoring of apple orchards using a hyperspectral sensor for perceiving the fruit condition and an image sensor for navigation. The main component of the system was a robotic wheeled platform (1 in Figure 1) powered by 500 W electric motors and 30 Ah lithium-ion batteries, lasting for at least 2 h. The overall dimensions of the platform were 1.5 × 1.5 m, and the load capacity was 200 kg, which is enough to carry additional equipment, e.g., a manipulator for fruit sampling or harvesting (5 in Figure 1). As a model ground, we employed an intensive industrial apple orchard (Ligol variety on the rootstock 62–396, a 4.5 × 1.2 m planting) located in Tambov region, Russia (52.8800906, 40.488069). Information about the geographical location of the orchard and the topographic characteristics of the site can be obtained from yandex maps by clicking on the following link: https://yandex.ru/maps/225/russia/?l=sat%2Cskl&ll=40.490031%2C52.880102&mode=searse&sll=40.488069%2C52.880091&text=52.880091%2C40.488069&z=18 (accessed on 10 September 2021). A circle on the map (discussed below in the text) marks a garden plot in the vicinity of the city of Michurinsk (Tambov region, Russia).

In the orchard, the platform is guided by a computer vision system based on a Zed 2i (Stereolabs, San Francisco, CA, USA, https://www.stereolabs.com/zed-2i/ (accessed on 17 August 2021)) stereo camera (2 in Figure 1) with a built-in function for inertial navigation (gyroscope, magnetometer, etc.) and an industrial computer AIE100-903-FE-NX (Axiomtek, Taipei City, Taiwan) (4 in Figure 1). A pre-built 3D map of the orchard is also required for the navigation. The platform also supports manual remote control for building the map when an operator guides the platform along each row of the orchard both in the forward and reverse directions to collect the images as well as the magnetometer and gyroscope readings. In the process of moving the platform around the garden using the Zed2i camera, the following information is collected: left and right rectified/unrectified images; depth data; IMU data; visual odometry: the position and orientation of the camera.

The collected data in the form of an SVO file was transmitted to the cloud storage, and the 3D depth map was generated on a local computer with an nVidia RTX3070 GPU (nVidia, Santa Clara, CA, USA). 3D point cloud generation is performed automatically using the ZED ROS2 wrapper package (https://www.stereolabs.com/docs/ros2/ (accessed on 24 August 2022)), launched under ROS2 Foxy Fitzroy (https://docs.ros.org/en/foxy/Installation/Ubuntu-Install-Debians.html (accessed on 6 September 2022)). The resulting 3D map is loaded onto the platform’s onboard computer and the operator’s tablet. The original application installed on a tablet transforms the 3D map into a 2D map, which is more convenient for the user in marking the route for the platform. The conversion of a 3D map to 2D is carried out using the package Rtabmap_ros (https://github.com/introlab/rtabmap_ros (accessed on 6 September 2022)). The pre-set route in the form of a coordinate array is transmitted to the onboard computer of the platform.

To monitor the plants, the platform is equipped with a hyperspectral sensor (3 in Figure 1) represented by an FX10 line scanner (SPECIM, Oulu, Finland) driven by a two-phase stepper motor, YK286EC156A1 (Yako, Shenzhen, China), controlled from the computer via an SSD2608H (Yako, Shenzhen, China) driver. The hyperspectrometer featured a 400–1000 nm spectral range and a 1024-pixel line sensor (the spectral band width was 2.6 nm); the shooting speed was 330 lines per second. The light source was a 50 W tungsten-halogen lampmounted on calipers for vertical movement. To obtain a 2D hyperspectral image, the camera moved linearly in the vertical direction; the lines recorded during the camera movement were programmatically joined into two-dimensional hyperspectral scans.

2.2. Robotic Platform on Board the Control System and the Sensors

The platform was controlled by a high-performance industrial computer, AIE100-903-FE-NX (Axiomtek, Taipei City, Taiwan), equipped with an NVIDIA Jetson Xavier NX module with a 6-core 64-bit processor and a 384-core NVIDIA^® Volta™ graphics processor architecture. This setup was capable of operation outdoors in real orchards in the −30 °C to +50 °C temperature range.

The platform control system is shown in Figure 2. To navigate the platform, a Zed 2i stereo camera (IP66 protection class) was connected to the onboard computer via a USB Type-C interface. Control signals were transmitted to the electric motors of the wheels through low-current control and power relays; the digital control signals were relayed through an optocoupler and a system of transistor switches.

The platform was also equipped with a remote control (RC) based on the Heltec ESP32 (Heltec, Chengdu, China, https://heltec.org/?s=ESP32, accessed on 8 August 2022) microcontroller with LoRaSX1276 (Semtech, Camarillo, CA, USA) modules equipped with a joystick and buttons. The RC was used to guide the platform when building a 3D orchard map or during platform staging.

The main component of the navigation system of the robotic platform was a stereo camera Zed 2i, which was protected according to the IP66 class. It takes images with two RGB cameras with a 120° FoV; it also uses IMU sensors for positioning and spatial orientation. The stereo camera is mounted on a device to stabilize its position in the horizontal plane. The images were processed with an onboard industrial computer, AIE100-903-FL-NX (Axiomtek, Taipei City, Taiwan), with an NVidia Jetson Xavier NX GPU.

The software on the onboard computer processes the stereo images from the Zed 2i camera together with the readings of the magnetometer, gyroscope and barometer. It calculates the current coordinates ($x, y, z$) of the platform in the coordinate system $\mathrm{O}{x}_0{y}_0{z}_0$ of the pre-built orchard map, as well as the rotation angles $\alpha , \beta$ and $\gamma$ of the platform coordinate system relative to the axes $z_0$, $y_0$ and $x_0$, respectively.

Disregarding the difference in the elevations, the trajectory of the platform movement in the orchard can be represented as a broken linear function in the horizontal plane $x_0-y_0$ (Figure 3). Considering the general case when the platform is located at point A, the coordinate system of the platform $x_1{y}_1{z}_1$ is rotated relative to the axes $z_0$, $y_0$ and $x_0$ at angles ${\alpha }_A$, ${\beta }_A$ and ${\gamma }_A$, and the optical axis of the camera coincides with the axis $x_1$. To proceed to point B, the platform must turn counterclockwise at an angle of ${\phi }_A$ and then move in the direction of the vector $\overrightarrow{\rho }$ until the platform occupies a circle with a center in point B which is equal to the largest dimensions of the platform.

Provided that the stereo camera is stabilized in the horizontal plane and the $z_0$ and $z_1$ axes remain parallel, the following Equation (1) is valid for determining the rotation angle of the platform:

$${\phi }_A=atan\left(y_{B1},x_{B1}\right)$$

(1)

The coordinates of point $B$ in the coordinate system of the platform are determined from Equation (2):

$$\overrightarrow{\rho }=M^{-1}\left(\overrightarrow{R}-\overrightarrow{r}\right)$$

(2)

where $M$ is the rotation matrix of Equation (3):

$$M=\left(\begin{array}{ccc}\cos {\alpha }_A& -\sin {\alpha }_A& 0\\ \sin {\alpha }_A& \cos {\alpha }_A& 0\\ 0& 0& 1\end{array}\right)$$

(3)

It is at an arbitrary angle ${\alpha }_A$ around the axis $z_1$ relative to the coordinate system $\mathrm{O}{x}_0{y}_0{z}_0$;$R$, $r$—the coordinate vectors of the points B and A in the coordinate system $\mathrm{O}{x}_0{y}_0{z}_0$.

The platform navigation system works as follows. It takes a pre-built 3D map of the orchard as an input together with a travel route specified as a matrix of ($x, y$) coordinates of nodal points (A, B, …) comprising the route. The platform software continuously analyzes the data received from the stereo camera Zed 2i and its built-in 9DoF IMU sensors determining the current coordinates and orientation of the platform in the coordinate system of the 3D map. Based on these data and the coordinates of the nearest route point, the resulting rotation matrix M is calculated, along with the vector $\rho$ of the coordinates of the nearest route point in the coordinate system of the platform and the angle of rotation of the platform $\phi$. The platform turns in the given direction and moves, following the specified course, to the vicinity of the nearest route point. After this, the next closest point of the route is taken, and the procedures of orientation and movement are repeated until the end of the route.

3. Results and Discussion

3.1. Construction of 3D and 2D Maps of the Orchard

For successful platform navigation, a 3D map of the orchard has been built by the manual guiding of the platform along each row (Figure 4), recording, with the Zed 2i stereo camera, a depth map, as well as information about the camera orientation angles in space and camera coordinates relative to the initial waypoints. The soil of the rows on which the platform was moving was covered with weeds. The raw data collected in this way were transferred to the cloud and processed as described above (Figure 5). To process the raw data received from the Zed2i camera, we used the ZED ROS2 wrapper (https://www.stereolabs.com/docs/ros2/ (accessed on 19 August 2021)), launched under ROS2 Foxy Fitzroy (https://docs.ros.org/en/foxy/Installation/Ubuntu-Install-Debians.html (accessed on 6 September 2022)) on a cloud computer running the Ubuntu operating system 20.04 (https://releases.ubuntu.com/ (accessed on 8 August 2022)). The ZED ROS2 package used allows us to receive the following data from the Zed2i camera: left and right rectified/unrectified images; depth data; colored 3D point cloud; IMU data; visual odometry: position and orientation of the camera. The resulting 3D map was uploaded, through a cloud storage, to an operator’s tablet for the creation of a route for the platform.

The resulting 3D map was transformed into a 2D map (Figure 6) suitable for specifying the starting and ending points of the route. To build a 2D map, a 3D binarization operation is performed by zeroing the z coordinates of the points above the specified threshold (0.1 m) and setting the $z$ coordinates of the remaining points to unity, i.e., $P_{x,y,z}=\left\{P_{x,y,0}, if z>\mathrm{threshold}; P_{x,y,1}\right\}$. The resulting 2D map of the orchard depicts tree rows as obstacles (in black) and the spaces between them as passable roads (in white). The specified operation is performed automatically using the RTAB-Map (Real-Time Appearance-Based Mapping) package, detailed instructions for the use of which can be found at the following link: https://wiki.ros.org/rtabmap_ros/Tutorials/StereoOutdoorMapping (accessed on 24 August 2022). The resulting map is loaded onto a tablet computer, where the user can specify the finish point and the trajectory of movement.

Figure 6a shows the constructed 2D map of the orchard, and Figure 6b shows a fragment of the map on which the shortest trajectory of the platform movement from the starting point to the end point is marked with a red line, and the blue lines indicate possible directions for moving. The calculation of the shortest trajectory was carried out using the Probabilistic Roadmap (PRM) pathplanner of the Robotics System Toolbox software package. The PRM pathplanner constructs a roadmap in the free space of a given map by using randomly sampled nodes in the free space and connecting them with each other. The output of the PRM is an array of ($x, y$) coordinates of points comprising the path of platform movement.

3.2. Hyperspectral Monitoring of Fruit Condition

Figure 7a shows an image obtained as a result of the pixel-by-pixel subtraction of images obtained in the channels 666 nm and 782 nm (showing the greatest contrast between leaves and fruits) and normalized by the minimax algorithm [33,34,35]. Due to the fluctuation of the solar light intensity during image recording, e.g., in cloudy vs. sunny weather, the image can be either too dark or bright. In the example shown in Figure 7, the image is quite dark, so its histogram (Figure 8a) is shifted to the left (it lacks pixels with values above 60). The use of filters or grading transformations to stretch the histogram is difficult in this case since they require the manual adjustment of parameters such as the aperture or constants of grading depending on the level of illumination of the scene.

To solve this problem, adaptive histogram equalization was performed using the CLAHE (Contrast Limited Adaptive Histogram Equalization) method, which can be implemented without the intervention of the operator. The equalization was carried out after splitting the original image into blocks of 32 × 32 pixels. The result is shown in Figure 7b, and the corresponding histogram is shown in Figure 8b. The presence of peaks at brightness levels of 60, 140 and 190 corresponds to three objects: background, foliage and fruits in the image.

The target region of interest encompasses fruits (the foreground objects). If the scene is uniformly illuminated, these objects are easy to pick by performing a global thresholding operation with a manually selected threshold. However, in a real-life orchard under fluctuating solar light, the threshold value should be individually selected for each image; hence, an algorithm for the automatic selection of the foreground area is required. The technique of detecting near-field objects was tested at the landfill under conditions close to real ones. Tests have shown that for the sort “Golden”, the proposed approach allows for a detection accuracy of at least 95%. At the same time, taking into account the limited field of view of the SPECIM FX10 camera and the need for the 2D reconstruction of the image, it is advisable to use a multispectral camera or an RGB camera with optical filters at 666 and 782 nm to monitor near-field objects.

Such an algorithm includes some operations with constant parameters independent of illumination intensity. First, the image is broken into the tiles of 505 × 505 pixels. Then, adaptive mean thresholding is carried out for each tile, and the threshold value is automatically calculated as the arithmetic mean of the values of the pixels comprising the corresponding tile. The resulting image contained a lot of noise (Figure 9b), which was suppressed by applying the operation of erosion followed by dilation (in three iterations, by the convolution of the image obtained at the previous step with a kernel of 9 × 9 pixels, Figure 9c). After this operation, the images were still noisy, so the Distance transform was applied, yielding binary images (Figure 9d) with pixel values representing the distance to the nearest pixel of the background. The brightest pixels in the resulting image corresponded to the central part of the foreground objects. To isolate the entire object around the central (Figure 9e), the threshold transformation and dilate operation were consistently applied. The threshold was set to the maximum of 0.6·PD_t, where PD_t is the pixel value of the Distance transform image, and the expansion was performed in 20 iterations by convolving the binarized image with a 15 × 15 pixel kernel. Finally, after bitwise multiplication of the resulting image by the original image (Figure 9a), an image of the foreground area with apples has been obtained (Figure 9f).

After the adaptive threshold transformation for each image tile, the foreground object mask is obtained (Figure 10). Bitwise multiplication of the mask and the image of the scene allows for the selection of the area occupied by the image of fruits in any spectral channel.

3.3. Detection of Apple Damages in Hyperspectral Images

The area of the apple image was further analyzed for the possibility of the classification into the following groups: healthy apple (class 0), rotten (class 1), Jonathan spotting (class 2), insect damage (class 3) and scab (class 4). Apples of the following sorts were studied: “Imrus”, “Orel striped” and “Spartan”. These apples were obtained from experimental gardens of the Federal Scientific Center named after I.V. Michurin (Michurinsk, Tambov region, Russia). For these classes of apple surfaces, using the Specim FX10 camera, more than 52,000 spectrograms of optical radiation reflected from their surface were obtained. At the same time, a halogen lamp, R7S Haloline Eco 64690 Osram, 80W2900 K (China), was used as a backlight. The camera sensor signals were normalized to the maximum value and then averaged for each class of apple plant tissue. It was found that, at certain wavelengths, the listed classes of plant tissues have common patterns of radiation reflection (Figure 11a–e), which makes it possible to use multidimensional discriminant analysis to classify apples by defects.

Discriminant functions were used in the classification, containing vegetation indices as predictors and taking into account the patterns of radiation reflection at wavelengths of 500, 510, 550, 680, 700, 750, 905 and 962 nm. The values of the coefficients of discriminant functions were found in the process of learning from more than 1000 reflection spectra from sections of plant tissues of apples of each class of the above sorts. In addition, 150 spectrograms were obtained for each apple sort for all classes of plant tissue, which were later used to independently verify the workability of the proposed classification method.

As a result of the application package SPSS Statistics, the following classification functions were obtained:

$$F_i=k_1\left(\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$R{I}_{510}$}\right.-\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$R{I}_{700}$}\right.\right)+k_2\left(\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$R{I}_{550}$}\right.-\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$R{I}_{700}$}\right.\right)+k_3\raisebox{1ex}{$\left(R{I}_{680}-R{I}_{500}\right)$}\!\left/ \!\raisebox{-1ex}{$R{I}_{750}$}\right.++k_4\raisebox{1ex}{$\left(R{I}_{750}-R{I}_{680}\right)$}\!\left/ \!\raisebox{-1ex}{$\left(R{I}_{750}+R{I}_{680}\right)$}\right.+k_5\raisebox{1ex}{$R{I}_{905}$}\!\left/ \!\raisebox{-1ex}{$R{I}_{962}$}\right.+k_6\raisebox{1ex}{$\left(R{I}_{680}-R{I}_{640}\right)$}\!\left/ \!\raisebox{-1ex}{$R{I}_{680}$}\right.+k_7$$

(4)

where i is the class of the plant tissue of apples, and RI_λ is the intensity of reflected radiation with a wavelength λ from the surface of the apple.

Table 2. Examples of the values of the coefficients of the classification.

Sort	Class	k1	k2	k3	k4	k5	k6	k7
IMRUS	0	−90,772.2	72,274.5	432.2	224.9	113.8	−141.8	−198.1
	1	−24,073.2	33,071.5	226.1	139.6	116.9	−39.36	−143.5
	2	−7872.68	15,772.93	229.67	95.6	132.42	−61.36	−171.70
	3	−18,502.0	19,299.0	231.7	147.8	97.2	−28.3	−113.0
	4	−27,195.1	21,123.5	290.0	214.5	103.7	−18.6	−145.4

shows, as an example, the values of the coefficients of the classification function for apples of the IMRUS variety obtained for a training sample of spectrograms.

Table 2 shows examples of the values of the coefficients of the classification function for apples of the IMRUS sort obtained for a training sample of spectrograms. At the same time, the value of the Wilkes lambda criterion is less than 0.007, which shows a significant difference in the average values of discriminant functions in the studied classes.

The prediction results from the control sample of the spectrograms show that the classification accuracy is not worse than 80%. The prediction accuracy of healthy plant tissue without defects is no worse than 98%. The class of the surface of the plant tissue of apples is assumed to be equal to the index of the classification function, which has the maximum value, with the values of predictors obtained experimentally

4. Conclusions

Here, we report on a complex solution for the proximal sensing of apple fruits and trees in an industrial orchard. The developed solution consisted of a robotic platform with an electric drive and an algorithm for its combined (visual and inertial) navigation based on the input from diverse sensors including a stereo camera. The detection of fruit damages by phytopathogens was accomplished by the use of a hyperspectral camera; special attention was paid to the pre-selection of the region of interest containing images of fruits. Overall, this solution would find use in monitoring and decision support systems in precision horticulture. Its potential applications include the timely scheduling of the agrotechnical and protective measures to ensure the high quality of the fruit crop.