**Contents**


#### **Athos Agapiou**


Reprinted from: *Drones* **2021**, *5*, 46, doi:10.3390/drones5020046 .................... **391**


### **Preface to "Feature Papers of Drones-Volume II"**

The present book is divided into two volumes (Volume I: articles 1–23, and Volume II: articles 24–54) which compile the articles and communications submitted to the Topical Collection "Feature Papers of Drones" during the years 2020 to 2022 describing novel or new cutting-edge designs, developments, and/or applications of unmanned vehicles (drones).

Articles 1–8 are devoted to the developments of drone design, where new concepts and modeling strategies as well as effective designs that improve drone stability and autonomy are introduced.

Articles 9–16 focus on the communication aspects of drones as effective strategies for smooth deployment and efficient functioning are required. Therefore, several developments that aim to optimize performance and security are presented. In this regard, one of the most directly related topics is drone swarms, not only in terms of communication but also human-swarm interaction and their applications for science missions, surveillance, and disaster rescue operations.

To conclude with the volume I related to drone improvements, articles 17–23 discusses the advancements associated with autonomous navigation, obstacle avoidance, and enhanced flight planning.

Articles 24–41 are focused on drone applications, but emphasize two types: firstly, those related to agriculture and forestry (articles 24–35) where the number of applications of drones dominates all other possible applications. These articles review the latest research and future directions for precision agriculture, vegetation monitoring, change monitoring, forestry management, and forest fires. Secondly, articles 36–41 addresses the water and marine application of drones for ecological and conservation-related applications with emphasis on the monitoring of water resources and habitat monitoring.

Finally, articles 42–54 looks at just a few of the huge variety of potential applications of civil drones from different points of view, including the following: the social acceptance of drone operations in urban areas or their influential factors; 3D reconstruction applications; sensor technologies to either improve the performance of existing applications or to open up new working areas; and machine and deep learning developments.

### **Diego Gonz´alez-Aguilera and Pablo Rodr´ıguez-Gonz´alvez**

*Editors*

### *Article* **Pasture Productivity Assessment under Mob Grazing and Fertility Management Using Satellite and UAS Imagery**

**Worasit Sangjan 1, Lynne A. Carpenter-Boggs 2, Tipton D. Hudson <sup>3</sup> and Sindhuja Sankaran 1,\***


**Abstract:** Pasture management approaches can determine the productivity, sustainability, and ecological balance of livestock production. Sensing techniques potentially provide methods to assess the performance of different grazing practices that are more labor and time efficient than traditional methods (e.g., soil and crop sampling). This study utilized high-resolution satellite and unmanned aerial system (UAS) imagery to evaluate vegetation characteristics of a pasture field location with two grazing densities (low and high, applied in the years 2015–2019) and four fertility treatments (control, manure, mineral, and compost tea, applied annually in the years 2015–2019). The pasture productivity was assessed through satellite imagery annually from the years 2017 to 2019. The relation and variation within and between the years were evaluated using vegetation indices extracted from satellite and UAS imagery. The data from the two sensing systems (satellite and UAS) demonstrated that grazing density showed a significant effect (*p* < 0.05) on pasture crop status in 2019. Furthermore, the mean vegetation index data extracted from satellite and UAS imagery (2019) had a high correlation (*r* ≥ 0.78, *p* < 0.001). These results show the potential of utilizing satellite and UAS imagery for crop productivity assessment applications in small to medium pasture research and management.

**Keywords:** grazing density; nutrient; pasture management; forage grass; remote sensing

#### **1. Introduction**

Pasture management is vital to ensure adequate forage quantity and quality in support of domestic animal production. In addition, several countries have utilized the integration of livestock into cropping systems to contribute to the ecological sustainability of agriculture [1–4]. Domestic livestock is grown in pastures, grasslands, and natural areas; well-managed grazing results in reducing soil erosion from tillage and heavy grazing, improving soil fertility through the application of manure, maintaining ruminants' natural digestive systems, and converting otherwise unusable plant material into more nutritious animal products, such as meat and milk [5–7].

Livestock grazing over a large area requires dynamic decision-making to appropriately allocate pasture forage for animals; this must consider pasture growth's spatial and temporal variation associated mainly with the weather, soil nutrients, and grazing management [8,9]. Systematic monitoring of plant community and soil health to inform this decision-making in larger grazed areas can be time-consuming and labor-intensive. Moreover, destructive sampling can limit the amount of vegetation available as animal feed. Thus, novel methods and techniques to gather information on the forage quality of paddocks using less labor could improve the management of large swaths of land [10,11].

Remote sensing technologies offer distinctive advantages, providing high spatial and temporal resolutions; these techniques are inexpensive, time-effective, non-destructive, and applicable in areas inaccessible to humans [12–15]. Satellite and unmanned aerial

**Citation:** Sangjan, W.;

Carpenter-Boggs, L.A.; Hudson, T.D.; Sankaran, S. Pasture Productivity Assessment under Mob Grazing and Fertility Management Using Satellite and UAS Imagery. *Drones* **2022**, *6*, 232. https://doi.org/10.3390/ drones6090232

Academic Editors: Diego González-Aguilera and Pablo Rodríguez-Gonzálvez

Received: 26 July 2022 Accepted: 28 August 2022 Published: 2 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

system (UAS) are the typical platforms used for acquiring remotely-sensed imagery [16,17]. Applications include grassland vegetation mapping and phenomena detection related to crop-livestock research/management, such as yield and biomass at large spatial scales, to optimize process/output efficiency [18,19].

Satellite imagery has been commonly utilized to observe large pastures, such as advanced very high-resolution radiometer–AVHRR [20,21], moderate resolution imaging spectroradiometer–MODIS [22,23], the Landsat fleet [24,25], and Sentinel-2 [26,27]. In addition, synthetic aperture radar (SAR) sensors have also been applied from TerraSAR-X [28,29] and Sentinel-1 sources [30,31]. UAS-mounted cameras/sensors can offer highresolution spectral data, providing more details to describe the plant/crop status in the field. Nevertheless, the issue of coverage area and the efficiency of resource utilization (personnel, travel) for continuous monitoring restrict the wide usage of UAS applications in rangeland systems [32,33]. In the same way, although the satellite platform can provide medium to high temporal resolution (~1–15 days), the spatial resolution is inadequate to delineate forage performances of small to medium pasture parcels (<1 ha) [34,35]. This research, therefore, aims to expand our understanding of the inter-relationships between these two remote sensing data for productivity evaluation in small to medium-sized pasture paddocks.

In this study, the primary objective was to utilize remotely sensed indices derived from satellite and UAS imagery to assess the biophysical connections/transitions of diverse grazing density and fertility management on a sustainably managed ranch pasture. The specific goals were to (1) examine the effects of two strategies of planned grazing management differing by the density and timing, and four soil fertility treatments on pasture productivity through replicated research, applying related remotely sensed data to investigate the sensitivity/quality of the two sensing platforms for estimating pasture biophysical changes; and (2) to evaluate relationships between digital traits extracted from satellite and UAS data to determine the applicability of different remote sensing platforms in small to medium-sized pasture paddocks for pasture characterization, research, and management.

#### **2. Materials and Methods**

#### *2.1. Study Area*

The study was established in 2015 in an existing irrigated pasture on a commercial ranch in Cheney, Washington (Figure 1a). The location of the study area (GPS: 47◦29 25.2 N, 117◦43 26.6 W) was on the edge of the semi-arid region leading up to the Selkirk Mountains foothills, and the soil type was gravelly silt loam, with a significant phosphorus deficit. The area has a Mediterranean climate with Köppen's climatic type of Csb; the summers (mid-June to mid-September) are short and warm with an average maximum temperature of 29–30 ◦C in July, and winters (mid-November to the end of February) are cold with an average minimum temperature of −4.50 ◦C in December. The wet season is from late October to early June (maximum precipitation in December at about 83.80 mm average and minimum in July at about 12.70 mm average). The weather data during the research period (2015–2020) was acquired from the National Oceanic and Atmospheric Administration (NOAA)–National Centers for Environmental Information (NECI) of the U.S. Department of Commerce (https://www.ncdc.noaa.gov/cag/; accessed on 20 July 2021) is presented in Figure 1b.

**Figure 1.** Location and weather data of the study area (**a**) the study location in Cheney, WA, United States, and (**b**) monthly meteorological data of Spokane County in Washington State, US, from January 2015 to December 2020 acquired from NOAA–NECI.

Plant species composition in the study area was evaluated in June 2015 before grazing, which was clearly driven by soil type and moisture. Cheatgrass and drought-tolerant annual forbs dominated with remnant perennial grasses in the drier soil on the eastern side. More mesic soil was found towards the west side (near a wetland) of the study area, where perennial grass and mesic pasture forbs dominated. Dominant species by weight were primarily western wheatgrass, tumble mustard, cheatgrass, and alfalfa.

#### *2.2. Experimental Design and Grazing Methodology*

The study area's experimental design was a strip plot consisting of four replicates (four blocks) and eight treatments over 3.24 ha (Figure 2). Cattle grazing was managed using planned rotational grazing at two different stocking densities: (1) low density (LD); and (2) high density (HD), using an approach referred to as "mob grazing" at high stocking densities of livestock for a short period of time (hours or days). At both livestock densities, cattle were moved into a grazing area with the aid of electrical fences, allowed to graze and trample aboveground forage biomass, and removed. The land was then left ungrazed until the following year [36,37]. These two strategies were overlaid with four soil fertility management to address pre-existing phosphorus and sulfur fertility limitations in this irrigated pasture: control (no fertility supplement), one year aged manure (2800 kg dry manure ha−1, supplying 55 kg ha−<sup>1</sup> N, 22.40 kg ha−<sup>1</sup> P2O5, 80 kg ha−<sup>1</sup> K2O, 8 kg ha−<sup>1</sup> S), organic registered phosphorus and sulfur fertilizers (22.40 kg ha−<sup>1</sup> S as gypsum and 22.40 kg ha−<sup>1</sup> P2O5 as bonemeal), and non-aerated compost tea (supplying 5 kg ha−<sup>1</sup> N, 2 kg ha−<sup>1</sup> P2O5, 7 kg ha−<sup>1</sup> K2O). Fertility treatments were applied annually.

**Figure 2.** Experimental design of the study area consisted of two different grazing managements: LD—low density and HD—high density with four soil fertility management: N—no fertility supplement (control), M—partially-aged manure, P—organic forms of phosphorus, and sulfur fertilizers, and C—non-aerated compost tea.

The HD grazing was applied once in June or July each year from 2015 to 2019 when the fields were predominantly covered by forage. Each 0.40 ha HD paddock was separated by electric fencing. Grazing duration in HD was in accordance with mob grazing methodology, using a target utilization level as a trigger of timing to move animals to the next strip [38]. The LD grazing represented a seasonal-sustainable grazing rate from June to September each year. Thus, LD removed approximately 40% of available forage, and HD targeted 80–90% forage removal and trampling at each grazing event, monitored by an experienced rancher's visual estimation.

Throughout the study, cattle (400–500 kg mature body weight) were used as forage "harvesters" to a targeted level of forage removal for each grazing density in each paddock. The rancher used 100 to 119 animal units (454 kg equivalent live weight (LW) per animal unit) of cattle in HD. All cattle were applied on one 0.40 ha HD paddock at a time for a stock density of 113,000 to 135,000 kg LW ha−<sup>1</sup> for 12 h. After all HD paddocks were grazed, two cow-calf pairs or 2.40 animal units grazed all LD paddocks together (1.60 ha), a stock density of approximately 700 kg LW ha<sup>−</sup>1, for 100 days.

#### *2.3. Image Acquisition*

#### 2.3.1. Satellite Imagery

Two remote sensing platforms, satellite, and UAS, were utilized to acquire raw data. Satellite imagery was the PlanetScope Analytic Ortho Scene (Level 3B) provided by Planet Labs Inc. (San Francisco, CA, USA). PlanetScope Dove satellite comprises over 180 CubeSats 3U form factor (10 × 10 × 30 cm) on the constellation having the capability to image all of the Earth's land surface each day. The ground sampling distance of PlanetScope imagery is 3.70 m at a reference altitude of 475 km, and pixel size is 3 m after the orthorectified

process. Ortho scenes are radiometrically-, sensor-, and geometrically-corrected, including atmospherically corrected using the 6S radiative transfer model with ancillary data from the MODIS for surface reflectance 4-band imageries that were utilized in the study (https: //www.planet.com/products/planet-imagery/; accessed on 20 July 2021). Images from 2017 to 2019 of cloud-free PlanetScope scenes were acquired on dates before the grazing period and downloaded through the Planet's Education Research Program (https://api. planet.com; accessed on 20 June 2020). The information about its specific attributes and raw image acquisition date are shown in Table 1.


**Table 1.** Parameters and information for sensors applied in the study.

<sup>1</sup> RGB: Red-Green-Blue spectral bands; <sup>2</sup> Raw images were acquired at 25 m UAS flight altitude; <sup>3</sup> Raw images were acquired at 50 m UAS flight altitude; <sup>4</sup> NIR: Near-Infrared.

#### 2.3.2. UAS Imagery

High-resolution UAS imageries were acquired from two quadcopters. DJI-Phantom 4 Pro with an onboard visible camera (DJI Inc., Los Angeles., CA, USA), as described in Table 1, was utilized to collect raw RGB images. Pix4Dcapture (Pix4D S.A., Lausanne, Switzerland) was used for mission planning, such that the images were captured by setting a flight pattern as a single grid with 80% front and 70% side overlap and speed at about 2.50 m s<sup>−</sup>1. In order to receive good image accuracies, UAS flight altitude was set at 25 m, and images were captured with two missions. The high-resolution RGB imagery was utilized to accurately geolocate/georeference each pasture plot of the multispectral imagery from satellite and UAS sources.

ATI-AgBotTM (Aerial Technology International, Oregon City, OR, USA) mounted with a RedEdge camera (Table 1) (Micasense Inc., Seattle, WA, USA) was employed to capture multispectral images. Similar mission planning with3ms−<sup>1</sup> UAS speed and 50 m flight altitude using Mission Planner software (http://ardupilot.org/planner; accessed on 20 July 2021) was established. Before each flight, 80 × 50 cm boards that could be seen in the resulting UAS orthomosaic images were placed at each ground control point position to assist in georeferencing process. In addition, a 30 × 30 cm white reference panel having 99% reflectance from RGB to NIR spectral range (Spectralon® Diffuse Reflectance Targets, SRS-99-120, Labsphere Inc., North Sutton, NH, USA) was also placed in the field during image acquisition of both UAS missions for radiometric correction.

#### *2.4. Image Pre-Processing*

#### 2.4.1. Pre-Processing on Satellite Imagery

Figure 3 summarizes the satellite imagery pre-processing steps performed utilizing an open-source Python 3 (Python Core Team, 2015) program to fully spatially align and precisely evaluate the pixels or areas transforming through time, as provided in [39]. In this process, AROSICS, the open-source image co-registration software for multi-sensor satellite imagery that uses the Fourier shift theorem to perform intensity-based registration and identify sub-pixel shifts [40], was used. All images in the time series were compared against the first image (reference image), which was the satellite image from 2019, as this image could be georeferenced with the UAS RGB image. Later, the multivariate alteration detection (MAD) algorithm [41] was employed to find invariant pixels between the target (2017 or 2018 satellite image) and the reference image (2019 satellite image) for normalizing the radiometry between the images. The MAD was selected due to its robustness against diverse atmospheric conditions and its appropriateness as a precursor to various normalization techniques.

**Figure 3.** Image processing pipeline used to process both satellite and UAS imagery.

Based on [39,41], the MAD component images at a no-change probability threshold of 95% were set to select the representing invariant features, which were the highest likelihood pixels and their spectral values. This process, combined with a linear regression approach, was used to generate a transformation between the two radiometries of the reference and target images. Moreover, 65% of invariant pixels were used to train the model, and the remaining were for testing purposes. In order to minimize the residuals and compensate for spectral values offset between the two images, both slope and intercept were automatically adjusted through the algorithm. With the linear regression model, the image appearance was preserved and not over-corrected, which simplified the image/result interpretability. Therefore, utilizing the MAD algorithm, the spatial resolution of the images in the series was maintained, and any parameters on the reference image were transformed to the images in series, but the atmospheric or cross-sensor normalization variation was compensated.

#### 2.4.2. Pre-Processing on Satellite Imagery

Raw RGB and multispectral images from UAS platforms were pre-processed (Figure 3), applying the structure-from-motion software Pix4Dmapper (Pix4D S.A., Lausanne, Switzerland) to derive an orthomosaic image of the study area with 0.62 cm and 3.44 cm spatial resolution, respectively. In the process, the software improved the radiometric quality of the images automatically by considering the scene illumination, reference panel, and sensor specifications to create reflectance imagery. The 2019 surface reflectance images from the satellite and UAS platform were rectified to the correct location between different images using the Georeferencer tool in open-source software QGIS (QGIS.org, 2021, version 3.10.4), similar to those described in [42].

#### *2.5. Image Processing*

Vegetation indices (VIs) were constructed utilizing the algorithm created in Python 3 using the Rasterio library (https://rasterio.readthedocs.io/en/latest/#; accessed on 20 July 2021). The satellite images after normalization and multispectral surface reflectance images from UAS after georeferencing were used in this method (Figure 3). The VIs, especially those commonly used in agricultural applications with a potential to estimate pasture productivity or aboveground biomass or yield estimation, were derived (Table 2).



<sup>1</sup> EVI: Enhanced Vegetation Index = 2.5 <sup>×</sup> *NIR*−*Red NIR*+(6×*Red*)−(7.5×*Blue*)+<sup>1</sup> , ref. [53].

The OSAVI [51] was utilized to estimate a threshold to create a soil mask that was eliminated from each VI image. Then, the polygons defining each pasture study plot were digitized in a \*.shp format using QGIS software as it is complex to generate the shapefile using an algorithm because the plots were not of uniform grid pattern (Figure 2). The shapefile of plot segmentation was imported to the created algorithm, and the Python libraries: NumPy (https://numpy.org/; accessed on 20 July 2021) and Rasterstats (https:// pythonhosted.org/rasterstats/#; accessed on 20 July 2021) were applied to extract statistical data (maximum, mean, and median) of each VI image from each plot and to export the data in a comma-separated values file format.

#### *2.6. Data Analysis*

The effect of grazing density and fertility management on pasture productivity were assessed from all calculated VIs in each year independently and also the comparison between the year of study (2017–2019) using analysis of variance (ANOVA) in R programing language (version 3.2.5, R core Team, 2017). Agricolae (version 1.3-3, ref. [54]) in R was utilized to test the variance in strip-plot design, divided into three parts: horizontal-factor analysis (grazing density), vertical-factor analysis (fertility management), and interaction analysis. Fisher's least significant difference (LSD) test was followed to compute the significant differences among the mean data (α = 0.05 and adjusting probability value method = Bonferroni). The tests created multiple comparisons of treatments by means of Fisher's LSD and a grouping of treatments. The analysis method described above was applied for the two datasets—first, the original dataset, the originally extracted VI data from all treatments. Second, the normalized dataset was the original VI data normalizing based on VI data from a no fertility supplement (control) paddock in each block of the study area to minimize the influence of different meteorological conditions each year on the crop growth and development and thus the investigated VI data. Furthermore, the linear relationship of extracted VIs between satellite and UAS imagery was evaluated using Pearson's correlation analysis.

#### **3. Results**

#### *3.1. Pasture Productivity over Time*

Satellite and UAS data were acquired before grazing to study the effect of applied treatments on the plots. Figure 4 displayed some of the spatial variability (satellite and UAS) of vegetation indices such as NDVI, EVI2, and MCARI2, representing the pasture productivity from the years 2017–2019. The plot segmentation layer presented the pattern of the pasture productivity in different blocks of treatments changed over the study period. Moreover, the range of VI data from satellite and UAS images was dissimilar, which could be because of the difference in spectral bands, as described in Table 1; however, the patterns of VI spatial distributions were similar.

The digital traits generated for each image demonstrated the highest data values for areas where high pasture productivity occurred as a dark green area. The crop productivity differences in the years 2017 and 2018 were low. However, in 2019, especially as observed from EVI2 and MCARI2 from satellite images, revealed a high vegetation probability of over 40–50% in the HD area. In addition, for UAS data, the VIs map also correspondingly distinguished that 40–50% of HD area had high VI values.

A comparative analysis utilizing box-and-whisker plots of the extracted mean statistic of EVI2 and MCARI2 from the satellite dataset (Figure 5) revealed the applied treatment effects, different treatment comparisons, and pasture productivity trends through the studied time period. Figure 5 demonstrated that the treatments involving 2015 and 2016 did not affect the 2017 pasture productivity data as the mean of the EVI2 and MCARI2 from different treatments (grazing density and fertility management) were consistent and showed low variability. Similar results were observed for the year 2018. Nevertheless, the mean values of EVI2 and MCARI2 from the two datasets (original and normalized) in 2019 revealed the difference in mean values between low and high grazing density. Notably, the mean VI values were most different between LD and HD, where organic phosphorus and sulfur fertilizers had been applied.

#### *3.2. Treatment Effects*

ANOVA in both the original and normalized datasets indicated that the grazing density significantly affected the digital traits (VIs) extracted from satellite and UAS imagery in 2019 (Figure 6). In contrast, fertility management and the interaction between both treatments were not significant in 2017–2019, as observed using remote sensing data. However, the extracted median of CIgreen value calculated from satellite data showed a significant interaction of both treatments' effects (*p* < 0.10) in the original dataset. In the normalized dataset, especially in UAS data, the fertility management effect was found to be significant from the extracted maximum of GLI (*p* < 0.05) and also in the extracted median of LAI (*p* < 0.10) and MSAVI2 (*p* < 0.10).

**Figure 4.** Vegetation index maps of the study area constructed from satellite (2017–2019) and UAS (2019) imagery.

The satellite data results revealed that the different grazing densities affected the VIs values (vegetation characteristics) after four seasons (2015–2018) of applying the treatments. Moreover, all selected VIs from both sensing platforms in 2019 from the original dataset displayed similar results, especially from the extracted mean VI data. The VIs showed the potential for observing differences/effects among treatments (at least *p* < 0.10), excluding GLI from UAS imagery. Correspondingly, the normalized dataset demonstrated the extracted mean and median VI values from the satellite sensing platform, including the extracted maximum (except NDVI and WDRVI) and mean VI values (except CIgreen and WDRVI) from UAS in 2019 could estimate variation in vegetative cover.

**Figure 5.** Box-and-whisker plots of extracted mean statistical VI data, EVI2 and MCARI2, created from satellite images from 2017 to 2019 to represent the pasture productivity changing on different grazing intensities and fertility management over the study period between original data and normalized data.

The EVI2, LAI, MCARI2, MSAVI2, and OSAVI extracted from satellite and UAS imagery (original dataset) were consistent in differentiating the grazing density effect. The normalized dataset revealed minor differences, especially using median data from UAS imagery. Nevertheless, EVI2, MSAVI2, and OSAVI showed a consistent and high impact from grazing density than other VI data (*p* < 0.05; excluding the extracted mean VI values from the UAS imagery had *p* < 0.10). EVI2 and MCARI2 data showing the effect of grazing density are shown in Figure 7. The results demonstrated that the mean of HD was significantly different from LD (both sensing platforms and datasets). This comparison indicated that implementing high grazing density could result in high canopy vigor in pasture systems, as observed from the remote sensing data.

**Figure 6.** Grazing density effects on VI data extracted from satellite and UAS imagery. F: main effect from fertility management; I: interaction effect from grazing density and fertility management.

**Figure 7.** Effects of grazing density on the mean EVI2 and MCARI2 data extracted from satellite and UAS imagery in 2019. Different letters above each bar indicate statistically different means between grazing density treatments (*p* < 0.05, except the normalized data of UAS imagery *p* < 0.10; ANOVA; Fisher's LSD test).

#### *3.3. Correlation between Satellite and UAS Data*

The congruence between satellite and UAS imagery for evaluating pasture productivity was investigated on the normalized dataset using mean and median VI values (Figure 8). Pearson correlation coefficients were stable and high (*r* ≥ 0.75, *p* < 0.001) between VIs extracted from the two sensing systems, especially CIgreen, MCARI2, and NDVI. Furthermore, the Pearson correlation coefficient of the extracted mean data, except GLI and WDRVI, revealed high correlations (*r* ≥ 0.78, *p* < 0.001) between the VIs from satellite and UAS imagery. Given the difference in resolution (3.40 cm for UAS and 3 m for satellite imageries), these findings are encouraging.

**Figure 8.** Scatter plots demonstrating the relationship of the extracted VI values from satellite and UAS sensing platforms in 2019 (**a**) mean data and (**b**) median data.

#### **4. Discussion**

The results from this study indicate that both the satellite and UAS imagery have the potential to assess pasture productivity (based on applied treatments and/or natural variability). The benefits of utilizing satellite imagery over UAS imagery would be the efficient utilization of resources (no equipment/training required to collect data, saved travel time) while capturing data across a larger region/area. One of the key challenges in this study was identifying the treatment areas across the imagery (especially satellite), where the higher resolution UAS imagery was useful. If satellite imagery were utilized for similar studies, ground control points or checkpoints with accurate GPS information on the boundary are recommended for the accurate positioning of the target plots and more straightforward image processing. Nevertheless, the co-registration and normalization during the pre-processing of the satellite images allowed the consolidation of data for the same location and spectral range across the different years in the study. Similar methods can also be deployed for evaluating the long-term evaluation of pasture health using historical data.

Figure 4 shows the higher spatial image resolution of UAS imagery in comparison to satellite imagery. In general, higher resolution imagery (such as UAS imagery) reduces the mixed pixels in the images (contain more pure pixels), resulting in better discrimination between vegetation and soil. Contrarily, the satellite imagery showed some spectral mixing. The mixing of vegetative and soil pixels with a high concentration of organic matter may influence and increase the vegetation index data [51,55], as observed in Figure 7.

The normalization with respect to control treatment was performed to eliminate the effect of different weather conditions across the years. Nevertheless, the results showed similar patterns as observed from the trends in the original and normalized datasets. The location of different paddocks did influence the vegetation development. The drier soil on the east side and the field entrance for the cattle on the northeast side displayed lower vegetation than the west side, which was closer to the wet area, as presented in Figure 4.

This study demonstrated that the high spatial resolution satellite imagery could be used for small to medium pasture field research/management, as the processed images are able to recognize the variation in vegetation growth or crop status. Furthermore, the product offers daily temporal coverage and a viable cost-affordable option in terms of spatial resolution (3 m pixel resolution), thereby allowing the researchers/farmers to check their crop/plant status over time. However, the satellite imagery used in this study did not have a red-edge spectrum that can be utilized to estimate leaves' chlorophyll content over a canopy [56,57], which may be useful to assess crop stress. More recent images (SuperDove satellites) have 8 spectral bands, including red edge bands.

The two sensing platforms (satellite and UAS) required different image acquisition and processing approaches to get crop information [58,59]. The data acquisition using UAS required travel to the imaging site, planning of the flight mission, and operating the UAS to collect data. The UAS platform can provide high resolution data that can be applied to monitor specific/interested crops and cropping regions, with plausible application of precision agriculture technologies. In contrast, satellite imagery acquisition and operation are more established than the UAS, where the imaging can be tasked through prior arrangement with the satellite imagery providing company. The database of satellite images is also vast, and historical data can be acquired for several agricultural applications, including the study of variability and changes in the cropping areas and overall health, as presented in this study.

Progress in remote sensing technology from both sensors and platforms (satellite and UAS) provides considerable benefits to pasture/grassland science. The production of grassland in terms of spatial and temporal patterns has been evaluated using vegetation indices extracted from the sensing imagery [60–62]. In addition, vegetation indices integrated with ground-truth data have been used to develop empirical models for estimating the productivity/biomass on pastures [63–66]. The most utilized remote sensing data for pasture management for large grassland areas were from satellite sources such as Formosat-2, Lansat, MODIS, and Sentinel-1. The applications included classification, detection, and analysis of applied management practices such as mowing, grazing, or a combination of the two operations [67–71]. Thus, with the capability of high-resolution satellite imagery (as used in this study), pasture research/management in small to medium areas can be a suitable application.

#### **5. Conclusions**

This research paper presents the application of vegetation indices extracted from satellite and UAS imagery for assessing crop status/vigor of a small to medium pasture under different grazing density and fertility management. The major findings can be summarized below.


These results offer great benefits to farmers, ranchers, and researchers alike as pasture management can be investigated and evaluated using sensing data. These tools can be resource-efficient, allowing short and long-term assessment of crop health and productivity.

**Author Contributions:** Conceptualization, W.S. and S.S.; methodology, W.S. and S.S.; software, W.S. and S.S.; validation, W.S., S.S. and L.A.C.-B.; formal analysis, W.S.; investigation, W.S., S.S., L.A.C.-B. and T.D.H.; resources, S.S. and L.A.C.-B.; data curation, W.S.; writing—original draft preparation, W.S.; writing—review and editing, W.S., S.S., L.A.C.-B. and T.D.H.; visualization, W.S.; supervision, S.S., L.A.C.-B. and T.D.H.; project administration, S.S. and L.A.C.-B.; funding acquisition, S.S. and L.A.C.-B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Washington State University's Center for Sustaining Agriculture and Natural Resources BioAg Program (project ID 184) and the U.S. Department of Agriculture-National Institute of Food and Agriculture (USDA-NIFA) hatch project (accession number 1014919).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Acknowledgments:** The authors would like to sincerely thanks Maurice Robinette from Lazy R Ranch for managing the treatments in the field site. We would also like to thank Chongyuan Zhang and Milton Valencia Ortiz for their support during the UAS data collection.

**Conflicts of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as potential conflict of interest.

#### **References**


### *Review* **A Review of Unmanned System Technologies with Its Application to Aquaculture Farm Monitoring and Management**

**Naomi A. Ubina 1,2 and Shyi-Chyi Cheng 1,\***


**Abstract:** This paper aims to provide an overview of the capabilities of unmanned systems to monitor and manage aquaculture farms that support precision aquaculture using the Internet of Things. The locations of aquaculture farms are diverse, which is a big challenge on accessibility. For offshore fish cages, there is a difficulty and risk in the continuous monitoring considering the presence of waves, water currents, and other underwater environmental factors. Aquaculture farm management and surveillance operations require collecting data on water quality, water pollutants, water temperature, fish behavior, and current/wave velocity, which requires tremendous labor cost, and effort. Unmanned vehicle technologies provide greater efficiency and accuracy to execute these functions. They are even capable of cage detection and illegal fishing surveillance when equipped with sensors and other technologies. Additionally, to provide a more large-scale scope, this document explores the capacity of unmanned vehicles as a communication gateway to facilitate offshore cages equipped with robust, low-cost sensors capable of underwater and in-air wireless connectivity. The capabilities of existing commercial systems, the Internet of Things, and artificial intelligence combined with drones are also presented to provide a precise aquaculture framework.

**Keywords:** drone technology; aquaculture; precision aquaculture; aquaculture monitoring; aquaculture drones; internet of things

#### **1. Introduction**

Fisheries and aquaculture play an essential role in feeding the growing population and are critical for the livelihood of millions of people in the world. Based on the long-term assessment by the Although the Food and Agriculture Organization (FAO) has assessed the continuous declination of marine fish resources [1], many interventions were made by government institutions, private organizations, and individuals to increase awareness of the importance of the world's fishery resource. Strict implementation of fishing regulations and water environment conservation has increased fishery production and sustainability. Despite these developments and with the expected increasing population of 8.5 billion by 2030, the increase in demand for marine commodities cannot be sustained any longer by wild fish stocks. Aquaculture is involved in farming of fish, shellfish, and other aquatic plants and have been a great help in food security. In the past years, it is the fastest-growing product in the food sector [2] and is emerging as an alternative to commercial fishing [3]. With this trend, the expansion of aquaculture plays a significant role in ensuring food sufficiency, improved nutrition, food availability, affordability, and security.

In 2018, world aquaculture reached 114.5 million tons of production record [1], making this industry marketable and promising. However, with the increasing global population, aquaculture production must also continue to increase to meet the food demand of the growing population. With this significant contribution of the aquaculture industry in alleviating poverty [4–6] and increasing income [5,6], employment [3,7], economic growth [8–10],

**Citation:** Ubina, N.A.; Cheng, S.-C. A Review of Unmanned System Technologies with Its Application to Aquaculture Farm Monitoring and Management. *Drones* **2022**, *6*, 12. https://doi.org/10.3390/ drones6010012

Academic Editors: Diego González-Aguilera and Pablo Rodríguez-Gonzálvez

Received: 30 November 2021 Accepted: 31 December 2021 Published: 6 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

reducing hunger for food, and increasing the nutrition of the population [9,11,12], one of the main challenges in aquaculture production is sustainability [13].

#### *1.1. Challenges in Aquaculture Production, Supervision and Management*

One of the indicators of the success of an aquaculture venture depends on the correct selection of the aquaculture site. Aquaculture farm types vary from small-scale rural farms to large-scale commercial systems. For choosing a farm location, a good quality water source is a must since surface water such as river, stream, or spring is prone to pollutions. They are also intermittently available since it is affected by weather such as drought or typhoons. Aquaculture farms are in lakes, rivers, reservoirs (extensive aquaculture), coastal lagoons, land-based coastal, in-shore, and offshore areas [14].

Coastal lagoons are shallow estuarine systems; they are productive and highly vulnerable [15]. Aquaculture in coastal lagoons is more heterogeneous in terms of cultivated species, techniques, extent [16], which can lead to reduced water quality, habitat destruction, and biodiversity loss which limits or restricts fish and shellfish farming concessions [17]. Land-based farming is also becoming famous due to less environmental impact on coastal areas and reducing the cost of transportation. Compared with open-water fish farming, monitoring is easy due to accessibility, and quick adjustments can be made to achieve optimal living conditions of aquaculture products [18]. Despite this, land-based coastal aquaculture is more constrained [14], and mass mortalities due to disease spread is fast, and sudden change in water temperature is also apparent.

In-shore farm locations are close to the open fishing grounds with minimal shore currents. However, concerns such as wind and wave protection currents brought by small boat fishers [19] are also evident. Offshore aquaculture farms' locations are in the deep-sea water. Since they are far from the shore, this reduces the negative environmental impact of fish farming. Despite the higher investment requirements for this farm location and some requiring importation of cages and equipment from other countries [14], its utilization offers a great potential to expand the industry in many parts of the world. Currents and greater depths generally increase the assimilation capacity and energy of the offshore environment and offer vast advantages for aquaculture farming. Since offshore cages are far away from the coast, there is an increased cost in terms of management and daily routine operations for farm visits and monitoring [20]. Recent technological innovations in offshore cage systems make it possible for aquaculture operations in the open ocean, and this industry is rapidly increasing in different parts of the world.

Aquaculture production is very costly considering the requirement in terms of human labor and feeds. The big aquaculture farms are located offshore in deep and open ocean waters, allowing them to produce with a large number. Many of the offshore fish cages are submerged in water and they can only be reached by boats and ships. This method limits the accessibility with additional capital costs [21]. Meanwhile, feeds have the highest share during the production period [22]. Farming systems are also diverse in terms of methods, practices, and facilities. The presence of climate change highly affects the quality of aquaculture production (e.g., change in water temperature, water becomes acidic); it has now become a threat to sustainable global fish production [2]. Global food loss and waste are also severe problems and concerns. Proper handling from production, harvest to consumption is also essential to prevent the identified problems and preserve the production quality [1].

Aside from feeding, farming in the grow-out phase involves tasks such as size grading and distribution of fish to maintain acceptable stocking densities, monitoring water quality and fish welfare, net cleaning, and structural maintenance. All these operations are significant to obtain good growth to ensure fish welfare. Attaining profitability and sustainability in production requires a high degree of regularity in all these operations [23].

Offshore aquaculture farms that have large-scale productions require high manual labor and close human interactions to perform monitoring and management. Proper farm management requires regular monitoring, observation, and recording. For example, to monitor the growth of the fish, the farmer must evaluate the utilization of the feeds utilized and assess the fish growth to optimize stocking, transfer, and harvests. According to FAO, the extent of farm monitoring depends on the educational level and skill of the farmer, the farmer's interest in good management and profit, the size and organization of the aquaculture farm, and the external assistance available to farmers. Commercial farms need a close monitor of fish stocks. Farmers should also be aware of various parameters for growth measurement, production, and survival of aquaculture stocks. In ensuring this achievement, farms visits should be at least once a day to check if water quality is good and if fish are healthy. Close fish monitoring determines growth, the efficiency of feeding, and adjustment of daily feeding ratio to save feed costs. Checking the adequacy of the stocking rate will enable the transfer of larger fish or marketed immediately and if the stock has reached the target weights, production and harvesting schedule can be changed [24].

According to Wang et al. [25], intelligent aquaculture is now moving beyond data toward decision-making. Intelligent aquaculture farms should be capable of carrying out all-around fine control on various elements such as intelligent feeding, water quality control, behavior analysis, biomass estimation, disease diagnosis, equipment working condition, and fault warning. It is significant to collect data from the aquaculture site to monitor and use technologies, such as sensors and unmanned systems to integrate artificial intelligence (AI) for a smarter fish farm. As an example, with feeding management considerations, feed cost has the highest share in the production period [22]. So, there is a need to reduce the cost to maximize the profit by making sure that the fish is not overfed, which is an added cost, or making sure that fish is not underfed, which affects the fish growth and density, thus, affect the production quality. Bait machines help automate the feeding process, but for it to be fully optimized, information is required of the level of fish feeding satiety or hunger. Information such as disturbance on the water surface can be a basis to determine the level of fish hunger or feeding intensity. Such information can be captured by the UAV using its camera sensors and sends the information to the cloud to perform data analysis using AI services such as deep learning techniques to evaluate the fish feeding intensity level. The analysis results will be forwarded to the baiting machine to determine how much food to dispense. If fish feeding intensity is high, the feeding machine continues to give food, and otherwise, when it is none, it will stop giving food [26].

#### *1.2. Aquaculture's Technological Innovation for Precision Farming*

With the challenges mentioned for aquaculture production, there is a need to identify and adopt various strategies. To address these previously mentioned issues, technology integration in the past decades has become famous for automating or helping aquaculture farmers monitor and manage their farms for improved aquaculture sustainability. Technological innovations (such as breeding systems, feeds, vaccines) and non-technological innovations (e.g., improved regulatory frameworks, organizational structures, market standards) have enabled the growth of the aquaculture industry. Radical and systemic innovations are necessary to achieve the ecological and social sustainability of aquaculture [27]. Integrating smart fish farming as a new scientific method can optimize and efficiently use available resources. It will also promote sustainable development in aquaculture through deep integration of the Internet of Things (IoT), big data, cloud computing, artificial intelligence, and other modern technologies. A new mode of fishing production is achieved with its real-time data collection, quantitative decision making, intelligent control, precise investment, and personalized service [28]. Various technological innovations are already available to improve aquaculture production and management [29]. The availability of unmanned vehicles equipped with aerial cameras, sensors, and computational capability is very famous for site surveillance [30].

Precision fish farming described by Føre et al. [31] aims to apply control engineering principles in fish production to improve farm monitoring, control, and allow documentation of biological processes. This method makes it possible for commercial aquaculture to transition from a traditional experience-based production method to a knowledge-based

production method using emerging technologies and automated systems that address the challenges of aquaculture monitoring and management. Precision fish farming aims to improve the accuracy, precision, and repeatability of farming operations. The preciseness facilitates more autonomous and continuous biomass/animal monitoring. It also provides higher reliable decision support and reduces dependences from manual labor and subjective assessments to improve worker safety and welfare. Furthermore, O'Donncha and Grant [32] described precision aquaculture as a set of disparate and interconnected sensors deployed to monitor, analyze, interpret, and provide decision support for farm operations. Precision farming in the ocean will help farmers respond to natural fluctuations and impact operations using real-time sensor technologies and will no longer rely on direct human observations and human-centric data acquisition. Thus, artificial intelligence (AI) and IoT connectivity now support farm decision-making.

Unmanned vehicles or aircraft is one of the emerging technologies for various personal, businesses, and governments, particularly in the military field intended for different purposes. Recently, it has become well-utilized in agriculture and aquaculture in managing and monitoring fish due to its availability and affordability [33]. They are capable of reaching remote areas requiring a small amount of time and effort. Users can control the flight or navigation using only a remote control or a mobile application. When UAVs were introduced around the 20th century, their intended function was for military purposes [34–36]. However, in the last few years, drones' capability has prospered and is now capable of accomplishing multiple and simultaneous functions. Such capabilities are aerial photography [37], shipping and delivery [38–40], data collection [41,42], search and rescue operations during disasters or calamities [43], agricultural crop monitoring [44], natural calamity monitoring, and tracking [45]. UAVs were also successfully integrated into marine science and conservation. In the paper of de Lima et al. [46], the authors provided an overview of the application of unoccupied aircraft systems (UAS) to conserve marine science. As part of their study, they used electro-optical RGB cameras for multispectral, thermal infrared, and hyperspectral systems. Their applications of UAS in marine science and conservation include animal morphometrics and individual health, animal population assessment, behavioral ecology, habitat assessment and coastal geomorphology, management, maritime archaeology and infrastructure, pollutants, and physical and biological oceanography. Some of these mentioned applications could also be utilized in the aquaculture environment.

Today, drones have been successful in collecting environmental data and fish behavior at the aquaculture site for monitoring [47]. In the work of Ubina et al. [30], an autonomous drone performs visual surveillance to monitor fish feeding activities, detect nets, moorings, cages, and detect suspicious objects (e.g., people, ships). The drone is capable of flying above the aquaculture site to perform area surveillance and auto-navigate based on the instructions or commands provided. The autonomous drone can understand the position of the target objects through the information provided by the cloud, which makes it more intelligent than the usual drone navigation scheme. It becomes an intelligent flying robot to capture distant objects and valuable data. The drone can also execute a new route based on the path planning generated by the cloud, unlike the non-autonomous drone, which only follows a specific path [30]. Their autonomous capability reduces the need for human interactions; actual site monitoring and inspection activities can be controlled or reduced [23].

The paper is organized as follows: Section 2 is the methodology; Section 3 provides the unmanned vehicle system platforms. Section 4, on the other hand, presents the framework of the aquaculture monitoring management using unmanned vehicles while in Section 5 is the unmanned vehicles capability as communication gateway and IoT device data collector. Section 6 provides how unmanned vehicles are used for site surveillance, Section 7 is for aquaculture farm monitoring and management function, and Section 8 contains the regulations and requirements for unmanned vehicle system operations. Lastly, Section 9 is the challenges and future trends, and Section 10 is the conclusions.

#### **2. Methodology**

This paper's purpose is to conduct a review of literature and studies conducted for unmanned systems' applicability to perform aquaculture monitoring and surveillance. The majority of the literature search was made using the Web of Science (WOS) database. Factors considered in the preference of articles include relevance to the related keywords provided for the search and the number of paper citations. There were no restrictions on the date of publication. Figure 1 is the taxonomy used for keyword extraction in the Web of Science database to determine the trend and the number of works involving unmanned vehicle systems for aquaculture. The authors also used Google Scholar, IEEE Xplore and Science Direct to search for related works.

**Figure 1.** Taxonomy for keyword extraction in the database search.

The articles from the keyword search were the basis in identifying the capabilities, progress, gaps, and challenges of unmanned vehicle systems for aquaculture site monitoring and management. We also conducted data analysis based on the search results from the WOS database to know the trend or research interest based on the number of published journal articles for each year. Graphs were generated to present the result of the analysis. Samples of the results are in Figures 2–4.

**Figure 2.** Publication result by year using the keyword aquaculture precision farming.

**Figure 3.** Publication result by year using the keywords "aquaculture precision" and "unmanned vehicle" or "unmanned system".

**Figure 4.** Publication result by country using the keywords "aquaculture" and "unmanned vehicle" or "unmanned system".

#### **3. Unmanned Vehicle System Platforms**

Unmanned vehicles can improve mission safety, repeatability and reduce operational costs [48]. The tasks performed by unmanned vehicles are typically dangerous or relatively expensive to use humans to execute. In addition, they are assigned jobs that are simple but repetitive and less expensive to implement without humans [49]. Low-cost, off-the-shelf systems are now emerging, but many still require customization [48] to meet the specific requirement for aquaculture monitoring and management. The work of Verfuss et al. [50] provides the detail of the current state-of-the-art autonomous technologies for marine species observation and detection. Although it does not focus on aquaculture, underlying principles, and requirements can be adopted in aquaculture monitoring.

In this paper, the authors describe the capabilities and limitations of unmanned vehicle systems to perform monitoring and management of aquaculture farms. The functions are to assess water quality, water pollutants, water temperature, fish feeding, water currents, drones as a communication gateway, cage detection, farm management, and surveillance of illegal fishing are content of this review paper as a mechanism to achieve precise aquaculture. There are different classifications of unmanned vehicles considered in this paper for aquaculture monitoring and management: unmanned aircraft systems, autonomous underwater vehicles, and unmanned surface vehicles. Each of the unmanned vehicle systems has its respective capabilities and limitations. However, they can be used together to collaborate and attain the goal of aquaculture monitoring and management. The strength of unmanned vehicles can address the issues or limitations of the other types to increase robustness and efficiency.

#### *3.1. Unmanned Aircraft Systems (UAS)*

Unmanned aircraft systems (UAS) or unmanned aerial vehicles (UAVs) provide an alternative platform that addresses the limitations of manned-aerial surveys. According to Jones et al. [51], UAS does not require hundreds of hundreds of dollars to perform surveillance and works best for geospatial accuracy of the acquired data and survey repeatability. A potential advantage of UASs is lower operating costs and consistency of flight path and image acquisition. UAS should be small, with an electric motor, easy to use, affordable, and record and store onboard data to prevent data loss or degradation from the transmission [50]. For real-time monitoring, UAS should send data using its wireless capability. Since they are pilotless aircraft, they can operate in dangerous environments inaccessible to humans [52]. For surveillance and monitoring, they have sensors such as cameras flying into the sky to monitor the target interests [53]. Cameras installed in UAVs can also serve as data collectors and send them into a repository system. Additionally, recent developments in UAS provide longer flight durations and improved mission safety. Although UAS has strong potential for aquaculture monitoring, its success still depends on various factors such as aircraft flight capability, type of sensor, purpose, and regulatory requirements for operations for a specific platform [54].

At the highest level, the three main UAS components are unmanned aerial vehicles, ground control, and the communication data link [55]. Low-cost or multi-rotor drones are easy to control and maneuver with the ability to take off and land vertically. Multirotor UAS has lightweight materials such as plastic, aluminum, or carbon fiber to increase efficiency, and wingspans range from 35 to 150 cm. They can be ideal for small areas and can be controlled from the deck of a small boat [56], but they are limited in terms of flight time and capacity to withstand strong wind conditions. An alternative to multi-rotor drones is single rotor or helicopter drones [57]; they are built for power and durability, with long-lasting flight time with heavy payload capability. However, single rotor drones are harder to fly, and they can be expensive and with more complex requirements.

Fixed-wing drones can travel several kilometers and fly at a high altitude and speeds and cover larger areas for surveillance. They can also carry more payloads, have more endurance which can perform long-term operations. They can be fully autonomous and do not require piloting skills [58]. Like the single rotor drones, fixed-winged drones are expensive and need the training to fly non-autonomous aircrafts. Aside from being difficult to land, they can only move forward, unlike the other two drones that can hover in the target area.

#### *3.2. Autonomous Underwater Vehicles (AUVs)*

Autonomous underwater vehicles (AUVs) or remotely operated underwater vehicles (ROV) are waterproof and submersible in the water as they are equipped with cameras to capture images and videos and other sensors to collect data such as water quality. Some of the capabilities of sensors in ROV can perform data collection such as water temperature, depth level, chemical, biological, and physical properties. They are equipped with lithiumion batteries that enable longer or extended time [59] for navigation or data collection. AUVs are now preferred to use human divers to perform underwater inspections, which is lesser in cost and provides better safety. They can provide a 4D view of the dynamic underwater environment capable of carrying a wide range of payloads or sensors. As the ROV moves to the water, its sensors can perform spatial and time-series measurements [60].

One of the challenges of AUVs since it is submerged underwater is high navigational precision [61], communication, and localization due to the impossibility of relying on radio communications and global positioning systems [62]. There are many devised alternatives in dealing with these challenges. One of them is the integration of geophysical maps to match the sensor measurements known as Geophysical Navigation [63]. In addition, UV navigation that uses a differential-Global Navigation System (DGPs) is with high precision. When submerged in water, its position is estimated by measuring its relative speed over

the current or seabed using an Acoustic Doppler Current Profiler (ADCP). For more precise navigation, an inertial navigation unit is used with positioning from a sonar system [60].

Vehicle endurance is also one of the requirements of AUVs and should be less dependent on weather and water current or pressure. AUVs should be equipped with reliable navigation to perform surveillance functions such as fishnet inspections and fish monitoring. In the paper of Bernalte Sánche et al. [64], the authors presented the summary of navigation and mapping of UAV embedded systems for offshore underwater inspections where sensors and technologies are combined to create a functional system for improved performance. Niu et al. [60] listed in their paper the specifications of candidate sensors embedded in AUVs such as salinity, hydrocarbons, nutrients, and chlorophyll.

#### *3.3. Unmanned Surface Vehicles (USVs)*

Unmanned surface vehicles (USVs) or autonomous surface craft [65] operate on the water without human intervention. They were developed to support unmanned operations such as environmental monitoring and data gathering [62]. USVs should be easy to handle and durable in the field environment. USVs can get up close to objects or targets to quickly close to gather high-resolution images. It is also fast-moving, can cover large accurate sensors, and execute run-time missions [66]. However, the autonomy level of USVs is still limited when being deployed to conduct multiple tasks simultaneously [67]. For USV to form immense heterogeneous communication and surveillance networks, they can cooperate with other UVs such as UAVs. One unique potential of USV is to simultaneously communicate with other vehicles located either above or below the water surface areas. USVs can also act as relays between vehicles operating underwater, inland, in air, or in space [68].

Combining various unmanned vehicle systems can maximize their strengths to collaborate and perform more expansive tasks and coverage to address the limitations of each type. In its collaboration, UAVs and USVs can cruise synergically to provide richer information functioning as an electronic patrol system. A USV-UAV collaborative technique can perform tasks such as mapping and payload transportation. In this way, it can handle more complex tasks with increased robustness through redundancy, increased efficiency by task distribution, and reduced cost of operations [66]. These heterogeneous vehicles can work collaboratively to achieve large-scale and comprehensive monitoring. Although there are still many open research issues for heterogeneous vehicle collaboration [69], the possibility of its exploration should increase performance, adaptability, flexibility, and fault tolerance [66].

#### **4. Unmanned Vehicles and Sensors**

Unmanned systems' navigation and monitoring capabilities concerning several quantities in their environment strictly depend on their sensors [70], measurement systems, and data processing algorithms. Sensor fault detection is also essential to ensure safety and reliability. UVs have different numbers, types, and combinations of sensors mounted in various ways to measure information using specific, diverse, and customized algorithms. Therefore, finding an optimal sensor that can perform various tasks, applications, and types is an unsolvable problem. Individual sensor specifications and characteristics affect the performance of UV aside from other factors such as operating conditions and environment [71]. One of the primary purposes of the sensor is to collect data relevant to a mission beyond plat-form navigation. Examples of data collected by sensors include acoustic profiles, radar, and infrared signatures, electro-optical images, local ocean depth, and turbidity. Major sensor subtypes are sonar, radar, environmental, and light or optic sensors [72]. Generally, aerial systems rely on electro-optical imaging sensors, while underwater and surface vehicles mostly rely on acoustic methods [48].

UAV's flight position and orientation are determined by combining accelerometers, tilt sensors and, gyroscopes [71]. Aside from GPS, and based on Table 1, USVs can also use radars or inertial navigation systems (INS) if the satellite signal is unavailable. Since UAVs

are vulnerable to weather conditions such as rain or wind, they should be equipped with wind-resistant equipment. Visual cameras can have shockproof and waterproof casings for protection. Extreme wind, rain, or storms can cause UAVs to deviate from intended missions, or small UAVs cannot operate in such weather conditions. UAVs must adapt to atmospheric density and temperature changes to preserve their aerodynamic performance.

**Table 1.** Navigational payloads characteristics.


The most common sensor payload is cameras. Although smaller cameras are lighter and easier to deploy, larger cameras provide better image quality. RGB digital cameras provide high-spatial-resolution. The spatial resolution of the RGB sensor determines the quality of the acquired images [71]. The work of Liu et al. [83] provided a detailed discussion of the various sensors shown in Table 2.


**Table 2.** Characteristics of exteroceptive sensors; adapted from Balestrieri et al., Liu et al. and Qiu et al. [71,83,84].

One of the challenges to facilitate image and video collection in the underwater environment is data quality, and AUV should be capable of collecting high-definition data for monitoring. Image captures of AUV are affected by the amount of light available in the underwater environment is poor due to the scattering light or turbidity for shallow coastal water [71]. AUVs are not capable of GPS signals; instead, they depend on acoustics, sonar, cameras, INSs, or combinations of such systems to navigate. For sonars, they are highly utilized for detection, tracking, and identification, but it is limited since sound propagation depends on temperature and salinity, and calibration is also required [72].

For unmanned surface vehicles, the water environment is affected by wind, waves, currents, sea fog, and water reflection [85]. There are remedies or solutions to dealing with these environmental disturbances to make the USV more robust. Monocular vision is strongly affected by weather and illumination conditions, which requires a high amount of calculating costs when obtaining high-resolution images [71]. Image stabilization, image defogging, wave information perception, and multi-camera methods are some solutions to deal with the factors affecting image quality due to weather conditions. For stereo vision, its lenses can calculate the flight time to generate a depth map that serves as an obstacle map for near-field collision avoidance. They can also extract color and motion from the environment but can be affected by weather and illumination conditions such as a narrow field of view. Likewise, infrared visons can operate during day and night since they can overcome problems caused by light conditions (night and fog). Omnidirectional cameras can have a large field view but require high computational cost; images from this type of camera are affected by illumination and weather conditions, as well. Infrared cameras also have good quality performance at night but are limited to providing color and texture information, and their accuracy is low. Event cameras are good in reducing the transmission and processing time but generate low-resolution outputs, and like others, it is affected by weather and illumination conditions [85]. Table 3 shows the advantages and limitations of various USV sensors.


**Table 3.** Advantages and limitations of various sensors for USVs.

To determine underwater quality sensors, factors to be considered are physical, chemical, and biological parameters [90]. In the paper of Bhardwaj et al. [91], the authors enumerated the requirements for aquaculture sensors. First, sensors should sense data over long periods without being cleaned, maintained, or replaced. Second, they should have a low energy demand to maximize the energy or power of the UV to perform longer monitoring. Third, sensors require waterproof isolation or the requirement such as avoiding corrosion and biofouling. Fourth, since organisms in the sea can alter the sensor surface and change the transparency and color, the potential flight path must be properly designed. Fifth, sensors should have no harmful effect on the fish. Avoid sensors that use ultraviolet light, acoustic beams that can be felt by the fish, and magnetic fields that can disturb fish activities. In addition, sensors should not alter fish swimming or feeding activities. Sensors must then be low maintenance, low cost, low battery-consuming, robust, waterproof, non-metallic, withstand biofouling, and have no effects on organisms. Modern real-time water quality sensors such as optical and bio-sensors have higher sensitivity, selectivity, and quick response time with the possibility of real-time analysis of data [92].

Although sensor fusion is possible, it could add cost to the operation and UV pay-loads. Its integration will complement the various strengths and capabilities to achieve higher accuracy and increased system robustness. When selecting a sensor, one must consider the cost, specifications, application requirements, power, and environmental conditions.

#### **5. Framework of the Aquaculture Monitoring and Management Using Unmanned Vehicles**

The architecture presented in Figure 5 provides a framework on how a drone works with sensors such as underwater cameras and water quality devices. These sensors are installed in the fish cage to collect data through a WIFI communication channel to transmit data to a cloud system. The cloud server serves as a repository and is equipped with data processing and analytics capabilities using AI-based techniques (e.g., computer vision, deep learning). The enormous amount of data collected from the underwater environment using sensors provides a non-invasive and non-intrusive method. This approach can achieve realtime image analysis for aquaculture operators [47]. Different data can be collected from the aquaculture site using these sensors to monitor the behavior of fish and the water quality of the aquaculture farm. The collected data informs the aquaculture farmers and enables them to provide immediate farm interventions to ensure farm produce and processes are optimized and of high quality to help increase production and income. The data collected, such as the level of food satiety of fish, as a specific example, are analyzed and transformed into meaningful information to dispense food from the smart feeding machine. A high level of satiety means continuous dispensing of food, while a low level of satiety means the amount of food dispensed is reduced or stopped. Real-time information with these mechanisms will achieve optimal aquaculture performance.

**Figure 5.** Architecture for aquaculture monitoring and management using drones.

In addition to the ability of the unmanned vehicles to capture or collect data from the aquaculture site, its mobility could be used as a communication channel connecting underwater cameras and sensors to the cloud as a Wi-Fi gateway that provides more services for precise aquaculture. Since the cameras installed in aerial drones have limitations and cannot capture underwater events, fish cages are equipped with stationary cameras (e.g., sonar, stereo camera systems) and other sensors to perform specific tasks. The drone now eliminates long cables for connection with improved reliable connection and communication [30]. Aerial drones work best for functions that involve mapping, site surveillance, inspection, and photogrammetric surveys. AUV and USV, on the other hand, can do other monitoring and assessment functions such as water quality and conditions that cannot be fully addressed by aerial drones. There are additional costs and technical requirements for this method. However, one can take advantage of its more extensive area and scope for monitoring functions.

With this ability, users, such as aquaculture farm owners, can remotely monitor their aquaculture farms and assess fish welfare and stock. With the vast and varied amount of data collection from the aquaculture site, data-driven management of fish production is now possible. This scheme improves the ability of farmers to monitor, control, and document biological processes on their farms so that they can understand the environmental conditions that affect the welfare and growth of fish [93].

#### **6. Unmanned Vehicles as Communication Gateway and IoT Device Data Collector**

In developed countries where access to the Internet is not a problem, the Internet of Things (IoT) is helpful to farmers. This new connectivity help increase production, reduce operating costs, and enhance labor efficiency. The Internet of Things (IoT) has made promising and remarkable progress in collecting data and establishing real-time processing through the cloud using wireless communication channels. With the presence of 5G technology, it is a great advantage to combine UAVs and IoT to extend coverage to rural or remote areas [94], which are the locations of aquaculture farms; thus, it is just appropriate to exploit this capability. The presence of LTE 45/5G networks and mobile edge computing now broadens the coverage of UAVs [95] and is even capable of performing real-time video surveillance [96].

The drone as a flying gateway is equipped with LTE cellular networks to base stations and a lightweight antenna to collect data. UAVs acts as the intermediate node allowing data collection from sensors and transmitting them to their target destinations. The drone then flies to the location of the IoT devices to offer additional coverage or support to the aquaculture farm in case there are problems with the wired connection of the devices. The gateway can receive sensor data and send these collected data to the servers [94] to integrate additional processing strategies, such as artificial intelligence and deep learning techniques. A drone can also serve as a node of the wireless sensor network where IoT communication is not available to receive the collected data from the node. Then it moves to an area where wireless IoT communication is possible and transfers the data to the IoT server [97]. Various sensor devices can connect to aquaculture cages and farms, such as underwater cameras and water quality sensors; Arduino [94] and Raspberry Pi can be embedded as part of an IoT platform, as shown in Figure 6.

**Figure 6.** IoT platform using drone as a communication gateway.

In maximizing the drone's capability, it is significant to optimize its energy consumptions. The work of Min et al. [97] proposes a dynamic rendezvous node estimation scheme considering the average drone speed and data collection latency to increase the data collection success rate. Many devices can be embedded on the drone to provide a better wireless communication network. The Lower Power Wide Area Network (LPWAN) gateway onboard can be installed in the UAV. The LoRa gateway is famous for its coverage and lower power consumption in its deployment.

Short-range communication devices are convenient to enable communication between sensors and gateways, such as Bluetooth, ZigBee, and Wi-Fi. However, with drones as a communication gateway, Lower Power Wide Area Network (LPWAN) is much of an advantage to provide extended communication coverage. The different types of LPWAN in Table 4 present their advantages and disadvantages. A comparative study with LP- WAN technologies for large-scale IoT deployment and smart applications based on IoT is utilized [98,99]. In the work of Yan et al. [100], a comprehensive survey was made on UAV communication channel modeling, taking into account the propagation losses and link fading, including the challenges and open issues for the development of UAV communication.


**Table 4.** Comparison of LPWAN wireless technologies.

With this capability of drones as a communication gateway, it can now serve as a medium to help achieve the goal of precise aquaculture. The drone can now provide wireless communication for IoT devices to send data to the cloud for processing, thus acting as a data collection medium. Data acquisition using UAVs is less expensive and more convenient than hiring manned aircraft, especially in remote and inaccessible places such as offshore aquaculture farms. UAVs, when combined with deep learning, can provide tremendous innovation for aquaculture farm management.

With all the identified potentials of drones as a communication channel [106], cameras, and sensors (e.g., stereo camera system, sonar devices) to capture the underwater environment is promising. The drone collects and then sends the data to the cloud to employ AI services using computer vision and deep learning techniques. The processed information provides information to users about the current conditions of the aquaculture farms. Fish survey activities [107] that can be performed includes fish behavior detection such as schooling [108–110], swimming [111–113], stress response [110,114,115], tracking [116,117], and feeding [112,118,119]. To determine the satiety or feeding level of fish used for demand feeder includes fish feeding intensity evaluation [26,120] and detection of uneaten food pellets [120,121]. The video collected from the aquaculture site through the drones can help estimate fish growth [122,123], fish count [124–126], and fish length and density estimation [127–131] as a device to transmit this information to the cloud for processing and data analytics to make predictions or estimates [132,133].

#### **7. Aquaculture Site Surveillance Using Unmanned Vehicles**

Illegal fishing is a global problem that threatens the viability of fishing industries and causes profit loss to farmers. On-the-ground surveillance is the typical way to monitor or minimize this practice [134], but with a very high operational cost. Submersible drones and UAVs are now capable of detecting illegal fishing activities [135] and are lower in terms of cost [136,137].

An unmanned system surveillance composed of fish farmers, vessels, and fish stocks was used to detect unauthorized fishing vessels [138] with an advantage in speed and size, making them capable of being unnoticed when performing surveillance. Automatic ship classification is relevant for maritime surveillance in detecting illegal fishing activities, which immensely affects the income of aquaculture farmers. Gallego [139] uses drones to capture aerial images for the detection and classification of ships. In the work of Marques et al. [140], aerial image sequences acquired by sensors mounted on a UAV detect vessels using sea vessel detection algorithms. A surveillance system framework was proposed using drone aerial images, drone technology, and deep learning [141] to eliminate illegal fishing activities. The ship is detected to identify its position and then classify the hull plate vessels to determine among them are authorized or not. The drone provides visual information using its installed camera. Additionally, crabs are highly valued commercial commodities, and also used drones with infrared cameras to detect crab traps and floats [134,135] to prevent illegal activities.

Remote sensing platforms or technologies with global positioning system capabilities, such as drones, have the ability for marine spatial planning to provide a wide spatialtemporal range for marine and aquaculture surveillance [142]. The drone is also applied to 3D mapping [143], aerial mapping [144], and low-altitude photogrammetric survey [145]. A semantic scene modeling was integrated to manage aquaculture farms using autonomous drones and a cloud-based aquaculture surveillance system as an AIoT platform. The scene modeling algorithm transfers information to the drone using the aquaculture cloud to monitor fish, persons, nets, and feeding levels daily. The drone acts as an intelligent flying robot to manage aquaculture sites [146].

The UAV with an onboard camera was also used for cage detection. The UAV's GPS is a guide to approximate the location of the cages, and applying image recognition methods follows to obtain the fish cage and the relative position of the UAV. This collected information will be the basis of the drone to adjust its position and proceed to the target object [147]. Additionally, UAVs could also be used for cage farming environment inspection [29] without requiring the installation of a hardware system in each cage which entails a higher cost in farming. Even a single UAV system can fly around all fish cages to capture data of the aquaculture cage environment, thus, a drastic reduction of the aquaculture operation cost. An inventory of salmon spawning nests is executed using UAVs to capture high-resolution images and videos to identify spawning locations and habitat characteristics; its abundance and distribution are metrics to monitor and evaluate adult salmon populations [148].

In Japan, they developed an agile ROV to perform underwater surveillance that provides real-time monitoring. The designed ROV is for easy transport, short startup time, effortless control, capable of high-resolution images at a low cost [149]. Drones are also applied to fishery damage assessment of natural hazards. It can survey fish groups, assist in salvage operations, and conduct aquaculture surveys and management after disasters [150]. In India, an autonomous AUV replaced the expensive sonar equipment to perform surveillance and relays the data and the global positioning system location. The drone provides a mechanism to serve as a bird's eye view to monitor the surrounding ocean surface like a person with normal vision can see [151]. Autonomous vehicles are also applied to increase spatial and temporal coverage. They can transit remote target areas with real-time observations with more potential than traditional ship-based surveys. Unmanned surface vehicles with two sail drones (USVs) were equipped with echo sounders to perform acoustic observations [152].

In the work of Livanos et al. [153], an AUV prototype was proposed as an IoT-enabled device. Machine vision techniques were incorporated to enable correct positioning and intelligent navigation in the underwater environment where GPS locations are limited due to its physical limitation to transmit communication signals through wireless networks. The AUV was programmed to record video and scan the fish cage net area and save this information in its onboard memory storage. Its navigation scheme is based on a combined optical recognition/validation system with photogrammetry as applied to a reference target of known characteristics attached to the fishnet. The AUV captures video data of the fish cage area under a relatively close distance successively to address the fishnet consistency

problem. The AUV architecture is cost-effective to automate the inspection of aquaculture cages equipped and accomplished a real-time behavior capability.

In the work of Kellaris e al. [154], drones were evaluated as monitoring tools for seaweeds using a low-cost aircraft. Compared to satellites and typical airborne systems as sources of images, drones achieve a very high spatial resolution that addresses the problems on habitats with high heterogeneity and species differentiation, which apply to seaweed habitat. A sample of the captured image for aquaculture site surveillance using a drone is in Figure 7. With the application of drones in surveying, it is now more accessible with a more large-scale range and scope of integration to aquaculture, fisheries, and marine-related applications. Table 5 shows the different types of drones and the embedded sensors for site surveillance and their corresponding applications.

**Figure 7.** Aerial view of in-land aquaculture site with scene modelling with detected objects such as fish pen, cages and house for site surveillance.



As much as possible, the position of offshore aquaculture cages is relatively close to onshore facilities to minimize distance-related costs of transport and maintenance services [155]. Table 6 provides the characteristics of the three aquaculture farm locations: coast, off-coast, and offshore based on physical and hydrodynamical settings. In the table, the work of Chu et al. [156], they provided a review on the cage and containment tanks designs for offshore fish farming and Holmer [157] provided the characteristics. The limitations in terms of accessibility to aquaculture farms is affected by weather conditions.

**Table 6.** Characteristics of coast, off-coast, and offshore aquaculture farms; adapted from Chu et al., Holmer and Marine Fish Farms [156–158].


The data provided in the table, especially the distance of the cages from the shore, are significant since they help determine the capability of the unmanned vehicle to perform navigation and monitoring. In Taiwan, the distance from the shore to the offshore cages range from 2 to 11 km, while the inshore cages are one kilometer away. The distance of the fish cages from the shore is significant in terms of the amount of time the unmanned vehicle needs to travel. Commercial UAVs are widely used for inspection since they are low-cost, but they are limited in terms of flight hours and payload capacity. Table 7 shows the characteristics of the UAV's performance measures.

**Table 7.** Characteristics of UAV types; adapted from Gupta et al., Fotouhi et l., Shi et al. and Delavarpour et al. [58,159–161].


Since the battery life of UAVs to perform extended navigation is limited most especially those with small size [162] (16 to 30 min for commercial drones), this restricts its operational range. For example, DJI Mavic Air 2, a quadcopter drone UAV that costs approximately \$800, has only 34 min flight time. Meanwhile, military drones have longer flying times, but cost millions of dollars. Fixed-wing drones with longer flight hours (120 min), such as Autel's Dragonfish [163], cost around \$99,000. Hybrid drones such as the SkyFront Perimeter 8 multirotor can fly up to 5 h [164]. UAV's flying time is also affected by the payload it carries; the fewer payloads, UAV will have a longer navigation time.

UAVs are also limited in their capacity to fly during bad weather. There are commercially available drones that can fly in windy conditions. But this scenario can be extremely difficult and challenging. One has to undergo a drone training course to make sure that setups are optimized to fly in difficult conditions, or one has to purchase high-end AUVs that cost hundreds of thousands of dollars, but many could not afford or might find it not practical. There are consumer-grade drone models that are available for windy conditions. The DJI Mavic Pro 2 can handle up to 15 mph though there are claims that it can reach a wind resistance up to 24 mph. Some commercially available drones can still fly in windy conditions but cannot withstand a tropical depression or a typhoon with at least sustained gusts of 30 mph. Although there are many efforts and studies for commercial-grade unmanned vehicle systems to advance their robustness and adapt to harsh weather conditions, this vision remains a challenge.

The capability of commercial-grade UVs to perform long-term mission is a challenge as well. The locations of coastal farms are close to the shore, so the flight time is shorter, and more time to perform navigation and its assigned mission compared to offshore farms, which are kilometers from the shore. In the case of offshore farms, if a UV takes off from the shore, it can no longer maximize its power once it reaches its destination since the battery is consumed for traveling. Thus, only limited time is available to perform its supposed function. However, there are many ways to extend and maximize their performance, such as lower altitude and smaller payloads. Instead of taking off from the shore or land area, they can take off from the barge. To assist the smart feeding machine for the fish feeding process, as an example, UV can take off from the barge or ship and does not need to travel a long distance from the shore. The operator can fly or control the UAV from the barge; it can return when finished monitoring. With this, there will be more time for the desired or target mission.

Aquaculture farms need to be visited at least once a day, and this is done during feeding time. The duration of a UV's mission depends on the function it must perform. Performing a water quality will not require some hours since the UV can get a water sample and perform analysis right away if it is equipped with sensors to measure water quality. On the other hand, monitoring the feeding activity requires longer hours; large offshore aquaculture farms have 24 cages where each cage is 100 m (standard size) in terms of the circumference. For each cage, there is an approximate distance of 5 m away from each other. To perform feeding in such conditions, it takes around 15 to 20 min to feed one cage, and 24 cages require almost a day of feeding activity. With the amount of time to monitor the feeding of the fish, one commercial-grade UV is not sufficient since it has limited power. Thus, multiple vehicles are needed to carry out a complete monitoring mission and data collection. During harsh weather conditions, fish cages are submerged in the water, and no fish feeding activity is carried out.

#### **8. Aquaculture Farm Monitoring and Management**

Drones are capable of monitoring fish farms in aquaculture, especially on offshore sites. Its affordability and mobility have allowed for a more open scope and access to difficult areas to reach and with high risks. The continued mechanization and automation of farm monitoring using drones, sensors, and artificial intelligence will enable farmers to inspect their farms, acquire more information needed for decision making, manage and interact with their farms efficiently. Furthermore, with the rapid growth of the aquaculture industry, drones will enable the monitoring of the growing farm site. Drones can replace the supply and demand for laborers and high-cost work in the aquaculture industry, thus ensuring that the management of the fish farm becomes stable by reducing farm deaths. To enable monitoring of the growing environment at the aquaculture farm site, using a drone as an image collection device, an integrated controller for posture stabilization and a remote device to control drones can capture underwater images in real-time [165].

An aquatic platform [166] composed of USVs and buoys has a self-organizing capability performing a mission and path planning in the water environment. This platform can

communicate with other devices, sense the environment (water or air), and serve as a communication channel using data gateways stations (DTS). The data taken by the USVs and Buoys using the attached sensors are forwarded to the server to be accessible to aquaculture workers to improve or maintain the aquaculture performance. Sousa et al. [167] designed and developed an innovative electric marine ASV (autonomous surface vehicle) with a simplified sail system controlled by electric actuators. This vehicle is capable of exploration and patrolling. Aside from reducing cost, since no fuel is required, it will be capable of endless autonomy, maximizes the limited energy to manage sails using propulsion power using solar cells and wind generators.

Aerial and underwater drones also have enormous potential to monitor offshore kelp aquaculture farms. Giant kelps with their same growth rate and versatility make them an attractive aquaculture crop that requires high-frequency monitoring to ensure success, maximize production, and optimize its nutritional content and biomass. Regular monitoring of these offshore farms can use sensors mounted to aerial and underwater drones. A small unoccupied aircraft system (sUAS) can carry a lightweight optical sensor. It can then estimate the canopy area, density, and tissue nitrogen content based on time and space scales, which are significant to observe changes in kelp. To provide a natural image of the kelp forest canopy, sUAS have sensors such as color, multispectral and hyperspectral tcameras [168].

An integrated system to count wild scallops based on vision was developed by Rasmussen et al. [169] to measure population health. Sequential images were collected using AUV and used convolutional neural networks (CNNs) to process those collected images for object detection. The images used as a dataset were captured by a downwardpointing digital camera installed in the nose of the AUV. In the work of Ferraro [170], UAV was also used to collect color photos and side-scan sonar images of the seafloor to perform a quantitative estimate of incidental mortality using a precise and non-invasive method for sea scallops. AUV was also used to capture a reliable image of the seafloor to determine the density and size of the scallops, thus providing an accurate set of data for site surveys. It also offers an efficient and productive platform to collect sea scallop images for stock assessment since it can be quickly deployed and retrieved [171].

Oysters were also detected and counted using ROVs for small-size aquaculture/oyster farms with robotics and artificial intelligence for monitoring. The ROV's front is mounted with a camera and two led lights. The camera feed streams to the remote machine, then used by the operator to perform underwater navigation. Additionally, the ROV was equipped with an additional GoPro camera and LED lights to view the seafloor. A graphic user interface called Qground Control (QGC) was installed to acquire underwater images of oysters by the ROV. The QGC sends commands to the device and receives the camera and other sensory information on the ground station machine or remote machine; the ROV can be controlled manually or automatically controlled. For manual control of the ROV, control commands are sent to the QGC through a wireless controller [172]. The Argus Mini, an observation class ROV built for inspection and intervention operations in shallow waters and can be used in offshore, inshore, and fish farming industries. It is equipped with six thrusters in which four are placed in the horizontal plane, and two are in the vertical plane to guarantee actuation in 4 degrees of freedom to resist water surges, sways, heaves, and yaw. The ROV is equipped with sensors to perform net cage inspection [173].

An underwater drone was developed integrating 360 degrees panoramic camera, deep learning, and open-source hardware to investigate and observe the environment such as the sea, aquarium, and lakes for fish recognition in real-time. The drone was also equipped with Raspberry Pi to compute module with GPU for processing and achieving real-time panoramic image generation [174]. Other application of UV includes periodic fish cage inspection [175], fish behavior observation [176], salmon protection [177], and fish tracking [178]. Table 8 presents the different application of unmanned vehicles for aquaculture farm monitoring and management.


**Table 8.** UVs and its application to aquaculture farm monitoring and management.

#### *8.1. Fish Feed Management*

The welfare of fish in aquaculture comes from improving standards and quality for fish production technologies and aquaculture products. The well-being of fish has direct implications for production and sustainability. Fish under good welfare conditions are less susceptible to disease, hence, manifest better growth and higher food conversion rate providing better quality [179]. There are many indicators to assess fish welfare, such as fish behavior and characteristics.

Many developed technologies can automate processes, such as underwater cameras to observe fish behavior and characteristics and provide visual observations in fish cages. However, installation and configuration of underwater cameras are laborious, particularly in an offshore area. They should be equipped with cables for communication and transmission and power source for continuous data collection. There are underwater cameras that are equipped with batteries but can only work for a limited time. For such cameras, it is necessary for physical installation, and it will be difficult to keep changing and charging the battery now and then. For underwater cameras with a power source (e.g., solar power), when the source malfunctions, these devices cannot perform data collection and surveillance. With these limitations, drones become helpful as an alternative or added support for underwater cameras to provide visual functionalities for fish behaviors and characteristics.

Feeding management in aquaculture is a challenging task since the visibility of the feeding process is limited, and it is laborious to have a precise measurement. Machine feeders became available to assist fish farmers in dispensing food. However, such a mechanism, when not accurately monitored, would lead to food waste and profit loss. Feeding using pellets that floats above the water should be observed when to discontinue or continue feeding. In the work of Ubina et al. [26], a drone equipped with an RGB camera captures the surveillance video of the water surface using optical flows to measure fish feeding level as shown in Figure 8. The authors conducted various experiments such as the different altitudes and viewing angles to determine the best visuals and features of the fish feeding. The images were processed using a deep convolutional neural network to classify

the different feeding levels. The drone provides a non-invasive way for fish observation, which is more reliable than human investigations and observations.

**Figure 8.** Image capture of the drone to evaluate fish feeding intensity using four different feeding intensity levels and the detected optical flow [26].

For a typical fish feeding to offshore locations, the feeds are transported in a boat or ship (see Figure 9). Then the pellets are dispensed using machine feeders, creating an annular feed distribution pattern across the water surface, and covering a limited percentage of the surface area. As an alternative method to determine the distribution of the pellets in the water surface, a UAV of Skøien et al. [121] was used to observe and characterize the motion and measure the spatial distribution of the pellets of the feed spreaders in sea cage aquaculture where the camera is always perpendicular to the water surface. The UAV also recorded the pellet surface impacts from the air together with the position and direction of the spreader. For this work, the UAV is fast with minimal equipment installation and a viable alternative in collecting pellets which can help farmers achieve feeding optimization.

**Figure 9.** The aerial footage using UAV to facilitate optimized feeding using feeders transported in boats.

To estimate the spatial distribution of feed pellets in salmon fish cages, a UAV provides a simple and faster setup, as it covers a large area of the surface of the sea cage. The UAV captures the aerial videos using a 4K camera from a top-view position of the hamster wheel in the fish cage during the feeding experiment. The UAV used for this work was DJI Inspire 1 and was positioned above the rotor spreader. But images taken outdoors are challenging, and it needs immediate adjustment to lighting conditions changes. These difficulties are induced by the reflection of the clouds on the water surface area and sometimes caused by slight variations in the camera position. For accurate estimations, the splashes of the dropping pellets must be identified and extracted to count or measure the splashes relative to the spreader in the image. A technique was integrated using top-head imaging as a processing step to extract brighter pixels from the image corresponding to splashes [180].

#### *8.2. Fish Behavior Observation*

A bio-interactive (BA-1) AUV monitors fish interactively and can stay in the environment where the fish resides. It can be swimming together with the fish to monitor their movements in a pen-free offshore aquaculture system. The vehicle can provide a stimulus to the fish and observe their behavior caused by stimulation. The UAV was designed to have hovering and cruising capability with bio-interactive functionality with an LED lighting system. It can also operate simultaneously with other BA-1 AUVs as its multiple AUVs capability feature. The BA-1 is equipped with sensors to perform navigation, collision avoidance, localization, self-status monitoring, and payload. The device was tested in tanks and aquaculture pens with sea bream species. Once the fish becomes familiar with the vehicle, it can come close to the demand feeding system to receive the bait [181] and assist in the smart feeding process.

A UAV device with GoPro cameras for its video recording tracks monitors the behavior in space and time of GPS-tagged sunfish. For communication, the vehicle uses Wi-Fi or GSM/HSDPA. Remotely sensed environmental characteristics were extracted for each position of sunfish and used as parameters to determine their behavioral patterns [182]. Spatial movements of fish are vital in maintaining fish populations and monitoring their progress. A multi-AUV state-estimator system helps determine the 3D position of tagged fish, also its distance and depth measurements. The system is composed of two AUVs with a torpedo-shaped vehicle. The attached rear propeller in the UAV determines the location, and the four fins control the pitch, raw, and yaw of the device. It is also equipped with two processors that communicate with the sensors and actuators [183]. A stereovision AUV was utilized to assess the size and abundance of ocean perch in temperate water. The AUV hovers above the target area with a constant altitude of 2 m and with a slow flying speed above the seafloor as it captures images using a pair of downward-looking Pixelfly HiRes (1360 × 1024 pixel) digital cameras [184].

#### *8.3. Water Quality and Pollutants Detection and Assessment*

Fish are in close contact with water, which is one of the most critical factors for fish welfare, which requires continuous and close monitoring. Poor water quality can lead to acute and chronic health and welfare problems, so water quality should be at optimal levels. Aquaculture is also significantly affected by climate change which results in changes in abiotic (sea temperature, oxygen level, salinity, and acidity) and biotic conditions (primary production and food webs) that will significantly cause disturbance in growth and size [1]. Parameters that reflect water quality [179] include temperature, conductivity, pH, oxygen concentration, and nitrogenous compounds such as ammonia, nitrate, and nitrite concentration. Traditional water assessments and predictions collect water samples and submit them for laboratory inspections, or some have physical-chemical test devices carried [185]. This method is a burdensome one and requires a physical presence to conduct water quality assessments. Many aquaculture farms rely on mechanical equipment to ensure water quality, which includes oxygenation pumps, independent rescue power systems, and aeration/oxygenation equipment. Although they are helpful, they have

limitations when installed in open-sea cages or offshore aquaculture sites and require additional configurations and setup. Drones have become very helpful to perform onsite water monitoring, sampling, and testing due to their high mobility, reliability, and flexibility to carry water quality sensors. A combination of UAV and wireless sensor network (WSN) in the work of Wang et al. [186] was designed for a groundwater quality parameter and the acquisition of drone spectrum information. Their proposed approach provides a new mechanism on how remote sensing with UAVs can rapidly monitor water quality in aquaculture.

An electrochemical sensor array to predict and assess water quality data using the pH of the water, dissolved oxygen, and ammonia nitrogen is carried by a floating structure UAV in T shape that can take off and land on the water surface. The sensor bears the capability of real-time detection and transmits its result to the sever backstage using the cloud server through a wireless network [185]. Furthermore, catastrophic events such as spills of hazardous agents (e.g., oil) in the ocean can cause massive damage to aquaculture products. To detect similar leaks like the fluorescent dye in the water, Powers et al. [187] used USV by mounting a fluorescence sensor underneath for detection. An unmanned aircraft system (UAS) visualized the fluorescent dye, and the USV takes samples from different areas of the dye plume.

Water sample collection based on in situ measurable water quality indicators can increase the efficiency and precision of collected data. To achieve the goal of preciseness, an adaptive water sampling device was developed using a UAV with multiple sensors capable of measuring dissolved oxygen, pH level, electrical conductivity, temperature, and turbidity. The device was tested using seven locations and was successful in providing water quality assessment [188]. In addition, in the works of Ore et al. [189], Dunbabin et al. [190] and Doi et al. [191], UAVs were used to obtain water samples that require less effort and faster data collection.

An extensive study on how drone technology assists in water sampling to achieve the goal of biological and physiological chemical data from the water environment can be found in the work of Lally et al. [192] and was characterized mainly using remote sensing. Spectral images captured by UAV were also used to assess water quality, such as algae blooms, to determine the chlorophyll content of the water [193], turbidity, and colored dissolved organic matter [194]. Other studies also show the use of drones with attached thermal cameras, such as miniaturized thermal infrared [195], to capture images for measuring surface water temperature, and environmental contamination [196].

The work of Sibanda et al. [197] shows a systematic review to assess the quality and quantity of water using UAVs. In Table 9, dissolved oxygen, turbidity, pH level, ammonia nitrogen, nitrate, water temperature, chlorophyll-a, redox potential, phytoplankton counts, salinity, colored dissolved organic matter (CDOM), fluorescent dye, and electrical conductivity were among the collected parameters for water monitoring. Additionally, the DJI brand of drones is the commonly used commercial type. Some UAVs have sensors specific to their functions (e.g., dissolve oxygen sensors test dissolve oxygen). Many customized UAVs were also used to perform a water quality assessment to meet the specific needs of each work and as an improvement to existing commercial capabilities such as navigation, strength, and mobility capabilities.


**Table 9.** Type of UVs and parameters used for water quality assessment and monitoring.


**Table 9.** *Cont.*

#### *8.4. Water Quality Condition*

Aquaculture farms have raised environmental concerns, and an increase in aquaculture production will pose a huge environmental challenge. Climate change is considered a threat to aquaculture production [21]. Sea-level rise, frequent and extreme weather (e.g., winds and storms) events are also projected to increase in the future [1]. For sustainable growth in aquaculture production, it is necessary to adapt to climate to produce more fish, and environmental impacts could not affect its operations.

UVs are commonly applied for image acquisition in the field of geophysical science to generate high-resolution maps. There is an increasing demand for high-performance geophysical observational methodologies, and UV technology combined with optical sensing to quantify the character of water surface flows is a possibility. Water surface flow affects the growth and health of aquaculture products with its environmental impacts from sea lice, escaped fish, and release of toxic chemicals and organic emissions to the water area [204]. It is also essential for farming fish in cages for replenishment of oxygen and removal of organic waste [156]. Water velocity also has a profound impact on fish metabolism, growth, behavior, and welfare. A higher velocity can boost the growth of farmed fish. In the work of Li et al. [205], it determines the protein content of the fish muscle using moderate swimming exercise. Using moderate water velocity exhibited a higher level of the protein content of the fish muscle. The growth performance of Atlantic salmon was also monitored using lower salinity and higher water velocity with positive effects on the growth of the salmon [98]. Another positive influence of higher velocity on fish welfare is in the work of Tauro et al. [204], where improvements of flesh texture, general robustness, and lower aggression lead to a reduced stress response. On the other hand, very high velocities increase oxygen need and anaerobic metabolism and cause exhaustion, reduced growth, and affect fish welfare. Moreover, excessive current flow causes the fish to excessively use its energy in swimming. Outrageous waves in an offshore environment, on the other hand, damages cage structures and moorings and can cause fish injury. A severe wave condition can be a hazardous situation and can cause an interruption in the routines or operations of farmers [156].

With the mentioned importance of measuring water surface flow and velocity for fish growth, drones can be integrated to perform such functions. Flying drones [204] were used to observe the water environment to produce accurate surface flow maps of submeter water bodies. This aerial platform enables complete remote measurements for on-site surveys. To measure the water velocities that integrate UVs, the work of Detertm and Weitbrecht [99] shows its effectivity to perform such function. A technique on how a drone can retrieve a two-dimensional wavenumber spectrum of sea waves from sun glitter images was proposed by [206], which shows the potential of the drone to investigate the surface wave field. Airborne drones were compared with satellite images to determine the state of the sea in the ocean and the dynamics of the coastal areas. Optical technologies that use spatial resolution optical images derive anomalies in the elevation of the water surface induced by wind-generated ocean waves [207].

In Table 10, UVs are equipped with cameras to collect data from the water environment. The majority of UV used are the commercial DJI Phantom, which is famous for its affordability and is sought-after but is reported to have a small amount of image distortion that can affect the images. According to Streßer et al. [208] and Fairleyet al. [209], some fixes were made with the gimbal pitch to make it independent of the aircraft's motion.


**Table 10.** Application of UVs to perform water condition monitoring.

#### **9. Legal Regulations and Requirements for Unmanned Vehicle Systems**

Potential users of unmanned vehicle systems, especially unmanned aerial systems (UAS), should be aware of the current and proposed regulations to understand their potential impacts and restrictions. The permitted sites for UAS should be first determined; flight restrictions for UAS in the offshore locations of aquaculture sites should be within the allowable time of the day. One of the challenges to consider when using UAS is that regulations are not fully established and are currently changing. The user must always check the updated rules in advance [54] of the scheduled flight or mission.

#### *9.1. Standards and Certifications*

New policies and regulations for UAS must be planned and implemented to ensure there is a safe, reliable, and efficient use of the vehicles. Developing standards is one of the most crucial issues for UAS since UAVs should be interoperable with the existing systems. In managing the electromagnetic spectrum and bandwidth, it is critical of UAVs not to be operating in crowded frequency and bandwidth spectrum. It is also essential to be aware of the published standardization agreements by NATO for UAVs. This standard defines the standard message formats and data protocols. It provides a standard interface between UAVs and ground coalitions. It also represents the coalition-shared database that allows information sharing between intelligent sources. In the US, the Federal Aviation Administration (FAA) has provided certification for remote pilots, including commercial operators [221]. UAVs used for public operations should have a certificate from the FAA; operators must comply with all federal and laws, rules, and regulations of each area, state or country [222].

#### *9.2. Regulations and Legal Issues*

In Canada, drones weighing from 250 g to 25 kg must be registered with Transport Canada, and pilots must have a drone pilot certificate. Pilots must mark their drones with their registration number before flying and drones should be seen at all times. While flying, they should be below 122 m in the air. The places where drones are prohibited to fly include 5.6 km from airports or 1.9 km from heliports. In the US, each state has its respective laws and regulatory requirements. In Taiwan, drones are prohibited to fly in sensitive areas such as government or military installations. Drone flights are permitted only within a visual line of sight and are limited to daylight hours between sunrise and sunset without prior authorization. A drone operator permit is required if the drone weighs more than 2 kg. In Germany, drones weighing more than 5 kg should obtain authorization from the aviation authority. When applying for permission, a map indicating the launch area and operating space, consent declaration from the property owner, timing, technical details about the UAS, data privacy statement, and a letter of no objection from the competent regulatory or law enforcement agency [223].

UAV regulations and policies of different countries have some common ground. However, they still differ in many aspects in terms of requirements and implementation. When used for a specific purpose, according to Demir et al. [222], aviation regulations determine the rules for the AUV minimum flight requirements. In most countries, UAVs are used in separate airspace zones. National regulations are also laid out to ensure safe operations of different UAVs in their respective national airspaces.

The operation requirements for unmanned maritime vehicles are also not yet clearly defined and regulated in terms of current domestic law or international conventions. There is no definite legal framework exists to regulate its use since permits and licenses are required based on a few narrow circumstances. The growing population and popularity of unmanned vehicles do not indicate causing danger to the oceans, in the future, but with is a possibility of potential implications of their widespread. Although there are regulatory gaps, there are options available to obtain permission for AUV operations to make the ocean a safer place for humans and animals [224]. Additionally, due to the varied types of AUV and their wide range of applications, it is also challenging to know their respective legal status for different operations, as their regulations vary significantly [225]. Operators should be aware of the prohibitions of such vehicles to avoid future problems or legal implications of their actions. The moral and ethical use of unmanned systems should also be considered by potential users to ensure that UAVs do not participate in illegal activities or morally doubtful operations.

#### **10. Challenges and Future Trends**

Unmanned systems have shown significant contributions to aquaculture management and monitoring to attain precision aquaculture. Table 11 shows the different functionalities identified in this paper with their strengths and limitations. However, despite all the functionalities, unmanned systems still have drawbacks and shortcomings and how improvements and modifications can be made to improve their performance.

**Table 11.** Application of drones to aquaculture management and monitoring to achieve precise aquaculture with its corresponding advantages and disadvantages.


UAVs utilized for wireless perspective can act as a base station in the cellular network providing communication links to terrestrial users or functioning as a relay in a wireless communication network. However, drones for wireless sensor networks have low transmission power, and many may not wirelessly communicate for a longer range or duration. There are technical challenges of providing a communication link between sensor nodes

and drones, such as network planning, sensor positioning, drone battery limitations, and trajectory optimization. With those challenges, there is a need to optimize the drone flight path planning based on the locations of the sensors to minimize flight time and overcome battery limitations [101]. To optimize path planning capability, algorithms such as the traveling salesman problem, A Star (A\*) algorithm [226], Dijkstra algorithm [226], and modified and improved Dijkstra algorithm [227,228] could be utilized. Optimizing the drone's flight capability would reduce cost, faster execution of missions, and increase navigation time, so there is a need to improve existing path planning algorithms to optimize the drone's navigation time.

To increase the battery life, the installation of more batteries may be a solution. However, such a remedy will increase the weight of the aircraft [58]; UAVs are designed to be lightweight for efficiency, so they operate longer and can cover a wider area. Adding a load to the drone can affect its weight shifting or create a disproportion of its structure. To increase UAV navigation time and for prolonged flight endurance, solar-powered aircraft can also be considered. With solar-powered batteries, there is no need to charge and refuel. This scheme reduces drone operational costs, but heavy and bulky solar panels to collect solar energies are not feasible for drones. In addressing such limitations, there are already available next-generation solar panels that are flexible, thin, and lightweight called gallium arsenide (GaAs) solar cells, which are highly efficient solar cells [229]. In the future, we can see more developments of power-solar drones using next-generation solar panels. In maximizing UAV's potential, using low-cost components can be considered; programmable microprocessors can connect the solar power source and a battery power source. In addition, there should be more investigation on auto-pilot settings such as airspeed, altitude, and turning radius to optimize flight endurance [230].

A docking station for drones as future development enables these vehicles of automated inductive charging of batteries at sea level. This station has a very narrow depth within the fish cage that will act as a power supply and data up-loading/transfer from the AUVG to the external servers for data processing. Once completed a mission or when the battery level becomes critical, the AUV will be directed to the docking station. Without any physical malfunctions, drones can permanently reside in fish cages and provide near real-time information on the condition of aquaculture farms [153].

Satellite images have the widest coverage compared to drone-captured but with the lowest quality and resolution. Although satellite images are best for mapping, they are strongly affected by clouds and fogs. AUVs provide image captures with better resolution and image quality for aquaculture site surveillance and monitoring. Many drones perform in situ surveillance, but they are lightweight and small with limited computational resources. Integrating AI and deep learning techniques could be computationally demanding and increase the drone's power consumption. It escalates its capacity for processing, the required analysis shifts to the cloud for processing, and the drone now becomes a 24-h surveillance system [231] with an increase in navigation time [232] and functionality [233]. Now that the high-volume data processing is eliminated, the drone can promptly collect a high volume of data in just a few hours.

For UAVs with attached camera sensors used as image capture devices, there are problems with the quality of the detected images. Raw captured images have low contrast [169], and small image size, which requires a post-processing procedure to improve image quality. One of the challenges of drone captures is weather conditions, where image capture is under suboptimal conditions, are highly variable, and is hard to predict. The sunglint effect also affects the water surface. Image enhancement and corrections are needed to improve the image quality and reduce noise [154]. Each image captured based on its specific function can employ explicit techniques to address a particular issue. For example, to solve the limitations of detecting objects, such as scallops, post-processing techniques specifically for small-sized images could be integrated. Despite the availability of image enhancement algorithms, underwater captured images continue to be a big challenge since they suffer from low contrast, low visibility, and blurriness due to light attenuation [234]. Water surface environments are active, and they continue to move, shrink, expand, or change their appearance with time [235]. Addressing these difficulties could employ and combine various techniques to process both underwater and above-water images. Each sensor type (e.g., sonar, stereo camera) also requires different processing techniques, which adds challenges to image enhancement integration. The use of sonar cameras depends on the wavelength of sounds, and the images generated are low contrast, and objects are blurred. For stereo camera systems, adjustments such as camera calibration are necessary. The use of deep learning is a well-proven technique to improve the image quality of surface water and underwater images to achieve a high precision rate. In practice, underwater video cameras are the most affordable data collector and highest quality and resolution for underwater surveillance, but they are difficult to install and configure.

There are many challenges in using unmanned systems to capture water movements to measure water velocity, such as camera shakes that affect the distortions of images or videos taken [99]. Physical instability of UAV induces motion in acquired videos that can significantly affect the accuracy of camera-based measurements, such as velocimetry. There are data-processing techniques or methods to deal with drone instability. The digital image stabilization (DIS) method uses the visual information of the videos in the form of physically static features to estimate and then compensate for such motions. In the work of Ljubiˇci´c et al. [236], seven tools were carefully investigated in terms of stabilization accuracy under various conditions, robustness, computational complexity, and user experience. Future work should aim to provide stability to aerial devices. Sensors carried by drones to perform meteorological surveillance combined with IoT, artificial intelligence, and cloud technology connected through a mobile communication channel provide optimal impact to the aquaculture industry, making it more sustainable and profitable.

One of the challenges of unmanned systems is to withstand typhoons with strong winds, heavy rains, and other calamities to increase their autonomous capabilities. Unmanned underwater vehicles (UUVs) should operate in harsh environments under high ocean currents and heavy hydraulic pressure; their navigation and maneuverability can still be strongly affected by oceans and water density [71]. Commercial graded are low-cost UAVs but are limited since their design is for operation in a stable or controlled environment. Commercially graded UAVs are low-cost in terms of acquisition, but only a few are with the capability to operate in such conditions. One of the few claims that their product is capable of such bad weather conditions is bbcom secure Deutschland GmbH [237], a company based in Germany. The company designed the unmanned aerial system (UAS) to be easy to use with low operating cost and capable of real-time video up to 17 sea miles away from the shore with 4 h of safe flying operation time even in harsh weather conditions. It is also capable of a maximum speed of 90 mph and can perform vertical take-off and landing and remote control with easy handling.

In the work of Elkolali et al. [238], a low-cost and solar-powered USV was designed for water quality monitoring that can operate in conditions that are dangerous and risky for human safety. However, adverse weather and water conditions such as rain and extreme wind or rough and choppy water can strongly affect USV's mission results and operations. Many business solutions are offering specialized packages combining highquality unmanned vehicles and customized software applications for aquaculture farm and water monitoring. Blueye [239] has a complete package including underwater drones and software to perform aquaculture monitoring to reduce the risk and minimize the use of divers to inspect aquaculture cages. The mini ROV has four powerful thrusters combined with a unique hull design to perform high-quality underwater inspections in tough weather conditions where very few ROVs are capable of doing it safely. Saildrone [220] developed a USV that is a capable, proven, and trusted platform for collecting high-quality ocean data for a wide variety of applications with uncrewed wind-powered vehicles using renewable energy, wind, and solar. Their vehicles are equipped with state-of-the-art sensors for data collection, and they can cover more than 500,000 nautical miles in the most extreme weather conditions. Deep Trekker's ROV is battery operated and ensures no contamination to the

environment or fish health. It was tested in several locations where ROVs faced extreme weather and sea conditions daily. Water samples can still be collected under the ice at various depths [240]. FIFISH PRO W6, an industrialized class ROV platform, is equipped with an all-new powerful and patented Q-Motor system, a depth of 350 m of dive, with an intelligent stabilization system against strong currents [241].

#### **11. Conclusions**

This paper assesses progress and identifies opportunities and challenges of utilizing unmanned systems to manage and monitor aquaculture farms. The different capabilities of drones were identified as a communication gateway and data collector, aquaculture site surveillance, and aquaculture farm management and monitoring. Some of the challenges for offshore aquaculture site management and monitoring were also part of this paper. The utilization of technological innovation using unmanned vehicle systems addressed these difficulties to achieve the goal of precision aquaculture.

We also presented three platforms for unmanned vehicles with corresponding functions and limitations. UAS or UAVs are best suited for aerial surveys, site surveillance, monitoring and inspection, and photogrammetric surveys. However, there were also some UAVs for water observation, such as the surface flow map. Unmanned vehicles equipped with LTE cellular networks and LPWAN technologies can act as a communication gateway and IoT data collector. Fairweather condition is a requirement for surveys and inspections. Most AUVs have difficulties operating in a strong wind environment, and many cannot fly during harsh weather conditions. AUVs capable to operate in such condition is very expensive and highly complicated as it also requires government certifications and formal training for operation.

AUVs, ROVs, and USVs equipped with sensors can collect data for analysis using water temperature, depth level, chemical, biological, and physical properties. Some relevant parameters to monitor water quality are temperature, oxygen level, salinity, acidity, conductivity, pH level, oxygen concentration, and nitrogenous compounds such as ammonia, nitrate, and nitrite concentration. USVs are widely utilized to monitor water conditions such as surface flow and velocity measurement. DJI Phantom commercial unmanned system is the most preferred type based on the collected works of literature. There were also some customized unmanned systems. The common sensors used for UAVs are acoustic cameras but there are also some vehicles equipped with thermal cameras. To provide motion stability for data capturing using cameras, gimbal pitch can be added although this concern should be further investigated to provide better stability, most especially for AUVs. For water velocity captures, camera shakes are evident that causes distortions to images. The capacity to operate despite a strong water current or pressure should be fully considered in selecting an underwater vehicle. There are AUVs and ROVs that were designed for this condition, but it comes with a higher price. Others might consider choosing low-cost vehicles with fewer capabilities and strengths for economic considerations. Furthermore, UAVs are more sensitive to unpredictable weather conditions such as strong winds and rains since they operate in the air.

Many unmanned system performances have limitations in terms of power or battery, which affects the mission or operation due to longer navigation time and slower mission execution. Many countermeasures were devised to optimize the navigation time of UVs. Some integrate flight path planning to reduce flight time, sensor positioning, and trajectory optimizations. There are also solar-powered UVs with efficient solar cells for an increased power source for longer navigation coverage. Multiple drones could also be used during surveillance to address the drone's limitations in terms of navigation time. To correct image blurriness, low contrast, low visibility, and small-sized captures, image enhancement, and corrections to improve quality and reduce noise; deep learning and computer vision techniques and algorithms are capable of such functions.

There is no unmanned system capable of performing all aquaculture operations and functions. These systems can collaborate to perform complex tasks to increase robustness and efficiency. Collaboration of heterogeneous vehicles can achieve larger scale and comprehensive monitoring. Despite many open issues for such kind of collaboration, the possibility of exploring its capability can help achieve high performance, adaptability, flexibility, and fault tolerance.

Different sensors were also presented, including their corresponding characteristics and limitations. Sensors are also susceptible to harsh weather conditions. For AUVs, sensors are affected by winds, waves, sea fog, and water reflection. There are various restoration methods in dealing with these concerns, such as incorporating image stabilization or image defogging. For water quality sensors, factors to consider in its integration can be low maintenance, low cost, low battery consumption, robust, waterproof, non-metallic, resistant to biofouling, and have no effects on organisms. The possibility of sensor fusion can be exploited to take advantage of UVs potentials and achieve higher precision.

Practicing awareness and continuous updates on the regulations must be practiced to avoid the legal implications of not following the law. The standardized policies for UVs operations are still not mature since regulations are different in each country although there are some common grounds. The wide range and varied types of UVs and their applications is an added challenge that requires operators and owners to be aware of the legal status and regulations of each operation. With a various and wide range of commercially available UVs in the market, compromise and trade-offs between the type of vehicle, installed sensors, power, manpower requirement, and cost are for the user's decision to weigh how to achieve maximum performance and potential based on their corresponding functions. To maximize the potential of a UV, each type should be maximized based on its strength and capabilities. There is no single unmanned system that can perform all the desired functions at once for aquaculture management and monitoring. Thus, each type can collaborate to achieve a bigger coverage for aquaculture monitoring and management. The integration of unmanned systems can be exploited to serve as a cutting-edge technology to provide robust, timely, efficient, reliable, and sustainable aquaculture. As these systems integrate more and more technologies, they can extend their functionalities to perform more capability for aquaculture production. UVs can be combined with sensors and robotics with artificial intelligence and deep learning techniques to process big data.

Unmanned systems are already widely used in fisheries science and marine conservation, such as megafauna, but the literature and research work on the application of such system in aquaculture can still be further explored to achieve maturity; more undertakings should be made for successful integration of such systems in the field of aquaculture. Although there were successful implementations that were stated in this work, state-of-the-art technologies and devices should continue for unmanned systems to provide better and more powerful aquaculture precision farming functionalities.

**Author Contributions:** Conceptualization, N.A.U. and S.-C.C.; methodology, N.A.U. and S.-C.C.; validation, S.-C.C.; formal analysis, N.A.U. and S.-C.C. investigation, N.A.U.; resources, N.A.U. and S.-C.C.; data curation N.A.U.; writing—original draft preparation, N.A.U.; writing—review and editing, N.A.U. and S.-C.C.; visualization, N.A.U.; supervision, S.-C.C.; project administration, S.-C.C.; funding acquisition, S.-C.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** No funding support for this paper.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


### *Article* **The Relationship between Drone Speed and the Number of Flights in RFID Tag Reading for Plant Inventory**

**Jannette Quino 1, Joe Mari Maja 2,\*, James Robbins 3, James Owen, Jr. 4, Matthew Chappell 5, Joao Neto Camargo <sup>6</sup> and R. Thomas Fernandez <sup>7</sup>**


**Abstract:** Accurate inventory allows for more precise forecasting, including profit projections, easier monitoring, shorter outages, and fewer delivery interruptions. Moreover, the long hours of physical labor involved over such a broad area and the effect of inefficiencies could lead to less accurate inventory. Unreliable data and predictions, unannounced stoppages in operations, production delays and delivery, and a considerable loss of profit can all arise from inaccurate inventory. This paper extends our previous work with drones and RFID by evaluating: the number of flights needed to read all tags deployed in the field, the number of scans per pass, and the optimum drone speed for reading tags. The drone flight plan was divided into eight passes from southwest to northwest and back at a horizontal speed of 2.2, 1.7, and 1.1 m per second (m/s) at a vertically fixed altitude. The results showed that speed did not affect the number of new tags scanned (*p*-value > 0.05). Results showed that 90% of the tags were scanned in less than four trips (eight passes) at 1.7 m/s. Based on these results, the system can be used for large-scale nursery inventory and other industries that use RFID tags in outdoor environments. We presented two novel measurements on evaluating RFID reader efficiency by measuring how fast the reader can read and the shortest distance traveled by the RFID reader over tag.

**Keywords:** speed; RFID; inventory; drones; labor; forecast

#### **1. Introduction**

According to the United States Department of Agriculture's National Agricultural Statistics Service [1], 91.1 million acres of land are projected to be used for plant production for 2021. Specialty crops, including floriculture and nursery products, accounted for \$13.8 billion in sales in 2019; the nursery industry is a multibillion-dollar enterprise that relies on inventory and monitoring to forecast sales, production requirements, and quality improvements [2]. The information collected in an inventory is used for planning that includes labor requirements, space requirements, production timing, and sales and demand trends, including product pricing [3]. However, obtaining individual plant information about the location or number of plants in the field is labor intensive and time-consuming. Since this process is done manually, there may be inefficiencies, including missing data, due to human error. Furthermore, it is difficult to avoid mistakes due to a lack of reliable equipment to gather data [4].

**Citation:** Quino, J.; Maja, J.M.; Robbins, J.; Owen, J., Jr.; Chappell, M.; Camargo, J.N.; Fernandez, R.T. The Relationship between Drone Speed and the Number of Flights in RFID Tag Reading for Plant Inventory. *Drones* **2022**, *6*, 2. https:// doi.org/10.3390/drones6010002

Academic Editors: Diego González-Aguilera and Pablo Rodríguez-Gonzálvez

Received: 24 November 2021 Accepted: 17 December 2021 Published: 22 December 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Increased personnel or labor alone cannot alleviate the current inventory inefficiencies. The number of hours required in the field is considerable, and workers will be scarce as the horticultural industry's labor supply declines. The decline of workers is because most entry-level employees would prefer to work in a position that requires less physical exertion [5]. Similarly, just 20% of the workers in the nursery are permanent, with the remaining 80% being temporary [6]. Thus, accessibility to laborers will be a persistent issue.

Small unmanned aircraft systems (sUAS) can fly at low altitudes and carry various sensors to gather real-time data; thus, being a viable alternative for ground-based data collection in nursery systems [7]. Aside from inventory data, sUAS is used for crop scouting and loss assessment, yield estimation and monitoring, irrigation and drainage planning, sampling of plant pathogens in the air, diagnosis of herbicide injury in crops, and efficient use of chemicals and pesticides [8]. Additionally, researchers have worked on RFID used for identification, harvesting, and crop histology [9–11]. An RFID system was used to track, capture remotely, and handle data in a vineyard [12]. RFID systems and Global Positioning System (GPS) used to monitor and track plant information can be updated in real-time. Geographic Information System (GIS) software can be used to visualize information location to edit and manage data changes in the area. RFID technology combined with GPS was used in plant pathology and genomics [13] and citrus fruit harvesting in Florida [14].

Passive RFID tags and sUAS can be combined to gather information about nursery plants in the field. The passive tag at the Ultra-high frequency (UHF) band ranges from 0 distance to 12 m and has a faster data rate [15]. However, UHF is more susceptible to interference. It usually works between 860–960 MHz band [16]. Preliminary results showed that the drone flying at a speed of 6 km per hour (kph) at an altitude of between 4.5 and 6.0 m can scan tags on a static position with the front of the tags orientated to face upward toward the antenna [17]. Knox Nursery in Winter Garden, Florida, operates a greenhouse using a ground-based Ultra-High Frequency (UHF) RFID tag platform [18]. As a result, there was an increase in productivity and the number of hours required to update the inventory report has decreased from three days to two hours, demonstrating a drop in labor costs. In addition, individual plants can be tagged or labeled using RFID; allowing them to be compared to geo-referenced data from transplant machines.

The deployment of mobile platforms such as sUAS to integrate RFID technology will ensure the ability to collect information over large areas, encompassing numerous products, taxa, or stock-keeping units (SKU), while further reducing the amount of labor and time spent in the field. In the long run, it can improve efficiency by lowering the operational cost of obtaining nursery data, resulting in higher productivity, better inventory management, and subsequent product forecasting. The following are the objectives of this experiment:


#### **2. Materials and Methods**

#### *2.1. Study Site, Drone and RFID Tags*

Clemson University Edisto Research and Education Center (EREC) in Blackville, SC, was the study site location. The Matrice 600 Pro (Matrice 600 Pro, Shenzhen DJI Sciences and Technologies Ltd., Shenzhen, China) (Figure 1a) sUAS carried the 2.6 kg RFID–Reader Module (RFID-RM) (Figure 1b) used to scan the RFID tags [17]. The sUAS can handle a maximum payload of 6 kg, including six TB47S batteries. Figure 2 shows the RFID tag used in this experiment based on prior work. Figure 3 shows the placement of the tags at the study site.

**Figure 1.** (**a**) DJI Matrice 600 Pro, and (**b**) RFID Reader Module (RFID-RM).

**Figure 2.** The RFID tag design (Avery Dennison Corp., Glendale, CA, USA) used in the experiment.

**Figure 3.** RFID tags at the study site facing upward.

#### *2.2. RFID-Reader Module (RFID-RM) and Dashboard*

A Lithium battery of 12-volt, 2200-mAh, 3-cell supplied power to the RFID-RM. The RFID-RM, once powered, sent all information to a remote computer via 450 Mhz transceivers. The RFID-RM reader chip was set to use the maximum power and operates at a frequency of 900 Mhz [17]. The current firmware was written in C and can be easily updated to change the polling frequency and duration of the reading.

A dashboard was developed to display RFID tag ID, GPS, Received Signal Strength Indicator (RSSI), Phase Angle, and battery information of the RFID-RM. The original dashboard in Figure 4a was developed to display all tag information without a map. The new dashboard in Figure 4b has a view map option button to show the map and the approximate location of the tags. It also has an indicator that the GPS signal is fixed. This information is an indicator that the system is now ready to collect data. Another new feature is the ability to display the approximate location of the tags on a map and the plotting of the reader coordinates in reference to the tag locations (Figure 4c). A logging area is also added to view the count summary of the new tags per pass, and the RFID tag or Electronics Product Code (EPC) Tagbytes are displayed in the order they were scanned. Furthermore, when a user clicks the disconnect button, the data collected are automatically saved in a folder with a timestamped filename.

(**c**)

**Figure 4.** The (**a**) old Dashboard version, (**b**) new version of the Dashboard, and (**c**) Map showing the approximate location of the different RFID tags in the field.

#### *2.3. Weather Condition*

The weather conditions were obtained from the Clemson University- EREC Station Summary of Weather Conditions (www.edistorecweather.net, accessed on 19 August 2021). Table 1 shows the weather report during the data collection.

**Table 1.** Clemson University-EREC Station Summary of Weather Conditions.


#### *2.4. The Tag Layout*

Twenty identical tags were attached to a 1.8-m-long round wood dowel. There were four dowels in total, with six tags attached to the first three dowels and the final two tags on the fourth dowel. The dowel was marked with black stripes to arrange the tags from tag 1 to tag 20. The dowels were placed on top of a tripod stand 1.2 m above the ground. Prior to the drone flights, the dowel was rotated so that the tags face upward due to our previous finding on the best position of the tag. To keep the dowel from rolling, it was placed on top of the foam and taped down. Figure 5 depicts the tags spread sideways by 0.3 m from the center of one tag to the center of the adjacent tag. The total distance measured between tags 1 and 20 was 5.8 m. The spacing of the tags (0.3 m) was based on the normal distance of the pots in nursery production [19].

**Figure 5.** The tags spaced at 0.30 m, facing upward.

#### *2.5. The Number of Tags Scanned*

The distance between the drone and the tag was held at 4.5 m. The flight altitude was set to 5.7 m with the tags positioned 1.2 m from the ground. Two flag sticks were used as a guide for the drone flight path. The RFID-RM was attached to the drone with the antenna pointing downward to the tags. Due to the limitations of the drone's battery, the trips were reduced to three flights per speed to accommodate four trips for a total of eight passes per speed. A single trip consists of two passes, one forward and one backward (Figure 6). Three drone speeds were evaluated; 2.2 m/s (8 kph), 1.7 m/s (6 kph), and 1.1 m/s (4 kph) to compare the scan rate. During the data collection process, some tags were scanned or read multiple times in a single pass, while others were not scanned at all. Tags that are scanned for the first time after being listed in the log area are considered new or unique tags.

**Figure 6.** Drone flying at 5.7 m AGL toward the tags.

#### *2.6. Inverse Rate*

Drone travel time and the number of tags scanned were used to determine the inverse rate (seconds per tag [spt]) of each pass. The spt is a value that describes how long it takes to scan a tag in each pass. The drone travel time is the time it will take the drone to travel from tag 1 to tag 20 at a set speed (Equation (1)). As shown in Table 2, the drone travel time is 3.5 s to complete one pass covering 5.8 m at a speed of 1.7 m/s. Consequently, spt was calculated by dividing the number of tags scanned per pass by the drone travel time (Equation (2)).

$$v = {}^d\!/\_t$$

where:

*ν* = drone speed (m/s);

*d* = total distance (meters) flown by the drone in one pass;

*t* = drone travel time (seconds).

**Table 2.** Drone travel time per pass.


$$spt = \frac{t}{T} \tag{2}$$

where:

*spt* = seconds per tag;

*t* = drone travel time (seconds);

*T* = new tags scanned per pass.

#### *2.7. RFID-RM Efficiency*

To determine the efficiency of the RFID-RM in reading tags at a fixed speed, it is important to determine the distance traveled by the drone to scan the next available tag (meters per tag [mpt]). The mpt is calculated by multiplying the spt with the drone speed (Equation (3)).

$$mpt = spt \times v \tag{3}$$

where:

*mpt* = meters per tag

*spt* = seconds per tag *ν* = speed (m/s)

#### *2.8. Statistical Analysis*

A single factor Analysis of Variance (ANOVA) without replication is used to analyze the data. The hypothesis is that the speed of the drone and the number of passes will influence the number of new or unique tag readings.

#### **3. Results**

#### *3.1. The Number of New Tags Scanned*

Figure 7 shows the total number of tag readings, which includes tags that have been scanned multiple time as well as newly scanned tags. A forward speed of 1.7 m/s resulted in the greatest number of tags read (451) and 2.2 m/s yielded the fewest (281). Figure 8 shows the percentage of new tags read for each pass based on the three drone speeds. The first pass resulted in the highest percentage of tag read across all three speeds. Values ranged from 25% to 30%. Starting with the fifth pass, the percentage of news tags read never exceeded 5% for any drone speed. In general, across all drone speeds, the percentage of new tags read decreased after the fourth pass. Figure 9 summarizes the cumulative total of tag readings at each pass and at three drone speeds.

**Figure 7.** The effect of drone speed on the number of total tags reading (includes multiple reads per tag).

**Figure 8.** Percent of new tags read at three speeds of the drone for eight passes.

**Figure 9.** The cumulative tag readings in each pass at three drone speeds.

#### *3.2. Inverse Rate*

At a speed of 2.2 m/s, the inverse rate ranged from 0 at pass number seven to 0.146 at pass number six as shown in Figure 10. An inverse rate of zero indicates that no tags were read. At a speed of 1.7 m/s, the inverse rate ranged from 0.039 at pass number four to 0.128 at pass number seven. The inverse rate at a speed of 1.1 m/s ranged from 0.048 at pass number four to 0.580 at pass number five.

**Figure 10.** The number of seconds per tag (inverse rate) for each pass and each drone speed as calculated using Equation (2).

Figure 11 summarizes the effect of drone speed on the number of seconds it takes to read one new tag on each of the eight passes. Results indicate the inverse rate for new tags is between 0.439 at pass number one and 2.632 at pass number four, five, six and seven. At a speed of 1.7 m/s, the inverse rate is between 0.578 at pass number one and 3.467 at pass number five. The inverse rate of 1.1 m/s is between 1.043 at pass number one and 5.216 at pass numbers three, six and seven. Contrary to Figure 10, Figure 11 only shows the unique tag read on multiple passes. Tags which were already read on the first few passes were not counted.

**Figure 11.** The time required to read each new tag for each pass at three drone speeds.

#### *3.3. RFID-RM Efficiency*

Figure 12 shows that at a speed of 2.2 m/s, the RFID-RM efficiency in terms of readings all tags is between 0.080 at pass number one and 0.322 at pass number six. Furthermore, at a speed of 1.7 m/s, the RFID-RM efficiency is between 0.064 at pass number four and 0.214 at pass number seven. The RFID-RM efficiency of 1.1 m/s is between 0.054 at pass number four and 0.643 at pass number five.

Figure 13 shows that at a speed of 2.2 m/s, the RFID-RM efficiency for unique tags was between 0.96 at pass number one and 5.79 at pass number four, four, six and seven. At a speed of 1.7 m/s, the RFID-RM efficiency is between 0.96 at pass number one and 5.8 at pass number five. While the RFID-RM efficiency of 1.1 m/s is between 1.158 at pass number one and 5.79 at pass number three, six and seven.

**Figure 13.** The distance covered to read unique tag for each pass at three different speeds.

#### *3.4. Statistical Analysis*

The ANOVA evaluates whether there is a statistically significant difference between each pass reading and how the RFID reader detects tags at variable speeds. The ANOVA parameters are the source of variation, Sum of Squares (SS), degree of freedom (df), Mean Squares (MS), F statistical (F), (*p*-value) a measure of the probability of the test results gathered on how likely that the same tag readings would occur by random chance, and F critical (F crit). Furthermore, F, *p*-value, and the F crit values are the values that would determine the significance level of the source of variation. The value F > F crit is the same as *p*-value < alpha, which rejects the null hypothesis. In this experiment, the source of variation is the number of passes and the drone speeds. The SS, df, MS are values used to compute the F, F crit, and the *p*-value. The SS is derived by getting the count for each speed reading multiplied by the difference between the speed and grand mean. The grand mean is the average of all the readings, adding the total new tag readings divided by the total count. In addition, the df is the difference between groups minus 1 (8 passes − 1 = 7). The MS is the square of the difference between the mean, and the data collected, while the F results from the mean square of the number of passes divided by the mean square in each pass. The *p*-value of 0.000126 measures the probability that an observed difference could occur is computed using F, df between groups as the numerator and df within groups as the denominator. F crit is computed using three arguments, the alpha value, numerator df between groups (between each pass), and the denominator df within each pass.

The number of passes made by drone scanning new/unique tags has an influence on the number of tags read (*p*-value = 0.000126); however, the three speeds used for this experiment are not significant (*p*-value = 0.941182) (see Table 3).

**Table 3.** Anova results for scanning a new/unique tag.


#### **4. Discussion**

Prior to these experiments, the hypothesis was that the speed of the drone and the number of passes has an influence on the reading of unique or new tags. Results from our experiments suggest that only the number of passes will affect the number of new tags read. At 2.2 m/s, the label scanning is between 0.037 and 0.146 spt. At 1.1 m/s the label scanning is between 0.048 and 0.580 spt. The result also shows that the total time required to collect tag information is determined by the drone's speed and the length of the flight path [20,21]. In addition, at speeds of 1.1 to 2.2 m/s, the scan time ranged from 0.01 to 0.15 has a common overlap. Based on our results, the effect of the number of readings when the drone changes speed is not significant.

Furthermore, the first pass results in the highest percentage of new tags scanned across all speeds. On the fifth pass, the scan rate begins to drop. However, 1.7 m/s scanned a total of 95 percent of new tags on the fifth pass, the highest percentage among the three speeds. The scan rate trend increases from the first to the fourth pass and then a decreasing trend from the fifth to the eighth pass. In addition, after the scan rate reaches 70% of the tags scanned, the scan rate also shows a decreasing trend. The decreasing trend was since only 30% of new tags are still available for reading.

There were instances where readings were not reported, starting at the pass 5 onwards. In Figure 9, no readings were recorded at a drone speed of 2.2 m/s for pass 7 and 1.1 m/s for pass 8. Figure 10 also shows that at pass 5, at the speed 1.1 m/s did not register any readings, but readings were registered on passes 6 and 7. The same instances can be found in Figures 11 and 12, where no readings were registered at various speeds. This could be attributed to how the RFID-RM is responding to tag readings. Every tag read goes into an interrupt service routine where it collects all available information before the reader will be available for reading another tag. Although the tags are of the same manufacturer, some tags may respond quicker and thus create multiple instances of the interrupt, thus, delaying the reader. Moreover, the physical constraints feature of radio frequency poses a barrier to advanced anti-collision techniques for RFID chips, due to the interference from the transition signal's scattering and reflection [22].

We presented two novel measurements on evaluating RFID reader efficiency by measuring how fast the reader can read and the shortest distance traveled of the RFID reader reading a tag. These two measurements can be used to compare other RFID readers developed by different manufacturers.

#### **5. Conclusions**

Based on results from this experiment, the first three flights have the highest number of unique tags scanned. The time required to scan tags repeatedly is 0.04 s per tag (spt), and the minimum distance is 0.064 m per tag (mpt) at a speed of 1.7 m/s. New tags can be scanned in the first five passes at a percentage between 70 to 95 percent. The first pass results in the most readings (30%), and gradually decreases as the number of passes increases. The decline is attributable to a reduction in the quantity of new scannable tags.

Finally, when scanning new tags, the pass number is essential. At least two flights equivalent to four passes will get a scan rate of 70% to 90%. It takes 0.578 to 3.467 spt to scan at a speed of 1.7 m/s with a minimum distance of 0.96 mpt. It is important to note that due to the placement of plants in the field at close proximity, this experiment showed that the drone must perform multiple passes on the same location to gather an accurate inventory. This limitation has been shown in the results of this work. The distance of the tags was based on the distance between the pots in a real nursery production.

This technology is a potential tool in obtaining automated, aireal based nursery inventory data that would reduce labor hours in the field for data retrieval. Once the pandemic has subsided, we will conduct the field testing in large nurseries.

**Author Contributions:** Conceptualization, J.M.M. and J.Q.; methodology, J.Q.; software, J.N.C. and J.Q.; validation, J.Q.; formal analysis, J.Q.; investigation, J.Q.; resources, J.M.M., J.R., J.O.J., R.T.F. and M.C.; data curation, J.Q.; writing—original draft preparation, J.Q., J.M.M. and J.R.; writing—review and editing, J.M.M., J.R., J.O.J., R.T.F. and M.C.; visualization, J.Q.; supervision, J.M.M.; project administration, R.T.F.; funding acquisition, R.T.F., J.M.M., J.R., J.O.J. and M.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was partially supported by Horticultural Research Institute (HRI) Grant number 5935985 and is based on work supported by NIFA/USDA under project numbers MICL02473, and SC-1700609.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors would like to thank Avery Dennison Corporation and R.A. Dudley Nurseries Inc. for their support and assistance in this project.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Comparison of RGB and Multispectral Unmanned Aerial Vehicle for Monitoring Vegetation Coverage Changes on a Landslide Area**

**Flavio Furukawa 1,\*, Lauretta Andrew Laneng 1, Hiroaki Ando 1, Nobuhiko Yoshimura 2, Masami Kaneko <sup>2</sup> and Junko Morimoto <sup>1</sup>**


**Abstract:** The development of UAV technologies offers practical methods to create landcover maps for monitoring and management of areas affected by natural disasters such as landslides. The present study aims at comparing the capability of two different types of UAV to deliver precise information, in order to characterize vegetation at landslide areas over a period of months. For the comparison, an RGB UAV and a Multispectral UAV were used to identify three different classes: vegetation, bare soil, and dead matter, from April to July 2021. The results showed high overall accuracy values (>95%) for the Multispectral UAV, as compared to the RGB UAV, which had lower overall accuracies. Although having lower overall accuracies, the vegetation class of the RGB UAV presented high producer's and user's accuracy over time, comparable to the Multispectral UAV results. Image quality played an important role in this study, where higher accuracy values were found on cloudy days. Both RGB and Multispectral UAVs presented similar patterns of vegetation, bare soil, and dead matter classes, where the increase in vegetation class was consistent with the decrease in bare soil and dead matter class. The present study suggests that the Multispectral UAV is more suitable in characterizing vegetation, bare soil, and dead matter classes on landslide areas while the RGB UAV can deliver reliable information for vegetation monitoring.

**Keywords:** landslides; unmanned aerial vehicle (UAV); multispectral; RGB; vegetation monitoring

#### **1. Introduction**

The evolution of remote sensing technology allows a feasible method for gathering detailed information for mapping land-cover changes [1], drought monitoring [2], and analyzing complex attributes [3,4] over space and time. This technology uses different types of sensor onboard satellites, airborne or unmanned aerial vehicles (UAVs), and provides different methods of vegetation classification at large and small scales. Remote sensing offers a practical approach to designing strategies for the management of forest disaster such as evaluating landslide-prone areas through airborne, UAV, and ground-based remote sensing [5], as well as for evaluating changes in vegetation cover after a wildfire for post-fire management by using satellite-based remote sensing and UAV [6].

To deal with the need to assess forest disasters for quick management decisions, the advancement of satellite-based remote sensing applications was initiated for detecting areas affected by natural disasters such as windthrow and landslide for forest restoration or forest disaster management purposes [7], assessing vegetation recovery [8], detecting and mapping [9,10] of landslide areas, and creating historical landslide inventories [11]. Although playing an important role in forest disaster management, satellite-based remote sensing has some limitations in terms of spatial and temporal resolution of the data.

**Citation:** Furukawa, F.; Laneng, L.A.; Ando, H.; Yoshimura, N.; Kaneko, M.; Morimoto, J. Comparison of RGB and Multispectral Unmanned Aerial Vehicle for Monitoring Vegetation Coverage Changes on a Landslide Area. *Drones* **2021**, *5*, 97. https:// doi.org/10.3390/drones5030097

Academic Editor: Giordano Teza

Received: 3 August 2021 Accepted: 10 September 2021 Published: 13 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Local cloudiness, low temporal and spatial resolution, and gaps on the image create a complex task for vegetation classification [2,12,13]. Recently, very high spatial resolution satellites are available, delivering data of around 30 cm per pixel [14]; despite a high spatial resolution, this could be a limitation in understanding changes happening on smaller scales [15]. A one-day temporal resolution satellite dataset is also available [16], but cloud cover can still be a hindrance to acquiring the desired dataset.

Nevertheless, the evolution of UAV technologies has brought RGB sensors and multispectral sensors to UAVs for more detailed information as compared to satellite-based remote sensing, making it possible to acquire centimeter-level imagery at any time. In terms of cost and availability, multispectral UAVs cost much more and have lower availability while UAVs coupled with RGB sensors are more affordable and accessible. However, RGB UAVs are limited for remote sensing analysis, especially on complex and heterogeneous forest-covered areas, due to the sensor having an RGB array filter [17]. Despite these limitations, Ruwaimana et al., [18] proved that the application of UAVs for vegetation classification on mangrove ecosystems provided higher accuracy concerning object-based and pixel-based classification compared to satellite imagery. The implementation of UAV systems gained attention not only due to their efficiency to map land cover [19,20] vegetation on a coastal dune [21] but also as an effective tool in mapping and characterizing burned areas affected by wildfires [22], as well as landslide displacement mapping [23].

Comparing the performance between satellite image and aerial photo for vegetation mapping [18], testing the applicability of UAVs for mapping and monitoring geohazard areas [24], as well as characterizing and monitoring landslides, [25] have been well documented. Yet there is still a gap in understanding how RGB and multispectral sensors on UAVs perform in assessing the regrowth of vegetation in an area affected by a natural disaster such as a landslide. In order to understand the condition of the affected area to make management decisions, it is important to determine the vegetation coverage to understand its regrowth on a landslide area on a small scale [26,27], and to evaluate the area's ability to undergo a natural regeneration process on a regional scale. Besides, the presence of debris including fallen logs and litter provides a potential for vegetation regrowth by sprouting and seedbanks [28] and by the colonization of early successional plant species [29–31]. Moreover, due to unstable bare soil conditions, vegetation regrowth is slow or non-existent on hillslopes [32].

Therefore, we mapped a landslide area considering three different classes (i.e., vegetation, bare soil, and dead matter) to assess the changes in coverage pattern focusing on vegetation growth throughout four months using two different types of UAV. This study aimed to compare the performance of an RGB UAV and a multispectral UAV using a pixel-based classification approach, to understand how the spectral resolution and the type of sensor can deliver precise information for vegetation mapping on a landslide area. The findings from this study can provide baseline information for forest managers and ecologists in selecting the applicable system and to assist in deciding on further management practice in the affected area, especially in understanding post-landslide regeneration. Thus, this study was designed for the following objectives: (1) to understand the differences between the UAV systems for vegetation mapping in a landslide area assessing the parameters that affect the datasets; (2) to monitor the monthly changes of vegetation, bare soil, and dead matter areas in landslides for the management of vegetation recovery.

#### **2. Materials and Methods**

#### *2.1. Study Area*

In 2018, the northernmost island of Japan, Hokkaido, was affected by the Hokkaido Eastern Iburi Earthquake, with a magnitude of 6.7 [33] and several aftershocks. The seism triggered over 4000 ha of landslides around different municipalities in western Atsuma town [34].

This study was conducted in an area of surface failure of approximately 8 ha in the Uryu District at Atsuma town (42◦43 20.3 N, 141◦55 22.5 E) (Figure 1). The area was characterized by moderate terrain with a predominant slope and an angle of less than 40 degrees, and the elevation ranged from 57m to 121m. Soil structure consists of Neogene sedimentary rock, i.e., sandstone, siltstone, mudstone, and a conglomerate that was covered by a thick pyroclastic fall deposit from the Tarumae Volcano [34,35]. The area was covered mostly by deciduous trees, fallen trees, and bare soil, an effect of the landslide, with grasses and shrubs such as Japanese sweet-coltsfoot (*Petasites japonicus* (Siebold et Zucc.) Maxim.), dwarf bamboo (*Sasa* spp.), and wild berries (*Rubus* spp.), etc.

**Figure 1.** (**a**) The study area located in Hokkaido, Japan, (**b**) at Atsuma town (black boundary), and in Uryu district (pink boundary) located at 42◦43 20.3 N, 141◦55 22.5 E (red star); (**c**) with the true color ortho-mosaic taken with the Multispectral UAV on 9 June.

#### *2.2. Datasets*

For acquisition of the aerial images to create the ortho-mosaics for analysis, two different UAVs were used: the DJI Phantom 4 Pro, and the DJI Phantom 4 Multispectral. The DJI Phantom 4 Pro has a 1-inch CMOS RGB sensor, which acquires the red, green, and blue wavelengths in the same sensor, delivering one 5472 × 3648 pixels RGB image per shot. On the other hand, the DJI Phantom 4 Multispectral, has six 1/2.9-inch CMOS sensors, one RGB sensor for visible imaging and five monochrome sensors for multispectral imaging in different spectral bands: blue, green, red, red-edge, and near-infrared. Each band generates one image of 1600 × 1300 pixels, totalizing six images per shot. The DJI Phantom 4 Multispectral also had a Real-Time Kinect (RTK) GNSS system built in for centimeter position accuracy, but for this study, we compared only the sensors of each UAV: the RGB sensor of DJI Phantom 4 Pro (RGB UAV) and the multispectral sensor from DJI Phantom 4 Multispectral (Multispectral UAV).

The data was taken in four different flight campaigns in 2021: 14 April, 12 May, 9 June, and 9 July, with all images taken in the morning. The weather condition on 14 April and 9 July was cloudy, while being sunny on 12 May and 9 June, with no clouds (Figure 2).

**Figure 2.** Cloud cover over the study site (red star) in each date, assessed using Modis M0D09GQ.006 Terra Surface Reflectance Daily Global 250 m, acquired in the morning [36].

For each flight campaign, we first flew the Multispectral UAV followed by the RGB UAV (around 14 min each flight), with 5 min in between flights to reduce the displacement of shadow areas. The UAVs were flown at 120 m of altitude, capturing images with 80% overlap and 80% side-lap to create the ortho-mosaics via photogrammetry processing. For the Multispectral UAV, images of a calibration reflectance panel were taken to be used on the calibration of the multispectral images inside the photogrammetry software [37].

To register the RGB and Multispectral ortho-mosaics, 15 ground control points (GCPs) made from plywood were placed along the study site and the position of each point was collected using the Drogger RTK GNSS system [38] connected to the ICHIMILL virtual reference station (VRS) [39] service provided by Softbank Japan [40]. The accuracy of each point position was around 2 cm.

For each flight campaign, a field survey was also conducted. Using the Drogger RTK system connected to an android tablet with the open-source application Open Data Kit (ODK) [41], we collected ground truth points to classify the ortho-mosaics and validate the classification results. Inside the ODK application, a questionnaire form was previously created containing the classes to be chosen on the field, and photos were taken with the tablet (Figure 3).

**Figure 3.** (**a**) The red dot is the vegetation class obtained by ODK with the RTK system accuracy (2 cm) on the Multispectral UAV ortho-mosaic in true color, and (**b**) the respective photo of a Japanese sweet-coltsfoot for verification on 12 May.

#### *2.3. Data Processing*

To create the ortho-mosaics, we used the photogrammetry technique for UAVs [42], where each image dataset was processed on Agisoft Metashape [43] with the GCPs taken on the field to improve the position accuracy of the ortho-mosaic. For the Multispectral UAV, the 5 monochrome images were automatically merged creating a multispectral orthomosaic, and the images were also calibrated in the software using the calibration reflectance panel images to convert the digital numbers into reflectance values. All ortho-mosaics were later uploaded into Google Earth Engine [44] and resampled to the same spatial resolution of 5.5 cm using the bilinear interpolation mode.

#### *2.4. Classification and Accuracy Assessment*

The processing workflow is shown in Figure 4. To identify vegetation cover in the study area, three different classes were established: vegetation, bare soil, and dead matter (dead leaves, fallen trees, and tree branches). To create the reference dataset, an empirical test was made and 30 samples for each class were selected to conduct the study. The reference dataset was composed of samples taken on the field and samples selected from a visual interpretation of the ortho-mosaic, totalizing 90 samples. For each date, the same reference dataset was used for the RGB and the multispectral dataset.

The classification and the assessment for this study were made by applying the crossvalidation method [45], using 5 k-folds inside Google Earth Engine. The built-in support vector machine classifier with the linear kernel type [46] was selected to classify the orthomosaics. This method was chosen due to its robustness in assessing the predictor model, which in this study was mainly influenced by the ortho-mosaic.

First, the reference data was divided into five different folds randomly, where four folds (80% of the reference dataset) were used to train the classifier and one fold (20% of the reference dataset) to test the classifier. A total of five iterations were made to test all folds.

For each iteration, we created a classification model based on the training dataset and the support vector machine classifier. Then, the classification model generated a prediction map which was put against the independent testing dataset to achieve a confusion matrix. The confusion matrix delivered three different results: overall accuracy, producer's accuracy (PA), and user's accuracy (UA).

**Figure 4.** The processing workflow for each dataset.

The final assessment values for each ortho-mosaic were created considering the mean of the accuracies of all five confusion matrices. To create the final classification map of each ortho-mosaic, an aggregation was made considering the majority of classes among the five iterations for each pixel; the final classification map presented a straightforward portrayal of confidence for the study site, which identified the model's fit and stability. Whilst not directly measuring mapping accuracy, the relative confidence of the methodology can provide valuable information to support the interpretation of the maps [47].

#### **3. Results**

#### *3.1. UAV Orthomosaics*

Figure 5 shows that the higher spatial resolution of the RGB UAV created orthomosaics with more details compared to the Multispectral UAV ortho-mosaic, even though the data were resampled to 5.5 cm.

**Figure 5.** (**a**) The RGB UAV in true color ortho-mosaic resampled to 5.5 cm, (**b**) the Multispectral UAV in true color ortho-mosaic resampled to 5.5 cm. The RGB UAV ortho-mosaic has a sharper image compared to the Multispectral UAV ortho-mosaic.

The RGB and Multispectral UAV ortho-mosaic colors and amount of shadow were also influenced by the weather condition (Figure 2). Due to the cloudy condition and rain on the previous days of 14 April and 9 July [48,49], the ortho-mosaics were generated with brownish soil and without any shadow effect. During the sunny condition on 12 May and 9 June, the ortho-mosaics were generated with whitish soil and shadow effects (Figure 6).

**Figure 6.** The RGB UAV and Multispectral UAV ortho-mosaics generated by Agisoft Metashape on 14 April, 12 May, 9 June and 9 July. The soil color on 14 April and 9 July was brownish with no shadow, while on 12 May and 9 June, the soil was whitish with shadow areas.

#### *3.2. Performance of the UAV's Imagery*

The performance of the UAV's imagery was accessed considering the overall accuracies calculated from the mean of all five K-folds of each dataset (Table 1). The Multispectral UAV delivered higher percentages (more than 95%) throughout the months. On the other hand, the RGB UAV presented slightly lower overall values, with the highest values on 14 April (94.44%) and on 9 July (90%), while for pm 12 May and 9 June, the values were 72.22% and 64.44% respectively.

**Table 1.** Overall accuracies for the Multispectral UAV and RGB UAV on each date with the respective weather condition.


Looking into the PA and UA of all classes (i.e., vegetation, bare soil, and dead matter) (Table 2), the RGB UAV had the highest values for the three classes on April 14th and July 9th, while lower values were found on 12 May and 9 June, mainly on bare soil and dead matter classes. The Multispectral UAV was more consistent compared to the overall accuracies in Table 1, and both PA and UA showed high values throughout the months for all three classes, above 90%.


**Table 2.** Producer's and user's accuracy of the vegetation, bare soil, and dead matter classes.

PA: Producer's Accuracy, UA: User's Accuracy.

#### *3.3. Classification Results*

The classification results created through the aggregation considering the majority classes for the five prediction maps are shown in Figure 7.

**Figure 7.** Classification results from the Multispectral UAV and the RGB UAV on each date.

Despite the high accuracy values on the Multispectral UAV, the visual interpretation showed some disparities when compared to the respective ortho-mosaics (Figure 8). Misclassification mainly occurred on the shadowed area (Figure 8a), where both bare soil and dead matter areas were misclassified as vegetation class (Figure 8b).

**Figure 8.** (**a**) The Multispectral UAV ortho-mosaic in true color on 9 June, (**b**) vegetation class (pink), misclassifying bare soil and dead matter areas (red arrows).

The RGB UAV generated more misclassification throughout the study area. On 12 May and 9 June, it was clear to see the misclassification of the dead matter class on bare areas (Figures 6 and 7). A closer look on 12 May (Figure 9) showed misclassification occurring even in no shadow areas.

**Figure 9.** (**a**) The RGB UAV ortho-mosaic in true color on 12 May, (**b**) the dead matter class (pink), misclassifying bare areas (red arrows).

The comparison among the classified maps in terms of class coverage (i.e., vegetation, bare soil, and dead matter) over the months, showed a similar pattern in the RGB UAV and the Multispectral UAV from April to June (Figure 10), where we found an increase in the vegetation class and a decrease in both bare soil and dead matter classes.

**Figure 10.** The graph shows the class coverage (%) generated from the (**a**) RGB UAV and (**b**) Multispectral UAV over time.

In the Multispectral UAV, the proportion for the vegetation class on 9 June was higher when compared to 9 July, while values for bare soil increased during the same period. This was due to the misclassification that happened in the shadowed area of 9 June (Figure 8). Another problem also occurred on the RGB UAV, where there was an increase in the dead matter class from 14 April to 12 May, misclassified by the inclusion of the dead matter class on bare areas (Figure 9).

Comparing the vegetation class of RGB UAV and Multispectral UAV, besides presenting high values of PA and UA, it was possible to see a similar pattern of vegetation growth around the already vegetated areas (Figure 11). On the other hand, for the bare soil and dead matter classes, the similarities were much smaller when comparing the RGB UAV and the Multispectral UAV (Figures 12 and 13), as expected by the low values of the PA and UA accuracies from these classes on the RGB UAV.

**Figure 11.** Change of vegetation class over the months from the RGB and multispectral UAV.

**Figure 12.** Change of bare soil class over the months from the RGB and multispectral UAV.

**Figure 13.** Change of dead matter class over the months from the RGB and multispectral UAV.

#### **4. Discussion**

#### *4.1. Comparison between the RGB UAV and the Multispectral UAV*

The evaluation of the performance of each UAV showed that the Multispectral UAV delivered more consistent results for every class, while the RGB UAV, even though more detailed (higher spatial resolution), suffered from the smaller number of bands and the type of sensor [50], generating a more speckled classification map. On the other hand, even though having five distinct spectral bands and higher accuracy values, the Multispectral UAV generated some misclassification, mainly on shadowed areas [51,52].

Apart from the misclassification of shadowed areas, the weather conditions played an important role in this study, mainly for the RGB UAV. Cloudy days with brownish soil had better results compared to sunny weather with whitish soil, delivering higher accuracy values for both RGB UAV and Multispectral UAV. This was also confirmed by Duffy et al. [51], which suggests that cloudy days had consistent lighting conditions, improving the homogeneity of the spectral signatures.

Even though the RGB UAV and the Multispectral UAV generated misclassifications, they could still provide valuable information regarding the monitoring of classes' coverage changes on a landslide area. The RGB UAV delivered impressive results, being able to monitor vegetation growth in detail despite the low cost of the system. Although the visual analysis showed a discrepancy between the RGB UAV classification map and the respective ortho-mosaics on the bare soil and dead matter classes, when comparing the area of coverage by the classes both UAV systems had similar patterns, with the vegetation class reflecting a gradual increase from April to June along with the decrease in bare soil and dead matter classes over these months.

Considering the pixel-based classification approach, the Multispectral UAV is recommended, due to its ability to acquire data on the red edge and near-infrared wavelengths, optimal for vegetation analysis. On the other hand, the higher spatial resolution of the RGB UAV could enable a more accurate visual inspection of the geohazard areas as reported by Rossi et al. [24]. Future studies using an object-based classification approach are suggested to understand the difference between the two UAV systems considering spatial resolution [18]. Therefore, both the RGB UAV and the Multispectral UAV proved suitable for evaluating the capability of the area to undergo a natural regeneration process, at a centimeter-level.

#### *4.2. Vegetation, Bare Soil, and Dead Matter Monitoring*

The results showed not only the possibility of monitoring changes throughout the months, but also locating where the changes happened. This is key since monitoring pattern changes from dead matter to vegetation class could provide an initial understanding of the potential of vegetation regeneration on the landslides area. The applied methodology also proved suitable for areas with a dominance of deciduous forest, where the identification of the dead matter was possible after the winter season when the trees had no foliage.

The vegetation growth around the already vegetated areas confirms that the condition of unstable soil after landslides, preventing seeds from nearby intact forests to germinate due to the erosion of soil, infertile soil, and other abiotic factors, slows down or impedes the regeneration process. The availability of decomposing material, i.e., fallen trees and leaf litter, favor the initial stage of plant succession on the landslide area [53,54] by protecting the seeds or saplings from rolling down due to soil erosion. as well improving soil fertility through the decomposition process.

The expansion in vegetation coverage observed during the four consecutive months could indicate that a post-landslide regeneration occurred in the affected area. This suggests that the increase in vegetation recovery on the landslides area might improve stability, especially on the bare soil area, in order to support seed germination and the growth of saplings, though this process would take a long time [53,55]. Thus, monitoring the pattern changes through time comparing the three classes, i.e., vegetation, bare soil, dead matter, contributes to a more detailed ecological research planning. Due to the role of

landslide areas in regenerating high vegetation species richness after disturbance [55,56], the annual vegetation growth dataset is suggested to infer the potential of the study area for dynamic regeneration.

#### **5. Conclusions**

Overall, the present study reveals that Multispectral UAVs are more applicable for characterizing vegetation, bare soil, and dead matter in areas affected by landslides, highlighting that cloudy weather and brownish soil are recommended to create a more reliable dataset. However, the RGB UAV can play an important role if the purpose is to monitor vegetation development, which is a positive achievement, especially in terms of accessibility and availability of the tool. In addition, the monitoring of vegetation, bare soil, and dead matter classes over four months suggests the initial recovery of vegetation on the landslide area. This indicates that the monthly annual dataset and multi-year dataset will serve a better understanding of the dynamic process of initial vegetation recovery. Future work is suggested using an object-based classification approach, in order to take advantage of the higher spatial resolution of the RGB UAV dataset.

**Author Contributions:** Conceptualization, F.F., L.A.L. and J.M.; data curation, H.A.; formal analysis, F.F.; investigation, F.F. and H.A.; methodology, F.F.; project administration, F.F.; resources, M.K.; software, F.F.; supervision, M.K. and J.M.; validation, N.Y.; visualization, L.A.L.; writing—original draft, F.F. and L.A.L.; writing—review & editing, F.F., L.A.L., N.Y. and J.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received JSPS KAKENHI Grant Number JP17H01516 and TOUGOU Grant Number JPMXD0717935498.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Quantifying the Spatial Variability of Annual and Seasonal Changes in Riverscape Vegetation Using Drone Laser Scanning**

**Jonathan P. Resop 1,\*, Laura Lehmann <sup>2</sup> and W. Cully Hession <sup>2</sup>**


**Abstract:** Riverscapes are complex ecosystems consisting of dynamic processes influenced by spatially heterogeneous physical features. A critical component of riverscapes is vegetation in the stream channel and floodplain, which influences flooding and provides habitat. Riverscape vegetation can be highly variable in size and structure, including wetland plants, grasses, shrubs, and trees. This vegetation variability is difficult to precisely measure over large extents with traditional surveying tools. Drone laser scanning (DLS), or UAV-based lidar, has shown potential for measuring topography and vegetation over large extents at a high resolution but has yet to be used to quantify both the temporal and spatial variability of riverscape vegetation. Scans were performed on a reach of Stroubles Creek in Blacksburg, VA, USA six times between 2017 and 2019. Change was calculated both annually and seasonally over the two-year period. Metrics were derived from the lidar scans to represent different aspects of riverscape vegetation: height, roughness, and density. Vegetation was classified as scrub or tree based on the height above ground and 604 trees were manually identified in the riverscape, which grew on average by 0.74 m annually. Trees had greater annual growth and scrub had greater seasonal variability. Height and roughness were better measures of annual growth and density was a better measure of seasonal variability. The results demonstrate the advantage of repeat surveys with high-resolution DLS for detecting seasonal variability in the riverscape environment, including the growth and decay of floodplain vegetation, which is critical information for various hydraulic and ecological applications.

**Keywords:** UAVs; lidar; streams; canopy height; roughness; vegetation density; change detection

#### **1. Introduction**

A riverscape is a spatially heterogeneous landscape representing a complex ecosystem that comprises the stream beds, banks, channels, riparian zones, floodplains, and basins spanning the many interconnected reaches of a river system [1–3]. Fausch et al. [1] proposed a "continuous view" of this riverscape environment is necessary at multiple spatial and temporal scales to properly study ecological processes. This "continuous view" is possible through state-of-the-art remote-sensing technologies, such as digital imagery and laser scanning, which can measure the physical properties of the riverscape at both fine and coarse scales [2]. Carbonneau et al. [2] combined a 0.03 m aerial image anda5m digital elevation model (DEM) over a 16 km stream reach to extract physical riverscape measures over multiple scales, such as channel width and depth, particle size, and slope, which were used to estimate the spatial distribution of salmon habitat patches. Dietrich et al. [3] used helicopter-based imagery and structure-from-motion (SfM) to derive a 0.1 m DEM for a 32 km stream reach, calculated metrics such as channel width, slope, and sinuosity along 3 m cross-sections, and explored their relationships with various land class and geomorphic variables. These studies demonstrated the need for multi-scale analyses for ecological studies involving riverscape environments and the benefit of applying remotely-sensed data over not just a large extent but also at a high resolution.

**Citation:** Resop, J.P.; Lehmann, L.; Hession, W.C. Quantifying the Spatial Variability of Annual and Seasonal Changes in Riverscape Vegetation Using Drone Laser Scanning. *Drones* **2021**, *5*, 91. https://doi.org/ 10.3390/drones5030091

Academic Editors: Diego González-Aguilera and Pablo Rodríguez-Gonzálvez

Received: 27 July 2021 Accepted: 5 September 2021 Published: 7 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Remote sensing technologies, such as imagery and lidar, rely on a range of platforms including satellite, aerial, terrestrial, mobile, and, more recently, drone systems for measuring physical riverscape properties. Farid et al. [4] classified riparian zone vegetation using 0.5 m resolution canopy elevation models and intensity rasters derived from aerial laser scanning (ALS) data. Heritage and Hetherington [5] scanned a 150 m reach with terrestrial laser scanning (TLS) to create a high-resolution (0.01 m) DEM. Resop et al. [6] applied TLS to a 100 m reach to classify cobble and boulders and calculate percent in-stream rock cover at 0.02 m resolution. Woodget et al. [7] used drone-based imagery and SfM to survey a 120 m reach and classify riverscape features, such as cobble, boulders, gravel, grass, and trees. Yang et al. [8] compared 0.25 m drone-based and 10 m satellite-based multispectral imagery to classify vegetation in a coastal area and found the higher-resolution drone data resulted in improved delineation of features, such as mangroves. Drone laser scanning (DLS), or unmanned aerial vehicle (UAV)-based lidar, is well suited in particular for measuring fine details in riverscapes due to its resolution and survey extent. Resop et al. [9] found that DLS (at 0.1 m resolution) was more accurate than ALS for measuring physical features in the riverscape and detecting micro-changes in the environment, such as streambank profiles and small herbaceous vegetation. While these studies demonstrated the ability of remote sensing for surveying and classifying riverscapes at multiple scales, they all take place over a single survey representing a single moment in time. More studies are needed that take advantage of repeat surveys for better measuring the temporal variability of riverscape features.

A number of studies have used lidar to detect physical changes in riverscapes. Huang et al. [10] combined 1 m ALS-derived inundation maps from 2007 and 2009 with 30-m Landsat images to monitor inundated area change. Anders et al. [11] estimated geomorphological changes based on 2 m ALS-derived digital terrain models (DTMs) from 2003 and 2011. While these studies demonstrate the benefits of lidar for annual change detection, many riverscape processes occur seasonally (in particular, those involving vegetation), which requires multiple surveys per year to properly monitor. The lack of widespread and continuous ALS data is a significant limitation for lidar-based change detection studies [12]. More often than not, locations only have a single ALS dataset at best, which means other remotely-sensed data, such as SfM or coarser-resolution satellite imagery, are required for change detection. In addition, ALS from high-altitude aircraft has limited resolution, typically around 0.5 or 1 m, which is not fine enough to detect small changes in vegetation [9].

A more flexible lidar platform than ALS, such as terrestrial, mobile, or drone, is required for more frequent surveys at higher resolution. Lidar platforms such as TLS and mobile laser scanning (MLS) have shown to be successful for measuring change over repeat surveys and have been used to measure streambank retreat, bluff erosion, and riparian vegetation change [13–18]; however, these studies typically scan a limited extent, such as a single streambank. For example, Resop and Hession [13] scanned an 11 m streambank six times over two years with TLS (at 0.02 m resolution) and observed seasonal variations in streambank retreat rates, demonstrating the advantage of repeat surveys. More research is needed to study the potential of DLS for detecting vegetation change along entire riverscapes at high resolution.

There are many metrics available to quantify riverscape vegetation, such as height, roughness, and density. These metrics have a history of being derived using lidar data, most commonly with ALS, for a range of ecological applications. Vegetation height, represented by a canopy height model (CHM), has been used to estimate tree height, above-ground biomass, bulk density, and canopy fuel weight over large extents [19,20]. Vegetation roughness, a measure of the level of smoothness or roughness [21], is related to a commonly used parameter in flood modeling, the hydraulic roughness, which is estimated over the channel and floodplain based on a combination of topographic and vegetative complexity [22]. Vegetation density, represented by metrics such as the laser penetration index (LPI), has been used to estimate percent canopy cover or leaf area index (LAI) [20,23]. These vegetation metrics could be quantified at high resolution using DLS and vegetation change could be measured seasonally with repeat surveys.

Many approaches have been used to estimate canopy height: (1) vector-based normalization of point clouds, (2) raster-based differences between digital surface models (DSMs) representing surface features and DTMs representing ground, and (3) full waveform lidar approaches. Vector-based methods normalize point cloud elevations to derive height above ground and are commonly used for tree detection [24–26]. Raster-based methods classify lidar points as vegetation or ground returns and rasterize them into DSMs representing maximum vegetation elevation and DTMs representing ground elevation with the difference resulting in a normalized digital surface model (nDSM) or CHM [9,20,27]. If full waveform lidar data is available, which is less common and generally has larger footprints, then canopy height can be estimated through decomposition of the waveform peaks [28,29]. Raster-based methods are most often used due to the flexibility of fixed-grid data formats for a variety of applications and because they are most practical for DLS due to the high resolution and discrete format of the point clouds.

Two approaches have been used to estimate vegetative roughness: (1) calculating roughness based on the variability of lidar point elevation within a moving window and (2) classifying lidar-derived rasters based on observed or calibrated roughness values. Mundt et al. [21] estimated roughness as a way to classify sagebrush distribution using the standard deviation of normalized ALS point heights within 4.6 m raster pixels. Dorn et al. [27] applied a supervised classification to ALS-derived CHMs to classify vegetation (i.e., grass, shrub, forest) corresponding to Manning's roughness (*n*) values from Chow [30]. Both methods have been effective at estimating roughness, but have only recently been explored with DLS. Prior et al. [31] estimated Manning's *n* from both DLSand SfM-derived CHMs through empirical methods as well as by classifying vegetation (i.e., grass, scrub, small trees, and large trees) and calibrating roughness with observed stream velocity data and 2D HEC-RAS.

Multiple metrics have been used to measure vegetation density, ranging from canopy cover, crown closure, gap fraction, or LAI. Lidar metrics estimating density take advantage of the physics of laser pulses passing through gaps in the canopy and include both vector-based and raster-based approaches. Vector-based methods classify lidar points as vegetation or ground and then calculate the LPI, defined as the ratio of ground points to total points in a given area [23,32]. Alternatively, raster-based methods derive a CHM at high resolution to define locations of canopy cover and then calculate the percent of canopy pixels within a coarser resolution grid or sample area [20,26]. Both approaches have a similar conceptual background and result in a raster representing the percent vegetation density or cover but have not yet been applied to DLS data.

The objectives of this study were: (1) to scan a reach of Stroubles Creek six times over two years (between April 2017 and March 2019) at high resolution with DLS; (2) to produce three lidar-based metrics of riverscape vegetation (i.e., height, roughness, and density); (3) to classify distinct vegetation classes in the riverscape (i.e., scrub and tree) and locate them with respect to the stream channel; and (4) to calculate and compare annual and seasonal changes in vegetation metrics spatially over the riverscape.

#### **2. Materials and Methods**

#### *2.1. Study Area*

The study area was a 0.65 km reach of Stroubles Creek located in Blacksburg, VA, USA (Figure 1). This reach has a history of agricultural use and underwent a stream restoration that was completed in May 2010 [33]. The restoration consisted of best management practices such as livestock exclusion, bank reshaping, and riparian vegetation planting [33]. The Stream Research, Education, and Management (StREAM) Lab at Virginia Tech continuously monitor water quality, stream flow, and weather at sampling bridges located along the reach as part of ongoing research efforts [34]. Resop and Hession [13] monitored streambank retreat on an 11 m bank of this reach using TLS between 2007 and 2009 and found

that TLS was more accurate than traditional surveying methods for measuring topographic change; however, the stationary nature of TLS limited the study to a single streambank and was not ideal for larger extents. Resop et al. [9] scanned multiple streambanks and the floodplain using DLS in 2017, demonstrating the potential of DLS to perform change detection over the entire riverscape. Prior et al. [31] used DLS and SfM data for this reach from 2018 to estimate hydraulic roughness based on vegetation height and velocity data.

**Figure 1.** The study area, a 0.65 km reach of Stroubles Creek, showing the approximate extent of the drone laser scanning (DLS) surveys. The actual extent varied slightly between scans.

#### *2.2. Lidar Data Collection*

The drone used for this study was an AeroVironment Vapor35 (Arlington, VA, USA). The lidar system on board was a YellowScan Surveyor Core (Saint-Clément-de-Rivière France). The drone weighed approximately 13.6 kg with a payload (i.e., the lidar system) of about 2.3 kg. The battery provides a practical maximum flight time of 40 min at an altitude

of 20 m above ground level (AGL). The lidar operated with a pulse rate of 300 kHz, used a near-infrared (NIR) wavelength of 905 nm and recorded up to two returns per pulse. Additional technical details for the drone and lidar systems can be found in Resop et al. [9], which used the same system to compare DLS and ALS for the same study area.

The reach was scanned six times over two years: 5 April 2017, 2 August 2017, 9 November 2017, 3 April 2018, 9 October 2018, and 20 March 2019. Three scans occurred during the leaf-off dormant vegetation season (April 2017, April 2018, and March 2019) and three scans occurred during the leaf-on season at varying stages of foliage (August 2017, November 2017, and October 2018; Figure 2). The extent was mostly similar for all six scans (Figure 1). Each flight consisted of six flightlines and accounted for overlap between swaths. All scans flew at a consistent altitude (about 20 m AGL), launch location, and speed to produce similar point densities, with the exception of the March 2019 scan, which tested a higher pulse density.

**Figure 2.** Photos of the study area, Stroubles Creek, taken downstream of the concrete bridge from an on-site tower camera around the same dates as the drone scans: (**a**) April 2017, (**b**) August 2017, (**c**) January 2018 (Note: Due to a malfunction in the tower camera, this is the closest photo to the November 2017 scan), (**d**) April 2018, (**e**) October 2018, (**f**) March 2019.

#### *2.3. Lidar Data Preprocessing*

The lidar data were georeferenced with the on-board GPS and local National Geodetic Survey CORS base station data [9]. For each scan, the DLS data was processed into six LAS files, one for each swath. The LAS files for each scan were added to a LAS dataset, projected to WGS 1984 UTM Zone 17N, and further classified and rasterized with ArcGIS 10.6 (Redlands, CA, USA) and LAStools (Gilching, Germany). Data analysis and visualization were performed with Python 3.7.7 and various open-source modules (e.g., numpy, pandas, matplotlib, sklearn).

The study area contained three prominent structures that served as control points to compare the DLS point clouds between scans: one concrete bridge used for vehicles at the north end of the reach and two wooden sampling bridges in the middle and south end of the reach (Figure 1). For each scan, lidar points representing bridges were manually classified as building with ArcGIS. Points representing other anthropogenic features (e.g., cars and people conducting the flights) were also classified as building so they would be ignored in further analysis.

During point cloud post-processing and georeferencing, an issue of concern occurred some LAS files were not properly aligned with respect to the others. Two misalignment

phenomena were observed: (1) within each scan, one of the six LAS files might be misaligned with respect to the other five, and (2) between each of the six scans, there might be an elevation bias. This issue was easily observed at bridges, which made it appear as if there were multiple bridges at the same location (Figure 3).

**Figure 3.** An example misaligned point cloud representing one of the wooden sampling bridges that cross the stream showing an elevation bias in the November 2017 scan (i.e., the green points) relative to the other scans before correction with CloudCompare.

Misaligned LAS files were corrected with the open-source software CloudCompare 2.10 (https://www.cloudcompare.org/ accessed on 2 July 2021). A two-stage process was used: (1) within each scan to correct misaligned LAS files and (2) between each scan and the April 2017 baseline scan to correct bias. The three bridges were used as control points during alignment. Points were finely registered with the "Iterative Closest Point (ICP)" tool to a root mean squared difference (RMSD) of 0.00001 m and a transformation matrix was calculated. The transformation matrix was then applied to the entire point cloud. After correcting all six scans, the elevation bias was calculated between all scans and the April 2017 baseline scan as a measure of relative accuracy.

The extent of each lidar scan was delineated with ArcGIS [9]. The "LAS Point Statistics as Raster" tool created a 0.1-m raster representing all pixels with at least one point. The "Expand" and "Shrink" tools performed a morphological closing and removed small data gaps. The final raster was converted to a polygon representing the scan extent. Key differences in extents include the November 2017 scan, which had a smaller extent south of the concrete bridge, and the March 2019 scan, which had a larger extent expanding out into the floodplain (Figure 4). The extent intersection was calculated with the "Intersect" tool to normalize comparisons between scans. The "annual extent intersection" (between April 2017, April 2018, and March 2019) was 11.02 ha and the "seasonal extent intersection (between all scans) was 8.29 ha (Figure 4).

**Figure 4.** The extent of each drone laser scanning (DLS) survey as well as the extent intersection of all six scans ("Seasonal Extent") and of only the annual leaf-off season scans (April 2017, April 2018, and March 2019; "Annual Extent").

#### *2.4. Lidar Data Classification*

With respect to the lidar data collected for this study, noise was rare and most likely represented data artifacts, such as "bird hits." The "Classify LAS Noise" tool in ArcGIS identified outlier points based on absolute minimum and maximum elevation thresholds. Afterwards, the point clouds from each scan were inspected with the "Profile View" tool and any remaining outliers were manually classified as noise.

After manually classifying building and noise, automated classification tools in ArcGIS were used to classify ground and vegetation points [9]. The "Classify LAS Ground" tool identified ground and "Classify LAS by Height" classified the remaining points as unassigned or vegetation based on a height threshold of 0.1 m (Table 1). Points with a height greater than 0.1 m were classified as vegetation and points less than 0.1 m were classified as unassigned. The unassigned class served as a measure of uncertainty, as these points were within the precision of DLS [35] and could represent ground or vegetation.


**Table 1.** Lidar point data classes used in this study.

Researchers have observed that point clouds from a YellowScan lidar, which is based on the Velodyne Puck (San Jose, CA), can result in a "thickness" of points of a few centimeters when scanning flat ground, producing surfaces that are not as "crisp" as they should [36]. This phenomenon was observed in the point clouds produced for this study. Upon investigation, a cross-section of points along the concrete bridge at the northern end of the study area showed an approximate 0.05 m thickness of what should have been a solid surface. To account for this thickness, the unassigned class, representing points with a height above ground less than 0.1 m, was used in addition to the ground class to define terrain for bare earth models (i.e., DTMs).

The automated classification algorithms produced two common misclassifications: (1) ground points misclassified as vegetation on high-gradient stream banks and (2) vegetation points misclassified as ground under dense canopy [9]. These misclassifications need to be corrected manually due to limitations in current lidar point classification algorithms. Unfortunately, the manual correction of large point clouds, especially from DLS, is very time-consuming and labor-intensive [9]. The April 2017 scan was corrected previously [9], but the other five scans were not fully verified. An investigation into a more efficient classification correction process was outside the scope of this study.

#### *2.5. Lidar Vegetation Metrics*

Three lidar metrics were selected to represent riverscape vegetation: (1) height (canopy height model; CHM), (2) roughness (vegetative roughness index; VRI), and (3) density (lidar vegetation index; LVI; Table 2). All three metrics were calculated in raster format for each DLS survey using data processing pipelines created with Model Builder in ArcGIS. All outputs had a pixel size of 0.1 m. Based on an average point spacing of 0.047 m over the six scans, this is about five lidar points per pixel.


**Table 2.** Metrics produced from drone laser scanning (DLS) data to represent different components of riverscape vegetation.

The CHM was derived from two rasters: the DTM (i.e., the minimum ground and unassigned point elevation per pixel) and the DSM (i.e., the maximum vegetation point elevation per pixel) [9]. The DTM and DSM were calculated with the "LAS Dataset to Raster" tool (Figure 5). The difference between DSM and DTM resulted in a normalized digital surface model (nDSM) [9]. Data gaps (e.g., water surfaces and lidar shadows) and pixels with a height less than 0.1 m (i.e., the smallest height of vegetation points), were assigned zero values to produce the final CHM. The CHM ranged from 0 m (i.e., bare earth) to about 13 m (i.e., the height of the tallest tree).

**Figure 5.** The workflow deriving the canopy height model (CHM) based on the maximum height above ground of lidar vegetation points in each raster pixel.

The vegetative roughness index (VRI) estimated the local variability of vegetation height. The VRI was calculated with the "LAS Height Metrics" tool as the standard deviation of vegetation point height per pixel (Figure 6). Points with a height less than 0.1 m were ignored. Low standard deviations corresponded to smooth surfaces while high standard deviations corresponded to rough surfaces. Data gaps were assigned a VRI of zero. The VRI ranged from about 0 m, representing relatively uniform vegetation, to 9 m, representing more complex vegetation.

**Figure 6.** The workflow estimating the vegetative roughness index (VRI) based on the standard deviation of vegetation height within each raster pixel.

The lidar vegetation index (LVI) represented the percentage of vegetation points per pixel. While CHM and VRI were both absolute measures, the LVI was the only relative measure. The number of vegetation points per pixel was calculated with the "LAS Point Statistics as Raster" tool and then the number of total points per pixel was calculated (Figure 7). The vegetation point count was divided by the total point count on a pixel-bypixel basis, resulting in a decimal percentage. Data gaps were assigned an LVI of zero. The LVI ranged from zero, representing no vegetation or open canopy, to one, representing heavy vegetation or dense canopy.

**Figure 7.** The workflow calculating the lidar vegetation index (LVI) based on the percentage of vegetation points within each raster pixel as a measure of density.

#### *2.6. Vegetation Classification, Distance to Water, and Tree Identification*

The CHM for each scan was classified into the following land classes: ground, scrub, and tree. These classes were defined with a simple threshold method based on a manual investigation of vegetation in the riverscape. Pixels with a CHM value (i.e., height above ground) less than 0.1 m were classified as ground, pixels between 0.1 and 2 m were classified as scrub, and pixels greater than 2 m were classified as tree.

The CHM for the April 2017 baseline was used to establish the stream location. A land class for water was defined by taking advantage of the physics of NIR lidar pulses, which are absorbed by water surfaces and result in "No Data" pixels. The "LAS Point Statistics as Raster" tool counted the lidar points in each 0.1 m pixel and those with more than three points represented ground. The "Expand" and "Shrink" tools were used to close small data gaps. At this point, "No Data" gaps represented the water surface. Once the water class was defined, the "Euclidean Distance" tool was used to determine the distance between each pixel in the riverscape and the stream.

To quantify the annual growth of vegetation, trees were manually identified from the April 2017 CHM based on pixels classified with a height greater than 2 m. The riverscape was inspected, crowns were identified, and stems were marked. The environment for this study had a fairly open canopy, so it was easy to manually identify trees. Each identified tree was verified by inspecting the point cloud. The "Buffer" tool created 0.5 m buffers around each tree stem and the maximum tree height was calculated from the CHMs for April 2017, April 2018, and March 2019.

#### *2.7. Annual and Seasonal Change Detection*

Annual and seasonal changes were calculated for each vegetation metric (Table 2). The overall two-year period was represented by the April 2017 to March 2019 scans. Annual change was measured between leaf-off scans (April 2017 to April 2018 and April 2018 to March 2019) within the annual extent intersection (Figure 4). Seasonal change was measured between all six DLS scans within the seasonal extent intersection (Figure 4).

An effective method of identifying pixel-level change is to calculate DEMs of difference (DoDs) by subtracting rasters representing two moments in time: the newer raster minus the older raster [12,37,38]. For CHMs, change represented vegetation growth or decay [15]. For VRIs, change represented increased roughness or smoothness. For LVIs, change represented increased or decreased vegetation density. Change was determined to be significant, as opposed to noise, by applying a minimum level of detection (LoD), representing the uncertainty of the lidar sensor [37,38]. An LoD of 0.05 m was used for the CHM and VRI DoDs [35], which is consistent with LoDs used by change detection studies involving lidar at a similar range [17]. For the LVI DoD, the unassigned class (Table 1) represented the point classification uncertainty.

#### **3. Results**

#### *3.1. Point Density Comparison and Elevation Bias between the Six Lidar Scans*

The point count varied significantly between the scans due to extent differences (Figure 4), ranging from 42.6 to 70.1 million points. When considering only the seasonal extent intersection the point counts were more consistent (Table 3). Most scans contained between 41.7 and 43.0 million points with the exception of the March 2019 scan, which contained 51.2 million points due to a higher pulse density. While most scans were consistent (491 to 502 pulses/m2), the March 2019 scan had a pulse density of 590 pulses/m2.

**Table 3.** Drone laser scanning (DLS) statistics for each scan within the extent intersection.


Agreement between the six DLS scans was determined based on the average elevation (Z) bias between DEMs produced from each scan and the April 2017 baseline. Bridge DEMs were created using points representing the concrete bridge and two sampling bridges. Before aligning the data in CloudCompare, the average Z bias was as large as 1.15 m (March 2019) with smaller biases of 0.23 m (November 2017) and −0.11 m (October 2018; Figure 3). After alignment, the average Z biases ranged from−0.03 m to 0.01 m over all six scans (Table 4), which is well within the precision of YellowScan lidar systems [35].

**Table 4.** The average elevation (Z) bias between each drone laser scanning (DLS) survey and the April 2017 scan after all point clouds were aligned with CloudCompare.


Between DTMs, the average Z biases had a wider range, −0.03 m to 0.25 m (Table 4). The scans with a higher bias included August 2017 (0.25 m), October 2018 (0.17 m), and November 2017 (0.06 m), which occurred during leaf-on seasons. The Z bias was negatively correlated to the time of year. Going forward in time from August to October to November, as the amount of vegetation decreased there was a lower likelihood of vegetation understory misclassified as ground, which decreased the bias. For the other leaf-off scans (April 2018 and March 2019), the bias was within the precision of the lidar system (−0.01 m and −0.03 m, respectively) [35].

#### *3.2. Annual and Seasonal Change of Lidar Point Classifications*

The three primary data classes were ground, unassigned, and vegetation (Table 1). Vegetation represented objects with a height above ground greater than 0.1 m that were not identified as a built structure, which allowed DLS to detect not just trees and bushes but also small herbaceous vegetation [9]. The scans were clipped to the seasonal extent intersection and the point statistics (percent terrain vs. percent vegetation) were compared. As expected, the percent vegetation varied periodically throughout the seasons as associated with the growth and decay of vegetation in the riverscape, with a maximum of 56% in August 2017 (leaf-on) and a minimum of 12% in April 2017 (leaf-off; Figure 8). During the leaf-on scans, there was a clear negative trend in the percent vegetation going forward in time from August (56%) to October (45%) to November (38%), likely due to the decay of leaves over the autumn season.

**Figure 8.** Drone laser scanning (DLS) point classifications within the extent intersection over all six scans showing the seasonal variation between terrain points and vegetation points.

There was a gradual increase in percent vegetation over time between the leaf-off scans going forward in time from April 2017 (12%) to April 2018 (15%) to March 2019 (23%; Figure 8). This positive trend was likely reflecting the annual growth of vegetation in the riverscape. The higher pulse density during the March 2019 scan (Table 3) may have contributed to the higher percent vegetation observed for this scan, but the level of influence is not clear.

#### *3.3. Annual Change of Maximum Tree Height*

A total of 604 trees were identified in the April 2017 CHM based on a manual inspection of areas classified as tree (CHM > 2 m; Figure 9). The average tree growth over the two-year period was 1.48 m. A majority of trees had positive growth (*n* = 570; mean = 1.62 m) while a few had negative growth (*n* = 34; mean = −0.96 m). Most trees with negative growth fell due to natural causes over the two-year period. Tree growth was greatest from 2018 to 2019 (mean = 0.83 m) compared to the period 2017 to 2018 (mean = 0.65 m). Based on the tree's distance to stream (i.e., pixels classified as water), trees within 20 m grew faster (*n* = 544; mean = 1.56 m) over the two-year period compared to trees farther than 20 m (*n* = 60; mean = 0.70 m).

**Figure 9.** Trees taller than 2 m were identified from the April 2017 canopy height model (CHM) and maximum tree height was measured for: (**a**) April 2017; (**b**) April 2018; (**c**) March 2019.

Not only did the trees show steady average growth (April 2017 = 4.37 m, April 2018 = 5.02 m, March 2019 = 5.84 m), but the variability or standard deviation of height increased over time as well (April 2017 = 1.28 m, April 2018 = 1.58 m, March 2019 = 1.90 m). This positive trend in standard deviation demonstrates not just an increase in tree height over time but also tree height diversity and variability (Figure 10). Overall, the annual scans showed a consistent, steady growth of woody vegetation over the riverscape.

**Figure 10.** Maximum tree height statistics of the 604 trees identified in the riverscape for each of the three leaf-off scans. The average and standard deviation of maximum tree height increased over the two-year period.

#### *3.4. Baseline Lidar Vegetation Metrics*

The vegetation metrics (CHM, VRI, and LVI) were derived for the April 2017 scan as a baseline (Figure 11). The CHM ranged from 0 m (bare ground) to 12.97 m (the height of the tallest tree). The VRI ranged from 0 m (smooth surfaces) to 8.41 m (rough surfaces). The LVI ranged from 0 (open terrain) to 1 (dense vegetation). The April 2017 scan was classified as: ground (CHM < 0.1 m), scrub (0.1 m < CHM < 2 m) and tree (CHM > 2 m; Figure 9). As expected, pixels classified as ground had a value close to zero for all three metrics. For pixels classified as vegetation, tree areas consistently had higher vegetation metrics than scrub areas. The biggest difference was between average height (CHM; scrub = 0.48 m and tree = 3.52 m), which is to be expected since the classes were determined based on height. The differences between roughness (VRI; scrub = 0.08 m and tree = 0.67 m) and density (LVI; scrub = 0.45 and tree = 0.61) were much smaller.

**Figure 11.** Lidar vegetation metrics for April 2017 showing: (**a**) height (CHM); (**b**) roughness (VRI); (**c**) density (LVI).

Very weak negative correlations were observed between CHM (R2 = 0.12) and VRI (R<sup>2</sup> = 0.07) with respect to the distance to the stream channel, but overall the taller and rougher vegetation tended to be located closer to the stream (Figure 12). There was no correlation between LVI (R<sup>2</sup> = 0.01) and distance to stream; however, it is important to note that within the stream channel itself the LVI ~ 1 due to the fact that the water surface absorbed the lidar pulses resulting in a lack of ground points measured by the lidar system (Figure 11).

#### *3.5. Annual Change of Lidar Vegetation Metrics*

The vegetation metrics for the annual leaf-off scans (April 2017, April 2018, and March 2019) were overlaid with each vegetation class (scrub and tree) and clipped to the annual extent intersection (total area = 11.02 ha). Both scrub and tree areas increased steadily over the two-year period (Figure 13). Scrub, as a percent of the total area, increased from 15.28% for April 2017 to 22.02% for April 2018 to 36.04% for March 2019 while tree area increased from 2.83% to 4.02% to 5.31%. In tree areas, all three metrics increased over the two-year period (Figure 13). An increase in height (CHM) was driven by annual tree growth (Figure 10). However, increases in roughness (VRI) and density (LVI) were likely also influenced by the growth of understory. On the other hand, scrub areas did not show any significant annual trends with respect to any metric (Figure 13). Annual variations in scrub vegetation were likely impacted by the growth of new scrub and decay of old scrub, while trees had more consistent growth without much change in the overall tree population.

**Figure 12.** The relationship between vegetation metrics and distance to stream for April 2017: (**a**) height (CHM); (**b**) roughness (VRI); (**c**) density (LVI).

**Figure 13.** Trends in annual vegetation change over the riverscape: (**a**) total land class area; (**b**) average height (CHM); (**c**) average roughness (VRI); (**d**) average density (LVI).

There was a clear increase in total scrub and tree area over each year (Figure 13). By looking at the 0.1 m pixel-level change, one can investigate how land class changed from one year to the next. From April 2017 to April 2018, scrub area increased from 1.68 ha to 2.43 ha; however, a majority (75%) of the April 2018 scrub area was previously ground (Figure 14) and a majority (62%) of the April 2017 scrub area became ground in April 2018. This same trend was observed between April 2018 and March 2019 when the scrub area increased from 2.43 ha to 3.97 ha and again a majority (69%) was previously ground (Figure 14) and a near majority (48%) of the previous scrub became ground. This demonstrates the volatility of scrub areas and could have dramatic effects on physical riverscape properties such as hydraulic roughness from year to year. In contrast, tree areas showed more consistency. From April 2017 to April 2018 to March 2019 the tree area increased from 0.31 ha to 0.44 ha to 0.59 m. Unlike scrub area, a majority of the tree area in both 2018 and 2019 was previously classified as tree, 54% and 65%, respectively (Figure 14).

**Figure 14.** Change matrices representing the pixel-level change in land class between: (**a**) April 2017 and April 2018; (**b**) April 2018 and March 2019.

Looking at the DoDs, one can observe the pixel-level growth and decay of vegetation spatially over the riverscape and temporally over the two-year study (Figure 15). The DoDs for height (CHM) and roughness (VRI) showed similar trends; although, the CHM had a greater overall magnitude of change compared to the VRI. Most of the positive change in vegetation occurred near the stream channel and was primarily due to the growth of trees. Farther into the floodplain, there was more spatial variability in the growth and decay of scrub. From year to year, some sections of scrub showed positive change and some showed negative change. Overall, tree areas showed greater growth in CHM and VRI over the two-year period compared to scrub areas (Figure 16). On the other hand, vegetation density (LVI) consistently had positive change over the entire riverscape (Figure 15) and the change was similar between scrub and trees (Figure 16). This consistent growth in vegetation density suggests that while scrub and trees are growing at different rates, the canopy density and vegetation understory is increasing more uniformly.

The vegetation metrics did not show much correlation with distance to stream. However, when selecting only tree areas, some annual trends were observed. There was a negative correlation for tree height and roughness with distance to stream (Figure 17). The negative correlation increased in magnitude over time from April 2017 to April 2018 to March 2019 (Figure 17). Generally, trees grew faster within 20 m of the stream compared to trees farther than 20 m. The average CHM growth from 2017 to 2019 was 1.03 m within 20 m and 0.07 m farther than 20 m. The average increase in VRI over the same period was 0.40 m within 20 m and 0.12 m farther than 20 m. While tree height and roughness showed a negative correlation with distance to stream, vegetation density showed a positive correlation over the two-year period (Figure 17). This positive correlation in tree areas decreased over time from 2017 to 2019 as the average LVI increased by +0.05 within 20 m but decreased by −0.09 farther than 20 m. Overall, the annual trend for all three metrics showed a greater increase in vegetation closer to the stream. While these annual trends were observed in tree areas, there were no significant correlations or trends observed in scrub areas. This is likely a result of the volatility of the scrub area over the riverscape that was previously described.

#### *3.6. Seasonal Change of Lidar Vegetation Metrics*

All six scans were clipped to the seasonal extent intersection (total area = 8.29 ha). Based on the change in vegetation area, two periods were associated with growth (April 2017

to August 2017 and April 2018 to October 2018) and three periods were associated with decay (August 2017 to November 2017, November 2017 to April 2018, and October 2018 to March 2019; Figure 18). These trends were similar to what was previously observed when looking at the classified lidar point clouds (Figure 8). Much like the observations of annual change for individual trees and the pixel-level change of the vegetation metrics, tree areas gradually increased in height (CHM) and roughness (VRI) over time (Figure 18). On the other hand, scrub areas were more constant for both metrics (Figure 18). Density (LVI) was highly variable for both scrub and trees (Figure 18).

**Figure 15.** DEMs of Difference (DoDs) over the two-year study from April 2017 to March 2019 representing pixel-level change in: (**a**) height (CHM); (**b**) roughness (VRI); (**c**) density (LVI).

**Figure 16.** The average pixel-level annual change in each metric, (**a**) height (CHM), (**b**) roughness (VRI), and (**c**) density (LVI), over each land class and over each year in the two-year study.

**Figure 17.** Linear regression models relating vegetation metrics and distance to stream annually over time for tree areas: (**a**) height (CHM); (**b**) roughness (VRI); (**c**) density (LVI).

**Figure 18.** Trends in seasonal vegetation change over the riverscape: (**a**) total land class area; (**b**) average height (CHM); (**c**) average roughness (VRI); (**d**) average density (LVI).

Temporal linear regression was used to remove the annual trend of each metric for scrub and tree areas and isolate the seasonal variability, which was represented by the normalized root mean squared error (NRMSE) of the regression residuals (Table 5). Out of all the combinations of lidar metric and land class, only height and roughness in tree areas had significant annual trends (both had R<sup>2</sup> = 0.70). Both scrub and tree areas demonstrated seasonal variability for height and roughness; however, the variability was greater for scrub (CHM NRMSE = 17.8%; VRI NMRSE = 25.6%) compared to trees (CHM NRMSE = 5.1%; VRI NMRSE = 9.2%). Vegetation density did not have any significant annual trends. However, it was highly seasonally variable for both scrub areas (LVI NRMSE = 23.1%) and tree areas (LVI NRMSE = 17.8%).

No temporal trends could be observed for the average vegetation density (LVI) when viewing the data sequentially (Figure 18). However, by reordering the scans as they would fall in a single growing season, a trend emerges representing vegetation decay (Figure 19). The scans were reordered August (2017) to October (2018) to November (2017) to April (2018). This time shift does not work well for height and roughness since these are absolute measures and are affected by annual vegetation growth. On the other hand, density is a relative measure based on the local percentage of vegetation points. By reordering the scans seasonally, one can observe a negative trend in LVI over both scrub and tree areas (Figure 19). This trend likely represents the seasonal decay of vegetation in

the riverscape as foliage falls and DLS is able to penetrate further through the canopy to record ground points.

**Table 5.** Regression results showing the annual trend (based on R2) and seasonal variability (based on normalized root mean squared error [NRMSE]) for each metric and land class.


**Figure 19.** Trends in seasonal vegetation decay over the riverscape going from August (2017) to October (2018) to November (2017) to April (2018): (**a**) total land class area; (**b**) average height (CHM); (**c**) average roughness (VRI); (**d**) average density (LVI).

#### **4. Discussion**

#### *Limitations with the Current Study and Future Research*

One of the limitations of raster-based analysis is that only a single Z value is allowed per pixel. When quantifying height or classifying different types of vegetation this limitation does not allow for an accurate representation of complex environments; in particular, when multiple vegetation types exist at the same location. For example, when grass or scrub grows under a tree canopy. With traditional image-based remote sensing, these distinctions are often impossible to make. However, since lidar can penetrate through the tree canopy, it is possible to derive lidar measures that represent this vegetation diversity. Many applications take advantage of this aspect of lidar data by generating metrics such as average height, standard deviation, point density, and height percentiles to represent forest structure [20]. The lidar metrics generated in this study, representing height, roughness, and density, allowed for different perspectives of vegetation structure. These metrics could allow for more complex vegetation classes, such as combinations of grass, scrub, and trees. This could be accomplished using machine learning algorithms [20], but would require additional field data for model training.

Another area of uncertainty is the impact of pixel size or resolution on the calculation of the vegetation metrics. It is expected that two of the metrics, VRI (which deals with roughness, defined by the standard deviation of height) and LVI (which deals with density, defined by the percentage of vegetation points) are both heavily influenced by the pixel size, as these metrics are dependent on the number of lidar points in each pixel. As the size of the pixel increases, more lidar points are included, which results in a more generalized rather than localized value for each pixel. A sensitivity analysis is needed to determine the effect of pixel size on each of these metrics, but this is an area of future study.

In this study, we quantified and classified riverscape vegetation at 0.1-m resolution using an active system (i.e., DLS). Other studies have classified vegetation using passive systems, such as drone-based imagery [7,8,31]. Woodget et al. [7] used drone-based imagery and SfM to create 0.02-m DEMs and classified images with a mean elevation error of 0.05 m, similar to the elevation errors observed in our study. Yang et al. [8] used multispectral drone-based imagery to create 0.25 m classified images. Prior et al. [31] created 0.1 m classified images from DLS and SfM, but the datasets were collected in different years. Both remote sensing technologies have advantages and disadvantages. Lidar is better suited for producing 3D point clouds and penetrating vegetation canopy. Imagery is better suited for collecting a range of spectral information. There is a long history of data fusion applications that have combined active and passive remote sensing datasets to take advantage of their respective strengths [21]. The results of this study could theoretically be improved by combining the DLS data with SfM data. However, such data fusion would require collecting both lidar and imagery data at each survey, presenting additional field management challenges, and was outside the scope of this study.

#### **5. Conclusions**

Comparing the annual and seasonal change of all three lidar vegetation metrics (height [CHM], roughness [VRI], and density [LVI]) over both vegetation classes (scrub and tree), the following trends and patterns emerge over the two-year period:


Based on these results, it is clear that all three metrics are representing different aspects of the riverscape vegetation as it changes both annually and seasonally. It is recommended that for any fluvial application able to utilize high-resolution survey data such as from DLS, whether it is for ecological modeling or hydraulic modeling, to consider integrating all three metrics as possible explanatory variables to describe the dynamic physical changes occurring to vegetation over the riverscape.

**Author Contributions:** Conceptualization, J.P.R.; data curation, L.L.; formal analysis, J.P.R.; funding acquisition, W.C.H.; methodology, J.P.R. and L.L.; supervision, W.C.H.; writing—original draft, J.P.R.; writing—review and editing, W.C.H. and L.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported by an Instrumentation Discovery Travel Grant (IDTG) from the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI), sponsored by the National Science Foundation (NSF). This work was also supported by the Virginia Agricultural Experiment Station (Blacksburg, VA, USA) and the U.S. Department of Agriculture (USDA) National Institute of Food and Agriculture (Washington, DC, USA).

**Acknowledgments:** Thanks to everyone with the Virginia Tech StREAM Lab (vtstreamlab.weebly.com/ accessed on 6 July 2021), including Charles Aquilina for performing field measurements and Alexa Reed for extracting photos from the on-site tower camera.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **UAV-Based Classification of Cercospora Leaf Spot Using RGB Images**

**Florian Görlich 1, Elias Marks 1,\*,†, Anne-Katrin Mahlein 2, Kathrin König 3, Philipp Lottes 1,4 and Cyrill Stachniss 1,4**


**Abstract:** Plant diseases can impact crop yield. Thus, the detection of plant diseases using sensors that can be mounted on aerial vehicles is in the interest of farmers to support decision-making in integrated pest management and to breeders for selecting tolerant or resistant genotypes. This paper investigated the detection of Cercospora leaf spot (CLS), caused by *Cercospora beticola* in sugar beet using RGB imagery. We proposed an approach to tackle the CLS detection problem using fully convolutional neural networks, which operate directly on RGB images captured by a UAV. This efficient approach does not require complex multi- or hyper-spectral sensors, but provides reliable results and high sensitivity. We provided a detection pipeline for pixel-wise semantic segmentation of CLS symptoms, healthy vegetation, and background so that our approach can automatically quantify the grade of infestation. We thoroughly evaluated our system using multiple UAV datasets recorded from different sugar beet trial fields. The dataset consisted of a training and a test dataset and originated from different fields. We used it to evaluate our approach under realistic conditions and analyzed its generalization capabilities to unseen environments. The obtained results correlated to visual estimation by human experts significantly. The presented study underlined the potential of high-resolution RGB imaging and convolutional neural networks for plant disease detection under field conditions. The demonstrated procedure is particularly interesting for applications under practical conditions, as no complex and cost-intensive measuring system is required.

**Keywords:** UAV-based monitoring; agricultural robotics; plant disease detection; RGB images; Cercospora leaf spot; CNNs; semantic segmentation; phenotyping

### **1. Introduction**

Our society relies on sustainable crop production for obtaining food, feed, and other resources [1]. Plant diseases can have strong negative effects on the achievable yield, heavily influencing the efficiency of farmlands. To address this problem in the short term, farmers need to detect the occurrence and spread of diseases in time to take adequate countermeasures. This is known as the concept of integrated pest management. To tackle the problem in the long run, plant breeders aim at developing new, more tolerant, or resistant varieties concerning yield-affecting plant diseases. Besides preventing the adverse effects on crop yield, cultivating these innovative crop varieties allows reducing the application of chemical plant protection products, resulting in benefits for the environment. Plant breeding considers the performance of varieties in diverse environments [2]. Therefore, the selection of preferable genotypes is time and cost intensive, often referred to as the so-called phenotyping bottleneck [3].

**Citation:** Görlich, F.; Marks, E.; Mahlein, A.-K.; König, K.; Lottes, P.; Stachniss, C. UAV-Based Classification of Cercospora Leaf Spot Using RGB Images. *Drones* **2021**, *5*, 34. https://doi.org/10.3390/ drones5020034

Academic Editor: Diego González-Aguilera

Received: 22 March 2021 Accepted: 29 April 2021 Published: 5 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

UAVs have been applied to many fields in recent years, and examples of these are cinematography [4], wind field estimation [5], and remote cameras [6]. Furthermore, UAVs are attractive platforms for monitoring agricultural fields and breeding plots as they allow for flexible image-based monitoring and enable the in-depth analysis of such image data retrospectively [7,8]. Therefore, UAVs represent a valuable tool for farmers and breeders to minimize the effort needed to detect and quantify diseases in crop fields.

An exemplary disease impacting yield in sugar beet is Cercospora leaf spot, caused by the fungal pathogen *Cercospora beticola (Sacc.)*. Causing yield losses approaching 40%, CLS is the most important foliar disease in sugar beet production. The first symptoms caused by the perthotrophic pathogen *C. beticola* are leaf spots with a reddish-brown margin of typically 2 to 5 mm in diameter [9]; see the bottom left of Figure 1. Later, the disease spreads, and the spots merge into a more extensive symptom distributed across entire leaves, as visible in Figure 1, bottom-right.

**Figure 1.** We aimed at predicting the occurrence of Cercospora leaf spot (CLS) for the entire field trial. We first separated the RGB orthomosaic into small plots that corresponded to the breeder's plot structure for variety testing. These plots were divided into eight strips, which are numbered in red. For each plot, we performed a pixel-wise classification into the classes CLS (pink), healthy vegetation (green), and background (no color). Finally, we illustrate the plot-based infection level for the entire field in semantic maps, where a red plot refers to high infection and blue refers to no infection. We flew a DJI M600 equipped with a Phase One IMX 100 megapixel RGB camera.

Finding new sugar beet varieties with CLS tolerance or resistance poses multiple challenges to the breeder. It requires a time-intensive observation of the breeding plots—a task that offers a great potential for UAV-based monitoring support. For the breeder, it is essential to know when and where outbreaks of the disease occur in the trial sites consisting of hundreds or thousands of trial plots and how each individual genotype is affected by the plant disease regarding several epidemiological parameters. In many cases, the disease starts from a primary inoculum in the soil and afterward spreads radially, generating so-called CLS nests [10]. Therefore, it is important to automize the detection and quantification of CLS among a vast amount of tested genotypes to capture the gradual

change of CLS resistance, as well as the detection of nests in breeding trials. Furthermore, early detection and identification in an automated and easy-to-execute way are key for the successful adoption of disease control in agricultural practice.

We addressed the problem of CLS occurrence detection and quantification in a breeding trial by analyzing UAV imagery. UAVs equipped with RGB cameras serve as an excellent platform to obtain fast and spatially detailed information of field environments such as the breeding trials we analyzed. An example is illustrated in Figure 1. We focused on detection at the pixel level to estimate the amount of CLS and provided this information in concise semantic maps to breeders. We aimed at dealing with different environmental conditions and agronomic diversity regarding plant appearances, soil conditions, and light conditions during capturing. Furthermore, our goal was to provide high performance also in unseen conditions, which can occur in unknown field sites and trials. To correlate our method with the approaches currently used for plant disease monitoring, we compared the predictions of our approach to the infection scores estimated by plant experts as the ground truth.

The main scientific contribution of this work was a novel, vision-based classification approach that uses fully convolutional neural networks (FCNs) that operate on RGB images to determine the occurrence of CLS in real fields. We proposed a detection pipeline that performs the pixel-wise classification of the RGB images into the classes CLS, healthy vegetation, and background. The FCN, on which our method was based, uses an encoderdecoder structure for the semantic segmentation of the images. We trained it end-to-end: the input to the network were raw images of the plots, and the loss was computed directly on the semantic maps. It neither relies on a certain pre-segmentation of the vegetation, nor any pre-extraction of handcrafted features. In contrast to Jay et al. [11], who targeted a leaf spot-level evaluation, we aimed at detecting CLS symptoms at the pixel scale to help breeders detect the infection earlier, allowing an improved evaluation of the temporal and spatial dynamics of the spread in breeding trials. Our approach was also intended for growers to quantify CLS at different levels to identify management thresholds in the context of integrated pest management. An exemplary output of our approach is illustrated in the center of Figure 1.

In sum, we make the following three key claims about our work. First, our approach can detect CLS with high-performance results when testing on data captured under similar field conditions, i.e., when trained and tested on images coming from the same agricultural field under similar conditions during a single flight. Second, our approach generalizes to unseen data when trained on datasets captured in different fields and under changing light and environmental field conditions. Third, we show that our proposed classification system's results correlate well to field experts' scoring results assessing CLS disease severity visually. This paper used several UAV datasets from diverse experimental field trials to explicitly evaluate our approach under these claims.

#### **2. Related Work**

In general, semantic segmentation can be applied in different fields of applications. These are, for instance, autonomous driving [12], medical analysis [13,14], video analysis [15], segmentation of 3D environments [16], joint semantic-instance segmentation [17], and many other domains.

Regarding classification approaches, Lottes et al. [18] proposed an FCN that can determine the localization of plant stems and a pixel-wise semantic map of crops and weeds at the same time. The difference from our approach is that we used UAV imagery instead of ground vehicle-captured images. In addition to this, another proposed network by Lottes et al. [19] sought to improve the previously presented crop and weed segmentation by performing a generalization.

Milioto et al. [20] proposed a CNN, which can classify sugar beet plants, weeds, and background in real time. They used images taken from a ground robot and did not tackle diseases. Another proposed approach by Mortensen et al. [21] aimed to classify several crop species within overloaded data. This task was also performed by semantic segmentation based on FCNs.

Various classification systems have been proposed to detect plant diseases such as CLS on sugar beet in the past. Most of these systems perform the classification task using classical machine learning methods such as clustering [22], spectral indices [23], or support vector machines (SVMs) [24–28]. More recently, the wide adoption of neural networks has led to classification approaches based on convolutional neural networks (CNNs) [11,29–34].

Jay et al. [11] proposed an approach focusing on comparing the capability of UGVs and UAVs to determine a CLS scoring, which refers to the infection of a plot by *C. beticola*. Therefore, spatially high-resolution RGB images for UGV and spatially coarser resolved multi-spectral images for UAV find use. In contrast to our approach, they used multispectral images captured by a UAV instead of the RGB imagery. For the assessment, the parameters disease incidence (occurrence of single symptoms on a specific unit) and disease severity are relevant. In Jay et al., both of these were addressed and were represented by the canopy cover (green fraction), as well as the spot density [11]. They found that using high-resolution RGB imagery by UGV capturing led to a good extraction of low and high CLS scoring values. In contrast, the spatially coarser multi-spectral imagery used in UAV capturing is only applicable for high scoring values [11].

Facing the problem of powdery mildew on cucumber leaves, Lin et al. [35] proposed a CNN for semantic segmentation, which enables the pixel-wise segmentation and determination of powdery mildew on leaf images. This related approach differs from ours because we used RGB images, which are not previously segmented as the network's input. Moreover, we used imagery captured by a UAV instead of images captured from a close point of view. Besides disease detection, also nutrient deficiencies can be identified using RGB data and deep learning as shown by Yi et al. [36]. This information can then be used to provide targeted fertilization and optimize field management.

In contrast to the aforementioned prior work, our network can detect CLS and differentiate symptomatic pixels from healthy sugar beet plants and background with highperformance results. Second, our method generalizes well to unseen data, using RGB images of various field conditions. To the best of our knowledge, we are therefore the first to propose an end-to-end learned semantic segmentation approach determining the occurrence of CLS within breeders' real fields.

#### **3. Classification System for Plant Disease Detection**

The primary objective of our work was to detect and quantify CLS in sugar beet trials robustly. With our approach, we can provide breeders with information about the disease severity within their fields on a plot basis. We developed a semantic segmentation pipeline that explicitly distinguishes between the classes CLS, healthy sugar beet, and background, i.e., mostly soil.

Our approach was designed to process three-channel RGB images as the input. The output is a pixel-wise pseudo probability distribution over the class labels mentioned above. We picked per pixel the class label with the highest probability and obtained the final class map with the same resolution as the original image.

Our fully convolutional network was based on the architecture that we proposed in our previous work [18] for semantic segmentation and stem detection. Fully convolutional DenseNets, proposed by Jégou et al. [37], inspired the architectural design for our semantic segmentation approach. Their FC-DenseNet architecture was based on DenseNet, which was introduced by Huang et al. [38].

In general, the architecture of our approach is structured into three different parts: preprocessing, image encoding, and feature decoding. Figure 2 illustrates our FCN approach's general processing pipeline.

**Figure 2.** Overview of the approach, developed with precision farming applications in mind. It briefly illustrates the FCN approach for the classification of CLS based on RGB images only.

#### *3.1. Preprocessing*

Our pipeline begins with an automatic preprocessing step of the network's input. By aligning the distributions of the training and test data, preprocessing the network's inputs can improve the classification system's generalization capabilities. It includes transforming the pixel values, typically ranging between [0, 255], into the range of [0, 1]. This transformation is carried out by subtracting each channel's mean value and dividing this result by each channel's standard deviation, i.e., a standardization. Afterward, we zero centered the data to [−0.5, 0.5].

#### *3.2. Encoder Structure*

The preprocessing step is followed by the encoder, which serves as a feature extractor. Our encoder incorporates five fully convolutional dense building blocks for the densely compressed feature extraction of the network's input. The basic building block in our FCN's encoder structure follows the idea of the so-called fully convolutional DenseNet (FC-DenseNet) [37]. It combines the recently proposed densely connected CNNs, which are organized as dense blocks [38], with fully convolutional networks (FCN) [39]. The dense connectivity pattern iteratively concatenates all computed feature maps of subsequent convolutional layers in a feed-forward fashion. These dense connections encourage deeper layers to reuse features that were produced by earlier layers. Additionally, it supports the gradient flow in the backward pass and thus a stable training. After passing the RGB input images through the encoder, they are transformed into a more compressed and high-level representation.

#### *3.3. Decoder Structure*

Our decoder structure closely follows the previously described encoder. It is used to bring the compressed feature representation of the input volume back to the original resolution. The decoder achieves this by applying learned transposed convolutions to the dense feature representations until the resolution of the original input image is matched. Additionally, we included skip connections between the encoder and decoder, supporting the restoration of the spatial information, which might get lost within the encoder structure. Each pixel contains, within the original-sized output, assignment probabilities of belonging to each possible class after passing through a final softmax layer.

The resulting output is a semantic map in which the pixel-wise final class labels are stated. The final class label assignment is based on choosing the class having the highest assignment probability.

#### **4. Experimental Evaluation**

Our experiments were designed to show our method's capabilities and support our key claims, which were: (i) our approach can detect CLS with high performance when testing under similar field conditions; (ii) our approach generalizes well to changing conditions; (iii) the classification system's results of our proposed approach correlate well to field experts' scoring results. Hence, our classification system can robustly classify CLS and is a valuable tool for the evaluation of genotypes in the breeding process or plant varieties in registration or recommendation trials.

#### *4.1. Experimental Setup*

Data assessment was performed in official breeding trials all over Germany. At each experimental site, several sugar beet genotypes are cultivated in a randomized plot design, and each plot consists of 3 rows. The data set was from different time points during the vegetation period, and all characteristic phases of CLS infection from healthy to severe were observed. Reference ground truth data for calibration in Experiment 3 were assessed at one trial side in North-Rhine-Westphalia in September. We show an illustration for one of these plots in Figure 1.

In the trials that we analyzed, the infection by *Cercospora beticola* was not inoculated, but appeared naturally as the plants grew on fields that are prone to this disease.

The manually labeled portion of the datasets consisted of around 80 sugar beet plots. Various visual appearances of infested and healthy sugar beets, changing light conditions, and environmental conditions were present in the datasets. In total, we recorded and manually labeled the datasets among 16 different locations across Germany.

The images were captured by a Phase One IMX 100 megapixel RGB camera attached to a M600 drone manufactured by DJI. This drone has a payload of 6 kg, so it can easily carry the employed camera.

#### *4.2. Parameters*

In our experiments, we trained all networks from scratch using previously generated random image patches with a resolution of 320 ×320 pixels. During training, we considered *K* = 2 pairs (*my*ˆ, *my*) within each batch. Hereby, *my*ˆ denotes the predicted semantic map, and *my* represents the ground truth labels of the image patches, each represented as a vector. Moreover, we applied randomized augmentations to the training image patches to reduce overfitting of the model.

We used a positive weighted cross-entropy loss function to evaluate the difference between the predicted semantic map and the ground truth semantic map. It is calculated as follows:

$$\begin{aligned} \mathcal{L}\_{\text{CE}}(\boldsymbol{y}, \boldsymbol{\hat{y}}) &= \frac{1}{K} \sum\_{k=1}^{K} \sum\_{c=1}^{C=3} \left( 1 - \boldsymbol{y}\_{c}^{k} \right) \cdot \boldsymbol{\hat{y}}\_{c}^{k} + \\ & \quad \left( 1 + (\omega\_{c} - 1) \cdot \boldsymbol{y}\_{c}^{k} \right) \cdot \\ & \quad \left( \log \left( 1 + \exp^{-\left| \boldsymbol{y}\_{c}^{k} \right|} \right) + \max \left( -\boldsymbol{\hat{y}}\_{c}^{k} \boldsymbol{\wedge} \right) \right), \end{aligned}$$

where *k* represents the current image. Additionally, *ω<sup>c</sup>* denotes the weight of class *c*; *y*ˆ*<sup>k</sup> c* is the predicted semantic map of image *k*; and *y<sup>k</sup> <sup>c</sup>* represents the provided ground truth semantic map of image *k*. The weights *ω<sup>c</sup>* depend on the occurrence of each class within the training dataset. We assigned 10, 1, and 1 for the respective classes of CLS, healthy vegetation, and background.

We trained our network through 150 epochs and chose 0.005 as the initial learning rate. We also introduced a learning rate schedule that improved convergence by decreasing the learning rate after a predefined number of epochs. To improve generalization, we used a dropout rate of 0.33. As our parameter initialization values, we sampled a random set from a uniform distribution with a lower boundary of 0.05 and an upper boundary of 0.15. For gradient-based parameter optimization, we used the Adam algorithm by Kingma et al. [40].

#### *4.3. Performance under Similar Field Conditions*

In this first experiment, we evaluated our network's performance results in detecting CLS within every single plot when testing on data captured in the same agricultural field (but in a different area) where the training data were acquired. In addition, we ensured that the used images were captured under similar light and environmental field conditions. Therefore, we used a dataset that contained 38 plot images recorded in the same sugar beet field within one flight. Exemplary images used within this experimental setup are visualized in Figure 3.

**Figure 3.** A single plots' RGB and corresponding ground truth information. The pixels labeled as Cercospora leaf spot (CLS) are in pink, whereas healthy vegetation is represented by green pixels. Note that this is a single plot from the field illustrated in Figure 1, but rotated 90◦.

We captured these plot images in the field shown in Figure 1. We split the total amount of 38 RGB images into a training and test dataset with a ratio of 50/50. Thus, we used 19 plot images for training and 19 pictures solely for testing. As previously mentioned, these performance results can be seen as the upper boundary of the experimental setup explained in Section 4.4, as the training and test data came from a rather similar distribution.

In Table 1, we show exemplary output results of our first experimental setup.

Visually, the performance results under similar conditions led to rather good classification results for the class CLS. We came to this view because, as depicted in Table 1, almost all pixels labeled as CLS within the ground truth were correctly predicted by our network as CLS. Hence, the class CLS's recall value should be high, which was confirmed by a corresponding value of around 85%. The performance results of this experimental setup are summarized in Table 2. As can be seen in Table 2, the high recall of 85% contrasts with a rather low precision value of around 33%. The reason for the low precision value is also visible in Table 1. Our network predicted the class CLS at more pixel locations within the prediction map than were actually labeled as infected in the ground truth. This is especially recognizable in the upper left corner of the test image in the last row of Table 1, where there are much more CLS-labeled pixels in the predicted semantic map in comparison to the ground truth. In the table, it can be seen that the network correctly predicted leaves that were totally damaged, while it tended to overestimate the spread of the infection in leaves that were only partially affected.

In general, the classification of the class background worked highly satisfactorily, recognizable by very the high-performance results in terms of the intersection over union (IoU), precision, and recall. This observation was confirmed both visually and with numbers, such as an IoU of 90%, a precision of around 98%, and a recall of around 91%. In contrast, the class vegetation only showed high-performance results when facing a precision value of 96%. That means almost all pixels predicted as vegetation were labeled within the ground truth as vegetation as well. The IoU value of 73%, as well as the recall of around 76% were not as good as the corresponding values for the class background, but this was because most pixels wrongly classified as CLS actually belonged to the vegetation class. The wrongly assigned pixels resulted in a low IoU and recall for the vegetation class. **Table 1.** Visualization of the resulting output of the CLS classification. In the first image from the left, we show the original RGB image. In the rightmost image, we visualize the agreement between the pixel-wise labels of the predicted semantic map (third image from the left) and the pixel-wise labels of the ground truth semantic map (second image from the left) using green pixels (agreement) and red pixels (disagreement). Within the semantic maps, we represent the class CLS as red pixels and the class healthy vegetation as green pixels, and all remaining pixels belong to the class background.

**Table 2.** Class-wise evaluation of the pixel-wise classification performance of the classes CLS, healthy vegetation, and background. Classification performance for the first experiment under similar field conditions. We evaluated using the IoU, precision, recall, and F1 scores. All presented results are given in %.


#### *4.4. Performance under Changing Conditions*

Within the second experiment, we examined whether our trained classification model achieved a good performance in detecting the CLS disease, which developed on a subset of the plants, even under changing light and environmental conditions during the capturing process. Hence, we aimed to reach a certain generalization capability of our network to different environment variables. The symptom development of CLS is dynamic. Early symptoms differ in size, color, and shape from a late symptoms [41]. Furthermore, different leaf stages (young or old leaves) can be diseased preferably. This poses a big challenge to the classifier by itself. Moreover, different lightning and points of view in the analyzed images made the classification problem even harder. Therefore, we used image data captured at 16 different locations at different times of the day to ensure a broad spectrum of environmental conditions during the capturing process. Figure 4 in comparison to Figure 3 shows one exemplary plot image, which illustrates that in differently located sugar beet fields, the light and environmental conditions, as well as plant appearances and soil conditions could vary dramatically.

**Figure 4.** The images show an exemplary plot at another location in comparison to the breeder's field visualized in Figure 1 and Table 1. In different sugar beet field locations, the conditions can strongly differ from one another.

Within this experiment, we used 38 plot images, captured at 15 different locations, for training. We then used 38 plots from another location as the testing dataset. By this data partitioning, we aimed at providing enough training information from different perspectives to the network. We, therefore, expected our approach to generalize well enough to correctly classify the test set acquired under unseen conditions.

In Table 3, we show exemplary output results of our second experimental setup.

**Table 3.** Visualization of the resulting output of the CLS classification. In the first image from the left, we show the original RGB image. In the rightmost image, we visualize the agreement between the pixel-wise labels of the predicted semantic map (third image from the left) and the pixel-wise labels of the ground truth semantic map (second image from the left) using green pixels (agreement) and red pixels (disagreement). Within the semantic maps, we represent the class CLS as red pixels and the class healthy vegetation as green pixels, and all remaining pixels belong to the class background.


The visual evaluation of these results showed that, generally, the classification of CLS-labeled pixels was characterized by a good performance regarding the recall. This was in line with the previous experiment, and we saw a certain generalization. As is visible in the top and bottom row of Table 3, almost all occurring CLS-labeled pixels were classified correctly by our network. Only a few false-negative pixels classified as CLS were recognizable. This generally happened only when the network predicted a small part of the damaged leaf instead of the entire infected surface. In the bottom row of Table 3, the network extended its CLS prediction to the background in some parts, especially in the upper left corner of the image. We imputed this to very similar colors of the soil and rotten leaves.

We can confirm this observation by considering the performance results regarding the IoU, precision, recall, and F1 score [42]. We show these results in Table 4. Concerning the class CLS, the pixels labeled with this specific class were recognized with a recall of about 67%. However, the precision was 33%, as in the first experiment. This was in line with our observations in Table 3, in which the predicted semantic map included most CLS-labeled pixels of the ground truth semantic map, but extended this prediction to certain soil and vegetation areas.

The classes healthy vegetation and background showed high-performance results with respect to the IoU, precision, and recall. Especially the class background with an IoU of around 89%, a precision of around 98%, and a recall value of around 90% was classified well. Furthermore, the class vegetation showed satisfying results with an IoU of 79%, a precision of around 93%, and a recall of 84%.

**Table 4.** Class-wise evaluation of the pixel-wise classification performance of the classes CLS, healthy vegetation, and background. Classification performance for the second experiment under changing field conditions. We evaluated using the IoU, precision, recall and F1 scores. All presented results are given in %.


In general, the achieved recall for the class CLS of around 67% was acceptable, but compared to the remaining two classes, the performance results were less precise. Considering the fact that the test images were captured under light conditions, as well as plant and soil appearances never seen by the network, the resulting recall value of the class CLS was quite good. This was especially true when considering the upper boundary, which we determined in our first experimental setup in Section 4.3. Regarding the precision, the performance results of the generalized network were identical to the upper bound value.

Regarding the classes healthy vegetation and background, both networks' resulting performances did not strongly differ from one another. That means the generalization of soil and vegetation did not seem to be challenging for our approach.

In conclusion, the performance results of the first two experimental setups, the classification of CLS, led to quite satisfying performance results regarding the recall. When considering the precision instead, the classification did not perform as well. In this regard, it should be noted that the detection of CLS is a difficult problem also for an expert human operator. Especially in early disease stages, symptoms are hard to detect accurately as they result in an appearance that is only slightly different from a healthy leaf. We, therefore, performed a third experiment, presented in the next section, to show that the performance of our approach led to a quantification of the spread of CLS, which was comparable to the one obtained by experts examining the plants visually on site.

#### *4.5. Comparison of the Estimated Level of Infection and the Experts' Scoring Results*

To correlate our network output with the experts' scoring results in the third experiment, we needed to derive a value that was comparable to the scoring values used by them. We further refer to these derived values as infection levels. They describe the severity of

infection by CLS on sugar beet plants within the individual plots via an ordinal rating scale. For this evaluation, an expert went into the field to determine the infection level of the plots. Therefore, the determination of the infection level is a time-consuming and expensive task for farmers and breeders. Since the fields were organized into plots, each of these single plots was assigned a score. This single number represented the infection level of CLS on the sugar beet plants within this certain plot. The infection level was a number in the range between 1 and 9. No infection or a very small infection was set as 1; 5 meant a mediocre infection; and 9 represented very heavily infested plants [43].

The ground truth data were provided by the breeding company Strube Research and were based on the experts' scoring of the infection level of CLS for 336 plots. The trial was comprised of 84 entries, which were laid out over 2 strips. The entries of the trial were replicated 4 times. Two of those replicates were treated with fungicide (Strips 3-4 and 5-6), and the other two were not treated with fungicide (Strips 1-2 and 7-8). We expected high infection levels in the strips where no fungicide was applied and lower scores for the other strips. The eight strips with 42 entries each are numbered in Figure 1. The data used in this experiment were collected by two experts with 12 and 25 years of experience, respectively. They used an evaluation scheme developed by Strube and used for 20 years to ensure a consistent interpretation of the symptoms.

In order to compare the classification system's results, we first needed to derive comparable values. Therefore, we analyzed the classified image pixels' distribution within the predicted semantic map, which we obtained after applying the previously trained classifier. We focused on the resulting semantic map we obtained using the classifier trained under similar conditions (Section 4.3) and the one obtained under changing conditions (Section 4.4).

In the infection level estimation process, we only considered the pixels that were predicted as CLS, since healthy vegetation and background were not relevant for this task. To convert the number of pixels predicted as CLS into the infection level score, we first defined the lower and upper boundaries. For the first boundary, we calculated the average occurrence of CLS pixels within all plots that were rated as Infection Level 1 in the ground truth defined by the experts. We then defined as 1 the infection level off all plots with a number of pixels classified as infected that was lower than or equal to the calculated average. We used this averaging instead of simply assigning the lowest value to the first score, which was more in line with the expert evaluation. The same procedure was applied by us to find the amount of CLS pixels relative to Infection Level 8, taking into consideration the most infected plots instead. We here used the score of 8 instead of the maximum possible infection level of 9 since the ground truth data contained only infection levels in the range between 1 and 8. This allowed us to actually define the plots with the lowest-rated infection level as 1 instead of interpolating them to a higher score. Based on our predicted infection levels' lower and upper boundary, we performed linear interpolation to find the occurrence frequencies of CLS corresponding to the different scores. By finding the score corresponding to an occurrence frequency that was closest to the one of a given plot, we could assign this score to the plot. This resulted in an estimated infection level for every single plot. Now, we could compare these predictions with the corresponding expert's ground truth scoring values.

Figure 5 contains the results obtained from the images of both the classifier trained under similar, as well as the one trained under changing conditions. In the leftmost image in the upper row, we see a histogram showing the difference between the ground truth value and the corresponding predicted infection level. Here, the estimated infection levels were based on the semantic maps predicted using the classifier, which was trained under similar field conditions (Section 4.3). Underneath, in the bottom row's leftmost figure, we show the histogram of the differences between the ground truth and the prediction based on the semantic maps obtained using the classifier trained under changing field conditions (Section 4.4). In the upper row's central figure, we present the difference between the ground truth and the predicted infection level for each plot, based on the output of the classifier trained under similar conditions. Below this plot, we show the counterpart based on the classifier trained under changing field conditions. We visualize the predicted infection level of the plots based on the two classifiers on the right-hand side.

**Figure 5.** Results of the infection level (IL) determination of CLS based on the predicted semantic maps. The upper line contains the results based on the semantic maps we obtained using the classifier trained under similar conditions, whereas the lower line consists of the results based on using the classifier trained under changing conditions.

As clearly visible within both histograms, the most occurring difference between the ground truth and the prediction was 0, equaling 29.8%. In addition, regarding the differences ±1 often occurring within the histograms, they could be observed in 51.5% of the plots. This led to the insight that the estimation of the infection levels based on our networks' semantic maps correlated well to the experts' scoring values. We came to this conclusion because in practice, the infection level determination is highly subjective and depends on each expert [44,45]. This led to the fact that, sometimes, an expert evaluates an area with a different scoring value than another expert would have done. Therefore, a deviation of ±1 of the experts' scores seemed reasonable. The resulting accuracy with a ±1 tolerance was 81.2%.

The plots on the right-hand side correlate to the ground truth data distribution we explained at the beginning of this section. Both classifiers predicted rather high infection levels for the first two rows of the breeder's field, small scores for Rows 3 to 6, and high CLS infection in Rows 7 and 8.

#### **5. Conclusions**

In this paper, we showed that we could determine Cercospora leaf spot (CLS) with a high recall, but a rather low precision, in the presence of mostly healthy sugar beet plants when using data that were captured under similar field conditions as the testing data. Additionally, we could classify healthy vegetation and background with F1 scores higher than 80%. Moreover, we demonstrated that we achieved a certain generalization regarding various light and environmental conditions during the data acquisition while classifying CLS's occurrence, underlined by a recall of around 66% for the class CLS. This generalization was also observable when considering the F1 score values of the classes healthy vegetation and background, which were higher than 88%. Considering the fact that this work was a proof-of-concept, the resulting performance of CLS detection was acceptable and showed that this approach is worth being studied further. Most of the false negatives were classified as healthy vegetation, which in our opinion were derived mostly from the difficulty of visually determining the CLS symptoms, which made it almost impossible to label the data with very high accuracy. Here also the resolution of the images played a big role. We showed that this approach is already valuable by verifying that estimated infection levels derived from the semantic map predicted by our network correlated well with the experts' scoring values. Still, there is room for improvement by using a bigger dataset, and experts could be involved in the labeling process to obtain a better ground truth. Furthermore, an increased resolution of the RGB images would most likely improve the result, especially regarding the detection of early symptoms.

**Author Contributions:** Conceptualization, E.M. and P.L.; methodology, F.G., E.M., P.L. and C.S.; software, F.G. and P.L.; validation, P.L.; formal analysis, F.G., A.-K.M. and P.L.; investigation, F.G., E.M. and P.L.; resources, P.L. and C.S.; data curation, F.G., K.K. and P.L.; writing—original draft preparation, F.G. and P.L.; writing—review and editing, E.M., A.-K.M., K.K., P.L. and C.S.; supervision, P.L. and C.S.; project administration, P.L.; funding acquisition, P.L. and C.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work has partially been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy, EXC-2070—390732324 (PhenoRob), by the Federal Ministry of Food and Agriculture (BMEL) based on a decision of the Parliament of the Federal Republic of Germany via the Federal Office for Agriculture and Food (BLE) under the innovation support programme under funding no 28DK108B20 (RegisTer), and under the support program for digitalisation in agriculture, grant number FZK 28DE104A18 (Farmerspace).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to ownership reasons.

**Acknowledgments:** This work was partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy, EXC-2070—390732324 (PhenoRob), by the Federal Ministry of Food and Agriculture (BMEL) based on a decision of the Parliament of the Federal Republic of Germany via the Federal Office for Agriculture and Food (BLE) under the innovation support program under Funding No. 28DK108B20 (RegisTer), and under the support program for digitalisation in agriculture, Grant Number FZK 28DE104A18 (Farmerspace).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Unmanned Aerial Vehicles for Wildland Fires: Sensing, Perception, Cooperation and Assistance**

**Moulay A. Akhloufi 1,\*, Andy Couturier <sup>1</sup> and Nicolás A. Castro <sup>2</sup>**


**Abstract:** Wildfires represent a significant natural risk causing economic losses, human death and environmental damage. In recent years, the world has seen an increase in fire intensity and frequency. Research has been conducted towards the development of dedicated solutions for wildland fire assistance and fighting. Systems were proposed for the remote detection and tracking of fires. These systems have shown improvements in the area of efficient data collection and fire characterization within small-scale environments. However, wildland fires cover large areas making some of the proposed ground-based systems unsuitable for optimal coverage. To tackle this limitation, unmanned aerial vehicles (UAV) and unmanned aerial systems (UAS) were proposed. UAVs have proven to be useful due to their maneuverability, allowing for the implementation of remote sensing, allocation strategies and task planning. They can provide a low-cost alternative for the prevention, detection and real-time support of firefighting. In this paper, previous works related to the use of UAV in wildland fires are reviewed. Onboard sensor instruments, fire perception algorithms and coordination strategies are considered. In addition, some of the recent frameworks proposing the use of both aerial vehicles and unmanned ground vehicles (UGV) for a more efficient wildland firefighting strategy at a larger scale are presented.

**Keywords:** unmanned aerial systems; UAV; autonomous systems; wildland fire; forest fires; fire detection

#### **1. Introduction**

Wildland fires are an important threat in rural and protected areas. Their control and mitigation are difficult as they can quickly spread to their surroundings, potentially burning large land areas and getting close to urban areas and cities. The occurrence of wildland fires results into substantial costs to the economy, ecosystems and climate [1]. Nevertheless, their frequency is on the rise. In fact, there has been an increase in the intensity and frequency of wildland fires in comparison to the past 10,000 years [2]. In the western U.S. alone, wildland fires increased by 400% in the last decades [3,4]. In 2018, 8.8 million acres (35,612.34 km2) were burned by more than 58,083 wildland fires in the U.S. [5]. In Northern California, a single fire, known as "Camp Fire", ended up killing 85 people. This fire was the most destructive in California history burning 153,336 acres (620.53 km2) and destroying 18,733 structures. Losses were estimated to \$16.5 billion [3]. Experts estimate that wildland fires will increase in the coming years mainly as a result of climate change [6].

With wildland fires being a multifaceted issue, many different elements are relevant to the efforts to reduce their impact. Aspects such as meteorology, drought monitoring, vegetation status monitoring can help the prevention and the preparation to wildland fires. Other aspects such as fire suppression actions and post-fire recovery strategies must also be taken into account after the appearance of fire. Many of these aspects have been studied with unmanned aerial vehicles (UAVs). However, in the literature, two elements

**Citation:** Akhloufi, M.A.; Couturier, A.; Castro, N.A. Unmanned Aerial Vehicles for Wildland Fires: Sensing, Perception, Cooperation and Assistance. *Drones* **2021**, *5*, 15. https://doi.org/10.3390/ drones5010015

Received: 3 February 2021 Accepted: 16 February 2021 Published: 22 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

seem more prominent in relation to UAVs. First, the time span between the start of a fire and the arrival of firefighters. This response time needs to be reduced to a minimum in order to decrease the chances of the fire spreading to unmanageable levels. The second key element is the evaluation of the extent of the event and the monitoring of the emergency response. As manual wildland fire assessment is rendered difficult by several factors (e.g., limited visibility), the consideration of this aspect is necessary in order to elaborate better fighting strategies. These two key elements can only be properly addressed through the development of reliable and efficient systems for early stage fire detection and monitoring. As a result of this need, interest has grown in the research community and led to a large number of publications on the subject.

Remote sensing has been widely researched in the field as it allows the observation of wildland fire events without unnecessarily exposing humans to dangerous activities. For instance, satellite images have been used to report the fire risks [7] and the detection of active fires [8,9]. Wireless sensor networks (WSNs) have also been proposed for wildland fire detection [10], monitoring [11] and risk assessment [12]. However, both types of systems have practical limitations. Satellite imagery has limited resolution. Therefore, the data relevant to an area are often averaged and constrained to a single-pixel making it difficult to detect small fires [13]. Furthermore, satellites have limited ground coverage and necessitate a significant amount of time before being able to resurvey the same region. Limited precision and the lack of real-time data reporting are therefore rendering satellite imagery unsuitable for continuous monitoring. As for WSNs, they operate as an infrastructure that needs to be deployed beforehand. As the sensors are installed in the forest, their coverage and resolution are proportional to the investment that is made in their acquisition and deployment. Moreover, in the event of a fire, the sensors are destroyed, leading to additional replacement costs. Maintenance difficulties, the lack of power independence and the fact that they are not scalable due to their static nature are all factors known to limit their coverage and effectiveness [14]. As a result of the previous systems' shortcomings, unmanned aerial vehicles (UAVs) have been proposed as a more convenient technology for this task. Their maneuverability, autonomy, easy deployment and relatively low cost are all attributes that made UAV the technology of choice for future wildland fire management efforts.

UAV technologies have seen an important progression in the last decade and they are now used in a wide range of applications. UAV has become smaller, more affordable and now have better computation capabilities than in the past making them reliable tools for remote sensing missions in hostile environments [15]. Furthermore, UAVs can fly or hover over specific zones to retrieve relevant data in real time with cameras or other airborne sensors. As a result, research has shown their benefits for surveillance and monitoring of wildland fire as well as tasks related to post-fire damage evaluation [16–20]. Additionally, UAVs have exhibited a positive economic balance in favor of their use in wildland fire emergencies [21,22]. This makes UAVs both a practical and an economical solution. Therefore, research efforts have been oriented towards the development of frameworks and techniques using UAVs with the goal of delivering optimal fire detection, coverage and firefighting.

The subject of this paper is a summarization of the literature pertaining to use of UAVs in the context of wildland fires. Research in this area revolve more predominantly around fire detection and monitoring, therefore the core of this review will be concentrated on technologies and approaches aimed at tackling these challenges. However, this paper also touches on other subjects when relevant such as fire prognosis and firefighting but less extensively as fewer works are available on the subject in the literature. The only other related works believed to exist are the work of Yuan et al. [19] and Bailon-Ruiz and Lacroix [23]. Yuan et al. [19] touch on subjects such as UAV wildland fire monitoring, detection, fighting, diagnosis and prognosis, image vibration elimination and cooperative control of UAVs. While the subject of this work overlaps with ours, it was performed 5 years ago and since then a lot of research has been produced on the subject. In fact, most of the papers reviewed have been published in 2015 or after and are not present in Yuan et al. [19]. Therefore, this work is much more current than Yuan et al. [19]. Bailon-Ruiz and Lacroix [23] have been published in 2020, and are therefore much more current. The authors discuss two components of the field of UAV wildfire remote sensing: system architecture (single UAV or multiple UAV) and autonomy level. The reviewed works are characterized by similar attributes (mission types, decision level, collaboration level, fielded) and include unique attributes such as information processing and airframe, while this paper also analyzes unique attributes such as sensing mode and coordination. Attributes such as information processing and airframe indicate that Bailon-Ruiz and Lacroix [23] put more focus on the type of UAV and the software that runs on it while this paper is focusing on sensing and communication. The most notable difference between both works is the depth of analysis of the reviewed works and the extent of the reviewed literature. While Bailon-Ruiz and Lacroix [23] discusses system architecture and autonomy level only, this paper discusses these topics as well as sensing instruments, fire detection and segmentation, available fire datasets, fire geolocation and modeling and UAV-unmanned ground vehicles (UGV) systems for wildfires. This paper also reviews more recent works (16 vs. 10 published in 2015 or after), more works in total (27 vs. 19), and this paper's reference count is more than three times higher (121 vs. 35) indicating a more in-depth discussion of concepts related to the reviewed works which in turn requires more referencing. Following these observations, it is believed that this paper is a significant contribution and is very relevant to the field.

The final goal of this review is to provide insight into the field towards the development of cooperative autonomous systems for wildland fires. Observations made after evolving for many years in the field indicate that the research community has provided many pieces of the solution to the problems that are wildland fires. However, these pieces, especially recent ones, often fail to come together in a unified framework to form a multifaceted solution to the underlying issue. A lot more could be accomplished by combining fire detection, monitoring, prognosis and firefighting under the same system. Therefore, this paper reviews fire assistance components, sensing modalities, fire perception approaches, relevant datasets and UAV/UGV coordination and cooperation strategies. In fact, this paper's review approach is to break apart the reviewed works in these categories instead of discussing all the aspects of a reviewed work in the same paragraph. The idea is to bring existing approaches into light in such a way that it would be easier in the future to combine them into more complete systems instead of seeing them as individual systems. These subjects lead to the last section of this paper where cooperative autonomous systems are discussed and where all previously discussed technologies come together under the umbrella of a single framework.

#### **2. Fire Assistance**

Remote sensing with aerial systems presents multiple advantages in the context of emergency assistance. Their high maneuverability allows them to dynamically survey a region, follow a defined path or navigate autonomously. The wide range of sensors that can be loaded onboard allows the capture of important data which can be used to monitor the situation of interest and plan an emergency response. The ability to remotely control UAVs helps reduce the risk for humans and remove them from life-threatening tasks. The automation of maneuvers, planning and other mission-related tasks through a computer interface improves distant surveillance and monitoring. Advances in these aspects have a direct impact on the firefighting resource management.

UAV fire assistance systems in the literature can be characterized using four attributes: sensing modalities and instruments, type of task performed, coordination strategies with multiple UAVs or with the ground control station (GCS) and the approach to experimental validation. Figure 1 is a visual representation of these characteristics and their implementation in the reviewed works. These components are designed to perform one or more tasks related to fire emergencies. Within the reviewed works, the most prevalent tasks are the fire detection and monitoring. Fire prognosis and firefighting are also present in some works, but have received less interest from researchers. Fire detection and monitoring

is based on recognition techniques, a field of research that has seen significant advances in the last decades. Meanwhile, fire prognosis and fighting has practical limitations that hinder research on the subject hence the imbalance in the research interest. Prognosis requires complex mathematical models that must be fed with data that can be difficult to acquire in real time and in unknown environments. Fire fighting, on the other hand, requires expensive combat equipment that is even more expensive for large wildland fires. Moreover, close proximity with fires can pose a significant risk for the vehicle integrity and lead to its loss. However, some initial research has been done to design a UAV capable of fighting fires [24–26] and more recently some drone manufacturers have steeped in to tackle this problem as well [27]. It is clear that more work remains to be done for these vehicles to be affordable and technically viable.

**Figure 1.** Characteristics of the reviewed works.

One key component of an airborne fire assistance system is the type of UAV used. UAVs have different sizes, maneuverability and endurance capacities. These characteristics in themselves have a strong influence on the overall architecture of the system. There

is a wide selection of aerial systems ranging from large UAVs with long endurance and high-processing capabilities to small UAVs with short flight times and limited processing capabilities [28,29]. Large vehicles are expensive but have higher payload and can carry more sensors and other instruments. On the other hand, smaller vehicles are more affordable but with limited payload. The instruments onboard the vehicles vary between the reviewed systems, but some are essential to navigation and localization and therefore found in almost all UAVs. Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS) fall into this last category. Furthermore, almost all of the vehicles in the reviewed works have at least one kind of imagery sensor used for different purposes including fire perception. Temperature sensors are also present in some of the proposed fire assistance systems, but as shown in Section 3, they are less common.

Sensor measurements are the inputs of fire perception algorithms that process the data to detect the presence of the fire. The processing can be performed either onboard the UAV or by a computer located at a GCS. Fire perception can also, in some cases, be performed by a human operator inspecting the data from a GCS. It seems that a lot of efforts in research are devoted to the automation of fire perception and the optimization of the processing while at the same time preserving the accuracy of the overall system. Computer vision and machine learning techniques are commonly used for this purpose.

The last component fire assistance systems is the coordination strategy, it provides the framework for the deployment of the flight missions. Surveillance missions are usually planned beforehand and aim to search wide areas, prioritizing areas with higher fire risks. These missions can be accomplished by humans manually operating UAVs or autonomously. The coordination strategy in itself becomes more critical during the monitoring of a fire propagation as it is necessary to adapt the flight plan to the fire spread. This is even more relevant if there are multiple UAVs collaborating to the mission during a fire emergency. For this purpose, multiple coordination strategies were proposed in the literature.

For example, a UAV could hover near a fire spot and alert the rest of the fleet to proceed with fire confirmation [30]. More complex planning is also possible, by requiring a consensus on the task to be performed by each unit [31] or by flying in a specific formation around the fire perimeter [32]. In both cases, a concrete description of the task and the autonomous decision scheme must be defined for the system to be effective. Section 7 gives more details about the coordination strategies using a single or multiple UAVs.

Tables 1–3 present an overview of the reviewed fire assistance systems. Table 1 contains the year of publication and the validation process used by the authors. Table 2 presents the sensing modalities used to perform fire perception and the tasks performed. Note that some of the works do not specify sensing instruments and the authors assume that the necessary instruments are available onboard the UAV. Table 3 contains the level of autonomy, the organization of the system and the coordination strategy. Not that in some works the system is only theorized and many assumptions are made and some information might not be specified as it is not relevant to the central subject of the work. This is especially the case with works validated in simulations that do not always define a specific hardware platform.


**Table 1.** Reviewed works' characteristics.

**Table 2.** Reviewed works' sensors and performed tasks.




**Table 3.** Reviewed works' system architecture.


#### **3. Sensing Instruments**

Sensors provide the necessary data for navigation and for firefighting monitoring and assistance. In outdoor scenarios, GNSS and INS provide real-time UAV localization. They are also used to georeference the captured images thus allowing geographical mapping of fires. While these sensors are of interest to localize fires, the following section will instead focus on sensors that are able to detect fires. Fires have specific signatures that can be composed of different elements such as heat, flickering, motion, brightness, smoke and bio-product [64]. These elements can be measured using suitable sensing instruments. Cameras are the sensing instruments that offer the most versatility in their measurement. Visual and infrared (IR) sensors onboard UAVs can be used to capture a rich amount of information. In relation to cameras, Table 4 provides a list of the spectral bands used in the literature reviewed in this section. The table is provided in hopes that it will help researchers identify pertaining spectral bands for their application or identify areas of the spectrum that needs more attention for future works.


**Table 4.** Visual and IR electromagnetic spectrum.

#### *3.1. Infrared Spectrum*

At room temperature, the radiation peak of matter is located within the thermal infrared band which ranges from 0.7 μm to 1000 μm. Specialized sensors are available and can capture images in different sub-bands of the IR spectrum. Wildfire temperatures can be as high as 1000 ◦C (1800 ◦F), leading to a peak radiation in the mid-wave infrared (MWIR) sub-band [64,65]. Therefore, a sensor operating in the MWIR spectral band is best suited for fire perception. However, until recently, the form factor of MWIR sensors and their cost limited their use for low-cost small and medium UAVs [66]. To overcome these restrictions in smaller aerial vehicles, recent fire detection systems are still using NIR, SWIR or LWIR sensors. The use of these sub-bands is possible due to the fact that the higher temperature of fires also shifts the distribution of the object radiation in shorter wavelengths. Therefore, it is not necessary to use MWIR sensors directly as the effect of the peak can be observed in these other bands as well. However, a disadvantage of using NIR and SWIR is that objects under sunlight are often reflecting radiation in these sub-bands creating false positives. In such conditions, fire hot spots still remain detectable but their contrast is reduced during day time flights [64].

Characteristics put aside, IR sensors remain the most commonly used sensors in fire assistance systems due to their ability to detect heat. Bradley and Taylor [38], Casbeer et al. [32], Hinkley and Zajkowski [18], Kumar et al. [39] and Yuan et al. [49] are among the authors who have proposed methods based solely on the IR spectrum (see Table 2).

#### *3.2. Visible Spectrum*

Visible spectrum cameras are widely available and commonly used in various applications. They come in a wide variety of resolutions, form factors and cost. Their versatility offers a valuable alternative in wildland fire research from both technical and commercial perspectives. Moreover, the ever continuing reduction in visible cameras size and weight makes them perfect candidates for UAVs.

Data provided by these sensors are images in grayscale or RGB format. This allows the development of computer vision techniques using color, shape, temporal changes and motion in images or a sequence of images. Some of the vision-based techniques are presented in Section 4. Although, they are versatile and widely available, visible light sensors must be carefully selected for night-time operations as some sensors perform poorly in low light conditions. Despite some of their limitations, they equip almost all UAVs and make them good candidates for wildland fires study.

Yuan et al. [50–53], Sun et al. [48] and Zhao et al. [56] are among the authors that propose systems that rely only on the use of visible spectrum cameras (see Table 2).

#### *3.3. Multispectral Cameras*

Using each spectral band alone comes with its limitations. To tackle these limitations some authors propose the use of multiple cameras and combine multiple spectra. This allows the use of data fusion techniques to increase the accuracy of fire detection in complex situations and under different lighting conditions. Esposito et al. [67] developed a multispectral camera operating in the LWIR, NIR and visible spectrum mounted on a UAV. In a NASA Dryden's project, Ikhana UAS [17,68], a Predator B unmanned aircraft system adapted for civilian missions, was built to carry a multispectral sensor that operates in

16 different bands from visible to LWIR spectrum. Despite their interesting characteristics, in both cases, the weight of these combined sensors limited their implementation to large airborne platforms only. To address this problem, other alternatives combine smaller sensors such as visible spectrum sensors and IR sensors. Martínez-de Dios et al. [30] used this approach to capture and project the IR data onto visible images. This generated a superposition of the data leading to pixels being represented with four intensity values red, green, blue and IR. The authors report improvements in fire detection with mixed segmentation techniques that make use of the four-channel values.

#### *3.4. Other Sensors*

Various sensors other than cameras have also been proposed to detect and confirm the presence of fires. Some authors proposed the use of chemical sensors which can detect concentrations of hazardous compounds [43]. Spectrometry measures is another approach that can be used to detect the characteristics of burned vegetation and confirm a fire [64]. Again, in both cases, the size of the sensors seems to limit their use.

Temperature sensors have also been used by Belbachir et al. [42,55] to generate heat maps and detect/locate fires. Lin et al. [54,63] also theorized the use of temperature sensors to estimate a fire contour and rate of spread in the context of fire modeling. Most of the authors are using temperature sensors in the context of a simulation and therefore assume their availability without referring to real hardware. However, Wardihani et al. [55] performed a real-world validation of their proposed solution and successfully demonstrated the use of a 2 × 2 pixel resolution non-contact infrared sensor with a field of view of five degrees to measure temperatures. While interesting, these sensors are limited in comparison to IR cameras that can provide richer data. This reflects on the reviewed works and the reported results are more limited than with IR cameras.

#### **4. Fire Detection and Segmentation**

Research has shown the effectiveness of UAVs as a remote sensing tool in firefighting scenarios [17,18,43]. They are very useful even in simple tasks such as observing the fire from a static position and streaming the video sequence to human operators. This simple use case already allows firefighters to have an aerial view of the spreading fire and plan containment measures. However, single man-controlled UAVs, even if they are useful for small emergencies, do not scale up in large scenarios. Therefore, the automation of the detection and the monitoring of fires can help deliver an optimal coverage of the fire area with the help of multiple UAVs and with less human intervention. Furthermore, the gathered data can later be processed to analyze the fire, estimate its Rate of Spread (ROS) [69], volume [40] or perform post fire damage evaluation [17].

To perform fire-related tasks autonomously, systems must address different subtasks such as fire geolocation, fire modeling and even path planning and coordination between UAVs. For that purpose, sensor data are often initially processed to detect fire and extract fire-related measures. The derived information is then passed on to the different subsystems. For fire detection, authors are usually able to directly extract fire-like pixels based on color cues or IR intensities and do not require further analysis. However, monitoring tasks usually require further analysis to estimate the fire perimeter or burned areas. In that context, computed measures (e.g., segmentation) are provided as input to fire models to estimate the fire propagation over time.

This section reviews some fire detection and segmentation techniques found in the literature.

#### *4.1. Fire Segmentation*

Fire segmentation is the process of extracting pixels corresponding to fire in an image. The criteria by which a pixel is selected vary from one method to another. The selection criteria are also the main factor affecting the accuracy of the detection. In general, fire segmentation uses the pixel values of a visual spectrum image (e.g., color space segmentation) or the intensities of an IR image. Motion segmentation can also be used to extract the fire using its movement over a sequence of images.

#### 4.1.1. Color Segmentation

Images are built of pixel units that can have different encoding (e.g., grayscale, color). In color images, pixels are composed of three values in the red, green and blue channels (RGB). Other color spaces are also possible such as YCbCr, HSI, CIELAB, YUV, etc. [70]. In IR images the pixels have one channel value representing temperature (MWIR and LWIR) or reflectance (NIR, SWIR).

In the COMETS project [30,34,35], the authors employed a lookup table with fire-like colors (RGB values) that were extracted from a learned fire color histogram. The image pixels were compared to the table and the values that were not found were considered as non-fire. A non-calibrated LWIR camera is used to capture qualitative images with radiation values relative to the overall temperature of the objects in the scene. The heat peak observed in the resulting image depends on the current scenario. A training process was carried out to learn the thresholds to be applied to the IR images for binarization. Images with and without fires were considered as well as different lighting conditions and backgrounds. This permitted the selection of the appropriate threshold to apply during deployment in known conditions. Ambrosia et al. [17] selected fixed thresholds for each IR spectral band. They also varied the bands used for day-and-night missions. During night-time, the MWIR and LWIR bands were used and during the day, the NIR band was added. The results show that fixed threshold adapts poorly to unexpected conditions but can be tuned to perform better in known environments.

Yuan et al. [49–53] used color space segmentation. The images are converted from RGB to the CIELAB color space before further processing. Sun et al. [48] proposed the use of YCbCr color space. In both cases, a set of rules were developed based on empirical calculations performed on captured fire images. For example, Sun et al. [48] considered pixels as fires if their values followed the following rules: *Y* > *Cb*, *Cr* > *Cb*, *Y* > *Ymean*, *Cb* < *Cbmean* and *Cr* > *Crmean*. The mean sub-index indicates the channel mean value of the corresponding image. Otsu thresholding technique [71] was used in [49] to segment IR images.

Color value rule-based segmentation approaches are computationally efficient, but lack robustness during detection. Results show that objects with a color similar to fire are often mislabeled as fire and trigger false alarms. A combination of rules in different color spaces and the addition of IR can increase the detection accuracy. More complex algorithms that are time and space aware have also been shown to increase the accuracy of the fire detection [72–82]. The majority of them have not been integrated with UAVs.

In recent years, deep learning algorithms have shown impressive results in different areas. Relating to UAVs, past work using deep convolutional neural networks (CNN) dealt mainly with fire detection [56,83,84]. Deep fire segmentation techniques proposed recently have shown the potential of developing an efficient wildland fire segmentation system [85]. The used dataset in this last work included some aerial wildland fire images [86]. Deep segmentation of wildland fires is still lacking in UAV applications.

#### 4.1.2. Motion Segmentation

Fire segmentation using static images help reduce the search space, but often objects with a similar color to fire can be detected and lead to false positives. Yuan et al. [49–53] and Sun et al. [48] proposed the use of Lukas-Kanade optical flow algorithm [87] to consider fire movements. With the detection of corresponding feature points in consecutive image frames, a relative motion vector can be computed. The mean motion vector matches the UAV's motion except for moving objects in the ground. Fire flames are among those objects because of their random motion. By detecting feature points within regions with both random movements and fire-like colors, the fire can be confirmed and the false alarm rate reduced.

#### *4.2. Fire Detection and Features Extraction*

The data fed to a detection system are analyzed in order to find patterns that confirm the occurrence of an event. Patterns are recognized by computing different features which can be strong or weak signatures for a specific application. In the case of fire detection with UAVs, the most popular features are color, brightness and motion. Research focusing on fire detection considers the fusion of more features to obtain better results in the classification stage. These features can be categorized by the level of abstraction at which they are extracted: pixel, spatial and temporal.

Color cues are widely used in the first step to extract fire-like pixels. This reduces the search space for further processing with more computationally expensive detection algorithms. For example, the RGB mean values of a Region of Interest (RoI) and the absolute color differences (|*R* − *B*|, |*R* − *G*|, |*B* − *G*|) can be thresholded [88] or used to train a classification algorithm [89]. In the work of Duong and Tinh [90], the authors further added the intensity mean, the variance and the entropy values of the ROI to the feature vector. Other features used in the literature include color histograms of ROI [75] and color spatial dispersion measures [73].

After the detection of the ROI, other features can be extracted. Some authors consider spatial characteristics to determine the fire perimeter complexity by relating the convex hull to the perimeter ratio and the bounding rectangle to perimeter ratio [91]. The distance between the blob centroid position within the bounding box has also been considered in this work.

Texture is another spatial characteristic often used for fire detection. The main texture descriptors proposed for this task are Local Binary Patterns (LBP) [92–95] and Speeded Up Robust Features (SURF) [75,78]. These operators characterize local spatial changes in intensity or color in an image and return a feature vector that can be used as input for classification. SURF [96] is computationally expensive but allows for scale and rotation invariant matching. LBP [97] needs less processing power and extracts the mean relation between pixels in a small area using the 8 neighbors of a pixel. Some authors [98,99] also used the Harris corner detector [100], which is a computationally efficient feature point extractor.

Deep learning is another approach that has been used for fire detecting. It allows the automatic learning of low- and high-level features instead of hand crafting them as it was the case with the previous approaches described. Zhao et al. [56] developed such an approach in the form of a 15-layer CNN called Fire\_Net. The proposed architecture is inspired by AlexNet [101] and is made of a succession of convolutions, ReLUs and max poolings that end with a fully connected layer followed by a softmax layer. The approach is able to classify image patches as fire or not fire with a 98% accuracy outperforming many other similar deep learning or learning-based approaches tested by the authors on the same data. Jiao et al. [60,61] also proposed a deep learning approach but based on the YOLOv3 architecture [102]. The solution is an object detection approach able to provide bounding boxes around objects of interest. In this case, the network is trained on 3 classes: smoke, fire and combination of smoke and fire. Initially, the authors used a YOLOv3-tiny architecture and on-board computations. The system was able to reach a precision of 83% and a frame rate of 3 to 6 fps [60]. In a more recent contribution [61], the same authors were able to reach a detection precision of 91% and a frame rate of over 80 fps by performing the computation on a GPU located in the GSC instead.

The features reviewed above are extracted from single images. When a video sequence is available, the temporal variation in color, shape and position of some blobs can be extracted. In the work of Ko et al. [103], the fire blob shape variation is computed by a skewness measure of the distance from the perimeter points to the blob's centroid. Foggia et al. [104] measured shape changes by computing the perimeter to area ratio variation over multiple frames. The authors also detected the blob movements by matching them in contiguous frames and to compute the centroid displacement. Fire tends to move slowly upwards, thus blobs that do not comply with this rule can be discarded [72,103,105]. The centroid displacement can also be an input for further classification [72,91,106]. A similarity evaluation is employed by Zhou et al. [91]. They measure the rate of change of overlapping areas of blobs in contiguous frames. This gives a practical representation of the speed at which the region of interest is moving and if it is growing or decreasing in size. Fire flickering can also be identified by considering specific measures such as intensity variation [107], the number of high-pass zero crossing in the wavelet transform [108] or the number of changes from fire to non-fire pixels inside a region [109]. Wang et al. [110] implemented a long-term movement gradient histogram, which accumulates the motion changes. The histogram is fitted to a curve which is used to evaluate if the area corresponds to a fire or not. Kim and Kim [111] proposed a Brownian motion estimator that measures the correlation of two random vectors [112]. The vectors are composed of channel values, the first intensity derivative and the second intensity derivative. Therefore, the Brownian motion estimator describes the dynamic dependence between a series of regions across multiple frames. Temporal features consider a time window for the fire evaluation. Then, some empirical criteria are established to determine the optimal thresholds and duration of the events in order to trigger a fire alarm.

Among the features described so far, there are some features that are more oriented towards fire detection. Features such as color, blob centroid displacement and flickering are some of the most popular. Some novel approaches such as the Brownian correlation or the histogram of gradients have been less explored but are nevertheless interesting. A comparison of these different features and an evaluation of which one has a greater impact on the fire detection accuracy and false positive rate would be very useful. Unfortunately, such a comprehensive comparison does not seem to have been published yet. However, as most of these features are not computationally expensive, ensembling the features can improve the performance and reduce the false detection rate. Table 5 gives an overview of the features used depending on the input.



#### *4.3. Considerations in UAV Applications*

Additional features can improve the fire detection. Features that are obtained by temporal analysis evaluate the difference between contiguous frames. In simple scenarios, where the camera is static and the background is not complex, frame subtractions can help detect moving pixels. In the presence of complex and dynamic backgrounds, Gaussian mixture models and other sophisticated background modeling techniques can be considered.

However, the video streams from UAVs have fast motions and no classical background subtraction method would give satisfying results because of the assumption of a static camera. Even in a situation where the UAV is hovering over a fixed position, the images are still affected by wind turbulence and vibrations. Therefore, in order to be able to apply these motion analysis techniques, it is necessary to consider image alignment and video stabilization. The usual approach is to find strong feature points that can be tracked over a sequence of frames. Merino et al. [44], in their fire assistance system, used a motion

estimation approach based on feature points matching known as sparse motion field. From the matched points, they estimate a homography matrix that maps the pixels in an image with the pixels in the previous frame. This allows mapping every image to a common coordinated frame for alignment. SURF [96] and ORB [113] are two feature point methods that were used for extracting salient features prior to the image alignment. It seems that the impacts and the benefits of the image alignment have not yet been addressed in the literature relating to fire and smoke detection but some researchers such as [44] consider it important for their fire assistance system to work properly.

#### **5. Wildland Fire Datasets**

A large number of fire detection approaches use a classification method that relies on learning algorithms. The main challenges of machine learning is to build or to find a large enough dataset with low bias. Such a dataset should contain positive examples with high feature variance and negative examples consisting of standard and challenging samples.

Deep learning techniques need even larger datasets for training. Data augmentation techniques can help in this regard but it requires a sufficiently large dataset to start. Welldeveloped research fields such as face or object recognition have already large datasets that have been built and vetted by the community. These datasets are considered suitable for the development and benchmarking of the new algorithms in their respective fields. In the case of fire detection, no such widely employed dataset is available yet. Some effort has been made toward this direction. Steffens et al. [114] captured a set of 24 videos from hand-held cameras and robot mounted cameras. The ground-truth was defined by bounding boxes around the fire. Foggia et al. [104] compiled a collection of 29 videos of fire and smoke but did not provide ground-truth data. Chino et al. [93] gathered around 180 fire images to test their BowFire algorithm and made the dataset available with manually segmented binary images representing the ground-truth for the fire area. However, the main problem with these datasets is the lack of wildland fire samples. This could be problematic for the development of a fire detection module for wildland fire assistance systems. Aerial fire samples in the form of videos are also necessary for the development of UA-based systems.

In [86], the authors collected images and videos to build the Corsican fire database. This dataset is specifically built for wildland fires. It also contains multimodal images (visual and NIR images) of fires. The images have their corresponding binary masks representing the ground-truth (segmented fire area). Other information is also available such as smoke presence, location of capture, type of vegetation, dominant color, fire texture level, etc. The dataset contains some aerial wildland fire views, but their number is limited.

The wildland fire UAV research is still lacking a dataset that can help improve the development of the algorithms needed in a wildland fire assistance system. Table 6 contains a brief description of the main fire research datasets.



#### **6. Fire Geolocation and Fire Modeling**

In a wildland fire scenario, when a fire is detected, the vehicle must alert the GCS and send the fire's geolocation to deploy the firefighting resources. In the reviewed literature, two different levels of approach are studied for detection alert. Some authors are using a local approach where the position of the fire is reported at first contact. Other authors go

further by taking a global approach to the problem by identifying and locating the entire perimeter of the fire.

The simplest alerting approach is to directly provide the geographical coordinates of the UAV using the onboard GPS when a fire is first detected. This can be performed with good accuracy when the UAV is flying at low altitudes and has its data acquisition sensor pointing to the ground with a 90-degree angle. This approach is employed by Wardihani et al. [55] using a downward pointing temperature sensor to locate fire hotspots. A similar approach is possible with a camera located on the bottom of a UAV and oriented downward. However, for a camera located on the front side of the UAV, it is required to compute a projection of the camera plane onto a global coordinate system using an homography. This transformation allows mapping pixel coordinates to the ground plane. This approach performs well when the UAV pose estimation is reliable and when the ground is mostly planar. Some difficulties arise in the presence of uneven surfaces. Some authors [17,30,41,44,45] have circumvented this limitation by exploiting a previously known Digital Elevation Map (DEM) of the surveyed area. DEM allows for the estimation of the location from where a ray corresponding to a fire pixel originated and thus improves the fire location estimation. DEMs can induce some errors. To reduce these errors, a UAV fleet looking at the same hotspot can first detect the fire and then use different views of the UAV to refine the estimations [30].

In order to better characterize ongoing wildfires, some authors have studied fire modeling in order to provide global information such as the fire boundaries and its behavior. The simplest models are using an elliptic shape which is fitted to the fire and where each ellipse axis increases at some given rate. For example, Ghamry and Zhang [46,47], Ghamry et al. [69] applied an elliptical model to estimate the fire perimeter. Here, the rate at which the ellipse axis grows depends on the direction towards which the wind blows and its speed. This allowed the authors to estimate the perimeter of the fire and then define a UAV team formation for further monitoring.

More complex fire models with more variables and data inputs have also been studied. These more advanced models often try to estimate the rate of spread (ROS) of the fire based on wind speed and direction, terrain slopes, vegetation density, weather and other variables. These models are often tested in a simulation. For example, Kumar et al. [39], Pham et al. [57,58], Lin et al. [54,63] and Seraj and Gombolay [62] used the FARSITE model to test their coordination strategies under various scenarios. Some of these models were not suitable for real-time fire estimation because their complexity significantly increased the computation time. However, Lin et al. [54,63] proposed a convergent Kalman filter-based methodology to provide data to a scalar field wildfire model that is executable on-board a UAV and requires low computation resources. The proposed approach was able to provide estimations of the wildfire ROS and the fire front contour.

Some authors used a different approach to model and characterize the fire. For example, Martínez-de Dios et al. [40] used multiple images to extract geometric features from the fire such as the base perimeter, the height and the inclination. The extraction is performed using computer vision techniques (e.g., image segmentation). The authors propose the use of multiple visible-NIR multimodal stereo vision systems to extract the fire area. Each stereo system provides an approximate 3D model of the fire. The models captured using multiple views are registered to get the fire 3D model. This 3D model is tracked over time to compute different fire characteristics such as height, width, inclination, perimeter, area, volume, ROS and their evolution over time.

Bradley and Taylor [38] divided the environment into cells and assigned a fire probability to each cell using IR images. This method takes into account the uncertainty in the position of the UAV and therefore applies a Gaussian weighting scheme to the probabilities. The authors then apply a Sequential Monte Carlo (SMC) method to compose a Georeferenced Uncertainty Mosaic (GUM) which is then used to locate the fire. Belbachir et al. [42] model the fire as a static cone of heat sourcing from the fire center and dissipating with an altitude and a horizontal distance. Based on this assumption, they

construct a grid of fire probabilities with the temperature measures. The fire is detected when the probabilities are above a defined threshold. Lin and Liu [63] also generate an occupancy grid by using temperature sensors and by associating temperatures to cells. They also compute the gradient of the grid and estimate the fire center, ROS and perimeter.

#### **7. Coordination Strategy**

Coordination strategy is an important component when deploying autonomous UAVs. The coordination strategy establishes the procedure for communication, task allocation and planning procedures. Based on the communication links established during the mission, three main schemes can be distinguished. First, for a single vehicle, there is no coordination strategy as the UAV does not need to communicate with other UAVs. For multiple UAVs, the path planning and task allocation are often resolved by an optimization process or assumed to be so. One approach is to centralize the path planning and decision process in the GCS and only allows the UAV to communicate with it but not between each other. Another approach is to tackle the problem of coordinating multiple entities in a distributed and decentralized manner. Each vehicle can connect to other UAVs, allowing for distributed decision-making and communication with the GCS is only for reporting observations but not for planning.

#### *7.1. Single UAV*

The viability of single UAVs, either large airships or small aerial systems, has been evaluated for wildland fire surveillance and monitoring. The Ikhana UAS [17] was deployed in western US between 2006 and 2010. It was a single large and high endurance vehicle with powerful sensory systems for autonomous fire detection. The decision strategy and the path planning were performed by human operators. Similarly, Wardihani et al. [55] used small quadcopter UAV and manually defined flight paths using a mission planner software in order to survey a region and detect hotspots. Pastor et al. [41] proposed a semi-autonomous system in which a single UAV would sweep a rectangular area, locate hotspots and then return to a nearby ground station. A human could control the UAV and order it to stay over the hotspot location to confirm visually if it corresponds to a real fire or not. Martins et al. [33] used an entirely autonomous navigation system where the UAV only received waypoints from where to start surveillance. When a hotspot is detected, the UAV approaches the source, hovers over the target and confirms the fire. The experimental tests showed very interesting results for fire detection and monitoring tasks.

While single UAV strategies are interesting for their simplicity, they remain very limited in relation to large-scale wildland fires. For this reason, the more advanced and mature solutions use team-based systems that help increase the coverage area.

#### *7.2. Centralized*

The addition of more UAVs to the mission increases the area covered by the systems. In a centralized team strategy, all UAVs are coordinated by a single GCS. This scheme can lead to a more accurate fire georeferencing and less false alarms by allowing for a global situation awareness at all times. Another advantage of centralized communications is that it makes centralized processing easy and therefore makes it possible to use smaller and more affordable UAVs as they do not require high-processing power. The main drawback of this approach is the need for a functional communication network that can connect to all UAVs at all times which is not always possible when the fire areas are remote.

Martínez-de Dios et al. [30] proposed a simple centralized approach where data from multiple UAVs is combined to correct and reduce the uncertainty of fire georeferencing. After a fire is detected by a unit, nearby vehicles are sent to the same region to perform a fire confirmation.

Belbachir et al. [42] proposed a greedy algorithm for fire detection using a probability grid. For this purpose, each UAV selects, in a greedy way, the path that provides more information. The UAVs visit cells that have not yet been visited and which are within the direction where the temperature increases.

Ghamry and Zhang [46] distributed the UAVs uniformly around the fire perimeter using an elliptical formation. This allows the UAVs to keep their paths at even angles around the estimated fire center. Ghamry and Zhang [47] added the ability to restructure the formation if a UAV is damaged or has to leave for refueling. To achieve this faulttolerant behavior, when a UAV needs to leave the formation, all communications with it are stopped. Other vehicles automatically notice the missing UAV and start performing a reformation process. In this system, prior to the monitoring task, the fleet flies in a leader–follower formation where the leader gets a predetermined flight path and the rest follow it at specific distances and angles. In the work of Lin et al. [54], Lin and Liu [63], UAVs are directed to fly uniformly in formation around an estimated fire center. In this approach, a Kalman filter is used to estimate the fire contour and the fire center movements, allowing the UAVs to fly and adapt their formation accordingly.

While incomplete, initial results by Aydin et al. [26] are worthy of mention as it is one of the only works to tackle fire fighting directly. The authors theorized a collaboration model where scout UAVs would spot wildfires and monitor the risk of spread to structures. Relay UAVs would then be used to extend the communication range and allow the scouts to contact firefighting UAVs carrying fire-extinguishing balls. It is believed that 10 UAVs each carrying 10 1.3 kg fire-extinguishing balls would be able to extinguish an area of approximately 676 m<sup>2</sup> per sortie. While the extinguishing capacity of the fire-extinguishing balls was validated, the UAV coordination strategy has not been tested yet. However, this approach remains promising for wildfire fighting.

#### *7.3. Decentralized*

In a decentralized communication scheme, the UAVs are communicating between each other in order to collaborate for path planning and optimal area coverage. The interaction with the GCS is reduced to a minimum and usually only happens at the beginning of a flight to receive initial flight coordinates or at the end of a flight for observation reporting and data transfers. The system is able to perform more tasks in an autonomous manner and even to cover larger areas by using some UAVs as communication relays. The main advantages of such an approach are reliability as a link with the GCS is not required to be active at all times and the possibility for operations in remote areas where global communication links are impractical. However, the added complexity imposes new challenges as distributed coordination algorithms need to be developed and implemented. In the literature, these systems were mainly used for optimal fire perimeter surveillance and task allocation.

Alexis et al. [37] describe a UAV rendezvous-based consensus algorithm which aims to equally distribute the path length of the UAV around the fire perimeter. UAVs depart in pairs and in opposite directions around the fire perimeter. They set rendezvous locations where they share knowledge about the traveled paths, the current state of the fire perimeter and other units encountered. If the update shows that the fire perimeter has evolved, then each UAV will select new rendezvous locations in such a way that the distance traveled by each of the UAVs is almost the same. The authors have shown through simulations that the algorithm converges and the recomputing of rendezvous points allows efficient adaptation of the UAV formation to an evolving fire perimeter. The optimal distribution of UAVs around a fire perimeter has also been studied by Casbeer et al. [32]. They demonstrated that in order to reduce the length of time between data uploads to the GCS, the UAVs must depart in pairs, travel in opposite directions and be evenly spaced around the perimeter. To achieve optimal perimeter tracking, they designed a control loop to keep half of the bottom-facing IR camera over hotspot pixels and the other half over non-fire area.

For monitoring, Pham et al. [57,58] proposes a collaborative system in which UAVs are sent to monitor a fire and optimally cover the fire area. This formation is achieved by detecting neighboring UAVs and reducing camera view overlaps while considering the location of the fire front. The UAVs are also allowed to increase or decrease their altitude in order to control the resolution of the captured imagery to provide optimal observational capabilities. This behavior is accomplished with the application of a force field-based algorithm that simulates the attraction of a UAV by the fire front and its repeal from the other UAVs. The attraction and repulsion forces are adapted by considering the fire front confidence and the estimated field of view of each UAV. One problem with this approach is that the visibility reduction induced by smoke is not taken into account which can put the vehicle in a dangerous situation.

Another coordination strategy was proposed to perform optimal task allocation within a team of UAVs. The tasks can be surveillance, monitoring or firefighting. Ghamry et al. [31] proposed an auction-based firefighting coordination algorithm. In this algorithm, a fire is first detected and then the UAV must coordinate themselves to act upon each known fire spot. To achieve this task, each vehicle generates a bid valued by a cost function of its distance from the fire spot. In this manner, the UAV with the best offer for the task will be assigned to it. Sujit et al. [36] also proposed a similar auction-based collaboration algorithm but with the ability to consider a minimal number of UAVs to watch each hotspot. Both contributions distributed the UAVs equally around the fire perimeter.

Decentralized approaches have also been used for direct fire fighting using fire suppressants. Kumar et al. [39] proposed such a coordination protocol where the planned path of each UAV is optimized to minimize the distance to a detected fire perimeter. As a second phase, the path of UAVs carrying fire suppressants is optimized by minimizing the distance to the fire center. This allows the solution to monitor a fire situation and provide optimal fire suppressant delivery.

Recently, new control approaches based on deep reinforcement learning (DRL) started to appear in the literature. One of the very first with such an approach for wildfire monitoring has been proposed by Julian and Kochenderfer [59]. The authors first formulated the problem as a partially observable Markov decision process (POMDP) solvable with DRL. A simulation environment being required for DRL, they also defined a simplified stochastic wildfire model using a 100 × 100 fire presence grid. This environment was used to train a simulated fixed-wing agent with a decision process based on a CNN. Multiple agent using the same CNN can be spawned in the same environment to simulate a multi-UAV system. While the authors defined different DRL approaches, the best performing approach used a collaborative belief map shared and updated by all agents indicating the state of the wildfire. A reward function rewarding newly discovered burning cells by any aircraft is used to encourage good fire monitoring and collaboration between agents. An aircraft proximity penalty is also added to encourage aircraft separation. Simulation results show that the approach is able to outperform a baseline receding-horizon controller, scale with different numbers of aircraft and adapt to different fire area shapes. However, the approach remains limited as the environment modeling is oversimplified, the UAVs are assumed to maintain a steady altitude, a constant speed and fly at different altitudes as collision avoidance is not implemented.

While new approaches are interesting, research on objective function optimization-based distributed control frameworks is still very active and continues to generate state-of-the-art results. This is the case with the approach proposed by Seraj and Gombolay [62]. The authors used a dual-criterion objective function based on a Kalman uncertainty residual propagation and a weighted multi-agent consensus protocol. An adaptive extended Kalman filter (AEKF) is used to leverage the fire propagation model (FARSITE) and the observation model. The approach includes an uncertainty-based controller built through the combination of a fire front location uncertainty map and a human uncertainty map. This allows the system to take into account GPS-equipped human firefighters on the ground in order to ensure their safety while considering the fire front location like other similar methods. A second controller (formation controller) is encouraging the UAV team to maintain a formation consensus for maximizing the coverage. The approach is using the theory of artificial potential field to generate artificial forces to pull or push on the UAV in order to attain an optimal state. Following a simulation, the solution was able to outperform both a state-of-the-art model-based distributed control algorithm and a DRL baseline strongly confirming the relevance of the approach.

This paper only reviews decentralized communication frameworks used in wildland fire contexts. However, many solutions in the literature are presented as general communication solutions without corresponding applications. This is the case with the work of Pignaton de Freitas et al. [115] that proposed a multipurpose localization service to inform all UAVs in the formation of the other UAVs position. One interesting and rarely seen feature of the system is its ability to estimate the position of UAVs that are not received due to communication errors. This illustrates that researchers in the field should not only refer to wildfire-related works when the time comes to design new systems and that some works outside of the field may be important to consider.

#### **8. Cooperative Autonomous Systems for Wildland Fires**

UAVs can play an important role in the detection and monitoring of large wildland fires. Multiple UAVs can collaborate in the extraction of important data and improve firefighting strategies. Moreover, aerial vehicles can cooperate with unmanned ground vehicles (UGV) in operational firefighting scenarios.

One type of cooperation can consist of the use of UGV to carry small short endurance UAVs to detected fire areas and be used as refueling stations. Ghamry et al. [69] proposed such a system, where a coordinated leader–follower strategy is used. UAVs are carried by UGV to a desired location and deployed to explore preassigned areas. If a UAV detects a fire, an alert is sent to the leading UGV and to the rest of the fleet. The leader computes new optimal trajectories for the UAVs in order to monitor the fire perimeter. Phan and Liu [116] present another firefighting collaborative UAV-UGV strategy. A hierarchical UAV-UGV system composed of a large leading airship and cooperative UAV and UGV is proposed. When a fire is detected, the vehicles are deployed for fire monitoring. In this scenario, UAVs and UGVs are supposed to have the capacity to carry water and combat fire. The UAVs are deployed in an optimal flying formation over the fire front area. UGV are sent to prevent the fire propagation and limit its spread using water and fire retardants. Auction-based algorithms are implemented to allocate the tasks to each vehicle. Viguria et al. [117] also proposed the use of task allocation by an auction-based algorithm. In their framework, the vehicles can perform various tasks such as surveillance, monitoring, fire extinguishing, transportation and acts as a communication relay. A human or the GCS can generate a list of tasks that need to be fulfilled. Each robot sends a bid for each task and the one with the best offer wins and can proceed to execute the task. The offers are based on specific cost functions for each task that consider the vehicle distance, fuel level and capabilities.

Akhloufi et al. [118] proposed a multimodal UAV-UGV cooperative framework for largescale wildland fire detection and segmentation, 3D modeling and strategical firefighting. The framework is composed of multiple UAVs and UGVs operating in a team-based cooperative mode. Figure 2 illustrates the proposed framework [118]. The vehicles are equipped with a multimodal stereo-vision system such as the ones developed for ground-based fire detection and 3D modeling [119–122]. The stereo system includes multispectral cameras operating in the visible and NIR spectrum for efficient fire detection and segmentation. Each stereo system provides an approximate 3D model of the fire. The models captured using multiple views are registered using inertial measurements, geospatial data and the extracted features using computer vision to build the propagating fire front 3D model [119–121]. Based on the 3D model of the fire, the UAVs and UGVs can be positioned strategically to capture complementary views of the fire front. This 3D model is tracked over time to compute different three-dimensional fire characteristics such as height, width, inclination, perimeter, area, volume, ROS and their evolution over time. The extracted three-dimensional fire characteristics can be fed to a mathematical fire propagation model to predict the fire behavior and spread over time. The obtained data make it possible to alert and inform about the risk levels in the surrounding areas. The predicted fire propagation can be mapped and used in an operational firefighting strategy. Furthermore, this information can be used for the

optimal deployment of UAVs and UGVs in the field. This type of framework can be combined with other firefighting resources such as firefighters, aerial firefighting aircraft and future fire extinguisher drones.

(**a**) The acquisition of 3D fire front data. UAVs and UGVs equipped with multimodal stereo cameras, IMU and GPS.

(**b**) The modeling and prediction. Registered 3D fire front, weather, topographic and vegetation data are used to predict the fire propagation and map it.

**Figure 2.** Unmanned aerial vehicle-unmanned ground vehicle (UAV-UGV) multimodal framework for wildland fires assistance.

#### **9. Conclusions**

This paper presents a survey of different approaches for the development of UAV fire assistance systems. Sensing instruments, fire perception algorithms and different coordination strategies have been described. UAVs can play an important role in the fight against wildland fires in large areas. With the decrease in their prices and their wider commercial availability, new applications in this field will emerge. However, some limitations remain such as autonomy, reliability and fault tolerance. Further research is needed to overcome these limitations. Security is also a concern, as there are risks associated with having UAVs flying over firefighters or close to aircraft carrying water and fire retardants. Nevertheless, the benefits of using UAVs are significant and this could lead to innovations aiming to solve these problems.

On the perception side, most of the developed techniques rely on classical computer vision algorithms. However, the emergence of some work in the field of deep learning has been witnessed in recent years, especially for fire detection, but it remains in the early stages of development. Furthermore, some datasets containing wildland fire images that can be used for the development of computer vision algorithms were presented. Unfortunately, only a small number of them contains aerial views of wildland fires. In addition, the lack of a large dataset limits the development of advanced deep learning algorithms. Such datasets would be important for the future of the field as they can serve to benchmark approaches and compare them quantitatively. Therefore, deep learning and the construction of new large-scale aerial wildfire datasets represents interesting research opportunities for future contributions by researchers in the field.

In this work, frameworks proposing cooperative autonomous systems where both aerial and ground vehicles contribute to wildland firefighting were also discussed. While these frameworks are mostly theoretical and limited to simulations, they provide interesting ideas about a more complete wildland firefighting system. Future research in these areas can provide new approaches for the further development of autonomous operational systems without or with little human intervention.

**Author Contributions:** Conceptualization, M.A.A.; writing—original draft preparation, N.A.C., M.A.A. and A.C.; writing—review and editing, A.C. and M.A.A.; visualization, A.C. and M.A.A.; supervision, M.A.A.; funding acquisition, M.A.A.; All authors have read and agreed to the published version of the manuscript.

**Funding:** Natural Sciences and Engineering Research Council of Canada (NSERC), reference number RGPIN-2018-06233.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Acknowledgments:** This work has been partially supported by the government of Canada under the Canada–Chile Leadership Exchange Scholarship.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


*Article*

## **Spray Deposition on Weeds (Palmer Amaranth and Morningglory) from a Remotely Piloted Aerial Application System and Backpack Sprayer**

### **Daniel Martin 1,\*, Vijay Singh 2,**†**, Mohamed A. Latheef <sup>1</sup> and Muthukumar Bagavathiannan <sup>2</sup>**


Received: 11 August 2020; Accepted: 15 September 2020; Published: 19 September 2020

**Abstract:** This study was designed to determine whether a remotely piloted aerial application system (RPAAS) could be used in lieu of a backpack sprayer for post-emergence herbicide application. Consequent to this objective, a spray mixture of tap water and fluorescent dye was applied on Palmer amaranth and ivyleaf morningglory using an RPAAS at 18.7 and 37.4 L·ha−<sup>1</sup> and a CO2-pressurized backpack sprayer at a 140 L·ha−<sup>1</sup> spray application rate. Spray efficiency (the proportion of applied spray collected on an artificial sampler) for the RPAAS treatments was comparable to that for the backpack sprayer. Fluorescent spray droplet density was significantly higher on the adaxial surface for the backpack sprayer treatment than that for the RPAAS platforms. The percent of spray droplets on the abaxial surface for the RPAAS aircraft at 37.4 L·ha−<sup>1</sup> was 4-fold greater than that for the backpack sprayer at 140 L·ha−1. The increased spray deposition on the abaxial leaf surfaces was likely caused by rotor downwash and wind turbulence generated by the RPAAS which caused leaf fluttering. This improved spray deposition may help increase the efficacy of contact herbicides. Test results indicated that RPAASs may be used for herbicide application in lieu of conventional backpack sprayers.

**Keywords:** UAV; UAS; RPAAS; aerial application; backpack sprayer; spray deposition; droplet spectra; palmer amaranth; morningglory

#### **1. Introduction**

Weeds are one of the major limiting factors to the production of agricultural crops and cause significant yield loss in crop farming systems throughout the world [1–3]. It is estimated that weeds in corn and soybean alone would reduce yield by 50%, costing growers \$43 billion in economic loss annually in the United States and Canada according to a recent study conducted by the Weed Science Society of America in conjunction with Kansas State University spanning over a seven-year period [4–6]. In general, broadleaf weeds are more competitive than grasses and early germinating weeds reduce yield more than weeds which emerge later in the growing season [7]. In many parts of the world, weed control with herbicides has gained traction in lieu of tillage and it is purported to improve environmental conditions including a reduction in soil erosion, fuel use and greenhouse gas emissions [8].

Recently, the use of unmanned aerial application systems (UASs) or remotely piloted aerial application systems (RPAASs) for field mapping, weed classification by species, plant stress detection, biomass and field nutrition estimation and application of pest control products in small-farm operations and site-specific management of crop pests in difficult terrains not easily accessible to manned aircraft has received increased attention around the globe [9–12]. RPAASs have the potential to occupy this niche because of their ability to fly at low altitudes and to hover close to plant canopies at different application heights and ground speeds with precision and safety. RPAASs are remotely piloted telemetrically and can fly autonomously using preprogrammed georeferenced flight paths. In precision agriculture, data on soil variability and crop characteristics to optimize field applications of seed, fertilizer and irrigation can be collected by remote sensing [13–15]. Huang et al. [16] developed UAS systems for digital imaging to identify glyphosate-resistant weeds and to determine crop injury from dicamba herbicide. Göktogan et al. [ ˇ 17] used a rotary wing UAS to locate, classify and map alligator weed and *Salvinia* in an inaccessible aquatic habitat. Using multispectral cameras, Castaldi et al. [18] acquired aerial images of weed patches in maize fields and developed prescription maps for herbicide application. Peña et al. [19] found strong correlation between on-ground weed coverage and that estimated by aerial remote imagery captured by an UAV (r2 = 0.89). Moreover, Peña et al. [19] reported that the determination of weed-free and low to moderate weed density areas would likely facilitate the growers to reduce herbicide application in maize fields. Although several researchers have reported that spray drones offer good potential for site-specific herbicide application in commercial farms or for monitoring weed populations over much larger areas [20], hardly any data exist on the use of RPAAS in controlling weeds. Exceptions to these reports was that of Ahmad et al. [21], who recently reported that the operational parameters, 2 m application height and 2 m s−<sup>1</sup> flight speed provided the highest average herbicide spray deposition on weed canopy.

Backpack sprayers are the preferred method for applying herbicides in small-farm operations, in small plot weed science research trials and in rangelands where selective application to patches of invasive species is required [22,23]. Research data comparing applications of pest control products made by backpack sprayers with spray drones are limited. Spray application rates for backpack sprayers are usually between 94 and 318 L·ha−<sup>1</sup> at 207 to 345 kPa pressure, with nozzle flow rates varying between 0.4 and 0.8 L·min−1. The walking speed of the operator is usually held at 1.4 m·s−<sup>1</sup> [22,24]. However, RPAAS vehicles are typically programmed to apply pesticides between 2 and 4 m application height and 1 and 7 m·s−<sup>1</sup> ground speed. The spray application rates usually vary between 19 and 38 L·ha−<sup>1</sup> [25]. These key differences can affect spray pattern uniformity, droplet spectra and application rates and efficacy of pest control products [22,26–28]. It is, therefore, essential to characterize and compare spray deposition and droplet spectra characteristics produced by these two delivery systems under field conditions. Such fundamental studies are required to assess whether RPAASs could be used in lieu of backpack sprayers for herbicide applications.

This research was designed to evaluate conventional ground and novel aerial spray technologies for herbicide applications against weed populations. This study was conducted in a soybean field pre-seeded with Palmer amaranth (*Amaranthus palmeri* S. Watson) and ivyleaf morningglory (*Ipomoea hederacea* (L.) Jacq.), summer annual and most problematic weeds [29–31].

#### **2. Materials and Methods**

The field experiments were conducted at Texas A&M research farm near College Station, TX, USA (30◦32 17 N; 96◦25 19 W). Four blocks, each 15 m wide × 12 m long, with 3 m between each block, were established with a 5 m strip of land earmarked at random for each treatment (Figure 1). The experimental units were assembled in a randomized complete block design with four replications to overcome heterogeneity in field conditions, relative to weed density and edaphic conditions between replicated blocks. Soybean was drill seeded at 320,000 seeds·ha−<sup>1</sup> with 76 cm row spacing on 15 May 2018. Weed seeds were broadcasted after soybean planting and were lightly incorporated into the soil. Weed density and size were recorded before spray application. Palmer amaranth and ivyleaf morningglory densities were 19 and 28 plants·m<sup>−</sup>2, respectively at the time of spray application.

The widest areas of the leaf blades of Palmer amaranth were ca. 7 cm long and 4 cm wide, while those of ivyleaf morning glory were ca. 9 cm long and 7 cm wide when the test was conducted.

**Figure 1.** Field plot layout of the study on Palmer amaranth and ivyleaf morningglory. Backpack spray application was evaluated only for the 6 m × 5 m sections of the plots indicated in blue.

Two spray treatments (18.7 and 37.4 L·ha<sup>−</sup>1) were applied with a RPAAS (model V6A, Homeland Surveillance and Electronics, Seattle, WA), and one treatment (140 L·ha−1) was applied with a custom-made CO2 backpack sprayer (Figure 2A,B), respectively, show V6A aircraft and backpack sprayer). These treatments are described in the text for brevity as acronyms, RPAAS-2, RPAAS-4, and BP-15, each representing spray application rates of 2, 4 and 15 gallons per acre or 18.7, 37.4 and 140 L·ha<sup>−</sup>1, respectively. This study was conducted on 27 June 2018 (Study 1) and was repeated in time and space two weeks later on 11 July 2018 (Study 2). The details about the spray treatment setups, operating parameters for the backpack sprayer and the RPAAS aircraft, including spray pressure, spray rates, nozzle type and orifice, aircraft ground speed, walking speed for the backpack sprayer and application height are provided in Table 1. Initially, four TTI110-015 nozzles were installed on the RPAAS. However, due to the low pump capacity, a pressure of only 345 kPa was achievable. For the RPAAS, the outboard nozzles were positioned 0.41 m away from the inboard nozzles, which were 0.82 m apart (Figure 3). Spray pattern testing was conducted with this setup according to the conventional pattern-testing technique described earlier [25]. Briefly, four spray passes were conducted with water and fluorescent dye and patterns from each of the passes were averaged to yield a representative spray pattern for the aircraft. The results showed a symmetrical pattern with an effective swath of 4.6 m (15 ft) (Figure 4). During the field study, only three of the four nozzles were operational (the left outboard nozzle was non-functional), likely due to the daylight visible dye used in the study for image analysis, which reduced system pressure. This pressure was not enough to open the check valves on all the nozzles. As a result, only three of the four nozzles were operational for Study 1, and a full spray pattern for each nozzle was not achieved. With an effective swath of 4.6 m and a ground speed of 4 m·s−<sup>1</sup> for three operational nozzles, the resulting spray application rate was 17.8 L·ha<sup>−</sup>1. Two of these passes would yield an application rate of 35.6 L·ha<sup>−</sup>1. For Study 2, only the two inboard nozzles were used to achieve a pressure of 414 kPa, which activated both nozzles and provided a full spray pattern for each nozzle. Spray pattern testing for this two-nozzle configuration was not conducted.


**Table 1.** Spray application setups for the backpack sprayer and the RPAAS aircraft in Study 1 and Study 2.

<sup>a</sup> TeeJet Technologies, Wheaton, Ill. <sup>b</sup> In Study 1, only three nozzles were operational for the RPAAS aircraft because of lack of adequate pressure due to large nozzle orifice size. In Study 2, the number of nozzles was reduced to two, which provided a full nozzle spray pattern at 414 kPa. The two nozzles used were the inboard nozzles which remained in their original positions. <sup>c</sup> This treatment was flown twice at this speed to achieve double the application rate. \* Please see Figure 2.

(**A**)

**Figure 2.** (**A**) V6A aircraft in flight. (**B**) The backpack sprayer with spray boom, nozzles, spray tank, and pressure gage.

**Figure 3.** Schematic of HSE V6A, showing rotor configurations and corresponding nozzle locations on spray boom. Circles represent rotor positions.

**Figure 4.** Pattern test results for RPAAS with the four-nozzle configuration. The top graph shows the pattern of individual passes. The bottom graph shows the average of the four passes. The table on the right shows the coefficient of variation (CV) at different swath widths. An effective swath of 4.6 m (15 ft) was chosen because it was the largest swath width with a CV < 15%.

The spray solution was comprised of a daylight visible fluorescent dye (Tintex Rocket Red, TX-13, DayGlo Color Corporation, Cleveland, OH, USA) at 10% *v*/*v* mixed with tap water. The fluorescent dye was used to quantify spray droplets on Palmer amaranth and morningglory leaves using the digital imaging technique documented previously [32,33].

#### *2.1. Sampling of Spray Deposition*

Water-sensitive paper (WSP) samplers (26 mm × 76 mm) (Spraying Systems, Wheaton, Ill.) were paper clipped to a metal plate (100 mm × 100 mm) and placed on a 0.3 m × 0.3 m wooden board. WSPs were oriented towards the upwind side of the metal plate to keep them flat and secured. There were four artificial samplers placed in each backpack treatment plot and five samplers in each RPAAS treatment plot. WSP samplers were diagonally placed in the test plots with the first sampling location 2 m in from the edge of each plot and subsequent locations 2 m farther down and 1 row over from the previous location.

After five minutes of drying time, WSPs were placed inside film negative sleeves. The spray droplets (Figure 5) captured on them were analyzed in the laboratory by the DropletScan™ scanner-based software system [34]. The droplet spectra parameters examined were Dv0.1, Dv0.5, Dv0.9, droplet density (droplets/cm2), percent area coverage and spray efficiency (proportion of spray relative to the target application rate). Dv0.1 is the droplet diameter (μm), where 10% of the spray volume is contained in droplets smaller than this value. Similarly, Dv0.5 and Dv0.9 are droplet diameters, where 50% and 90% of the spray volumes, respectively, are contained in droplets smaller than these values.

53\$\$6/āKD 53\$\$6/āKD

### %3/āKD

**Figure 5.** Spray droplet images captured on WSP samplers in each of the three treatments. The RPAAS at 18.7 L·ha<sup>−</sup>1, RPAAS-4 at 37.4 L·ha−<sup>1</sup> and Backpack Sprayer at 140 L·ha<sup>−</sup>1.

#### *2.2. Fluorescent Imaging*

Five leaves of Palmer amaranth and morningglory were collected from each replicated block close to where the wooden boards containing the WSP samplers were placed. A total of 20 leaf samples of each weed species were collected from each treatment for the RPAAS platform. However, four leaves of each species were collected for the BP-15 treatment, with a total of 16 leaf samples per species per treatment. Leaves were imaged in the laboratory using a digital single-lens reflex camera (Model Alpha 7R, Sony Corp., Tokyo, Japan), secured to an adjustable camera stand. The camera was equipped with a macro lens (Close-up + 4 Polaroid) and UV filter (49 mm) to help zoom in and obtain a closer view of the droplets, while maintaining high spatial resolution (*ca*. 30 μm). Each leaf was placed on an integrated platform at the base of the camera stand where the abaxial and adaxial surfaces of the leaves were imaged. A blue LED light at a wavelength of 470 nm (StellarNet Inc., Tampa, FL, USA) illuminated the droplets during the imaging process. After imaging, the photographs of both the top and bottom leaf surfaces were processed using ImageJ, a public domain, Java-based image processing software. The image processing procedure used in the study was similar to that described earlier by Martin [32]. However, some modifications were made to accommodate the larger droplet spectrum produced by the TT110-015 nozzles used in the study. Lab color space was used to detect the droplets, with the red threshold color chosen to align with the Rocket Red color of the fluorescent dye. In Lab color space, the 'L', 'a' and 'b' minimum and maximum values were set to 55 and 255, 179 and 205, and 52 and 115, respectively. Spray droplet particles were determined by setting the minimum and maximum pixel area size of the droplets between 10 and 4000 pixels. Circularity values were set between 0.00 and 1.0 to include all of the droplets, regardless of shape. The Show: Outlines option displayed the outlines of the individual droplets and the images were saved for the top and the bottom leaf surfaces. Figure 6 illustrates the enhanced (Figure 6A), selected (Figure 6B) and computer drawings (Figure 6C) of the image of spray droplets on the top surface of a Palmer amaranth leaf as an example.

(**A**) **Figure 6.** *Cont.*

(**B**)

(**C**)

**Figure 6.** Fluorescent images of a Palmer amaranth leaf processed by ImageJ software: enhanced image of spray droplets (**A**), selected image of spray droplets (**B**) and the computer drawing of spray droplets (**C**).

Thus, the fluorescent imaging provided data on the number of droplets found in each sample of leaves collected. Spray droplet density (droplets/cm2) on the adaxial and abaxial surfaces of the weed foliage was calculated using the area of each leaf determined by a leaf area meter (Model 3100, Li-Cor, Lincoln, NE, USA). Because the droplet density data were highly variable, the percentage of spray droplets on the adaxial and abaxial surfaces of the leaves was calculated for each sample collected from the designated locations in each treated block to compare the treatments using a common format.

#### *2.3. Data Analysis*

Spray deposits, as a percentage of the target application rate, were highly variable and comprised of values ranging from 0 to 100%. Because of the high variability of the data, they were transformed to arcsine √*p*, where *p* = original variates in proportions [35]. The spray droplet density of the fluorescent imaging data was transformed to log (*x* + 1), where *x* = the original variate. When the mean is positively correlated with the variance, the logarithmic transformation is likely to remedy the situation, and make the variance independent of the mean [36]. Figure 7 shows the functional

relationship between the variance and the mean, with an R<sup>2</sup> of 0.9033 (*p* < 0.0001; df = 14) and a highly significant slope coefficient (*b* = 70.7; *t* = 11.4; *p* < 0.0001), which indicates that the data do not meet the assumptions of the analysis of variance. The adequacy of the transformation in stabilizing the variance was achieved following transformation by calculating the correlation coefficient between the two parameters (Figure 8) as suggested by several researchers [36–39]. The coefficient of determination between the two parameters was 0.11, with a non-significant slope coefficient (*b* = 0.37; *t* = 1.32; *p* > 0.21). The transformed fluorescent droplet density data were used for statistical analysis. All other data were analyzed without transformation. The analysis of variance of the data was conducted using the Proc GLIMMIX procedure (SAS) and least square means were separated using the lines option at *p* < 0.05 when sample size was equal [40]. The replicated block effects were not significant for any of the data discussed in this study.

**Figure 7.** Relationship between the original variance and original mean of spray droplet density of the fluorescent imaging data.

**Figure 8.** Relationship between the transformed variance and transformed mean of spray droplet density of the fluorescent imaging data in log (*x* + 1) transformed scale, where *x* is the original variate.

#### **3. Results**

Meteorological data for each test/replication combination are presented in Table 2. Mean temperature and relative humidity remained relatively constant. Wind speed was variable; however, the angular deviation of the wind from row orientation was well within 40◦ used for studies designed to mitigate spray drift [41]. Moreover, each test plot was separated by 5 m from each other as shown in Figure 1 to mitigate the effect of cross winds that can move spray deposits between plots.


**Table 2.** Meteorological data during the test periods.

<sup>a</sup> Angular deviation of the wind from row orientation (215◦).

#### *3.1. Spray Droplet Spectra on WSPs*

The spray droplet characteristics revealed from the WSP samplers (Dv0.1 and Dv0.5) were not significantly different between treatments in Study 1 (Table 3). However, the Dv0.9 of the deposits was higher for the backpack sprayer than those for the RPAAS aircraft, as would be expected since the backpack sprayer was operated at a lower pressure. The spray droplet spectra (Dv0.1, Dv0.5 and Dv0.9) were not significantly different between treatments in Study 2, likely due to the small sample size collected. Numerically, the backpack sprayer again yielded a larger droplet spectrum. The spray droplet density (drops/cm2) and % area coverage were higher for the high-volume treatment (140 L·ha−1) applied with the backpack sprayer than those for the low-volume treatments (18.7 and 37.4 L·ha<sup>−</sup>1) applied with the RPAAS platform in both Study 1 and Study 2 (Tables <sup>3</sup> and 4).


**Table 3.** Spray droplet spectra parameters sampled by WSP collectors in Study 1 a.

<sup>a</sup> Least square means were separated using the lines option at *p* = 5%. Means followed by the same lower-case letter are not significantly different. ns = Not significant. \*, \*\* significant and highly significant, respectively. "a" and "b" represent means separation.


**Table 4.** Spray droplet parameters sampled by WSP collectors in Study 2 a.

<sup>a</sup> Least square means were separated using the lines option at *p* = 5%. Means followed by the same lower-case letter are not significantly different. ns = Not significant. \*\* significant and highly significant, respectively.

The spray application efficiency, which is the percent of spray deposits collected on artificial samplers relative to the theoretical application rate, in Study 1 and Study 2 is shown in Figures 9 and 10, respectively. The spray application efficiency for each treatment was the same for Study 1 (*F* = 1.94; *p* > 0.15; df = 2,49) as it was for Study 2 (*F* = 0.73; *p* > 0.48; df = 2,53).

**Figure 9.** Percent of spray deposits collected on WSP samplers relative to target application rate in Study 1. Data analysis was based upon arcsine transformation. Bars with original means ± SEM with the same lower-case letter are not significantly different (*p* > 0.05).

**Figure 10.** Percent of spray deposits collected on WSP samplers relative to target application rate in Study 2. Data analysis was based upon arcsine transformation. Bars with original means ± SEM with the same lower-case letter are not significantly different (*p* > 0.05).

#### *3.2. Fluorescent Droplets on Weed Leaves*

Droplet density, determined by fluorescent imaging on adaxial and abaxial surfaces of Palmer amaranth and ivyleaf morningglory leaves for the RPAAS platforms and the backpack sprayer in Studies 1 and 2, are shown in Tables 5 and 6, respectively. Transformed means are presented in original scale to maintain clarity. Droplet density varied between treatments and leaf surfaces for both Palmer amaranth and morningglory in Studies 1 and 2. Interactions between the treatments and the leaf surfaces were observed for both studies and for both weed species. Droplet density on the adaxial leaf surfaces of Palmer amaranth and morningglory leaves was higher for the BP-15 treatment than those for the RPAAS platforms in Studies 1 and 2. The abaxial leaf surfaces of both weed species had greater droplet density with RPAAS-4 compared to RPAAS-2 treatment and that the RPAAS-4 treatment resulted in greater droplet density on the abaxial leaf surfaces of morningglory than the BP-15 treatment. Droplet density test results for Study 2 were similar to Study 1 (Table 6).

**Table 5.** Fluorescent imaging of spray droplet density (drops/cm2) on the top and the bottom surface of weed leaves in Study 1.


Droplet density data were transformed to log (*x* + 1). Original means within each column followed by the same lower-case letters are not significantly different (*p* = 5%).

**Table 6.** Fluorescent imaging of spray droplet density (drops/cm2) on the top and the bottom surface of weed leaves in Study 2.


Data were transformed to log (*x* + 1). Original means within each column followed by the same lower-case letters are not significantly different (*p* = 5%).

Tables 7 and 8 describe the percent of spray droplets deposited in the adaxial and abaxial leaf surfaces of weed foliage in Study 1 and Study 2, respectively. In Study 1, the proportion of spray droplets on Palmer amaranth was significantly higher on the abaxial leaf surface when the spray rate was 37.4 L·ha−<sup>1</sup> (20.76%) than when the spray rate was 18.7 L·ha−<sup>1</sup> (9.30%) for the RPAAS platform. Likewise, the proportion of spray droplets on morningglory was significantly higher on the abaxial leaf surface when the spray rate was 37.4 L·ha−<sup>1</sup> (30.25%) than when the spray rate was 18.7 L·ha−<sup>1</sup> (9.50%). In Study 2, similar results were evident. The proportion of spray droplets on the abaxial leaf surface for the Palmer amaranth was 20.93% vs. 6.28%, while that for the morningglory was 27.54% vs. 12.47% for the 37.4 and 18.7 L·ha−<sup>1</sup> spray rates, respectively. The BP-15 treatment resulted in deposition of only 6% of the total droplets on abaxial surface (average of both weeds) in both studies. The RPAAS-4 treatment in both studies deposited approximately 25% of the total droplets on abaxial surface (average of both weeds), which was four times than that of BP-15 treatment (Tables 7 and 8).


**Table 7.** Fluorescent imaging of percent of spray droplets in the top and the bottom surface of weed leaves in Study 1.

Means within each column followed by the same lower-case letters are not significantly different (*p* = 5%).

**Table 8.** Fluorescent imaging of percent of spray droplets in the top and the bottom surface of weed leaves in Study 2.


Means within each column followed by the same lower-case letters are not significantly different (*p* = 5%).

#### **4. Discussion**

#### *4.1. Spray Droplet Spectra*

The percent of applied spray on WSPs did not significantly differ among treatments in either of the two studies. The total percent spray deposits on WSP samplers (mean ± SEM) were 31.7 ± 5.3, 40.3 ± 8.1 and 33.0 ± 2.3 for RPAAS-2, RPAAS-4, and the backpack treatments, respectively, in Study 1. The total spray deposits (mean ± SEM) were 42.4 ± 8.9, 59.8 ± 12.2 and 36.7 ± 9.1 for RPAAS-2, RPAAS-4, and the backpack treatments, respectively, in Study 2. Although these values were numerically different, statistical differences between treatments were not realized likely due to small sample size. However, test results indicate that although all of the spray application systems were equally effective in the delivery of spray deposits as sampled on artificial collectors, none of the systems were able to produce 100% of the targeted spray rates.

Our data agree with Wang et al. [42], who reported that the amount of imidacloprid (μg/cm2) deposited on wheat did not vary between an RPAAS and a conventional backpack sprayer. In another study on wheat, Wang et al. found that with higher spray volume (>18.8 L·ha−1) and with coarser nozzles, deposition of tebuconazole fungicide with a spray drone was similar to that of a conventional backpack sprayer, but the deposition was depressed at the lower spray application rate (<9.0 L·h<sup>−</sup>1) with fine nozzles. Hill and Inaba [43] studied the deposition of deltamethrin and permethrin formulations on WSP samplers and found a significant linear relationship between μg of the chemicals/WSP and spray droplets per cm2 with similar size droplet spectra. This indicates that droplet density from non-active spray applications should be a good indicator of active formulation spray deposits. Test results reported by Hill and Inaba and Wen et al. [44] indicate that the spray quality assessments with WSPs are reliable when using conventional hydraulic spray nozzles.

Despite the difference in operational parameters between the RPAAS platforms and the backpack sprayer, no significant difference in Dv0.5 between the RPAAS and the backpack treatments were observed. The operational parameters used in this study were closer to those used by Qin et al. [45], who reported testing a UAV flown at 3.5 m height and 4 m/s ground speed with a fungicide over a wheat canopy and compared that to a backpack sprayer operated at 0.5 m height and 0.8 to 1.0 m s−<sup>1</sup> ground speed. Although the spray volume (140 L·ha−1) applied with the backpack sprayer was 3.7 to 7.5 times greater than those of the RPAAS treatments, droplet size was not affected by the carrier volume. In a laboratory study, Creech et al. [26] evaluated the effects of nozzle type, orifice size, herbicide active ingredients, pressure, and carrier volume on the droplet spectra of the spray, and found that when averaged across all the experimental variables, the herbicide carrier volume had the least <sup>e</sup>ffect on Dv0.5 spray droplets. When the carrier volume increased from 47 to 187 L·ha<sup>−</sup>1, Dv0.5 increased only by 5%, indicating that the droplet size of the herbicides tested was not highly dependent on spray volume. Even though the 37.6 L·ha−<sup>1</sup> application rate was achieved by making two passes over the research plots, there is no expectation that this should change the resulting droplet spectrum.

#### *4.2. Fluorescent Droplets on Weed Leaves*

Data reported herein indicate that the backpack sprayer with a 140 L·ha−<sup>1</sup> spray volume produced only 6% of the total number of applied spray droplets on the abaxial surface, while the RPAAS aircraft with a 37.4 L·ha−<sup>1</sup> spray volume produced a 4-fold increase in the number of spray droplets on the abaxial surface. Thus, it is likely that this would help improve the efficacy of contact herbicides against weed foliage, because it increases the probability of a spray droplet occupying an active site on the under surface of the leaf. These data suggest that leaf fluttering caused by rotor downwash likely caused this increased spray deposition on the abaxial surfaces of Palmer amaranth and morningglory leaves.

The results reported here corroborate research data which indicate that downwash and wind turbulence created by rotor blades of RPAAS aircraft assist in droplet deposition and canopy penetration [46–52]. For instance, Qing et al. [48] studied the movement of spray plumes in the laboratory at different rotor speeds, using an 8-rotor RPAAS and found that the spray angle of the nozzles was reduced with an increase in the speed of the downwash flow and spray droplets tended to move towards the direction of the rotors. The reduction in nozzle spray angle is known to be highly correlated with an increase in air velocity past a nozzle. Songchao et al. [53] used a single rotor unmanned helicopter to study downwash distribution using a computational fluid dynamics model and found that the downwash covered an area equivalent to the rotor radius of 3.0 m in size with a boundary velocity of 0.5 m·s<sup>−</sup>1. This appears to indicate that the downwash effect extends beyond the immediate vicinity of the aircraft. Yang et al. [54] reported that the downwash airflow of the RPAAS rotor caused a pressure differential between the upper and lower surfaces of the leaf producing a torque and thus enabling the spray droplets to penetrate as much as 4-fold to the bottom surface of the leaf.

#### **5. Conclusions**

The spray application efficiency described as the percent of spray deposits collected on an artificial sampler for the RPAAS systems at the 18.7 and 37.4 L·ha−<sup>1</sup> spray rates was comparable to that for the backpack sprayer at a 140.0 L·ha−<sup>1</sup> spray rate. The spray droplets, Dv0.5, deposited on an artificial collector were statistically similar for both backpack and RPAAS treatments, although these spray platforms were operated at different heights and ground speeds. Test results suggest that if the nozzles were kept similar for the remotely piloted aerial and the conventional ground spray application systems, operational protocols (application height and ground speed) may not significantly influence the spray droplet spectra characteristics. Higher-volume treatment (140 L·ha−1) applied with the backpack sprayer resulted in greater fluorescent droplet density on the adaxial leaf surface compared to the lower-volume RPAAS treatments. However, the highest spray rate RPAAS platform (37.4 L·ha<sup>−</sup>1) resulted in the deposition of the largest proportion of droplets on the abaxial surface of weed foliage, relative to the total number of depositing droplets. This suggests that the rotor downwash and wind turbulence created by the RPAAS aircraft when operated at a ground speed of 4.0 m·s−<sup>1</sup> to achieve 37.4 L·ha−<sup>1</sup> spray application rate likely helped flip leaves over and enabled spray deposition onto the lower surface of weed canopy. It is important to note that the spray application rate (37.4 L·ha<sup>−</sup>1) was

achieved by flying the aircraft twice over the test plots at 4 m·s−<sup>1</sup> and this could have likely helped to increase spray deposition on the abaxial surfaces of weed leaves. However, if the 37.4 L·ha−<sup>1</sup> spray application rate was achieved with a single pass of the aircraft, it would have required a 2 m s−<sup>1</sup> ground speed, and the slower ground speed would have increased the residence time of the aircraft over the weed canopy, thus increasing the downwash effect of the rotors. Whether or not a single pass of the aircraft with a delivery volume at 37.4 L·ha−<sup>1</sup> would similarly increase under-leaf surface deposition remains conjectural and should be investigated. Test results suggest that RPAAS systems may be used for herbicide applications against post-emergence weeds in lieu of conventional backpack sprayers. Data reported here indicate that further research should be conducted to evaluate herbicidal efficacy of spray applications from the RPAAS platforms compared to backpack sprayers.

**Author Contributions:** Conceptualization, D.M., V.S. and M.B., methodology, D.M., V.S., post M.B.; formal analysis, M.A.L.; writing—original draft preparation, D.M. post M.A.L., writing—review and editing, M.A.L., D.M., post V.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Acknowledgments:** The authors would like to thank Al Nelson for allowing them to use the research plots at the Texas A&M University Research Farm.

**Conflicts of Interest:** The authors declare no conflict of interest. Disclaimer: The use of trade, firm, or corporation names in this publication is for the information and convenience of the reader. Such use does not constitute an official endorsement or approval by the United States Department of Agriculture or the Agricultural Research Service of any product or service to the exclusion of others that may be suitable.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

### *Article* **An Approach for Route Optimization in Applications of Precision Agriculture Using UAVs**

**Kshitij Srivastava 1, Prem Chandra Pandey 2,\* and Jyoti K. Sharma <sup>2</sup>**


Received: 23 July 2020; Accepted: 16 September 2020; Published: 18 September 2020

**Abstract:** This research paper focuses on providing an algorithm by which (Unmanned Aerial Vehicles) UAVs can be used to provide optimal routes for agricultural applications such as, fertilizers and pesticide spray, in crop fields. To utilize a minimum amount of inputs and complete the task without a revisit, one needs to employ optimized routes and optimal points of delivering the inputs required in precision agriculture (PA). First, stressed regions are identified using VegNet (Vegetative Network) software. Then, methods are applied for obtaining optimal routes and points for the spraying of inputs with an autonomous UAV for PA. This paper reports a unique and innovative technique to calculate the optimum location of spray points required for a particular stressed region. In this technique, the stressed regions are divided into many circular divisions with its center being a spray point of the stressed region. These circular divisions would ensure a more effective dispersion of the spray. Then an optimal path is found out which connects all the stressed regions and their spray points. The paper also describes the use of methods and algorithms including travelling salesman problem (TSP)-based route planning and a Voronoi diagram which allows applying precision agriculture techniques.

**Keywords:** precision agriculture; travelling salesman problem; UAV-based precision farming; Euclidean distance; Voronoi; site-specific fertilizer spray

#### **1. Introduction**

One of the main objectives of the United Nations is to have sustainable agriculture which can enhance agricultural productivity to meet the food supply-demand of the increasing population without overexploiting and wasting natural resources [1]. The estimated population is likely to reach nine billion from seven billion by 2050 [2,3]. Significant pressure is expected on agricultural systems to meet the continuously increasing needs of the population, thus escalating human pressures on the environment [3–6]. Farming is predominantly done on limited space or regions, which are decreasing day by day due to several factors, such as conversion to other land use categories, e.g., farmland to residential, commercial and industrial buildings [7,8]. With the limited land area available, agriculture will have more pressure for increased crop yield to enhance productivity as well as maintain the food quality to cope up with the increasing population's demands. Several parameters, such as soil moisture, nutrients, such as nitrogen, phosphorus, and potassium [9], water holding capacity, pH and other variables affect crop productivity and thus require continuous monitoring to prevent crop stress. At present, the identification process of stressed regions and preventive measures are performed manually, while the identification of stressed regions is assessed using visual image interpretation using remote sensing and GIS. Moreover, at most of the places, other agricultural applications, such as spraying of pesticides or fertilizers, are performed manually too. This leads to an increased cost and

inefficient use of resources, along with wastage of input resources. To maintain quality and enhance the productivity of crops, precision agriculture (PA) has to be adopted by farmers worldwide [10,11].

In recent decades, there has been an increase in the number of techniques and methods employed in agriculture to enhance crop yield and productivity. Pesticides have been used against insects and pests, while fertilizers are used to improve the fertility of the soil by adding major nutrients to enhance crop productivity [12,13]. Moreover, wastage of these agricultural inputs is more common during steps and processes which, overall, impact the expenses in agriculture. In recent years, there has been a gradual shift towards mechanization in agriculture for harvesting, spraying and drip irrigation which has helped to optimize the amount of water used and reduce the wastage.

In the present era, PA is a new concept that has been introduced in agricultural practices to improve agricultural crop yields and help in decision making using advanced geospatial tools and techniques for analysis [14,15]. This is used worldwide nowadays to reduce labor costs, minimize the time duration and assist in the proper management of fertilizers, pesticides and irrigation activities throughout the field in less time [16]. PA also helps in understanding the soil condition and its parameters over time and aims at addressing this soil spatial variability, but for these tasks, spectral images of the land are required [17]. Therefore, a new concept of PA has been introduced to tackle these problems, such as minimizing wastage of agricultural resources inputs and reducing labor time to increase the productivity of the crops [18]. PA is the science of improving crop yields with assisted management using high technology sensors and analysis tools. Advanced multispectral imaging equipment offers innovations in the practical application of PA techniques. This is useful for the valuation of crop stresses, quality of soils and the vegetative cover, as well as for yield estimation [19,20]. The technique also adopts a concept to minimize the wastages and effective management of fertilizers and pesticides for increased crop yields [18]. The everyday practices of agriculture can be observed by UAVs for soil monitoring, early weed detection, disease and pest control, nutrient assessment and fertilizer applications at different spatial scales [21–23]. These innovative technologies have been implemented and are in demand for new opportunities in PA by farmers to provide stressed maps, pest infestation maps, soil condition, disease control and yield maps [22–25].

The above concepts, when implemented in agricultural practices, undoubtedly result in increased agricultural productivity by protecting crops against pests, insects, disease and any stress. Concepts and methods to provide fertilizers at a particular stressed region effectively at appropriate times are also promoted. To realize this, spectral images and thermal images are obtained from the UAVs, which have a high spatial resolution compared to satellite images and are less susceptible to weather conditions [26–28]. These are useful for regional mapping and analysis where limited areas are being considered and are in focus.

Spectral and reflectance images from UAVs are incorporated for analysis and assessment in agricultural research and agricultural tasks, as well as decision making [29,30]. In 2008, Nebikera conceptualized the use of the spectral camera on drones for aerial imaging and analysis [31]. Research has demonstrated the usage of these spectral images to assess stressed regions as well as stressed crops based on different soil parameters, water content and plant chlorophyll contents [32]. This was recently demonstrated in our previous study [33] by the development of VegNet (Vegetative Network) software to assess crop stress in a large field by extracting different vegetative indices, such as NDVI (normalized difference vegetation index) [34], NDRE (normalized red difference index) [35], EVI (enhanced vegetation index) [36]. We also incorporated other reflectance indices, such as NDVI, TCARI/OSAVI (transformed chlorophyll absorption in reflectance index/optimized soil adjusted vegetation index), etc. These indices were derived from spectral images and demonstrated that they have a positive correlation to stress indicators (water or nutrients). Both TCARI/OSAVI are very sensitive to chlorophyll content variations which are useful to assess the stress condition of vegetation [37]. The spectral images were employed for the detection of stressed regions in the agricultural fields using the VegNet software application (please refer to [33]). VegNet is an application that was developed with the aim of providing the necessary tools to detect stressed crop locations using spectral images obtained from UAVs, and providing the

stressed crops' condition and location and the area covered by those stressed crops. It has been shown that VegNet has the flexibility to use any combination of reflectance indices, or even thermal indices, for detection of stressed regions.

In the past, UAVs have been used for monitoring agricultural fields with spectral images and then calculating the above-mentioned indices which indicate deficiencies in the field. Even the usage of UAVs in non-agricultural activities, such as monitoring natural resources, such as land, water and rivers [38–40], as well as in disaster management [41], has rapidly increased in the past few years. In 2015, Gao [26] demonstrated the use of UAVs for water stress assessment using thermal indices and canopy conductance measurements. Hence, both thermal imagery and spectral imagery obtained from UAVs providing reflectance indices and thermal indices have shown great potential to determine field stress heterogeneity. In 2019, authors developed an application based on a combination of reflectance indices to detect stressed regions in agricultural fields [33].

The soil parameters and use of spectral indicators for water content extraction, plant chlorophyll content and so on have been demonstrated in past research. This was done by extracting different vegetative indices, such as NDVI, NDRE and EVI. While these findings are very helpful in gaining new insights about a farm, these sophisticated vegetative indicators make little sense for a nonprofessional. Even if farmers get to know about the deficiency of nutrients in their crops and the locations of stressed regions, they will still be taking corrective steps manually, which is inefficient and time-consuming, hence optimized use of machinery is suggested. UAVs are also increasingly being used in PA activities, such as administering inputs, apart from monitoring. Thus, UAVs have been employed in supporting PA mapping to effectively manage and provide optimized inputs to agricultural fields, such as water, fertilizers and pesticides, to increase the quality and yield of crops [42,43]. There is still a lot of progress to be made in the way in which corrective measures are applied, such as spraying water, pesticides or fertilizers. Therefore, a system has to be brought to the forefront that addresses the technological gaps and applies the benefits of PA for increasing farm productivity.

Software and information technology (IT) solutions have been developed by researchers on a variety of aspects of precision farming [44–47]. Most of them have focused on PA by using it primarily for monitoring and analysis purposes and not focused on IT-based mechanization and automation [48]. One of the research papers has tried focusing on providing integrated UAV-based PA using an aerial farm robot that takes care of all agriculture-related activities, such as cropping, planting and fertilizing [49]. It used UAVs that were attached with detachable instruments for this purpose. In this system, they applied fertilizer to all parts of the field equally without any regard for spatial nutrient differences in the land. However, what was needed was for the fertilizer to be applied only on stressed regions and not to all the parts of the field, so there must be a feedback mechanism for nutrient deficient regions and application of fertilizer [48]. Analysis using spectral indices will help in identifying stressed regions and will help in site-specific input applications.

Recent studies have focused on studying the effects of several UAV design parameters on spraying outcomes. Qin et al. illustrated and discussed the influence of spraying parameters, such as the operation height and operation velocity of the UAV, on droplet deposition [50]. Studies also revealed spray factors, such as the speed and distance of UAVs for droplet size, distribution and distribution uniformity [51]. It studied factors, such as the speed and distance of UAVs for droplet size, distribution and distribution uniformity. Most of these papers have focused on the spraying while in motion. In contrast, the present paper suggests spraying fertilizers while the UAV is flying over the spray position. This method of spraying would reduce some of the factors associated with UAVs.

All the past research works have been done on spraying fertilizers and pesticides from UAVs on farms and stressed regions of particular shapes and sizes. This paper will open up the possibilities of working in farms and stressed regions of irregular shapes and sizes which were not addressed earlier. There are a few algorithms that can work on farms of all shapes and sizes, and our algorithm is one among them. One of the early research papers used multiple drones on a full field where the drones were spraying parallel to one another in various rows of the farm [52]. However, the method proposed was not site-specific and no algorithm was proposed for traversing the field. It required the crops to be planted in rows which are to be fed into the system for spraying. In 2013, a unique approach was given where centroidal Voronoi tessellation was used to spread swarm UAVs to cover an infected region. It claimed that the most effective solution, then, is to attack around the infection and contain the situation before fully eradicating the infection. Voronoi divided the region into many parts and drones started spraying from each Voronoi centroid [53]. It could attend to any stressed region, ensure to limit the amount of spraying over healthy plants and avoid overlapping of pesticide spraying. Nevertheless, this method required many UAVs for attending to a single stressed region, making it very resource-intensive. In 2017, another article looked at using UAVs to find stressed regions and then spraying on the stressed patches using a tractor with many spraying nozzles that could be turned on and off depending upon the patches [54]. However, the authors failed to provide an algorithm for spraying on the stressed regions. In 2019, another study looked into site-specific management during the vineyard spraying process. Here, the authors also used UAVs to find stressed regions. They divided the stressed regions into three different zones indicating different levels of canopy vigor. The application map was divided into small and irregular rectangles. The width of the rectangle was decided such that it corresponded to the working width of the sprayer [55]. However, the problem with this approach is that this division of stressed regions is done in the form of a rectangular shape, which is not the most appropriate approach since the spraying happens over a circular area. Hence, rectangular divisions should be replaced with circular divisions to ensure more effective dispersion. This paper combines all the deficiencies present in earlier research to come out with an algorithm for proper spraying of inputs by drones. It provides continuous feedback to the system to allow for spraying of fertilizers on all shapes of stressed regions. Finally, it brings out an optimized route covering all the stressed regions and spray points.

Nevertheless, so far, these UAVs may not have been following optimum paths and routes to deliver inputs in large agricultural fields, wasting time by visiting the same place repeatedly [42,43]. In the past, researchers have addressed this problem by initiating research on route planning for vehicles delivering resources [56] and also for energy saving schemes in machinery utilized in agricultural systems [57]. Cabreira et al. (2019) and Galceran et al. (2013) reviewed the important path planning algorithms used in UAVs [58,59]. Most of the algorithms, such as back-and-forth coverage path planning algorithms, were suitable for continuous spraying in a region. These algorithms used turning maneuvers which increased the time and energy spent at corners, thus giving irregular spraying at the corners of the regions [59]. Other algorithms used were grid-based algorithms, such as the Wavefront algorithm, which utilized regions made of grids for making paths [59]. But grid-based or rectangular divisions-based algorithms are not efficient for spraying as explained in the previous paragraphs. Therefore, to avoid wastage of resources, such as fertilizers or pesticides, by spraying the same places, the paths of UAV movement need to be controlled. Therefore, an optimal path and route should be devised which will allow UAVs to visit the identified region and spray points only once in their trip, thus reducing time and resources [60,61].

This paper removes all the problems associated with the previous research to come out with an algorithm for proper spraying of inputs by drones. The present study attempts to implement an optimal path and route to ensure the effective spraying of fertilizers and pesticides at the identified stressed regions using the optimal path and route quickly, for effective PA. The UAVs will be carrying the fertilizer spray or pesticide spray as per user needs or requirements with shortest optimum trip length covering all points with only a single visit by the UAVs. This article also demonstrates some practical experiences of a travelling salesman problem (TSP), Euclidean distance metrics and a Voronoi diagram, which have been applied to provide optimal paths, routes and points for spraying, using UAVs to deliver effective inputs to agricultural activities to increase crop yields and save time as well as resources from being wasted.

In the present work, Section 2 describes the study area briefly, while Section 3 deals with the materials and methodology adopted in the present study. The various shortcomings of previous research as well as many novel methods have been suggested which are significant improvements over past research. In Section 3, the paper discusses, in detail, the proposed algorithm used in this paper. In Section 4, the results are discussed, which were obtained with the proposed algorithm. In Section 5, we brief about the challenges faced with the proposed method, and in Section 6, we discuss future improvements that could be possible. At the end, the paper is concluded by recommending some ideas for achieving higher crop productivity.

#### **2. Study Area**

The study area chosen for the present study was an agrarian region located around Devagiri village in Haveri Tehsil, Haveri district, located in the state of Karnataka, India. The district has an area of 4851.56 square km and it lies between the latitudinal coordinates of 14.19–15.09◦ N and longitudinal coordinates of 75.01–75.50◦ E [33]. Agricultural activities dominate in part of the study area. Maize, paddy, sorghum, and chili are the major crops cultivated in this region. The total land under irrigation in the selected region in Haveri Tehsil is 86.2 km2, including all varieties of horticulture, vegetable, spice and plantation crops. Spectral images were acquired using UAVs covering the test site (with 490 sq. m), spanning across a length of 3 km in multiple drone flights. These images were acquired in the month of October in the year 2018.

#### **3. Materials and Methods**

This section provides the specifications of the UAVs and methodology adopted in the present study. Figure 1 illustrates the schematic diagram with the overall methodology adopted in the present study.

**Figure 1.** Schematic diagram illustrating the overall methodology adopted in the present study.

#### *3.1. Specifications of UAV*

The specifications of the UAV employed for capturing spectral images are discussed in this section and a detailed specification is illustrated in Table 1. A quadcopter UAV (as shown in Figure 2) was used for this study. It has four brushless direct current (BLDC) motors [62] with carbon fiber propellers and is powered by 10,000 mAh lithium-ion batteries. BLDC motors have the ability to deliver precise control, therefore making them suitable for flying drones. The autonomous flight path of the UAV was designed using mission planner software, and the flight parameters, points and paths were configured in such a way to ensure proper capturing of spectral images for the whole field [63]. Therefore, the necessary instructions were fed into the system to configure the autopilot system. The UAV consists of a Pixhawk PX4 flight controller (FC) having components attached, such as a power system, a global positioning system (GPS), an external compass, a radio control system, etc. The flight controller's function is to generate a control signal for each motor in response to a user's control input [64,65]. It uses sensors to determine vehicle state, which is needed for stabilization, and to enable autonomous control. The vehicle state includes its position, altitude, direction, speed, airspeed, rates of rotation in different directions and battery level of the UAV. The UAV was controlled using a radio communication (RC) controller [66]. This RC controller transmitted control signals to the receiver on the UAV. A GPS sensor was integrated into this flight controller so that the UAV could know the exact locations [67]. A light detection and ranging (LIDAR) sensor was integrated into the UAV so that it could estimate the height of flight [68].


**Table 1.** Specifications of the UAV employed in the present study.

**Figure 2.** Illustration of the UAV employed in the study.

A multispectral camera was mounted on the UAV for extraction of multi-spectral band imagery of the agricultural fields. The images were captured in sunny and cloud-free conditions at a height of 121 m from the ground level for a flight duration ranging from 27 to 30 min. An overlap triggering mode was used for capturing the images and the overlap percentage was set at 70% between images taken at a speed of 15 m/s and giving a resolution of 8 cm per pixel [69]. The camera had a downwelling light sensor (DLS) which measured the ambient light during flight and exposure settings were automatically optimized for each capture to prevent blurring or over-exposure [70].

#### *3.2. Algorithm*

This paper presents an algorithm which has been designed for automated spraying on stressed regions with fertilizers, and pesticides. The steps followed in the methodology are presented briefly here and discussed in the later section.


The next step of the process involved the identification of stressed and unhealthy regions from the spectral images. As discussed in the previous sections, we used the VegNet software to identify the stressed regions [33]. These stressed regions indicated some sort of deficiencies which could be reduced with the help of appropriate fertilizers. The proposed method of spraying with UAVs took into consideration that an equal quantity of spray would be administered irrespective of the degree of

deficiency. To decide the equipment to be used for spraying, parameters such as the radius and area to be covered in single spray and the type and quantity of fertilizer to be sprayed were considered.

When the thresholds were identified and applied to the spectral image, the stressed areas were obtained using VegNet software application. These stressed regions were small regions in the farm separated from each other by some distance. These were then extracted to get the individual stressed regions. The flood filling method is one of two popular methods for extraction of individual regions in a binary image. In this method, a region is filled in all directions starting from a single point within the region. This method searches for an unlabeled foreground pixel, labels it and marks it "visited" to all the neighboring pixels in the region [71,72].

In this method, the 8-connected component was used as a metric for the identification of connected image regions. Hence, by employing the flood filling algorithm, each stressed region could be obtained. Therefore, later on, the centroid of each stressed region could be obtained from these stressed regions.

The next procedure involved finding the shortest path from the starting point and traversing through each stressed region's centroid. Therefore, this method found the shortest path through each stressed region and then, for each stressed region, the shortest path for traversing each spray region was determined. The problem of finding the shortest path through each of these points is called the TSP. The travelling salesman is a classical problem in computer science and operations research. It can be described as a graph with N nodes. All the nodes are connected to each other with an edge that has a corresponding weight and cost attached to it. The cost describes how "difficult" it is to traverse this edge on the graph. The objective of the salesman is to visit all the N cities (nodes) by visiting each node only once, finishing where he started and keeping the traversal costs as low as possible. Since there are N factorial combinations for N nodes, it is impossible to go through all the possible combinations if N is large; therefore, some scientific methods have been formed for solving the problem. Some of the popular ones are the Branch and Bound algorithm [73] and local search algorithms, such as the 2-opt algorithm [74]. In 1975, Chisman used a variation of the travelling salesman problem called the clustered travelling salesman problem [75]. Here, a group of nodes (clusters) must be visited contiguously in an optimal order. There exist several sets of clusters within the problem. This method was developed for optimizing, simultaneously, the ordering of nodes within each cluster and the ordering of clusters. Therefore, in our problem, the stressed regions act as clusters while the spray points inside them are the nodes of each cluster.

In order to find the spray points for each stressed region, their boundary was calculated. This was performed with the help of mathematical morphological operations. The boundaries of the stressed regions inside spectral image A can be calculated as A—(A Θ B) where B is a 3 × 3 square structuring element. Here (A Θ B) denotes erosion of A by B where erosion is another mathematical morphological operation [76,77]. Figure 4 shows the boundary of the stressed region identified in the study site.

The boundary points obtained from each stressed region were used to find a convex hull for the points. A convex hull is the smallest convex set that contains a set of points enclosed within it. Figure 7 shows the boundary as well as the corresponding convex hull of the stressed region. A convex hull is a primary requirement for implementing the algorithm proposed in this paper. The aim of the convex hull is to make the structure of the stressed region into the shape of a polygon that resembles the original shape while containing all the stressed regions inside of it. The convex polygon shape would help in the storage, manipulation and analysis of these stressed regions as opposed to earlier work with images. There are many algorithms called convex hull algorithms that are used to achieve the convex hull. Some of the popular ones are the Graham scan algorithm proposed by [78] and the gift-wrapping algorithm developed by Jarvis [79]. The algorithm given below is the Graham scan algorithm.

#### *3.3. Graham Scan Algorithm*

In Algorithm 1, ccw (counterclockwise) is a function which denotes the counterclockwise rotation. ccw > 0 if three points make a counterclockwise turn clockwise if ccw < 0, and collinear if ccw = 0.

The method utilizes the unique approach of covering each region with circles, with the idea borrowed from the covering of convex regions by base stations towers used in telecommunications [80]. This idea was introduced in the context of mobile communication for efficient base station placement problems in a convex region such that each point in the convex region is covered by at least one base station of equal radius. The present research is based on the above idea, with the convex region being the stressed region and the base station being the location where the fertilizer has to be applied, optimizing the base station location for the k points [80].



Since finding the number of points for spreading is also a task, this method was applied iteratively and the points were increased one by one till almost all of the convex region was covered and the maximum circumscribing radius was obtained, approximately equal to the radius chosen for the spray. The algorithm initially started with a rough estimate of the number of points needed to cover the convex region, i.e., equal to the total area of the convex region divided by the area of the spray. Then, the number of points was increased iteratively and optimized until most of the stressed area was covered by spray, i.e., 97% of stressed region in our case, and the maximum radius of the enclosed circle was less than or equal to the radius of the spray. We chose 97% in our case because we wanted to find a balance between the minimum number of circles required to cover a region and to cover the maximum amount of area. The number of points, as well as their location, was the answer which was finally obtained. Table 2 shows the number of spray points and area of the stressed region not covered by spray in the selected region after completion of optimization processes. This table shows that eight points were selected, covering more than 97% of the stressed region and had a maximum circumscribing radius equal to the spray radius. Thus, eight points and their optimized locations became the answer for that particular stressed region. These points were further used for route planning, See Algorithm 2.

**Algorithm 2.** Function find\_optimum\_points(convex\_region, spray\_radius)

*Number of points (N)* = *floor(area of convex region*/*pi\* spray\_radius \* spray\_radius) while True: Points (P)* = *Find N random points inside the convex region Points (P), max\_radius* = *optimum\_location\_algorithm(convex\_region, points, spray\_radius) If ((area\_covered\_by\_points(convex\_region, spray\_radius)* > *97%) and (max\_radius* =< *spray\_radius)) return Points (P) Else: Number of points (N)* = *(N)* + *1 End*


**Table 2.** Number of spray points in the selected region and area of stressed region not covered by spray after completing the optimization process.

#### *3.4. Voronoi Diagram*

The fundamental data structure used in this method is the Voronoi diagram [81] of the point set P, (P is the list of points in the convex region), denoted by VOR(p) for formulating the update mechanism of the members in P to achieve optimum placement. VOR(p) divides the convex region Π into n disjoint convex polygon region such that:


In this paper, we consider vor(qi) as a closed convex region for each point. So, if a part of the region is outside the convex region then vor(qi)∩Π is used as vor(qi).

The way to optimize the points in the region is to find the positions of all the points P inside the region such that the maximum range required is as quickly as possible. The algorithm is iterative, so it perturbed the points P until it finally attained a local minimum. This algorithm is also called the Voronoi iteration algorithm or Lloyd's algorithm [82]. At each iteration, a circumscribing circle (Ci) for each vor(pi) was calculated using the algorithm [83]. Ri became the radius of the circumscribing circle (Ci). In order to cover a convex polygon by a spray with a minimal range, the spray should be placed at the center of the circumscribed polygon of the convex region and assigned a range of spray equal to the radius of the circle. As such, the maximum radius was calculated for all the circumscribing circles, which was equal to ρ. This maximum radius was minimized until the circle radius matched the spray radius and the number of iterations did not exceed the limit, i.e., cross the particular value of 40 in our case. We choose 40 in this case because most of the optimizations take place in the first few iterations; after that it converges to another value. The last iterations would lead to a very small decrease in the maximum radius, which is the value that we were optimizing (see Table 3). Therefore, 40 iterations ensured that only major iterations took place in the algorithm and reduced the time for running further optimization processes.

**Table 3.** Radius of the maximum circumscribing circle for all the iterative optimization steps when N = 8 points were selected for covering a stressed region.


The number of points was increased iteratively until most of the stressed area was covered with a spray, i.e., 97% of stressed region in our case, and the maximum radius of the enclosed circle was less than or approximately equal to the radius of the spray. If these spray point positions covered more than 97% of the area and had a maximum radius which was less than or equal to the spray radius, then these points became the spray points of the stressed region.

#### *3.5. Voronoi Iteration Algorithm or Lloyd's Algorithm*

Lloyd's algorithm starts with an initial placement of some number k of points. It then repeatedly executes the following relaxation steps:


This relaxation step terminated when the new set of points met some convergence criteria. Algorithm 3 was very crucial in obtaining the optimum location of these points so that the spray radius would become equal to the maximum circumscribing radius of the region.

**Algorithm 3.** Function optimum\_location\_algorithm(convex\_region, points, spray\_radius)

```
set of points P = -
                   p1, p2, .... pk  inside the convex polygon.
iter_count = 0
ρ (maximum radius) = 0
while iter_count < 40 and ρ > spray_radius
Find the voronoi diagram for the points P
Compute the circumscribing circle Ci f or each vor(pi)
ri be the radius of Ci
Move p i to the center of Ci and assign range ri o to it
ρ = = max{ri , i = 1, 2 ... }
iter_count += 1
return Points (P), (maximum radius) ρ
End
```
After finding the locations for the application of fertilizer for a particular stressed region, the TSP-based algorithm was, again, applied to obtain the shortest path traversing all the points. When the drone had traversed all the points in the stressed region, it then moved on to the next stressed region. Figure 9 shows the stressed region along with spray points and spray regions in the stressed region. Figure 11 shows the optimized path through all the stress points in the region.

Most drones today are configured with mission planner software which can be used to create automated missions. This is done by programming the micro-controller manually or programmatically to automated paths using the input of latitude, longitude, altitude and time delays. This automated mission ensures that these drones would not have to be controlled manually thus removing any risk of a crash or manual error. Ardu Pilot's Mission Planner software has the ability to create automated paths using Python scripts [84].

#### **4. Results and Discussion**

This section discusses the outcomes derived from the paper, i.e., finding the optimal points for spraying stressed regions and the optimal path to follow while covering all possible points in the field without re-visiting single points. For assessing the stress regions, a combination of spectral indices or thermal indices were obtained as per user needs and requirements, or the user could employ any techniques or methods to assess the stressed regions. Results will demonstrate the advantages of using TSP-based algorithms using UAVs within agricultural plots to reduce associated costs with labor and

fertilizer spraying, thereby realizing the goal of this study for optimal route to visit each stressed point only once, and cover the entire agricultural area in a minimum time interval.

As discussed in previous sections, VegNet software helped in providing the stressed location and regions of stress in the farm holdings (please refer to [33]). Figure 3 illustrates the stressed regions identified in the agricultural regions, highlighted with yellow color, and the blue color represents the non-stressed regions. Figure 3a represents whole agricultural fields while Figure 3b illustrates a very small part of the agricultural holdings. We thus had this small piece of agriculture field to which we applied the algorithm present in this paper. Figure 4 shows the optimum route to be taken to reach each stressed region, which was obtained after applying an algorithm for solving this problem of clustered TSP. Figure 4 demonstrates the application of the (Travelling salesman) TS algorithm on the stressed regions in the agricultural field in order to find the optimum path to each stressed region. Each stressed region is illustrated with a different color and the blue line shows the optimal path to the centroid of each stressed region.

**Figure 3.** (**a**) Stressed regions in the whole agricultural field and (**b**) a selection of stressed regions for a small region highlighted in yellow.

Using flood filling algorithms, we selected stressed regions from the cluster of stressed regions, as shown in Figure 5. Thereafter, the boundary of each stressed region was delineated from the selected region. The delineated boundaries of the stressed regions can be seen in Figure 6. Thereafter comes the role and importance of the convex hull, which uses the delineated boundaries of the stressed regions in order to get a polygon that resembles their original shapes. The convex shape helps in the storage, manipulation and analysis of stressed regions which is a primary requirement for implementing the proposed algorithm. A typical convex hull applied on the delineated boundary of the stressed region can be seen in Figure 7.

The algorithm then estimated the minimum number of circles required for covering the stressed region using the stressed region's area and the area of spray region. Let this number be N. Then, N random points were found in the stressed region. For these random points, a Voronoi diagram was made and the corresponding circumscribing circles (for each region of the Voronoi diagram) were obtained. Then began the iterative optimization process where the locations of the points kept on updating so that a minimum number of sprays was required to cover a stressed region. The number of points was increased iteratively until most of the stressed area was covered with spray, i.e., 97% of the stressed region, and the maximum radius of the enclosed circle was less than or approximately equal to the radius of the spray. If these spray points positions covered more than 97% of the area and had a

maximum radius which was less than or equal to the spray radius, then these points became the spray points of the stressed region.

**Figure 4.** Applying travelling salesman (TS) on the stressed region, with different stressed regions highlighted by different colors.

**Figure 5.** A stressed region selected from the cluster of stressed regions (as shown in Figure 3a,b).

Table 2 shows the number of spray points and area of the stressed region not covered by spray in the selected region after the optimization process. This table shows that as the number of points was increased, the percentage of stressed region area not covered by the spray region decreased. When eight points were selected, more than 97% of the stressed area was covered and had a maximum radius less than or equal to the spray radius. If the number of points was increased after this, then there was a very marginal decrease in the percentage of stressed region area not covered by the spray region. On the other hand, there would be a large requirement of spray liquid which would lead to overuse of fertilizer and pesticides. Table 3 shows the optimization steps when eight spray points were

selected. The maximum circumcircle radius kept on decreasing in each step until there was very little change in the last few steps where the algorithm ended. Figure 8 shows the Voronoi diagram and the corresponding circumscribing circles after optimization, where different regions of Voronoi are shown with different colors. Here, the maximum radius of the circumscribing circle was 5.046 units, i.e., 40.4 cm, which is shown in the last step of optimization in Table 3. The maximum radius of the circumscribing circle, i.e., 40.4 cm, was almost equal to the spray radius from the UAVs, i.e., 40 cm (1 unit = 1 pixel = 8 cm).

**Figure 6.** Boundary of the stressed region.

**Figure 7.** Convex hull on the boundary of stressed region.

**Figure 8.** Voronoi diagram and the corresponding circumscribing circles for Voronoi regions at an iterative optimization step.

This optimization provides optimal points in the stressed regions which uses these points to provide the optimal path and route to deliver services in the fields at appropriate points with accuracy. Thus, optimal points are required to ensure the route of the UAVs to reach at each point without re-visiting the same place in its complete trip. In our study, optimum route was identified using TSP based algorithm and Voronoi Diagram which has been illustrated in Figure 9. Thus, Figure 9 shows the coverage of the stressed crops with a fixed amount of radius i.e., radius of the spray from UAVs. In our case study, the optimal route has been shown in the application of UAVs for spraying the fertilizer at each point and spray radius of fertilizer by UAVs equal to 0.4 m (as shown in Figure 9). Here, all the circles will be of the same radius. We have also provided the different stressed regions in Figure 10a and illustrated how Drone is covering all the three types of stressed regions along with overlap areas. Table 4 provides the information about the overlap regions for these three stressed regions (based on the area covered by stressed regions). Therefore, this optimal points and optimal path route will use the shortest possible route to cover the agricultural fields completely in possibly less time frame. The advantages of this study is that it will provide the coverage of entire field in a minimum period with full coverage without re-visiting. Figure 11 indicates the optimal route through all spray points in the stressed region with a fixed spray radius of 40 cm. Hence, the optimised path is derived through all the spray points generated using TSP based algorithms.

A few important studies can be highlighted here that briefly demonstrated the spraying of fertilizers and pesticides on farms of a particular shape and size, which were easy to maneuver, and design algorithms. In 2013, one article demonstrated the optimization of maneuvering near boundaries and the loading and unloading of inputs of an agricultural machine [85]. The method suggested works only for specific field geometries with all parts of the field being sprayed irrespective of whether an area is stressed or not. Furthermore, previous papers divided the stressed regions into various small rectangles for spraying, but this shape is not the most appropriate one for spraying since the spraying happens over a circular area. Therefore, a circular shape was chosen for dividing the stressed region in this study. In comparison, some authors demonstrated easier methods of maneuvering with the help of UAVs. Similarly, Cabreira et al. [58] discussed coverage path planning to cover every region of interest considering the different area shapes [58]. Some others discussed algorithms, such as back-and-forth coverage path planning algorithms, which are suitable for continuous spraying in a region. These algorithms used turning maneuvers which increased the time and energy spent at corners, thus giving irregular spraying at the corners of the regions. One article discussed the development of a coverage trajectory with a minimal required time for UAVs for better navigation for

any specific task [60]. Here, the authors used rectangular grids to divide a stressed region and then used path planning algorithms to find an optimized path. However, it is not the best idea to divide a stressed region which has a very irregular shape into rectangular grids. This is because the grids might contain areas which do not require these inputs which leads to unnecessary spraying. These grid-based spray systems would not be optimal as applying a circular spray motion on a rectangular cell is not preferred. This may lead to missed spraying or over-spraying in some areas.

**Figure 9.** Fertilizer spray with radius of 0.40 m on the spray points in the stressed region (here, all the circles are of the same radius).

**Figure 10.** (**a**) Stressed regions in test site with different area marked as region I, II and III (based on the area covered) and (**b**) Path covered for all the stressed regions and their respective spray points (Total Distance covered = 227.73 × 0.08 m = 18.2984 m).

**Table 4.** Statistics for the overlap and area covered for the test site during the optimal route and field coverage (area for overlap for stressed wise percentage).


**Figure 11.** Optimized path through all the spray points in the stressed region.

This paper will help to spray inputs only on the stressed area where it is required and where there is nutrient deficiency. We also suggest spraying inputs while the UAV is flying still over the spray position. This method of spraying would reduce some of the spraying factors associated with UAVs. We have tried to combine all the shortcomings from earlier research to come out with an algorithm for proper spraying of pesticides by drones which provides continuous feedback to the system for allowing spraying of fertilizers on all shapes of stressed regions. Finally, the algorithm outputs an optimized route covering all the stressed regions and spray points. The tasks of administering pesticides and fertilizer, when done by traditional means, take a lot of time and are not efficient processes. However, by using the proposed method, the task of administering fertilizer and pesticide can be completed in a shorter time frame, with optimum resources and with a high level of accuracy.

#### **5. Challenges**

This section deals with the challenges and hurdles associated with the UAVs and discusses the future work scope, conclusion and recommendations. There are numerous challenges associated with UAVs which will be the main focus before delivering the application part for the optimal path and route. The first challenge is the usage of drones as they are more difficult to control as they move rapidly in all possible directions, and they can be difficult to control in adverse weather conditions, such as rain or high wind turbulence. However, most drones now come with automated navigation software that enables autonomous flight, requiring no input from the user apart from route instructions. The UAV's size and weight are also challenges to the user, along with their operational flight time, which depend upon the battery being used. Drones have a limited battery capacity; therefore, they have a short time flight. To compensate for this, larger batteries can be used which makes them more expensive. UAVs also cannot carry a heavy load; therefore, they would need to be refilled with inputs and their batteries recharged before flying again. Therefore, the resupply trips should be incorporated into the route planning, depending on the type of input being administered and size of the field. Drones also require a large memory for storing and processing high resolution spectral images. Drone operations are heavily regulated by most governments and require a license to be operated in certain regions and places due to security and safety issues. Along with challenges in the hardware, there exist some challenges with the software too, despite the presence of a large number of open-source libraries for the implementation of software and availability of affordable graphical processing units (GPUs). Irrespective of the above-mentioned challenges, UAVs are widely used by overcoming these challenges, somehow, either by using a short flight duration, low spectral resolution data (not acquiring hyperspectral data) or by spending money on expensive batteries to operate over large land holdings.

#### **6. Future Work**

This section deals with future research directions. In this paper, a simple method of fertilizer and pesticide application by drone has been discussed which automates the process of identification and application of fertilizer automatically, without any need for human decision-making and manual labor. This paper used only one of the many vegetative indexes available in the research literature for finding stressed regions with a deficiency of a particular nutrient. The future work of this paper could be the integration of various kinds of vegetative indices which indicate various kinds of nutrient deficiencies. These deficiencies could be used to predict the overall health of the crop. This paper proposes a method for spraying only one fertilizer at a time, but the method could be developed so that multiple fertilizers and pesticides could be sprayed onto the stressed region, thus improving the process efficiency and reducing the cost. This would result in a reduction in the number of steps which need to be taken to find and rectify each deficient nutrient, while a centralized system could be developed to address all the deficiencies with a single spray of fertilizers/pesticides. As agricultural machines spend a significant part of their time on non-productive operations, with more time spent on turns and repetitions, the technique described will help to reduce these non-productive operations.

This paper focused on administering an equal amount of fertilizer to the stressed region irrespective of the degree of nutrient deficiency. However, ideally, there should be a focus on applying the fertilizer based on the degree of fertilizer and nutrient deficiency. Therefore, a novel method will be much needed in the future which can administer fertilizer based on the degree of deficiency of fertilizers in respect to volume requirement. This method could lead to the development of more sophisticated software, with which farmers can estimate their total revenue and losses due to the stressed regions. This could be done by integrating various kinds of real-time global prices and real-time sensor data with the yield prediction data obtained through the drone. Another future scope is the usage of artificial intelligence which is being used in a wide range of complex tasks, ranging from speech processing to self-driving cars and, more recently, in geospatial applications (remote sensing and geographical information systems). Many artificial intelligence applications are currently being researched and developed to use hyperspectral data for prediction of the yield and health of crops. These data could be used for PA activities and would help in increasing the agricultural crop yields and income. Autonomous UAVs have great potential to exploit agricultural applications to improve and enhance crop yields and productivity, with improved accuracy in spraying route and higher time efficiency, as well proper coverage. This applicability can be applied to other research domains as well, such as forestry, ocean and defense. Another suitable application could be the use of this method for extinguishing forest fires that could be detected by UAVs. This method can help to timely extinguish fires and prevent them from spreading using optimal route calculation and maintaining the route of interest by not deviating from

optimum path. With the more advanced development of UAVs which are capable of lifting higher loads, this method can be used for precision irrigation so that the optimum amount of water can be used.

#### **7. Conclusions and Recommendations**

The paper discussed the results, challenges and future scopes of the methodology for using UAVs in precision agriculture. The results demonstrated the advantages of using TSP algorithms for UAVs within agricultural plots to reduce associated costs with labor and fertilizer spraying. This paper described a method by which automated UAVs can be used efficiently in providing inputs, such as pesticides and fertilizers, for precision agriculture. The method uses optimal points and route to provide these inputs to the stressed regions of the farm, thus covering the entire agricultural area using optimal amounts of inputs over a minimal distance and in minimum time. These optimal outcomes are generated from TSP-based algorithms and Voronoi diagrams implemented on spectral images acquired through UAVs. The spectral images are employed to locate and assess the stressed regions in large landholdings, using a spectral indices-based application (for details, refer to [33]). A few recommendations for future work using UAVs for PA applications that can be used by farmers to enhance crop yields (prevent crop damage) have also been given. Some of the recommendation points are listed below:


Implementation of the points outlined above would greatly improve the productivity of large landholdings and farms while decreasing the need for manual labor and repetitive concerns for stressed point location through remote sensing application instead of knowledge-based field visiting information. In this paper, we concluded with a key recommendation to employ UAVs to find an optimized route to completely cover stressed areas efficiently and without wastage of input resources during the spraying process which is tackled with the TSP solving technique. However, there is a need for evolving much more efficient algorithms for a true sense of PA in a sustainable way to meet the future requirements of PA.

**Author Contributions:** Conceptualization, P.C.P.; Data curation, K.S.; Methodology, K.S.; Project administration, P.C.P.; Resources, K.S. and P.C.P.; Software, K.S.; Supervision, P.C.P. and J.K.S.; Validation, K.S.; Visualization, K.S. and P.C.P.; Writing—original draft, K.S. and P.C.P.; Writing—review and editing, P.C.P. and J.K.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received OUR Shiv Nadar University funding.

**Acknowledgments:** The authors are thankful to Shiv Nadar University for providing the research facilities.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

1. United Nations. Sustainable Development Website—United Nations. Food Security and Nutrition and Sustainable Agriculture. 2019. Available online: https://sustainabledevelopment.un.org/topics/foodagriculture (accessed on 13 March 2019).


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

#### *Article*
