**Satellite and UAV Platforms, Remote Sensing for Geographic Information Systems**

Editor

**Alfred Colpaert**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editor* Alfred Colpaert University of Eastern Finland Finland

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Sensors* (ISSN 1424-8220) (available at: https://www.mdpi.com/journal/sensors/special issues/ UAV GIS).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-5361-0 (Hbk) ISBN 978-3-0365-5362-7 (PDF)**

Cover image courtesy of Augustine Gbagir.

© 2022 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

### **Contents**



### **About the Editor**

### **Alfred Colpaert**

Alfred Colpaert has been a professor of physical geography at the university of Eastern Finland since 2004. He was previously affiliated with the university of Oulu, Finland. His main research interests are the use of remote sensing (UAV and satellite platforms) for the monitoring and analysis of ecological changes on the Earth's surface. He has studied, amongst other issues, the effects of reindeer herding in northern Scandinavia, the effects of cattle and wildlife on vegetation in southern Africa, and the effects of anthropogenic pressure on ecosystems.

### **Preface to "Satellite and UAV Platforms, Remote Sensing for Geographic Information Systems"**

LAerial photography has been used since the First World War, and after the Second World War became an important tool of the Earth Sciences, as well as Biological and Geographical Sciences. Satellite Remote Sensing has been a standard research instrument since the launch of Landsat 1 in 1972. The digitalization of geographical work resulted in the first Geographical Information Systems in the 1960s. The rapid evolution of computer hardware and software has provided ever faster and better platforms for the analysis of digital geoinformation. The introduction of cheap and reliable unmanned aerial vehicles (UAV) in the last two decades filled the gap between global satellite imagery and digital aerial photogrammetry. Together with the advent of computer modelling and geo-statistical analytical capabilities, we have now an extremely powerful complex to analyze, model and predict changes in the environment. The effects of climate change, loss of biodiversity, and complex ecological questions are all possible fields benefitting from the integration of digital remotely sensed data and the analytical powers of geoinformatics.

> **Alfred Colpaert** *Editor*

### *Editorial* **Satellite and UAV Platforms, Remote Sensing for Geographic Information Systems**

**Alfred Colpaert**

Department of Geographical and Historical Studies, University of Eastern Finland, Yliopistokatu 7, P.O. Box 111, 80101 Joensuu, Finland; alfred.colpaert@uef.fi

Satellite and UAV (unmanned aerial vehicle) imagery has become an important source of data for Geographic Information Systems (GISs). Enormous volumes of remote sensing and GIS data are nowadays produced, stored and analyzed in high-capacity network-based geoinformatics systems. Satellite imagery in a wide range of spatial, spectral, and temporal resolutions provides the scientific community with rapidly available global data to be used as an integral part of spatial data structures and analyses. Remote sensing platforms, such as Modis and Landsat, have a unique historical record of providing tens of years of uninterrupted global data, provided by repositories such as NASA and ESA.

For local applications, the rapid evolution of unmanned aerial vehicles and lightweight sensors has provided the scientific community with a tool for acquiring data of extremely high resolution, covering areas that vary from several hectares to hundreds of square kilometers in size. These data can be used for precision farming, forestry, and environmental monitoring.

The present Special Issue on the integration of UAV and satellite imagery with GIS contains ten articles, which can be divided into three parts: UAV applications, satellite remote sensing and methodological work using RS data.

Three articles deal with pure UAV applications, two applying UAV for agricultural crop monitoring [1,2] and one paper [3] using GIS and computer vision to analyze UAV orthomosaics. The planning of high-altitude long-endurance pseudo-satellite missions is dealt with in this paper [4].

The integration of satellite RS data in GIS systems for vegetation monitoring is used in three papers: one paper dealing with winter stress on arctic understory vegetation [5], one on the application of Copernicus (CMEMS GlobColour-Merged CHL-OC5 Satellite Observations) satellite-derived data concerning the aquatic environment [6] and one on the application of MODIS NDVI data and GIS to assess the effect of wildlife upon tropical savannah vegetation [7].

The problem of mosaicking multiple high-resolution orthoimages (UAV or satellite) is dealt with in paper [8], introducing a novel method based upon the D-LinkNet Neural Network.

Precise satellite telemetry data can be used to monitor the minute deformations occurring in the Earth's crust. The paper by Jagoda et al. describes the use of high-precision laser altimetry data to assess the deformation of the Earth's crust caused by external planetary bodies (tidal forces) [9]. The last paper uses data derived from global navigation satellite systems (GNSS) to monitor the crustal deformation of the Earth [10].

**Funding:** This research received no external funding.

**Conflicts of Interest:** The author declare no conflict of interest.

**Citation:** Colpaert, A. Satellite and UAV Platforms, Remote Sensing for Geographic Information Systems. *Sensors* **2022**, *22*, 4564. https:// doi.org/10.3390/s22124564

Received: 14 June 2022 Accepted: 15 June 2022 Published: 17 June 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

### **References**


### *Article* **Assessing the Impact of Wildlife on Vegetation Cover Change, Northeast Namibia, Based on MODIS Satellite Imagery (2002–2021)**

**Augustine-Moses Gaavwase Gbagir 1,\*, Colgar Sisamu Sikopo 2, Kenneth Kamwi Matengu <sup>3</sup> and Alfred Colpaert 1,\***

	- kmatengu@unam.na

**Abstract:** Human–wildlife conflict in the Zambezi region of northeast Namibia is well documented, but the impact of wildlife (e.g., elephants) on vegetation cover change has not been adequately addressed. Here, we assessed human–wildlife interaction and impact on vegetation cover change. We analyzed the 250 m MODIS and ERA5 0.25◦ × 0.25◦ drone and GPS-collar datasets. We used Time Series Segmented Residual Trends (TSS-RESTREND), Mann–Kendall Test Statistics, Sen's Slope, ensemble, Kernel Density Estimation (KDE), and Pearson correlation methods. Our results revealed (i) widespread vegetation browning along elephant migration routes and within National Parks, (ii) Pearson correlation (*p*-value = 5.5 <sup>×</sup> <sup>10</sup><sup>−</sup>8) showed that vegetation browning areas do not sustain high population densities of elephants. Currently, the Zambezi has about 12,008 elephants while these numbers were 1468, 7950, and 5242 in 1989, 1994, and 2005, respectively, (iii) settlements and artificial barriers have a negative impact on wildlife movement, driving vegetation browning, and (iv) vegetation greening was found mostly within communal areas where intensive farming and cattle grazing is a common practice. The findings of this study will serve as a reference for policy and decision makers. Future studies should consider integrating higher resolution multi-platform datasets for detailed micro analysis and mapping of vegetation cover change.

**Keywords:** vegetation monitoring; drivers of deforestation; Zambezi region; land degradation; vegetation cover change; wildlife management; TSS-RESTREND; greening and browning; MODIS; Mann–Kendall

### **1. Introduction**

One of the persistent ongoing global environmental challenges is that of land degradation [1–3]. Land degradation is quite complex in nature and often involves the inter-play of biophysical, environmental, and socioeconomic factors [4]. There are several scientific debates on what constitutes land degradation but in this study, we adapt the general definition of land degradation by Barbier and Hochard 2018 [5], "as some measurable loss of the biological or economic productivity and complexity of rainfed cropland, irrigated cropland, or range, pasture, forest and woodlands ... arising from human activities and habitation patterns".

Anthropogenic disturbances have been identified as a major driver of land degradation globally [6–10] and are well documented [6,11–15]. The drivers of land degradation are many, complex, and unique across regions [8,13,14,16,17], but these have been categorized as direct and indirect [16,17]. Based on this categorization, the direct causes of land degradation include: (1) infrastructure development (e.g., roads and settlements),

**Citation:** Gbagir, A.-M.G.; Sikopo, C.S.; Matengu, K.K.; Colpaert, A. Assessing the Impact of Wildlife on Vegetation Cover Change, Northeast Namibia, Based on MODIS Satellite Imagery (2002–2021). *Sensors* **2022**, *22*, 4006. https://doi.org/10.3390/ s22114006

Academic Editor: Assefa M. Melesse

Received: 6 April 2022 Accepted: 9 May 2022 Published: 25 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

(2) expansion of agriculture (e.g., large- and small-scale farming and cattle grazing), and (3) wood extraction (e.g., fuelwood, pole wood, and charcoal production). While the indirect drivers of land degradation include: (a) demographic (e.g., population density, and migration/emigration), (b) economic (e.g., market growth and commercialization, and economic structures), (c) technology (e.g., agro-tech changes), (d) policy and institutional (e.g., formal policies, and property rights), and (e) cultural (e.g., public attitudes and beliefs, and individual and household behavior).

Anthropogenic activities such as wood extraction and conversion of woodland and forests into small and large-scale farming have been a major contributor to land degradation across different geographical regions [15,18]. In the tropics, particularly Africa and in Namibia, agricultural expansion, wood extraction, and infrastructure development are the key drivers of land degradation [9,12,16–18].

The impact of land degradation is quite severe in arid, semi-arid, and sub-humid regions [4,19]. Land degradation has a long history in Sub-Saharan Africa and has been well documented and researched [19–21]. In Africa, anthropogenic activities, including unsustainable land use practices (e.g., overexploitation of natural vegetation cover), are a major contributor to land degradation, in addition to other natural causes such as droughts [16,22,23]. In a country such as Namibia in south-western Africa, where about 22% of the land area is classified as desert, 70% as arid to semi-arid, and 8% as dry subhumid [24–26], any slight change or modification in the vegetation structure could have adverse effect on the environment, social well-being, and livelihoods of the people [27].

In Namibia, the contribution of anthropogenic activities to the loss of vegetation cover has been well documented [12,24,27–29]. The loss of vegetation cover is mostly driven by the conversion of forests and woodland into agricultural farmlands [12,24]. Even though this is the case, the interaction and contribution of wildlife to vegetation cover loss is less understood and needs to be studied in more detail. On the other hand, wildlife damage and human–wildlife conflict is an on-going topic of research and discussion amongst researchers, natural resources managers and various other stakeholders [30–32]. This study will focus on one aspect of land degradation: vegetation cover loss and how anthropogenic and wildlife interaction are driving land cover change in the Zambezi region.

Amongst all the human–wildlife conflicts, the African elephant (*Loxodonta africana*) is one of the most significant wildlife species causing structural changes and damage to vegetation [33]. Elephants are herbivores and bulk feeders and require large amounts of food resources to fulfill their nutritional requirements, which they receive from trees, shrubs, and grasses [33,34]. Even though elephants' consumption of vegetation to meet their dietary needs is natural, overexploitation and mechanical damage becomes destructive, causing vegetation browning and contributing to land degradation (e.g., leaving soils bare) [33–35]. The impact of elephants on structural changes in vegetation has been documented by several studies [34,36–42]. Unlike anthropogenic activities, the impact of elephants on vegetation cover is, to a large extent, confined to locations where elephants exist mostly within protected areas [35,40,43,44]. Anthropogenic restriction of elephant movement and access to space and resources is the main factor driving the browning of vegetation cover by elephants [45,46].

In Namibia, anthropogenic activities are the primary driver of land degradation [12,24], while elephants are mainly responsible for modifying the vegetation structure [47]. As such, we will limit our discussion in this study to the elephant as the major interacting nonhuman factor contributing to the loss of vegetation cover in Namibia. Additionally, we will use the Zambezi region as a test case, as it is one of the best areas suitable for agriculture in the whole of Namibia and, historically, is home to a wide variety of wildlife (both large and small) [28,48]. The region has the largest savannah woodland cover in Namibia [24,27,28] and is habitat to most elephants in Namibia [30,48]. It is well documented that elephants browse, break, pull and uproot woody species, thus causing structural changes in vegetation cover [44,49–51]. The movement of elephants depends on several interrelated factors such as food, water, elevation, density and human settlements [30,49]; they have a home range

of from 10 km to more than 8000 km [49]. Consequently, human–wildlife conflicts are a common occurrence in the region [32]. One of the identified reasons for these conflicts is the anthropogenic fragmentation of natural wildlife habitats [30,32,46]. Though this is the case, elephant-induced vegetation cover loss is most likely a secondary cause [51], the primary cause being the limited availability of resources driven by anthropogenic activities [25,46].

One of the solutions to these conflicts has been the establishment of wildlife reserves and national parks and the construction of fences and other barriers to keep wildlife at bay from human settlements [25]. While this has largely worked, the carrying capacity of these wildlife reserves is often not sufficient to sustain large herds of herbivores [25], thus putting pressure on available resources and causing the loss of vegetation cover [25].

Currently, in Namibia, specifically in the Zambezi region, there are projects to combat land degradation [48], conserve wildlife and manage human–wildlife conflicts [51–53]. In this study, we will use land degradation to mean the loss of vegetation cover with contextual meaning [54]. Additionally, we will use the terms greening and browning to refer to vegetation increase and decrease, respectively.

Presently, there are multiple satellites that provide datasets that can be used for different research purposes [55–57]. Satellite remote sensing is widely used in environmental monitoring, mapping of vegetation, and assessment of different land use and land cover changes [58–61]. Remote sensing is a popular mode of research as it is the cheapest and most efficient way to assess land use and land cover change [62]. Land use and land cover change assessment is still one of the most important areas of research because of the direct, immediate, and long-term impact of anthropogenic activities on the environment [3,60,63]. Thus, finding long-lasting and sustainable approaches to address land degradation is essential [3,64]. Additionally, it is important to understand that land degradation is contextual in nature and this should be taken into account during discussions [65].

In the Zambezi region, many studies have successfully used satellite remote sensing to assess and map changes in vegetation cover [24,28,29,66]. Although this is the case, to our understanding and best knowledge, assessing and characterizing the impact of wildlife on vegetation cover using remote sensing in the Zambezi region has not been attempted before.

Thus, improving our understanding of the dynamics and impact of wildlife on land degradation within the Zambezi region and beyond is important. Better understanding will provide better insight and tools to improve the management of wildlife and natural land resources in the region. One major challenge of implementing more effective wildlife conservation and natural resource management is continuous access to historical and upto-date land use and satellite data. Fortunately, the availability of historical satellite remote sensing data and the increasing improvements in analytical software provide opportunities to assess and map changes in vegetation cover and structure. In a 2019 study, the authors successfully applied remote sensing data to characterize regional vegetation cover change in the Zambezi region [24]. In that study, they used eight km resolution Global Inventory Monitoring and Modelling Studies (GIMMS) from the Advanced Very High-Resolution Radiometer (AVHRR) [67]. In the study, only results on a regional scale were obtained due to the coarse resolution of the data [24].

Although higher resolution satellite observations exist, there are drawbacks in using this data, such as (a) the exponentially increasing amount of data result in high computational costs for a long time series, and (b) the low temporal resolution and higher impact of cloudiness (especially in the tropics) [68]. The MODIS 250 m resolution dataset provides over 20 years of continuous global daily imagery, which has been resampled into a monthly NDVI nearly cloudless dataset. This monthly NDVI dataset allows us to use sophisticated geospatial trend analysis techniques.

In this study, we will assess and characterize the vegetation cover change in the last nineteen years (2002–2021).

The specific objectives of this study are to:


To assess the vegetation cover change, we will use historical remote sensing data from the 250-meter Moderate Resolution Imaging Spectroradiometer (MODIS) Terra satellite instrument. We will use time series and geostatistics, as well as geo-spatial analytical methods. Specifically, we will use the Time Series Segmented Residual Trend (TSS-RESTREND) method, which will allow us to separate human-induced land degradation from that caused by natural climatic factors [24,69–71]. A similar approach was used by the authors in their 2019 study [24]. Additionally, we will use the Mann–Kendall non-parametric test and Sen's Slope measure of direction and magnitude of vegetation change [72]. In addition, we will use the Kernel Density Analysis [73] to cluster the elephant GPS tracking data and correlate the results with the trajectory of vegetation cover change.

### **2. Materials and Methods**

### *2.1. Study Area*

The Zambezi region ((formerly Caprivi Strip) (Figure 1)) is part of the Kavango-Zambezi Transfrontier Conservation Area (KAZA TFCA), stretching across five countries: Namibia, Angola, Zambia, Zimbabwe, and Botswana [48], forming the second largest conservation area in the world [51]. The land area of the Zambezi region is 14,785 km2 [51], with a total population of 98,849 (2011 Census) [24]. Most of the vegetation in the region is woodland savanna and open grasslands [27,28,51]. The region contains three large National Parks, Bwabwata, Mudumu and Nkasa Lupala (formerly Mamili) [48,51]. Conservancies in the region include: Kwandu, Mayuni, Salambala, Sibbinda and Linyanti [25,29]. A conservancy is a legally defined area set aside and managed by local communities who have rights to live within, use, and manage wildlife and other natural resources for personal and tourism purposes (including trophy hunting) [74,75]. The region is an important migratory route and home to a high density of elephants [25,76], buffalos, and antelopes [48]. Additionally, the region is an important agricultural area, due to good soils and high rainfall [77,78].The major soil types in the region are poor ferralic arenosols containing high iron contents and fertile eutric fluvisols with high base saturation [77,78]. The yearly amount of rainfall in the region is the highest in Namibia (500–700 mm/year) [77,78] when compared to the national mean of <250 mm/year [19] and <50 mm/year in the southwestern and coastal areas [77]. The wet season in the region starts in November and ends in April. The average summer and winter temperatures in the region are 35 ◦C and 5 ◦C, respectively [78]. The region has a yearly evaporation rate of about 2500 mm [78]. The region shares common borders with Angola, Botswana, Zimbabwe, and Zambia [48]. The perennial Kwando (Cuando) River flows along the border between Angola and Zambia through the Zambezi region (with Bwabwata National Park on the west and the Mudumu National Park and the six conservancies on the east) south towards the swampy areas around Nkasa Lupala National Park. East of Nkasa Lupala is the Linyanti River that flows east through the seasonal Lake Liambezi into the Chobe River. The Chobe River flows eastward into the perennial Zambezi River, one of Africa's major and longest river systems. The Zambezi River flows from Zambia and forms the border between Zambia and Namibia in the Zambezi region.

### *2.2. Satellite and UAV Field Data and Image Pre-Processing*

We downloaded and processed the monthly 250 m Moderate Resolution Imaging Spectroradiometer (MODIS) Terra satellite instrument NDVI datasets. We resampled the NDVI index to a common 250 m grid (UTM-35S). Because the 2001 data set is incomplete, we used only raster images from 2002 to 2021.

**Figure 1.** Study area, main roads, rivers, conservation areas, and field work way points for high resolution (2 cm) UAV data in (**a**). In (**b**), the map of Namibia, and (**c**) map of major soil types.

The time series for precipitation and temperature data were monthly ERA5, available from 1979 (ERA documentation). The monthly 0.25 × 0.25 degree resolution data was downloaded from 1999 to 2021 (https://cds.climate.copernicus.eu (accessed on 6 May 2022)). Both temperature and precipitation were resampled to a common 250 m grid and re-projected to the UTM reference system. The temperature and precipitation data must start two years before the NDVI time series because this information is required by the processing algorithms during analysis to calculate the maximum rainfall accumulation months. Finally, we used 265 gridded monthly temperature and precipitation raster layers.

### *2.3. Field Sample Locations and Elephant Tracking Data*

During our field work, we used a hand-held GPS instrument [(Garmin GPSMAP 62ST), Garmin Finland] to collect the latitudes and longitudes of sample points. At each sample point (Figure 1), we recorded the location coordinates and geographical name. We used a DJI Mavic Pro Platinum drone to document the vegetation characteristics by taking aerial photos and videos at every sample point. At each sample location, we flew the drone at a height of 40–90 m and recorded a 360◦ view of the surrounding vegetation (pictures and videos).

The Government of the Republic of Namibia (Ministry of Environment Tourism and Forestry) provided the elephant tracking data (2010–2020). These data are a transboundary hourly GPS-collar tracking dataset, covering Namibia, Botswana, Angola, and Zambia. The data consisted of 31 individual elephants over a period of eight (8) years (2010–2020). The GPS-collared elephant data were collected by the Africa Wildlife Tracking company (https://awt.co.za/ (accessed on 6 May 2022)), based in Pretoria, South Africa. The GPS collars were put on the elephants by first using a tranquilizer dart from a helicopter to immobilize them. The brand of GPS collar used was the Iridium Satellite (IR-Sat) that collects and transmits continuous near real-time data. The data transmission and receiver of the IR-Sat covers a few hundred meters to multiple kilometers [79]. Table 1 presents a breakdown of the number individual elephants tracked and during which period. We downloaded additional crowd sourced wildlife observations, elephant and buffalo observations (one kilometer grid), using the Monad (1 km × 1 km) reference grid data from the Environmental Information Service Namibia (http://www.the-eis.com (accessed on 6 May 2022)).

**Table 1.** The tracked elephant data used in this study. The periods correspond to the 12 calendar months of the year. The total length of tracking period (in months) is shown in brackets.


### *2.4. Data Analysis*

To assess the vegetation changes, browning (decrease) or greening (increase), we used the Time Series Segmentation and Residual Trend analysis (TSS-RESTREND) method [69] to perform a pixel wise analysis. To achieve this, we created and used an R-script that iterates over each pixel across the image stack of the complete the time series. TSS-RESTREND is an improved method of the Residual Trends algorithm (RESTREND) [71] that incorporates the functionalities of Break For Additive Season and Trend (BFAST) algorithm [80,81] to look for break points in the time series.

RESTREND uses an Ordinary Least Squares Linear Regression model, fitted on the residual and time [1,24]. The equation is of the form:

$$\mathbf{y}\_{i} = \beta\_{0} + \beta\_{1}\mathbf{x}\_{\prime} \tag{1}$$

where x is time in years, β<sup>0</sup> is the intercept and β<sup>1</sup> is the slope.

BFAST fits a linear piecewise harmonic model using the ordinary least squares moving sum (OLS-MOSUM) to test for structural changes within time series data [24,82].

The decomposition model takes the following form:

$$\mathbf{Y\_{t}} = \mathbf{T\_{t}} + \mathbf{S\_{t}} + \mathbf{e\_{t}} \tag{2}$$

where Yt is the original observed data (TS) at time t, Tt is the trend, St is the seasonal, and et is the remaining unexplained variation within the TS, respectively [24,80].

TSS-RESTREND fits a multivariate regression between the VPR-Residual (vegetation precipitation) and a dummy variable (0 before a break point and 1 after). The model is of the form:

$$\mathbf{y}\_{i} = \beta\_{0} + \beta\_{1}\mathbf{x} + \beta\_{2}\mathbf{z}\_{i} + \beta\_{3}\mathbf{x}\_{i}\mathbf{z}\_{i} \tag{3}$$

where x is time in years, β<sup>0</sup> is the intercept, β<sup>1</sup> is the slope, β<sup>2</sup> is the offset at breakpoint position, β<sup>3</sup> is the change in slope at the breakpoint position and z is the dummy variable (0 or 1) [24,69].

In addition, we performed a pixel wise Mann–Kendall statistics test of the NDVI time series to determine the trend of total vegetation change in the Zambezi, and the Sen's Slope to determine the magnitude of the change [72]. Mann–Kendall is a non-parametric test and does not rely on a particular data distribution but rather on the relative magnitude of the sample data [83,84].

The Mann–Kendall statistics is of the form:

$$S = \sum\_{\mathbf{k}=1}^{n-1} \sum\_{\mathbf{j}=\mathbf{k}+1}^{n} \text{sign}\left(\mathbf{x}\_{\mathbf{j}} - \mathbf{x}\_{\mathbf{k}}\right) \tag{4}$$

where:

$$\text{sign}(\mathbf{x}\_{\text{j}} - \mathbf{x}\_{\text{k}}) = \begin{cases} & \text{1 if } \mathbf{x}\_{\text{j}} - \mathbf{x}\_{\text{k}} > 0 \\ & \text{0 if } \mathbf{x}\_{\text{j}} - \mathbf{x}\_{\text{k}} = 0 \\ & -1 \text{ if } \mathbf{x}\_{\text{j}} - \mathbf{x}\_{\text{k}} < 0 \end{cases} \tag{5}$$

xj and xk are the annual values in years j and k, respectively [29].

The non-parametric Sens Slope time series analysis was performed using the same pixel-wise moving window method to obtain the linear rate of change in the time series. During all pixel-wise analysis (TSS.RESTREND, RESTREND, Mann–Kendall and Sen's slope), we used a *p*-value parameter of 1 during the analysis, thus we obtained the change in every pixel, irrespective of the *p*-value. We took this approach because it provides for a synoptic spatial overview, showing gradual changes between distinct areas of significant degradation and vegetation increase, and areas of no change; the latter is associated with non-significant *p*-values. We observed that the changes that the algorithm interpreted as non-significant contain important information, e.g., areas of no change. This approach also provides a much more homogeneous and easier to understand cartographic map product. Although the general result of the different methods conforms very well, local differences are noticeable when comparing the results of different algorithms, therefore we made an additional ensemble analysis by combining the TSS.RESTREND, RETREND and Mann–Kendall algorithms and calculating the mean value of the results.

We computed a Kernel Density Estimation (KDE) of the elephant tracking point data in ArcMap 10.5.1. We then used the KDE and the ensemble mean to calculate a simple Pearson correlation analysis between the presence of elephants and vegetation changes. Before calculating the Pearson correlation, we used ArcMap 10.5.4 to create a grid of 1600 points over the whole Zambezi area where elephants are present (Figures 2 and 3). Additionally, we created an additional l600 points in two sub-sample grids, west of the Kwando River in Bwabwata park and around the Mudumu park (Figure 3B,C). We then extracted the values of the ensemble mean at these points and we excluded data points where the KDE was zero (no presence). We were left with 1506 points 1386 points in sub-sample locations one (Bwabwata) and 1482 points in two (Mudumu). We then used these points to compute a simple Pearson correlation in Microsoft Excel (version 2202).

To validate our results, we compared the UAV data we collected during our 2019 field trip with the outcome of the time series analysis. We used R (R Core Team, 2022) and ArcMap (version 10.5.4) for data analysis and to produce graphics. The R-code was run on the cPouta cloud services of CSC using 24 and 48 cores Ubuntu Virtual Machines (https://research.csc.fi (accessed on 6 May 2022)). We implemented Google Earth Pro for visual interpretation and verification of results.

9

**Figure 2.** (**a**–**e**) show the output of the TSS.RESTREND, Mann–Kendall, ensemble, and kernel density estimate (KDE) analysis over the whole Zambesi region. In (**a**), the Residual Change of the TSS.RESTREND and (**b**) is the Mann–Kendall Tau. In (**c**) is the ensemble of the mean values of RESTREND, TSS.RESTREND, (**d**) the spatial distribution of elephant sightings overlayed over the ensemble mean value from (**b**) above and (**e**) the overlay of the Kernel Density Estimate (KDE), and Mann-Kendal Tau from (**b**) above. No data and background values are displayed as grey.

**Figure 3.** Shows the locations of the sampled grid points in the Zambezi region of the Kernel Density Estimate and the ensemble. The grid points are overlayed on the ensemble (a) used previously in Figure 2. The results of the Pearson correlation in (**A**) correspond to all the sampled points in the Zambezi while (**B**), and (**C**) correspond to the sub-sampled areas (black polygons) above.

### **3. Results**

### *Pattern of Vegetation Trend: 2002–2021*

Based on remote sensing and GIS data, this study analyzed the human–wildlife interaction in the Zambezi region. Figure 2 shows the pixel-by-pixel vegetation change pattern in the Zambezi region during the period 2002–2020. The TSS-RESTREND residual change

(Figure 2a) highlights a mixed pattern of positive (greening) and negative (browning) vegetation changes. Similarly, the ensemble and Mann–Kendall Tau (Figure 2b,c) both clearly show a mixed pattern of positive and negative values. The positive and negative pixel values indicate vegetation change increase and decrease, respectively. The observed negative pattern of the TSS-RESTREND and RESTREND is attributed to factors other than climatic aspects because the variability associated with climate was removed during the analysis. Most of the browning pixels (land degradation) are along the Kwando River (Figures 2a, 3 and 4A), which is a major migration route for wildlife, specifically elephants [25]. On the eastern part of the Kwando River are two National Parks Mudumu and Nkasa Lupala, and six conservancies (Kwandu, Mayuni, Mashi, Balyerwa, Wuparo, and Malengalenga), (Figures 1, 2a and 3) [25]. On the western part of the Kwando River is the Bwabwata National Park, a part of the home range for large herds of elephants [32], and also contains a buffalo core area [25]. Most of the vegetation browning we observed was also taking place within Mudumu and Nkasa Lupala National Parks (Figure 2). Within Bwabwata National Park (west of the Kwando River), browning is mainly close to and along the Kwando River, while elsewhere most of the pixels show greening (Figure 2).

Figure 3 shows the Pearson correlation between the Mann–Kendall Tau and the Kernel Density Estimate of the elephant data. Over the whole Zambezi (Figure 3A), as well as in the western (Figure 3B) and eastern areas (Figure 3C), there is a clear negative trend (browning) indicated by the red straight line. In the whole Zambezi area (Figure 3A) and in both sub-sampled locations (Figure 3B,C), the negative trend (browning) is significant (*p*-values = 5.5 × <sup>10</sup><sup>−</sup>8, 0.0005, and 3.93 × <sup>10</sup>−24). Additionally, the results (Figure 3A–C) show that as the density of elephants decreases away from the KDE core areas, vegetation greening begins to occur (black polynomial line, Figure 3A–C). It is noteworthy that the polynomial line increase related to high KDE densities probably indicates that large herds are attracted to abundant vegetation resources. Moving further to the eastern part, we observed browning around and within the Salambala core area, located east of Lake Liambezi (Figure 2c). The Salambala core area is also home to elephant herds (Figure 2e). In addition, we also observed browning close to and around roads (Figures 2–4). These roads are locations of high-density human settlements where cattle grazing and extensive agricultural activities are a common practice. On the other hand, we also observed some relatively high greening, mostly around Lake Liambezi, communal areas, as well as some locations of the floodplains (Figures 2–4).

The browning pattern we observed is not only confined to the Zambezi region but extends across the border into neighboring countries (Figure 2). For example, the browning along the Kwando River continues across the Namibian border into Luina Partial Reserve (Angola) and Sioma Ngwwezi National Park (Zambia). We also observed a similar pattern across the border into Botswana, where we can see a clear difference along the 135 km veterinary fence which was constructed between 1991 and 1997 [25], (Figure 2c,e). On the eastern side of the fence, we observed high levels of degradation, while to the west we see relatively high greening values (Figure 2c).

We also ran the Mann–Kendall test on temperature and precipitation but did not see any significant trend, so the result was not shown here. An enlarged graphics of the Mann–Kendall NDVI Tau is shown in Figure 5 (Additional resources).

**Figure 4.** Shows aerial images from our 2019 field survey in the Zambezi region (**a**–**m**). In (A), the labels (**a**–**m**) correspond to the images shown with their respective GPS coordinates at the sample locations (red circle with black dot in the middle). Additionally, in (A), we use the same ensemble as in Figure 2b above for reference purposes.

**Figure 5.** Additional resources.

### **4. Discussion**

### *Potential Impact of Wildlife on Vegetation Cover and Land Degradation: 2002–2021*

This study found that human–wildlife interaction is driving vegetation cover change in the Zambesi region. Specifically, anthropogenic restriction of space and resources for wildlife is driving the observed accelerated vegetation removal by large wildlife herbivores, in this case elephants. Consequently, this interaction is a potential contributor to land degradation in the Zambezi region. Previous studies in the Zambezi region by Gbagir et al. 2019 [24] revealed that land degradation is driven by the interaction of multiple direct and indirect factors. These factors include: demographic [24,27], ecological (e.g., floods) [27,85], and environmental factors (e.g., topography) [24,28]. Specifically, subsistence farming, infrastructure expansion (e.g., roads), including settlements, and legal and illegal wood extraction for firewood were identified as the main drivers of land degradation in the region [24]. However, the studies by Gbagir et al. 2019 [24] were not able to clearly establish more specific causes for land degradation due to the nature of the data used [24].

In the present study, a more detailed pattern and trend of vegetation cover change was revealed and the impact of wildlife on land degradation was clearly established. The contribution of wildlife on land degradation in the region corresponds to previous base line studies and ongoing statistics from the region. Reports indicate that elephant populations in the Zambezi region are stable or growing [51,86]. In addition to elephants, other wildlife populations, e.g., buffalo, are present (Figure 2d), but since elephants clearly damage trees, our results and discussion are focused on these.

Baseline studies on the presence of elephants in Namibia were carried out in the early 1980s and 1990s [32,87]. Based on these studies, several subsequent surveys have shown that the population of elephants has been increasing steadily in Namibia [25,48] from 600 to 1000 in 1934 to 22,754 in 2016 [48]. Probably the elephant population can be assumed to be even larger in 2022. The Zambezi region (Namibian KAZA) hosts most of the elephants in Namibia and is the most important migration route [30,48]. Reports show that the number of elephants within the Zambezi region has more than tripled since 1995/1996 [48]. A current estimate of elephants in the region is reported as 12008 [88]. According to Chase and Griffin (2009) [25], the elephant population in the Zambezi region was 1468 in 1989, while in 1994, these figures were reported as 5804 [25], 7950 and 5556 [87,88].The differences between the figures are mainly due to the different sampling techniques used in those studies [25,88]. By 1998, these figures were down from 5804 to 4576 [25], but in 2005 the numbers had again increased to 5242 [89]. The decline in the elephant population was thought to be the result of the civil unrest in Angola [25,89] and the construction of the veterinary fence between Namibia and Botswana in western Zambezi [25,48,76], consequently restricting and cutting off the migration of wildlife (elephants, buffalos, wildebeests, zebras, etc.) [25,48,76].

Most of the browning we observed was along the Kwando River (Figure 2), an important migration corridor for elephants [32]. The negative impact (browning) on the surrounding vegetation is clearly visible compared to elsewhere in the Zambezi region, as elephants browse on and de-back and break down trees, causing structural changes to the surrounding vegetation [47,90]. The Kwando River and the Mudumu and Nkasa Lupala parks contain medium to high numbers of elephants [32]. The studies by O'Connell-Rodwell et al. 2000 [32] put the number of elephants present on the western Kwando River as 3000, while 400 and 600 were reported for Mudumu and Nkasa Lupala, respectively. Within Bwabwata national park (west of the Kwando River), browning is limited to the riverbanks and migration route, while elsewhere, there is less indication of high levels of browning. We also observed a similar pattern of browning within the Salambala core area, while most of the greening is occurring elsewhere in the area (Figure 2c). A 2019 survey of elephants reported the current population in the Salambala conservancy as 507 [88].

Other important factors contributing to land degradation in the Zambezi region are of anthropogenic origin, as has been established by previous studies [24]. However, in this study, we now see clearly how these anthropogenic activities have contributed to wildlife-induced vegetation browning in the region. The expansion of settlements and roads and the construction of artificial barriers (e.g., fences) has diminished the habitat of elephants and reduced their access to food and water resources [25,45,47,91]. As a result, more pressure is put on the remaining resources, propagating vegetation browning in the region [25,45,47,91]. Additionally, the shrinking of elephants' habitats has modified the behavioral pattern of these animals [46] and increased human–wildlife conflicts [53,66]. Most of the visible browning was observed within the protected areas (e.g., Mudumu, Nkasa Lupala, Salambala core) due to the high density of elephants concentrated within small restricted areas. Restricting the habitat of wildlife, specifically elephants, has impacted and contributed to the observed browning [45,91]. The Pearson correlation results (Figure 3A,B) also confirmed that the concentration of elephants within a certain restricted area was contributing to the observed gradual browning. However, the right side of the curve (Figure 3A,B) show that areas of vegetation browning do not sustain high animal populations; hence, the curve rises as the green areas attract large herds, thus indicating that the available space and resources may be beyond the carrying capacity of the current number of elephants with the protected areas. Additionally, taking into consideration that the arenosols soils are poor in nutrients [77,78], any slight modification in the vegetation cover will have a visible impact, which in this case is browning.

In addition to the elephant populations in this area, along the Kwando River there are also six conservancies (Kwandu, Mayuni, Mashi, Balyerwa, Wuparo, and Malengalenga) [25]. The presence of these settlements has given rise to clearing of land for farming and cattle grazing and increases in the road network. The presence of human settlements increases human–wildlife competition for land resources, making these areas hotspots for human–wildlife conflicts. Just like the six conservancies along the Kwando River, the Salambala core is also surrounded by several villages where farming and cattle grazing is a common practice [24,29,92,93], which could explain the high levels of vegetation browning (Figure 2c).

Apart from browning, there is also greening within the Zambezi region, most of the greening is occurring within the communal areas of Lake Liambezi and the Chobe River floodplains farther east. These greening areas have high human population densities and are locations of intensive farming and cattle grazing [48]. This we were able to verify during our field visit in December 2019 (Figure 4).

The greening of the Lake Liambezi is mostly due to the present drying (Figure 4j) which opens land for vegetation growth, farming, and grazing activities (Figure 4k). Large numbers of cattle are grazing on the eastern floodplains (Figure 4l) [48]. Recent reports in 2019 estimated the total number of cattle in the Zambezi region to be 135,878 animals [88].

The pattern of vegetation browning observed along the Kwando migratory route continues into neighboring Zambia (Sioma Ngwezi National Park) and Angola (Liuana Partial Reserve) (Figures 1 and 2). Additionally, this pattern of vegetation browning applies to the southern border with Botswana around the Chobe National Park (Figures 2–4). The Chobe National Park is where the majority of the 200,000 migratory elephant population is located [24,94,95]. Furthermore, it is quite clear that greening is mainly on the opposite side (south) of the northern buffalo fence, where access by elephants is restricted [9,35] (Figure 2c,e) [59].

Similar studies elsewhere have also linked elephants to the loss of vegetation in protected areas. Examples include: Samburu and Buffalo Springs National Reserves [39], Aberdare National Park [44] in Kenya, Addo Elephant National Park, Eastern Cape, South Africa [38], and the Serengeti National Park, Tanzania [40,42]. We anticipate that the results of this study will provide increased understanding of the interaction between wildlife and land degradation in the Zambezi region. This new additional information could potentially improve and inform policy formulation and decision-making regarding wildlife and natural resources conservation and management in the region and elsewhere in Namibia.

Future studies should consider detailed and micro-analysis, classification and mapping of vegetation cover change by combining and integrating higher resolution remote sensing datasets [56,61,63,96–98]. This form of information will provide even better data to improve the current integrated sustainable land use and management practices.

### **5. Conclusions**

This study assessed the impact of wildlife populations, specifically elephants, on vegetation browning in the Zambezi region during the last 19 years (2002–2021). Our analysis reveals that vegetation browning is mostly in locations with a high density of elephants. Most of the browning is along the migration corridor of elephants within national parks and conservation areas as a result of exclusion and harassment in areas with human settlements. Obviously, brown vegetation areas do not sustain high population densities of animals.

We also found that the expansion of settlements and the construction of artificial barriers (e.g., fences) has affected the movement and migration pattern (behavior) of wildlife populations, specifically elephants, in the region, which has led to the concentration of game animals within confined national parks.

Furthermore, the limited amount of space and resources for wildlife populations could potentially be a major factor contributing to vegetation browning in the region. This assumption is supported by the high incidence of ongoing human–wildlife conflicts within the region. On the other hand, our study found that most of the greening was occurring in areas with intensive farming; for example, around the shrinking Lake Liambezi, and within communal areas.

**Author Contributions:** Conceptualization, A.-M.G.G. and A.C.; methodology, A.-M.G.G. and A.C.; software, A.-M.G.G. and A.C.; formal analysis, A.-M.G.G. and A.C.; investigation, A.-M.G.G. and A.C.; resources, A.C.; data curation, A.-M.G.G., K.K.M., C.S.S. and A.C.; writing—original draft preparation, A.-M.G.G.; writing—review and editing, A.-M.G.G., K.K.M., C.S.S. and A.C.; visualization, A.-M.G.G. and A.C.; supervision, A.C.; project administration, A.C.; funding acquisition, A.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** We acknowledge the Government of Namibia (Ministry of Environment, Tourism and Forestry), especially the, University of Eastern Finland and the Kone Foundation for funding. CSC–IT Centre for Science, Finland (urn:nbn:fi:research-infras-2016072531) and the Open Geospatial Information Infrastructure for Research (Geoportti, urn:nbn:fi:research-infras-2016072513) for computational resources and support.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


*Article*

### **Soil Salinity Inversion of Winter Wheat Areas Based on Satellite-Unmanned Aerial Vehicle-Ground Collaborative System in Coastal of the Yellow River Delta**

### **Guanghui Qi 1,2, Gengxing Zhao 2,\* and Xue Xi <sup>2</sup>**


Received: 7 October 2020; Accepted: 12 November 2020; Published: 14 November 2020

**Abstract:** Soil salinization is an important factor affecting winter wheat growth in coastal areas. The rapid, accurate and efficient estimation of soil salt content is of great significance for agricultural production. The Kenli area in the Yellow River Delta was taken as the research area. Three machine learning inversion models, namely, BP neural network (BPNN), support vector machine (SVM) and random forest (RF) were constructed using ground-measured data and UAV images, and the optimal model is applied to UAV images to obtain the salinity inversion result, which is used as the true salt value of the Sentinel-2A image to establish BPNN, SVM and RF collaborative inversion models, and apply the optimal model to the study area. The results showed that the RF collaborative inversion model is optimal, R<sup>2</sup> = 0.885. The inversion results are verified by using the measured soil salt data in the study area, which is significantly better than the directly satellite remote sensing inversion method. This study integrates the advantages of multi-scale data and proposes an effective "Satellite-UAV-Ground" collaborative inversion method for soil salinity, so as to obtain more accurate soil information, and provide more effective technical support for agricultural production.

**Keywords:** sentinel-2A image; UAV image; remote sensing; soil salinity

### **1. Introduction**

Soil salinization is a form of soil degradation. Soil salinization will not only cause a series of problems such as ecological deterioration, but also have a negative impact on the growth of crops [1]. Therefore, it is of great practical significance for agricultural production to carry out research on soil salinization in coastal winter wheat planting areas and grasp the spatial distribution of salinization.

Traditional soil salt information is obtained mainly through field survey sampling and chemical analysis method. This method is relatively accurate, but it is time-consuming and labor-intensive, and has obvious limitations in terms of spatial globalization and effectiveness. In addition, field sampling can also cause damage to winter wheat and other crops. Satellite remote sensing data can make up for these shortcomings. Because of its high timeliness, economy and large-area simultaneous observation ability, it has become an important method for quantitative extraction of saline soil information in a large area. A large number of scholars have carried out related research and achieved excellent results. Most of them have realized a large-scale quantitative inversion of soil salt content based on multi-source satellite images [2–11], and they inverted soil salinity by studying the quantitative relationship between soil salinity and vegetation index. This is mainly because the vegetation cover

and growth can reflect the status of soil salinity. In addition, the satellite image detects mainly the vegetation coverage information, so it is hard to directly use the spectrum of bare soil to monitor the soil salinity in the vegetation coverage area on a large scale [12–15]. When constructing soil salinity inversion model based on satellite imagery, most studies use traditional linear regression methods and machine learning algorithms. After comparison, it is found that most machine learning methods are more accurate and therefore more widely used [16–19].

However, satellite imagery is greatly affected by factors such as fixed orbit, time phase, weather, etc., especially the low spatial resolution and imaging quality, and it often fails to meet the demand of high-precision and real-time monitoring of soil salinization [4]. In recent years, UAV technology has rapidly developed, due to its advantages of high image accuracy, simple operation and flexibility, it has been applied to soil salt monitoring by scholars [20–22]. However, UAV technology has some limitations in monitoring soil salinization on a large scale due to its limited observation range. Therefore, it is necessary to make use of the complementarity of "Satellite-UAV-Ground" multi-scale data to carry out collaborative inversion of soil salinity.

At present, scholars have carried out certain researches on the inversion of surface-related parameters by combining data from two platforms such as Satellite-Ground and UAV-Ground. For example, An et al. [23] built soil salinity inversion model by simulating Landsat8 band through actual hyperspectral measurements on the ground. Schut et al. [24] combined the vegetable index of UAV and satellite with the crop growth model to evaluate the yield and fertilizer response in the field of heterogeneous smallholders, which would improve the accuracy of yield and crop production assessment. Kattenborn et al. [25] built a random forest model based on the spectrum, texture information and canopy structure obtained from UAV data, and upscaled it to Sentinel satellite data to draw a large range distribution map of tree species; Hu et al. [26] found that the inversion accuracy of soil salinization using the combination of UAV hyperspectral data and GF-2 multi-spectral data was better than that of GF-2 multi-spectral data inversion. Daryaei et al. [27] used Sentinel-2 and UAV data to conduct fine-scale monitoring of vegetation in semi-arid mountainous areas focusing on riparian landscapes, and the accuracy has been improved. Zhang et al. [2] established the reflectance relationship between satellite and UAV data and applied the constructed UAV high-precision model to satellite imagery to realize the SPAD worth inversion during the recovery period of winter wheat in a large area. Studies have shown that the combination of data from the two platforms can effectively improve the accuracy of the inversion, but most of these studies are based on the spectral information relationship between the data samples of different platforms for data fusion to achieve collaborative inversion. Due to the large spatial scale and spectral differences between the samples, it will cause the spectrum information loss and errors, which will affect the accuracy of inversion results. How to realize the accurate and efficient connection of data from the three platforms of "Satellite-UAV-Ground", and then the high-precision and large-scale monitoring of ground surface information, especially soil salinization information, needs further research and exploration.

Therefore, this study selects typical coastal areas in the Yellow River Delta, and uses Sentinel-2, UAV and ground multi-platform data to explore its efficient and accurate collaborative method to carry out "Satellite-UAV-Ground" collaborative inversion of soil salinity in winter wheat planting areas. Thus, a rapid and accurate method for obtaining soil salt in the coastal area was proposed, which provided a scientific basis for the production and management of winter wheat.

### **2. Materials and Methods**

### *2.1. Study Area*

The study area is located in Kenli District (37◦24 –38◦10 N, 118◦15 –119◦19 E), which is the core area of the Yellow River Delta. It has a warm temperate continental monsoon climate, with sufficient sunlight but little precipitation and evaporation large, uneven droughts and floods, and obvious seasonal alternating wet and dry. The annual mean precipitation, evaporation and air temperature are 511.6 mm, 1928.2 mm and 12.4 ◦C, respectively [28]. The terrain of the study area is low and flat, decreasing from southwest to northeast. The source of surface water is natural precipitation and water from the Yellow River. The groundwater level is relatively shallow and the salinity is high. The main soil type is gleyic solonchaks with a high sand proportion and high salinization. Soil texture is light, organic matter is generally lack, nitrogen and phosphorus are less, soil overall nutrient is poor, the pH value of the soil is greater than 7 [29]. The main crops are winter wheat, corn, rice and cotton, but the overall management is extensive and the yield is low. Wheat planting areas are mainly distributed in the higher terrain area in the southwest and the Yellow River coast area in the northeast. The variation of soil salinization is obvious, which is an ideal area for this study. Based on the investigation of Kenli area, landform, soil and crop distribution, two test areas A and B (Figure 1) were selected for the concentrated distribution of winter wheat in the southwest and northeast of the study area respectively to carry out field soil salinity measurement and UAV flight test. Test area A was a square area of 200 m × 200 m, and a 100 m × 50 m area was selected to arrange sampling points. Test area B was a rectangular area of 200 m × 100 m, and a 50 m × 50 m area was selected to arrange sampling points. An investigation suggests that planting time, farming methods, and fertilization conditions are all basically the same in A and B test areas, while the growth of winter wheat there is obviously different, with all soil salt content levels distributed, making the experimental area more typical and representative.

**Figure 1.** Location map of the study area and test areas (**A**, **B**), the red dots are the sampling points in the test area. The study area is true color image of Sentinel-2A, and the test area (**A**, **B**) are false color images of UAV.

### *2.2. Data Acquisition and Preprocessing*

### 2.2.1. Acquisition of Ground Soil Salinity Data

There is less rainfall in spring in Kenli District, and the surface salinization is obvious and stable. Winter wheat is at the reviving stage, while other major crops are not sown, which is convenient for extracting spectral characteristics [30]. A field survey in the study area was conducted from April 10 to 16, 2019. To ensure uniform distribution, three samples were pre-arranged in the study area every 5 km × 5 km grid, and finally 77 winter wheat sample data were collected. At the same time, for the two test areas, a ground card was placed every 10m at the outer boundary of the sampling point area of the A and B test areas, and connect them with a measuring rope to form a 10 m × 10 m grid of sample points, taking the intersection of the grid as sampling points. This 10 m × 10 m grid is the same as the pixel size of Sentinel-2A. A total of 102 sample points was collected, including 66 in the A test area and 36 in the B test area. Four outliers in the sampling points were eliminated, and the remaining 98 samples were used to construct and verify the soil salinity inversion model of winter wheat. An EC110 portable salinity meter (Spectrum Technologies, Inc., Aurora, USA) equipped with a 2225FST series probe (conductivity temperature correction was performed) was used to make multiple measurements of the electrical conductivity (EC) of the soil surface layer 10 cm below the plant at each sample point and make a record after stabilization. The average of the measured values is taken as the EC value of each sample point, in ds/m. According to the results of earlier research which in our laboratory, the measured EC data were converted into soil salt content (SSC) in g/kg by using the formula SSC = 2.18 EC + 0.727 which was obtained by chemical analysis of soil in the same study area in spring [31]. At the same time, the orientation, topography, soil, wheat growth and other relevant information were recorded.

### 2.2.2. Acquisition and Processing of UAV Imagery

A multispectral camera (Parrot Sequoia, Parrot Inc., Paris, France) was mounted on a Dajiang Matrice 600 Pro UAV (loaded mass: 5.5 kg; flying time: 18 min) (SZ DJI Technology Co., Ltd. Shenzhen, Guangdong Province, China). The camera can receive a total of 4 bands of information, which are green light (G), red light (R), red edge (RE), near infrared (NIR). The wavelengths are 550 nm, 660 nm, 735 nm, and 790 nm, and the band widths are 40 nm, 40 nm, 10 nm and 40 nm. The Sequoia multi-spectral camera is mounted on the head of the UAV, and the radiation sensor is mounted on the top of the UAV to write the radiometric correction data into the image during flight.

The data collection time was from 11:00 to 15:00 on 14 April 2019. The weather was clear and cloudless with low wind force when the UAV was flying. Before takeoff, the Sequoia multispectral camera and radiation sensor were calibrated, and the ground standard whiteboard image was collected. The flight height was 50 m, the flight speed was 5 m/s, and the image acquisition interval was 1.5 s. After the data are collected, they will be imported into Pix4D Mapper software (Pix4D, S.A., Prilly, Switzerland) for splicing, radiation correction and other processing to obtain the high-resolution orthophoto image of the test area, with a spatial resolution of 4–5 cm. Finally, in ENVI5.3, the decision tree method is used to remove the soil background. In order to eliminate the random error caused by the reflectance of a single point, a 5 × 5 pixels image is taken with the sampling point as the center, and the average reflectance value is taken as the reflectance data of the sampling point.

### 2.2.3. Acquisition and Processing of Sentinel-2A Satellite Data

Sentinel-2 satellite is a multispectral imaging satellite with high resolution, revisit rate and update rate. It includes two small satellites A and B. The revisit period is 5 days. The main payload is MSI multispectral imager, covering 0.4–2.4 μm spectral range, including 10 m (four bands), 20 m (six bands), 60 m (three bands) ground resolution, which can monitor the growth, coverage and health of land vegetation, and obtain information on crop planting, land use changes, etc. In this paper, the Sentinel-2A products were downloaded from the ESA Copernicus data sharing website (https://scihub.copernicus.eu/). Considering the acquisition time of ground and UAV data and the quality of the image, the Sentinel-2A Level-1C multispectral image on 17 April 2019 was selected for modeling and inversion of the soil salt content, and the images of 3 November 2018 and 26 June 2019 were used for extraction of winter wheat planting areas.

The downloaded L1C data are orthophoto with geometric precision correction, without radiometric calibration and atmospheric correction. First, radiometric calibration and atmospheric correction are carried out by using Sen2cor, a plug-in published by ESA. Then, the data are re-sampled by the Sentinel Application Platform (SNAP) software to generate 10m spatial resolution images, and the data are exported in ENVI format. Finally, the Sentinel-2A true color images of the research area were obtained by splicing the images in ENVI 5.3 (Exelis Visual Information Solutions, Inc., Colorado, USA) and clipping the images using kenli District administrative boundary vector documents (Figure 1). In order to be consistent with the UAV image band, the green (G), red (R), red edge (RE), and near infrared (NIR) of Sentinel-2A image are selected in this study. The wavelengths are 560 nm, 665 nm, 740 nm, 865 nm, and the band widths are 45 nm, 38 nm, 18 nm and 33 nm.

### *2.3. Methods*

### 2.3.1. Calculation and Optimization of Vegetation Indices

Studies have shown that different levels of soil salinization have an impact on vegetation growth and morphology, plasma membrane permeability, photosynthetic pigments of leaves, gas exchange parameters, chlorophyll fluorescence characteristics, etc. [32]. Therefore, there are differences in spectral information of vegetation at different levels of soil salinization, which can indirectly reflect the level of soil salinization [33,34]. The vegetation index can highlight the characteristics of vegetation and effectively reflect the health and growth of vegetation. When the soil salt content increases, the reflectivity of visible red light of the salt-sensitive vegetation will increase, and the near-infrared reflectance will decrease [35].In order to better reflect the vegetation conditions, 8 vegetation indexes related to red light and near-infrared are selected in this study, including normalized difference vegetation index (NDVI), normalized difference red edge index (NDRE), optimized soil adjusted vegetation index (OSAVI), green normalized difference vegetation index (GNDVI), triangle vegetation index (TVI), difference vegetation index (DVI), Improved chlorophyll absorption vegetation index based on PROSPECT and SAILH radiation transfer model (MCARI2) and renormalized difference vegetation index (RDVI).

The multi-band spectrum collected by the UAV is used to calculate the 8 vegetation indexes, and the formulas are shown in Table 1. Then the correlation coefficient between each vegetation index and soil salinity was calculated, and the variance inflation factor (VIF) between vegetation indexes was calculated by using the formula VIF = 1/(1-r\*r) (r is the correlation coefficient between vegetation indexes) [12], excluding the low correlation or VIF > 10, which is the parameter that cannot be diagnosed by collinearity. The sensitive vegetation indexes are selected for soil salt modeling.


**Table 1.** Formulas and corresponding citation for vegetation indexes.

2.3.2. Construction and Optimization of UAV-Ground Collaborative Inversion Model of Soil Salinity in Wheat Field Based on UAV Images

The 98 samples were sorted from small to large, and the modeling set and the validation set were sampled at equal intervals in a ratio of 2:1 to ensure the same range and uniform distribution of the model samples and the validation samples. 68 samples were selected for modeling and 30 samples for validation.

Taking the sensitive vegetation index as the input variable of the model, three methods were used to construct the winter wheat soil salinity inversion model, namely, BPNN, SVM and RF. BPNN is a multi-layer feedforward neural network trained according to the error back propagation algorithm, and it has also been applied to the salt inversion problem [39,40]. SVM is a new machine learning method from linear separable to linear nonseparable based on the principle of minimizing structural risk according to the statistical theory. It has been widely applied in image recognition and classification and has also been applied in regression problems in recent years [41,42]. The RF algorithm is an integrated learning algorithm obtained by combining the bagging algorithm with the decision tree algorithm. In recent years, many scholars have applied it to remote sensing technology [43,44]. All the three methods were implemented in MATLAB R2016b (MathWorks, Inc., Natick, MA, USA). The BPNN sets the number of training iterations to 1000, the accuracy to 0.003, the learning rate to 0.01; The SVM method was set as V-SVR, Gaussian kernel function was selected, the best penalty factor C and kernel parameter gamma were selected through network search and cross validation, and the SVM model was trained. The RF method called MATLAB random forest toolbox, and the parameter leaf node Leaf = 5 and the number of trees Ntrees = 200 were finally determined by Bayesian optimization.

The accuracy of model modeling and verification was evaluated by the coefficient of determination (R2) and root mean square error (RMSE) [45–47]. R2 is used to measure the fitting degree of the model, and RMSE reflects the deviation between measured value and predicted value. The closer R<sup>2</sup> is to 1, the smaller the RMSE, which means the higher the accuracy of the model, the better the effect. The model with the best accuracy and effect was selected for soil salinity inversion of winter wheat. The degree of soil salinization is divided into five grades according to relevant criteria [48], non-salinization (<1 g/kg), mild salinization (1–2 g/kg), and moderate salinization (2–4 g/kg), severe salinization (4–6 g/kg) and saline soil (>6 g/kg), and we get the distribution map of soil salinity grade.

### 2.3.3. Information Extraction of Winter Wheat Planting Area

The planting area of winter wheat in the study area was extracted by using the time series features composed of the NDVI of the Sentinel-2A images on 3 November 2018, 17 April 2019 and 26 June 2019. According to the investigation and analysis of various vegetation types in the study area, only winter wheat was in the seedling stage in early November, its growth reached the peak stage in late April of the second year, and entered the maturity stage at the end of June. Its NDVI time series curve showed a rapid rise from early November to late April, while other vegetation types showed little change in NDVI, the NDVI of winter wheat declined rapidly from late April to the end of June, while the NDVI time series curves of other vegetations were in varying degrees of rising stages, as shown in Figure 2. The valley-peak-valley in the NDVI time series at the beginning of November, late April and the end of June is a significant feature that distinguishes winter wheat from other vegetation. Therefore, by calculating the changes in NDVI from early November to late April, and late April to the end of June, a decision tree is established to extract this feature. Combined with the training sample data, the following decision rules are established:

$$\text{T types of ground objects} = \begin{cases} \text{Winter wheel, } b\_2 > 0.1 \text{ and } b\_2 - b\_1 > 0.1 \text{ and } b\_3 - b\_2 < 0\\ \text{Other ground objects} \end{cases} \tag{1}$$

where *b*1, *b*2, and *b*<sup>3</sup> are the NDVI values of Sentinel-2A images on 3 November 2018, 17 April 2019, and 26 June 2019, respectively.

**Figure 2.** Changes of NDVI of different vegetation types in the study area with time.

Thus, the distribution of winter wheat planting areas on 17 April 2019 was obtained.

2.3.4. Construction of the Satellite-UAV-Ground Collaborative Inversion Model of Soil Salinity in Wheat Area Based on Sentinel-2 Images

Sentinel-2A images have mixed pixels, and it is difficult to accurately obtain the measured data of soil salinity within the corresponding range of pixels. However, the resolution of UAV images is up to centimeter level, so the measured data of soil salinity within the corresponding range of pixels are easy to obtain and accurate. Therefore, the measured ground salt data and the UAV image vegetation index are used to construct a high-precision inversion model to obtain the soil salt content of the test areas A and B, then the salt values corresponding to sentinel-2 image pixels in the test area were calculated and taken as the "salt true value" of the Sentinel-2A image construction inversion model. It is combined with the Sentinel-2A image vegetation index to construct three inversion models including BPNN, SVM and RF.

First, the 10 m × 10 m surface vector data corresponding to the size positions of Sentinel-2A image pixels were successively established in the test area A and the test area B. In order to ensure the objectivity of the data, in the formed surface vector data grid, every three rows and three columns of surface vector data are used as a unit to extract the surface vector data at the center of the unit. If the center position is not in the unit, the surface vectors at other positions in the unit are extracted. Figure 3 is the distribution map of the area vector data extracted from the test area A and a total of 75 surface vector data are extracted. The vegetation index corresponding to the Sentinel-2A image pixel of the surface vector was counted and entered into the attribute information, and the combination of the vegetation indexes were used as the input variable of the model. Secondly, A vector surface corresponds to 40,000 image UAV pixels, and the average salt value of 40,000 UAV pixels is calculated as the salt "truth value" of the surface vector data. The soil salt content of 75 surface vector data was recorded into the attribute information as the output variable of the model. Finally, BPNN, SVM and RF were used to construct soil salt content collaborative inversion model based on Sentinel-2A image. All three methods were implemented in MATLAB R2016b.The BPNN sets the number of training iterations to 1000, the accuracy to 0.003, and the learning rate to 0.02; The SVM method was set as V-SVR, and the training set cross-validation and network search method were used to optimize the parameters. According to the principle of minimum variance, the penalty coefficient was determined as C = 10000, γ = 0.01; the RF method called MATLAB random forest toolbox, and the parameter leaf node, Leaf = 5, and the number of trees, Ntrees = 300, was finally determined by Bayesian optimization.

**Figure 3.** Distribution map of the extracted surface vector data of test area A.

2.3.5. Soil Salinity Inversion Results and Accuracy Analysis in Wheat Area

Among the three machine learning models constructed, the optimal model was selected according to R<sup>2</sup> and RMSE. The optimal model was used to invert the soil salinity of the winter wheat planting area in Kenli District, and the soil salinity distribution map of winter wheat in Kenli District was obtained.

At the same time, this method was compared with the direct inversion method of satellite remote sensing, namely, based on Sentinel-2A images and ground-measured salt data, the optimal model was directly constructed and selected to invert the winter wheat soil salt distribution in Kenli District. In order to verify the accuracy of the two methods, the R<sup>2</sup> and RMSE of the inversion results and the data of 77 winter wheat soil sampling points in Kenli District were calculated for quantitative evaluation.

### **3. Results and Analysis**

### *3.1. Screening of Soil Salt-Sensitive Vegetation Index*

Correlation analysis was conducted between UAV image vegetation index and measured soil salinity content respectively, as is shown in Table 2. In the correlation matrix, the DVI, MCARI2, and TVI of the eight vegetation indices had low correlations with soil salinity. The VIF value of OSAVI and NDVI was greater than 10, and there was strong multicollinearity. Therefore, the combination of vegetation index NDVI, NDRE, GNDVI and RDVI was selected as independent variables for modeling.


**Table 2.** Correlation coefficient between vegetation index and soil salinity.

Significance levels: \*\* 0.01.

### *3.2. UAV-Ground Collaborative Inversion Model of Soil Salinity Based on UAV Images*

Soil salinity inversion model was established by taking 4 vegetation indexes of 68 modeling samples as independent variables and soil salinity content as dependent variables, and 30 validation samples were used to verify the model. The results are shown in Table 3.


**Table 3.** Accuracy of three inversion models based on UAV images. ˅

It can be seen that all three inversion models show good stability, with their Rm<sup>2</sup> and Rv <sup>2</sup> both exceeding 0.6, and there is no "overfitting phenomenon". The highest modeling Rm<sup>2</sup> of the RF algorithm is 0.878, which is 0.089 and 0.27 higher than the Rm<sup>2</sup> of the BPNN and SVM algorithms, and the RMSEm is the lowest of 0.511, which is 0.159 and 0.38 lower than the RMSEm of the BPNN and SVM algorithms, respectively. From the validation results, the highest Rv <sup>2</sup> of the RF algorithm is 0.827, which is 0.16 and 0.226 higher than the Rv <sup>2</sup> of the BPNN and SVM algorithms, and the RMSEv is the lowest 0.473, which is 0.216 and 0.340 lower than the RMSEv of the BPNN and SVM algorithms, respectively. The inversion model of the RF algorithm has the highest accuracy. Therefore, The RF inversion model was selected to conduct soil salinity inversion for UAV images in the test areas A and B, and the salinity levels were divided according to five levels to obtain the distribution diagram (Figure 4a,b). Figure 4a,b are the interpolation maps of the measured soil salinity in the sampling area. Generally speaking, the inversion results of the wheat field soil salinity are basically the same as the interpolation results of measured data, and the change trend of the area proportions of each grade is roughly the same, but in comparison, the reflection of the inversion results on spatial distribution of soil salinity is more refined. Therefore, the soil salinity inversion model of winter wheat based on RF is better and has the best stable effect.

**Figure 4.** *Cont.*

**Figure 4.** Salinity inversion map (**a**,**b**) and regional interpolation map (**a'**,**b'**) of sampling points in the test area.

### *3.3. Collaborative Inversion of Soil Salinity Based on "Satellite-UAV-Ground"*

Based on the 75 surface vector data of test areas, the NDVI, NDRE, GNDVI and RDVI of the Sentinel-2A images of 52 modeling samples were taken as independent variables, and the soil salt content estimated by UAV was taken as dependent variable to establish soil salt inversion model, and 23 validation samples were used to verify the model. Figure 5 shows the accuracy of the modeling set and validation set of the three models. By comparison, RF model has the highest accuracy, modeling set R2 = 0.886, RMSE = 0.456, validation set R<sup>2</sup> = 0.850, RMSE = 0.505, followed by SVM model, the modeling set and validation set R2 are 0.796 and 0.649 respectively, and BPNN model has the lowest accuracy, the modeling set and validation set R2 are 0.734 and 0.630 respectively. Therefore, RF model is the optimal model for collaborative inversion of soil salinity based on "Satellite-UAV-Ground".

**Figure 5.** *Cont.*

**Figure 5.** Accuracy of the three collaborative inversion models, (**a**) RF model, (**b**) SVM model, and (**c**) BPNN model.

### 3.3.1. Analysis of Inversion Results of Soil Salinity in Wheat Area

A large-scale inversion of the soil salinity in the winter wheat planting area in Kenli district was carried out through the optimal RF model of the "Satellite-UAV-Ground" collaborative inversion, and the soil salinity level distribution map was obtained, as shown in Figure 6. Table 4 shows the area ratio of each salinity level. It can be seen that the distribution of non-saline soil in the wheat planting area of the study area is very small, accounting for only 0.05%. The proportion of winter wheat soil area shows a trend of decreasing with the increase of salinization, which is consistent with the actual situation of the survey. Among them, mildly salinized soil in wheat area is widely distributed, accounting for 61.32% of the total area. It is concentrated in the southwest where the terrain is relatively high and the northeast area affected by the fresh water of the Yellow River. The moderately salinized soil area was the second, accounting for 19.53%, which was distributed in all wheat regions. Severe salinized soil and salinized soil accounted for 19.1% of the total area and were scattered in wheat area.



3.3.2. Accuracy Comparison between "Satellite-UAV-Ground" Collaborative and "Satellite-Ground" Direct Inversion

Figure 7 shows the soil salinity inversion results of the RF model constructed directly based on Sentinel-2A images and measured salinity data. The R2 of the model is 0.56, which is far lower than the RF model of collaborative inversion. From the comparison of Figures 6 and 7, it can be seen that the trend of soil salinity distributions obtained by the two inversion methods are basically consistent. However, Figure 7 is generally lower than the salt value in the survey. This is mainly due to the influence of the mixed pixels during the directly satellite remote sensing inversion method. Figure 6 has fewer non-saline areas and more saline soil areas, which is consistent with the field survey. As a result, using the UAV inversion result as the medium data can effectively reduce the influence of mixed pixels on the inversion result.

**Figure 6.** Inversion results based on "Satellite-UAV-Ground", and (a) is a partial enlarged view.

**Figure 7.** Inversion results based on satellite remote sensing, and (a) is a partial enlarged view.

In order to further compare the inversion results of the "Satellite-UAV-Ground" collaborative method and the direct satellite remote sensing method, the salt values of 77 sampling points in Kenli District at the two inversion results were extracted and compared with the measured data. As is shown in Figure 8. The R2 = 0.741 and RMSE = 0.776 of the "Satellite-UAV-Ground" collaborative inversion results and the measured values, while the R2 = 0.591 and RMSE = 0.831 of the satellite remote sensing method and the measured values, indicating that the results of the "Satellite-UAV-Ground" collaborative inversion are highly consistent with the measured salinity, and the direct satellite remote sensing inversion results have a large deviation.

**Figure 8.** Scatter plots of measured sample points and soil salinity inversion results by two methods in Kenli District.

Therefore, "Satellite-UAV-Ground" collaborative inversion effectively improves the accuracy of soil salt inversion in winter wheat planting areas and obtains a more reliable soil salt distribution.

### **4. Discussion**

The measured data on the ground is the basis for quantitative analysis of soil, UAV near-earth remote sensing is the link between the satellite and the ground, and satellite remote sensing is the platform for large regional inversion. Combining the three will be an important way to obtain soil salt information at present and in the future. Therefore, in this study, the three platforms of "Satellite-UAV-Ground" data were used to perform soil salinity inversion. Compared with the direct satellite image inversion results, the inversion quality was significantly improved. The inversion results of two models are verified with ground-based data. The results show that the R2 based on the "Satellite-UAV-Ground" collaborative inversion model has been significantly improved, and RMSE and distribution pattern is also improved, but not so obviously. It might be because the salinity inversion result of UAV image is used as an intermediate bridge, its scope is limited and it is hard to fully express the situation of the entire study area. We will continue to improve research on collaborative approach, models, vegetation spectra, data sampling, etc. to improve the performance in terms of scale, accuracy, and temporal and spatial resolution.

The selected test areas in this study were located in two concentrated winter wheat planting areas in the southwest and northeast of Kenli District. The growth of winter wheat was significantly different, and the soil salt content was distributed in all grades, which was typical and representative. In order to improve the spectral and salinity data accuracy of winter wheat, the grid intersection method was used to determine the location of ground data collection, which could ensure the uniformity and accuracy of sample points more than the traditional random sampling point determination method. In order to ensure the representativeness of modeling samples, the samples are sorted and sampled at equal intervals at the ratio of 2:1 between the calibration set and the validation set. Compared with the traditional method of random partition modeling set and validation set [5,49], the data is more uniform and effective, thus ensuring the strong universality and high stability of the model.

The soil salt contents in the study area vary greatly, so the smaller the pixel and the closer the sampling point, the more accurate the information reflected. In this study, an inversion model was built based on the measured ground data and UAV images. The salinity value obtained from the inversion was regarded as the "salinity truth value", one pixel of Sentinel-2A corresponds to 40,000 UAV pixels, the average value of the soil salinity corresponding to these 40,000 UAV pixels was calculated, which was used as the salt value corresponding to the pixel of Sentinel-2A. Compared with the salt value measured on the ground, the "true salt value" is a more comprehensive representation of the

true salt status in the Sentinel-2A pixel. Most of the previous studies have realized the up-scaling inversion method of UAV model by establishing the relationship of spectral information between UAV image and satellite image [5,50]. The method in this paper is a collaborative inversion model based on satellite images which is constructed by taking the UAV "salt truth value" as the bridge of "Satellite-UAV-Ground" collaborative inversion. It can better maintain the information content and spatial structure characteristics of the original remote sensing data, and better guarantee the accuracy of salt inversion. It is an effective way to realize the integrated inversion of "Satellite-UAV-Ground".

Previous research results show that the machine learning inversion model is superior to the statistical model [51–54]. The machine learning method can show strong nonlinear fitting ability and excellent data mining ability and can better simulate the complex nonlinear relationship between soil salinity and remote sensing image characteristics. Therefore, three machine learning modeling methods were directly adopted in this study, and the comparison found that the RF model was particularly effective, especially the "Satellite-UAV-Ground" collaborative inversion model, with R<sup>2</sup> and RMSE up to 0.886 and 0.456, respectively. Therefore, if there is a non-linear relationship between the predictor variable and the response variable, a non-linear model such as RF will usually have a better fitting effect and produce excellent estimation accuracy.

The growth of winter wheat in coastal salinized areas is mainly affected by soil salinity, while other factors such as soil texture, pH value, climate, water, nutrients, and fertility have a more balanced impact on crop growth, which is a systematic error. Therefore, salt content is mainly considered for the impact on vegetation index. The vegetation index can be used to invert soil salinity indirectly, which has been confirmed by previous research [11,55,56]. However, the relationship between the vegetation index and soil salinity is different under different environmental conditions. Consequently, the constructed model is only applicable to wheat fields in coastal saline soil in spring. In order to make the obtained soil salinity better support crop production, the next step is to perform salinity inversion for different crops in the study area in different seasons, and use satellite data at different times to verify the model to further improve the reliability of the model.

When salinity inversion is carried out by using data from different remote sensing platforms, matching between data is very important. The predecessors usually take the average of multiple measurements within the ground range corresponding to the satellite pixel as the salt value corresponding to the pixel [57,58]. In this paper, the salt value of the corresponding range of Sentinel-2A pixels is obtained by calculating the average value of 40,000 pixels of the corresponding UAV. Compared with the previous methods, the accuracy of salt value corresponding to satellite pixel is improved, thus the spatio-temporal matching accuracy of the data is improved. Due to the uncertainty of remote sensing data, the band response functions of different sensors are different. When building a collaborative inversion model, it is necessary to fully consider the radiation characteristics of the data and select similar bands to reduce the impact caused by radiation and improve the accuracy of data spectrum matching.

### **5. Conclusions**

In this study, satellite images, UAV images and measured soil salt data were combined to build an inversion model to realize the "Satellite-UAV-Ground" collaborative inversion of soil salt in the coastal area of winter wheat. The main conclusions are as follows:

(1) The correlation between 8 vegetation indexes based on the multi-spectral bands of UAV images in the winter wheat test area and soil salinity was all greater than 0.5. According to the correlation coefficient and variance expansion factor, four sensitive vegetation indices, including NDVI, RDVI, GNDVI, and NDRE, were selected for modeling, and three machine learning inversion models, BPNN, SVM and RF were constructed. The RF model modeling set has R2 = 0.878, RMSE = 0.511, and its accuracy is higher than BPNN and SVM. It is the best salt estimation model. The inversion results are in good agreement with the actual distribution of soil salt in the test

area. The model has good predictive ability and applicability for the estimation of soil salinity of winter wheat in spring in coastal salinization areas.


This study makes full use of the advantages of satellites, UAV images, and ground data to construct soil salinity inversion model, using UAV soil salinity inversion results as intermediate data, and machine learning modeling methods to obtain a grade distribution map of winter wheat soil salinity in the study area which is consistent with the actual distribution of soil salt. The study proposed an effective "Satellite-UAV-Ground" collaborative inversion method for soil salinity, which provides a scientific basis for accurately grasping the distribution of winter wheat soil salinity levels and guiding agricultural production in the study area.

**Author Contributions:** Conceptualization, G.Z. and G.Q.; methodology, G.Q. and X.X.; software, G.Q. and X.X.; validation, G.Z., G.Q. and X.X.; formal analysis, X.X.; investigation, G.Q. and X.X.; resources, G.Z.; data curation, X.X.; writing—original draft preparation, G.Q.; writing—review and editing, G.Z.; visualization, X.X.; supervision, G.Q. and X.X.; project administration, G.Z.; funding acquisition, G.Z. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was financially supported by the National Natural Science Foundation of China(41877003), the Shandong Province Key Science and Technology Innovation Project(2019JZZY010724), and the Funds of Shandong "Double Tops" Program(SYL2017XTTD02).

**Conflicts of Interest:** There are no conflict of interest exits in the submission of this manuscript.

### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

### *Article* **Analysis of UAV-Acquired Wetland Orthomosaics Using GIS, Computer Vision, Computational Topology and Deep Learning**

**Sarah Kentsch 1,2,\*, Mariano Cabezas 3, Luca Tomhave 2, Jens Groß 2, Benjamin Burkhard 2, Maximo Larry Lopez Caceres 1, Katsushi Waki <sup>4</sup> and Yago Diez 4,\***


**Abstract:** Invasive blueberry species endanger the sensitive environment of wetlands and protection laws call for management measures. Therefore, methods are needed to identify blueberry bushes, locate them, and characterise their distribution and properties with a minimum of disturbance. UAVs (Unmanned Aerial Vehicles) and image analysis have become important tools for classification and detection approaches. In this study, techniques, such as GIS (Geographical Information Systems) and deep learning, were combined in order to detect invasive blueberry species in wetland environments. Images that were collected by UAV were used to produce orthomosaics, which were analysed to produce maps of blueberry location, distribution, and spread in each study site, as well as bush height and area information. Deep learning networks were used with transfer learning and unfrozen weights in order to automatically detect blueberry bushes reaching True Positive Values (TPV) of 93.83% and an Overall Accuracy (OA) of 98.83%. A refinement of the result masks reached a Dice of 0.624. This study provides an efficient and effective methodology to study wetlands while using different techniques.

**Keywords:** ArcGIS; big data; blueberries; deep learning; image analysis; orthomosaics; segmentation refinement; UAVs

### **1. Introduction**

Recent changes in global climate conditions influence species composition and accelerating the presence of invasive plant species in natural environments. Species that spread outside their native habitat and rapidly and effectively adapt to new environments are known as invasive species [1]. The spread of invasive species often benefits from ecosystem changes and habitat disturbances that weaken the natural species and open an ecological niche for invaders. Hence, invasive species can influence the biodiversity, thus limiting the growth of natural plant species due to a higher occurrence of an invasive species, which could lead to ecosystem degradation [2]. The fast adaption to multiple stress factors in environments could also lead to a replacement of native species and it may increase economic costs due to production losses in agriculture and forestry [3]. In Europe, 11% of the 12,000 identified species have caused damage to the economy, society, and the environment [4]. Reference [5] states that hundreds of invasive species find their pathways through horticulture, agriculture, etc., and the linearly increasing trend of invasive species numbers (from 1970 to 2007) indicates higher impacts of invasive species in the future. Reference [6] pointed out that not only invasive species have an impact on native plants, since several factors often interact with the environment that influence species distributions. In recent years, the need to precisely understand the ecological impacts of invasive

**Citation:** Kentsch, S.; Cabezas, M.; Tomhave, L.; Groß, J.; Burkhard, B.; Lopez Caceres, M.L.; Waki, K.; Diez, Y. Analysis of UAV-Acquired Wetland Orthomosaics Using GIS, Computer Vision, Computational Topology and Deep Learning. *Sensors* **2021**, *21*, 471. https://doi.org/10.3390/s21020471

Received: 27 November 2020 Accepted: 7 January 2021 Published: 11 January 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

species in ecosystems has become a key in designing and prioritizing natural resource management approaches [2], since the behaviour and impact of invasive species is still not well understood [3,7]. Furthermore, a high number of invasive plant are species spreading in natural environments, which increases the demand of management practices [5].

In the past two decades, an explosive spread of North American blueberry hybrids (*Vaccinium corymbosum* x *angustifolium*) has been observed in several moors in the northern German geest area, endangering the natural development of these protected raised bog areas [8]. The starting point of the spread has been almost exclusively located at existing or former commercial blueberry plantations [9,10], which can be found near or in the immediate vicinity of bogs or former peat extraction areas, due to good soil and local climatic conditions. Most of the recipient habitats are pine forests and bogs in various stages of de- and regeneration. Because of these characteristics, the American Blueberry (*Vaccinium angustifolium* x *corymbosum*) has been classified as a potentially invasive neophyte by the *German Federal Agency for Nature Conservation* [11]. After the degradation of wetland areas due to anthropogenic activities, protection programs, called "Moorschutzprogramme", were established by the state government of Lower Saxony for the conservation and the development of rare animal and plant communities in these areas [12]. Furthermore, activities that could threaten the goals of the protection program are prohibited, which increases the difficulty of conducting relevant field studies [12]. However, maintenance and development measures are needed in order to rehabilitate protected and relatively sensitive wetland areas, into which the invasive blueberry species *Vaccinium corymbosum* x angustifolium has migrated. In 2011, 20 counties in Lower Saxony reported stands of spontaneously growing blueberry bushes [13]. The potential area that is occupied by spontaneously growing blueberry bushes can reach several square kilometres within a few years [14]. Previous studies, as presented in [10,14], used grids in the field in order to plot the distribution of blueberry bushes within wetlands. Both of the studies focused on sites near blueberry cultivation areas, as the biggest spread was found in close proximity to blueberry plantations [9]. Those studies show the limitations in the studied area and lack an overview. It is still unclear how far the blueberry bushes have already spread and in which areas they occur. In order to implement effective measures in these areas and minimise the disturbance of sensitive biotopes, it is necessary to locate the individual blueberry bushes as accurately and early as possible. In addition, the following questions arise: does a displacement of natural species occur and where should what measures be taken against a continuing invasion? According to [14,15], relatively simple counter measures can lead to good results and prevent further spread, especially when invasive blueberry bushes are identified early. Therefore, a suitable and non-invasive method for recording stock development and distribution is needed. A simple tool is needed to rapidly, cost-effectively, and precisely detect invasive species in wetlands to counteract their rapid reproduction. Wetlands are protected environments with limited ground accessibility making UAVs particularly appropriate for data collection. UAVs offer the possibility to cover large areas with high resolution images, and they have proved their usefulness in a variety of studies in agriculture [16,17] and forestry [18–20]. Still, UAV images present challenges like the pre-pocessing of the data. Large amounts of data need to be processed, labeled, and annotated by experts, which is usually time consuming, before the data can be further analysed. The use of deep-learning techniques reduces the amount of time that is needed to extract information from the data, which increases the benefits for several applications.

Images that are acquired by UAVs can be analysed while using computer vision and GIS techniques. Important results can then be obtained by reducing the complexity contained in the images (using different image interpretation strategies) and the findings can be presented in elaborate visualisations [21]. Persistent homology, a tool of topological data analysis, can help to understand complex datasets by analysing their large scale geometric features [22]. In our study, we have used persistent homology to measure the spread of invasive species. Processing images and creating orthomosaics allow for a fast analysis of

large amounts of data. Deep learning techniques additionally automatize classification and localisation processes, and make it possible to incorporate expert knowledge into the automatic image processing pipeline. This has the potential to increase the scale of the resulting studies to reach large regions that are significant in terms of country-level invasive species detection and management.

In this study, the incorporation of all the mentioned techniques from remote sensing, GIS, computer vision, computational topology, and artificial intelligence allows for us to study invasive species on a large scale, with minimum disturbances and the incorporation of expert knowledge. To the best of our knowledge, this is the first study that includes different techniques and UAV gathered data to increase the understanding of an invasive blueberry species in wetlands. Furthermore, locating and studying small bushes in large areas and at the single-bush level was done for the first time. Therefore, UAV-supported methods offer an efficient possibility to discover individual young plants on a large scale and detect propagation hotspots at an early stage.

The following objectives guided this study:


### *State of the Art*

In recent years, remote sensing techniques have been used in various natural environments with the goal of reducing the need for in situ measurements [23]. Low-cost data gathering, time saving, and larger study areas are the benefits. Furthermore, data can be directly used and processed within Geographical Information Systems (GIS) [23]. This has been done successfully, for environmental studies [24–26]. Closely related to the current work, Reference [27] proposed using GIS as a synthesising tool in invasive species management approaches. Reference [28] used satellite images of 1992 and 2002 in order to identify stress indicators and change detection in a wetland in Sri Lanka to quantify the conditions of the complex. The authors state that the inventory, mapping, and monitoring are needed to understand interactions in the ecosystem. Classification with the Maximum Likelihood Algorithm were performed, mapping and spatial analysis were used, and finally refined and verified with ground data. Their approach ([28]) reached 86% accuracy and provided detailed analysis. GIS in combination with remote sensing data was found to be an effective methodology for investigating wetlands. Reference [29] evaluated vegetation change detection while using the NDVI of remote sensing data and applied GIS in order to visualise the results. Landsat and Shuttle Radar Topography Missions were used to capture the Vellore District. Image interpretations were carried out using ERGDAS IMAGINE software in order to classify and detect changes in the vegetation. The differences in the NDVI values were used to analyse data sets of the years 2001 to 2006. The study provided information about the lowest decrease in the forest area by 6%, while agriculture land increased the most by 19%.

In comparison to satellite images UAV images provide a higher resolution and appear to be more suitable for wetland investigations, especially when focusing on invasive species. Higher resolution images allow for higher accuracies of image interpretations and feature extractions [30]. Several studies using UAVs in wetlands have already been carried out [31–34]. Reference [34]developed a method for detecting and mapping invasive species with UAVs. The authors acknowledged UAVs as suitable for monitoring eradication efforts in wetlands. Reference [33] realized the higher efficiency in gathering valuable and accurate information in comparison to field studies, when using UAVs and computer vision techniques to enhance classifications and health assessments in wetlands. Gandhi et al. [31] used UAV imagery for detecting invasive species and mapping their distribution and spread. This study also

compared data from two years (2009 and 2011) and detected an increase of 19.07%, which was confirmed by field studies with a total agreement of 94% and shows the suitability of UAV imaging for this kind of application. Reference [31] lanalysed the spread of *Spartina alterniflora* in Beihai in the years 2009 and 2011, using high resolution images acquired by UAVs. They captured images at a flight height of 800 m, generated orthomosiacs, performed multi-resolution segmentation by grouping homogenous pixels, and classified them. The target species was extracted by their pixel values. In a final step the accuracy was assessed and verified with field data by comparing three sample plots (a total of 166 samples) with the image results. A total accuracy of 94.0% could be achieved and, hence, provided information regarding an increasing spread of 19.07% from 2009 to 2011. The total infected area was, in 2011, 357.2 ha. Moreover, the image analysis provided the opportunity to identify areas with different levels of densities. Reference [35] collected UAV images of *Harrisia pomanensis* in the Limpopo province of South Africa. An area of 87 ha was captured by images that were taken at a height of 800 to 817 m. Orthomosaics generated with Agisoft Photoscan and pixel as well as object-based classifiers were used. The classification results of supervised and unsupervised classifiers were assessed. The supervised classification outperformed the unclassified one, and the object-based approach outperformed the pixel-based one. The best accuracy achieved was 86.1%.

Given the sizes of the data sets used and the need to detect and classify parts of the images that only trained experts can confidently tell apart the use of deep learning in problems that are closely related to the current work is of common use. For example, a recent study [36] used CNN (convolutional neutral network) in order to classify images from wetlands in an area of 700 km2. They used a fully trained and fine-tuned CNN with a limited amount of data. A comparison of the CNN with random forest was performed to evaluate the capacity and classification accuracies of CNNs. Canadian wetlands were captured with two RapidEye images with 5 m resolution in 2015. The validation data were sampled in 2015 and 2016 and four wetland classes were identified using 1000 samples. The network was trained with patches and 30,000 iterations and then tested in a second step. The CNN outperformed the random forests and an overall accuracy of 94.82% was achieved, varying between 76.65% and 98.74%. Deep convolutional neural networks were also used by [37] in order to classify AUV-acquired wetland images. The 677 m × 518 m study area was located in Southern Florida. The authors used processed orthomosaics and multi-view images and then compared them with the performances of random forests and support vector machines. Image segmentation was done with Trimble's eCognition by first segmenting objects, then extracting features, and finally training a classifier. The results of the study show the advantages of deep CNN, reaching an accuracy of 82.02%, when multiview images were used and with lower accuracy when orthomosaics were used (71.69%). A similar approach was used in [38]. 3800 images of the Brazilian national forest (Kaggle dataset) were used in order to identify *hydrangea* in the images. The dataset contained two-thirds of images, including the invasive species, a smaller fraction, where the invasive species appeared only in parts of the images, and a third small fraction that included no plants at all. The authors used three models: VGGNet, DenseNet, and Inception, which were pre-trained with ImageNet. Data augmentation was used and accuracies of 97.6% were reached.

### **2. Materials and Methods**

### *2.1. Study Area*

The study area "Lichtenmoor" is located in a wetland region about 60 km northwest of Hanover in Lower Saxony, Germany (52°43 06.2 N 9°20 41.5 E) (Figure 1).The total size of the moor area is 38 km2 [39]. Post industrial peat cutting characterises the central area of the studied wetland. As a subsequent use, some former peat cutting areas have been rewetted with the aim of regenerating raised bogs. In the surrounding area, parts of the Lichtenmoor have been designated as nature reserves. Former hand peat cutting sites are located at the edges and in parts of the nature reserves. Agricultural areas dominate the remaining areas, mostly grassland, dry to moist moorland forests, scrubby heather and moorland degeneration stages, pioneer stages of moorland rewetting, and peat extraction areas under cultivation. The case study area covers a total area of 62 ha and it is located in the central part of the Lichtenmoor region. To the northeast, it borders a pine plantation, while at its southwest border current peat extraction areas are located. Blueberry plantations can be found throughout the Lichtenmoor, especially on the outskirts of the localities Lichtenhorst, Heemsen, Sonnenborstel, and Steimbke.

**Figure 1.** Location of the study area in the north of Germany. In the upper right part of the figure, the study sites are marked with different colours. The bottom right part an example orthomosaic is shown with detail of the different classes.

### *2.2. Characteristics of the Blueberry Species*

Blueberries have been cultivated in commercial plantations in Lower Saxony since the early 1930's [14]. Since then, the area under cultivation has steadily increased. In 2005, the area with blueberry cultivation in Lower Saxony was approximately 1400 hectares [13]. In nature, blueberries are mainly distributed by birds and small mammals, who spread the seeds. Once established, plants can spread in a vicinity by clonal growth and the high regeneration potential of blueberries favours a strong spread [10]. The species prefers acidic locations, such as pine forests or wetlands. Especially, raised bogs in their degeneration stage provide ideal habitat conditions for invasive blueberries [14]. Thus, blueberries are growing increasingly wild in neighbouring areas. References [9,14] found a correlation between the density of blueberries and the distance to blueberry plantations; a maximum distance of 1700 m was recorded. Near cultivated areas, the feral blueberries form dense shrub stands with height of up to 2–3 m and have a ground coverage of up to >60%. With increasing distance, the degree of coverage decreases rapidly [8,9,13]. Since these studies were already conducted in the last millennium, a larger distribution must be expected today. Reference [10] describes a distribution of blueberry bushes over 4 ha in the "Krähenmoor", where the maximum distance to the plantation was identified at 100 m. Blueberry bushes

are low and can reach a height of 60 cm, but, occasionally, tall species can be found with a height of3m[10]. In the course of the increasing growth and dense shrub structures, other ground vegetation is displaced, since it cannot exist under the shade of blueberries. Other presumed effects of blueberry cultivation are reduced evaporation rates and the influence on nutrient cycles in wetlands, which, in turn, can have an impact on existing plant species. Therefore, human interventions are necessary to protect the sensitive rare structures and characteristic plants of wetlands [10].

### *2.3. Data Collection and Pre-Processing*

Image collection was performed by using a DJI phantom 4 UAV in autumn 2018, because of the seasonal red colouring of blueberry leaves, which makes them easily detectable (Figure 2). For sites B1 to B4, 490 to 584 images were collected, while 1346 images were taken for B6. The flight height was 50 m and front and side overlaps of 80% were chosen. These images were then processed while using the Metashape software [40] to align images in order to produce one orthomosaic and DEM (Digital Elevation Model) for each site (Figure 1). It should be mentioned that an overlap between the sites B1 to B4 was chosen, so the east and west borders of the orthomosaics B1 to B4 are overlapping. All of the obtained orthomosaics were annotated while using the open source image manipulation software GIMP [41]. For three of them (B1, B2, and B3), the whole orthomosaic was annotated and each pixel was given one of the following six labels: blueberries, trees, yellow bushes, soil, water, and dead trees (Figure 1). The class trees contains pine trees (*Pinus sylvestris*), the class yellow bushes contains shrubby birches (predominantly *Betula pubescens*, secondarily *Betula pendula*). Binary layers for each of the six classes were created for each of the three orthomosaics while using the pixel-level labels. These annotations were based on colour, shape, and context information contained in the orthomosaics. In the last two orthomosaics (B4 and B6), only blueberry bushes were annotated.

**Figure 2.** Workflow of the paper. Gray is our data base; purple boxes show softwares/programs used in this study; white boxes with a blue outline are generated files; dark blue are processes used for the deep learning (DL) classification and detection; light blue are tools in ArcGIS.

In order to train deep learning models to detect blueberry bushes orthomosaics annotated with all of the aforementioned labels were needed. Therefore, only B1 to B3 were used in the deep learning section of this study. These three orthomosaics, as well as the corresponding annotated binary layers were divided into axis-parallel patches of side length = 100 (hereafter "patch size"). This value was determined by taking the sizes of the blueberry bushes in the images, which ranged from 20 to 100 pixels in radius, into account. The classes in each patch were stored in a separate *label* list. In general, patches contained more than one class and, therefore, our problem can be defined as a multi-label patch classification problem.

### 2.3.1. Data Processing Using Arcgis

This section deals with data processing performed with ArcGIS pro 2.4.1 and python in order to identify parts of the orthomosaic containing blueberry bushes, in order to visualize them and analyse their characteristics. ArcGIS is a Geographic Information System software that visualises and comprehends geographic data. The software provides over 1000 tools to analyse real world data, including UAV-acquired images. The mapping options allow to visualise the gathered data within the correct location in an eligible base map. The primary purpose of this study was to provide information about the location and distribution of invasive blueberry species and map them for management.

In this context, the positions of the blueberry bushes in the five orthomosaics were digitised by hand as point data in an ArcGIS shapefile and several analytical tools were then used (Figure 2). The *kernel density* tool functions were used to calculate the magnitudeper-unit area from the blueberry points. Smaller search radiuses were used to show a detailed density raster. The tool *integrate* was used to group blueberry bushes that fall into a specified distance, as specified distance, 3 and 6 m were used. With the tool *collect event*, the number of points which were integrated before, were summarized in a new layer. Those steps were necessary to perform an *optimized hotspot analysis* of the blueberry abundance with the blueberry point shapefile. The hotspot analysis identifies the significant difference between the neighbourhood of a feature in comparison to the extent of the respective study area. Is the value of a feature significantly higher, it is considered to be a hotspot and the tool provides a feature map with three levels of confidence (90%, 95%, and 99%). As input for the hot spot analysis, the created layer of the tool *collect events* was used and analysis field counts were chosen. Furthermore, it also indicates the significantly lower features.

On the basis of the annotation made for all orthomosaics, several simple python codes and ArcGIS were used to perform image analysis. Pixels were counted for all orthomosaics, as well as all annotated layers. The number of black pixels in the annotated layers were specifically counted in order to obtain the percentage and area in m2 per class. Additionally, the overall area presented in the orthomosaics was calculated in hectares. Because the focus of this study was on the blueberry bushes, several statistical values were generated: the number of blueberry bushes was counted and the number of blueberry bushes per ha was calculated for each orthomosaic. Additionally, the total area, as well as the area per blueberry bush, were computed in m2. Furthermore, the proportion of blueberry bushes in relation to the vegetation was calculated in %. Finally, height values were computed on the basis of DEMs, annotations of the blueberry bushes, and annotations of ground points per site. Ground points were annotated close to blueberry bushes in order to increase the accuracy of the computed height. Maximum height values were estimated in a first analysis and the median height in a second analysis to evaluate the results.

### *2.4. Persistent Homology*

Persistent homology provides topological information of complex datasets [42] at different spatial resolutions. This information deals with the connectivity of nearby points and it can be computed in different dimensions. For the purpose of this study, we worked with 0-D persistent homology (usually expressed in the form of H0 diagrams). H0 homology can be seen as growing disk at uniform radius-increase speed around pre-defined data points. In order to use this tool, we first discredited the manually annotated blueberry

regions by uniformly sampling them. Subsequently, the radius around each sample point was increased to grow blueberry bush regions. When two blueberry regions were merged because of the expansion, one of the regions was considered to be dead because it was absorbed by the other region. As time passed, the number of connected blueberry regions decreased, and finally all of the regions were connected in one region.

The H0 diagram shows the change in the connectivity of the blueberry bushes by plotting the time/radius when blueberry regions get connected. False regions that were produced by our sampling of the annotated regions were discarded and only those parts of the diagrams that were obtained after all sampling points in each region had been joined together were considered. Based on this outcome, the radius was calculated until 1%, 10%, 50%, and 90% of the blueberry regions were connected and plotted. The diagram will vary greatly, depending on the number and position of blueberry bushes in the input image.

### *2.5. Deep Learning Techniques*

Deep learning is a trending field of machine learning that focuses on fitting large models with millions of parameters for a variety of tasks, such as image classification and segmentation. These approaches have been rapidly gaining attention in computer vision tasks, due to their recently increased accuracy. Deep learning models commonly learn from examples in a supervised manner. First, an *architecture* or a graph of connected nodes is defined. These nodes are often grouped in *layers* that perform a specific operation, and the combination of a large number of layers is referred to as a *deep neural network* (**DNN**).The typology of the nodes, the number of nodes per layer and the connections between them determine the behaviour of the network. In general, two main types of nodes are used: linear nodes, expressed as matrix multiplications and nodes that introduce non-linear functions (such as the sigmoid function). The weights in linear nodes are usually initialised with random values following a specific distribution. Afterwards, the network is given samples of the data, known as *training samples*, which contain instances of the problem (i.e., image intensities) with their corresponding solutions (i.e., labels). These samples are iteratively run through the network in order to evaluate its current accuracy and the weights are updated following an optimization process.

In this study, DNNs were used to locate and identify the six classes that are defined in Section 2.3, with a focus set on the blueberry class. The basis for this deep learning approach is described in previous studies [20,43], which led to the use of the algorithms that are described in this section.

Our approach is based on a patch classification model that uses the patches of 100 × 100 pixels described in Section 2.3. A patch of orthomosaic B1 and B3 covered; therefore, 700 cm × 700 cm and a patch of orthomosaic B2 500 cm × 500 cm. For each patch, a list containing the class labels was created from the binary maps for each class. This classification is usually referred as multi-label, since each input patch might contain different labels (i.e., a part of the patch may contain soil, while other parts of the same patch could also contain bushes or blueberries).

Deep neural networks for classification have two major components: a feature extraction stage and a prediction stage. At the first stage, convolutional operators are trained in order to extract salient and meaningful features (such as texture) while at the second stage these features are used to predict the final labels for the given input patch or image. In order to train general and robust feature extractors, a large pool of heterogeneous images with different properties (lightning, colour, view, etc.) is needed to capture all of the possible image variabilities. However, as proven by our previous work [20,43], transfer learning is a useful tool for image analysis applications, where the training dataset is too small to properly train these feature extractors from scratch.

In our case, only three different orthomosaics are available; hence, we decided to use transfer learning by loading a pre-trained ResNet50 architecture with weights from ImageNet, due to its accuracy and reduced training time. ImageNet is one of the largest

image databases for image classification research, with more than 80,000 labels and at least 1000 images for each label.

In order to perform the evaluation of the proposed model, two of the three orthomosaics were used for training and validation and the third one was used for testing. This cross-validation strategy is usually referred to as a leave-one-out strategy. All of the orthomosaics were used once for testing, training, and validation by rotating them for each experiment. Patches from the testing orthomosaic were not included for training or validation to avoid data leakage during training.

Two main approaches were used in order to obtain a higher detection rate for the blueberry class:


We considered labels for all the patches used and the relation between (1) predicted values resulting from our algorithm and (2) real values as stated in the manually-annotated ground truth in order to assess the predictive power of our algorithms.

We then broke all of the patches into the usual classification categories of:


We used the two following metrics (True Positive Rate or Sensitivity and Accuracy) in order to summarize the occurrences of the four categories.

$$TPR = SENS = \frac{TP}{TP + FN} \qquad ACC = \frac{TP + TN}{TP + TN + FP + FN} \tag{1}$$

Finally, we modified the algorithm to present the results in a way that was more usable for end-users. The 100 × 100 patches used for prediction managed to capture most of the occurrences of the Blueberry class (see Section 3.2 for details). However, these rather large patches also included large areas that actually contained no blueberries. While expert users could easily use these results as a starting point in order to quickly identify the exact location of blueberry bushes, we felt that refining our prediction using smaller patches would make their work faster, while also providing clearer information for nonexpert users. Consequently, we divided each of the predicted 100 × 100 patches into

16 25 × 25 patches, re-sampled each of these newly-made smaller patches to the image size that is used by our DNN and re-classified them. This resulted in a refined result made up of 25 × 25 patches. This process had the disadvantage that, if errors were made, some of the correctly predicted blueberry pixels might be lost. In order to evaluate this issue, we considered the *TP*, *FP*, *TN*, and *FP* status of each pixel in each patch and measured the percentage of positive pixels that were covered by our predicted patches as well as the Dice coefficient that gave us an indication of the relative weight of blueberry pixels inside of our predicted patches:

$$Dice = \frac{2TP}{2TP + FP + FN} \tag{2}$$

### **3. Results**

This section is divided in two parts: the first part of the analysis focused on the manual annotations of the wetland vegetation and, most specifically, of the blueberry bushes. GIS, computer vision, and persistent homology were used to describe and quantify the characteristics of the blueberry invasion in all of our test sites. In the second part of this section, the results of Deep learning techniques are presented. In this case, our goal was to assess to what extent these technologies can be used to automatically generate the annotations that were used in the first part to characterise the invasion. The general workflow can be seen in Figure 3.

**Figure 3.** Image analysis workflow. Consists of two parts, the manual approach using manual annotation, GIS and persistent homology, and the automatic approach while using deep learning and segmentation to analyse the blueberry invasion.

### *3.1. Analysis of the Blueberry Invasion*

In this part of the study, we focused on characterising and measuring the blueberry invasion of the wetland.

### 3.1.1. Quantitative Analysis of Blueberry Bushes

The distribution of the classes in the images (blueberries, trees, yellow bushes, soil, water, and dead trees, see Section 2.3) was analysed and the state of the invasion was

assessed by gathering information regarding the areas of the sites, the numbers of blueberry bushes, and the average area per bush.

One important aspect was to calculate the area of blueberry bushes within the orthomosaics. The area covered by the orthomosaics varied between 10.6 to 12.5 ha, only B6 was larger with 15.5 ha (Table 1). Together with the annotations made for B1 to B3 the area of each class was calculated (Figure 4). As can be seen in the orthomosaic (Figure 1) the main part of the image represented soils. This was validated by the area calculations: with 76% of the orthomosaic B2 and 89% of B3, soil represents the highest values of all classes. The second smaller pie shows the living vegetation varying between 4.7% in B1 to 18.6% in B2. Out of the living vegetation, 8.2% (B2), 15.0% (B3), and 21.1% (B1) are blueberry bushes. In most of the orthomosaics blueberries were the least frequent class (with 1 to 1.5%).

**Figure 4.** Distribution of annotated classes for the orthomosaics B1 to B3.


**Table 1.** Area and counting measures of blueberry bushes detected in the orthomosaics.

The number of blueberry bushes varied from 235 in orthomosaic B6 to 687 in B2 (Table 1). The site areas of orthomosaic B1, B3 and B4 were similar, while orthomosaic B6 is the largest site containing the least number of blueberry bushes and orthomosaic B2 contains the greatest number of blueberry bushes in the smallest area. The ratio could be confirmed by calculating the blueberry bushes per ha (Table 1). In another step, annotations were used in order to calculate the total area covered by blueberry bushes. In orthomosaic B6, an area of 278.07 m2 was covered by blueberry bushes, which represents the smallest area and it resulted in an area of 1.18 m<sup>2</sup> per blueberry bush. The largest area was covered by blueberry bushes in B2 with 1885.51 m2 . The average size of the bushes were similar for B2 and B3. In site B1 the average size of blueberry bushes was the highest with 3.55 m2, while the covered area was third lowest with 1331.41 m2.

Because the covered area and the average size of the bushes could be calculated, the next point of interest was the area and height per blueberry bush (Figure 5). Bushes were grouped into six to smaller than 10 m<sup>2</sup> and over 10 m2 because the percentage of blueberries decreased towards larger cover areas. B1 and B2 had approximately 30 bushes between 6 and 8 m2, which was the maximum of all sites. The mean areas were computed for all orthomosaics, indicating that B1 had a high number of large bushes with a mean area of 3.57 m2. The smallest blueberry bushes could be found in orthomosaic B6 indicated a mean value of 1.18 m2. In general, most of the blueberry bushes showed areas of up to 2 m2, a lower amount distributed between 2 and 4 m2 and the lowest numbers distributed in areas greater than 4 m2. B1 was an exception, with around 10% per class over 4 m2. The highest areas calculated range between 17 to 25 m2, with B1 containing four bushes in that range and 27 with areas above 10 m2. The lowest areas were found to be less than 10 cm<sup>2</sup> for B1/B4 and approximately 15 cm<sup>2</sup> for all other orthomosaics.

A similar distribution can be seen in Figure 5, where the number of blueberry bushes were plotted against the height. Classes were chosen for each 0.5 m starting with 0 m up to lower than 3.5 m and more than 3.5 m. This distribution was chosen due to the characteristics of shallow blueberry bushes that reach 60 cm and tall species that reach 3 m. Regarding the maximum height, no height was computed for 2.3% (B1) up to 15.5% (B6) while the numbers were higher when the median height was considered (11.0% for B1 up to 35.2% for B6). In general, the median height values were higher for the classes 0 m and 0.5 m in comparison to the maximum height, while the values are lower from 0.5 m.The lowest height values started from 0.01 m (B6), 0.03 m (B1), 0.07 m (B3/4), and 0.1 m (B2) for both max. and median height. In general, the maximum and the median height distribution of the orthomosaics was similar. Almost all of the blueberry bushes in orthomosaic B6 were within the class < 0.5 (83.3%). In B1 the number of blueberry bushes in this same class was 79% and 15.2% was between 0.5 and <1 m, which was similar to B6. Orthomosaic B2 to B4 showed a Poisson distribution, whereby B2 had the highest number in 1 < 1.5 m with 21.2 m, while 41.6 of the blueberry bushes in B4 had the highest value in <0.5 m. Furthermore, in B2 and B3, more than 50% of the blueberry bushes reached heights between 1 m and 3.5 m (57% and 57.7%).It has to be considered that the area was calculated on the shapefiles in ArcGIS, while our developed python code was used in order to calculate the height of the blueberry bushes. Because the input for the code was the annotations of the blueberry bushes that were stored as a PNG file, those bushes that were close together were grouped. Therefore, the height values were not always calculated for an individual bush, which resulted in a different number of blueberry bushes per site: 309 (B1), 519 (B2), 461 (B3), 394 (B4), and 219 (B6).

**Figure 5.** Distribution of the area and height values of blueberry bushes. From top to bottom: area, maximum height, and median height.

### 3.1.2. Analysis of Spread Patterns

In a second step of our quantitative analysis of the blueberry invasion, the concentration, density, and spread patterns were examined. GIS and persistent homology were used to assess these issues. Characterisations of concentrations and densities can indicate the number of blueberry bushes within a given region of the orthomosaic, which exceeds a simple location because the distribution of the bushes can be analysed precisely. Clustering bushes and mapping densities further increase the understanding of the distribution. Together with the persistent homology and hotspot analysis, the spread can be defined for all orthomosaics, which helps to characterise the invasion.

The first step was to cluster blueberry bushes by using specified distances, of which 3 and 6 m were chosen, due to the calculated area of the blueberry bushes. The average diameter was considered to be 2 m for the different sites, and therefore a diameter of 3 m was found to be appropriate to especially group blueberry bushes that were close to each other. The results of both distances were compared and they are listed in Table 2. When blueberry bushes were located in a range of 3 m, they were clustered with the following results. From orthomosaic B3 to B1, 35.51% to 39.87% were clustered. The highest number of clustered bushes were 33 in B3, followed by 25 in B2 and 10 in B1. In comparison to B1 to B3, B4 and B6 had around 28.3 % clusters with three and more blueberry bushes. The highest number within a cluster was nine for B4 and 13 for B6. After increasing the range to 6 m, the number of blueberry bushes clustered in the group 3 or more bushes increased to 69.57%, which is more than 30 percentage points. B1, B3, and B4 had a similar increase of around 15 percentage points and reached 56.63 % in B1, 50.43% in B3 and 44.35 in B6. With less than 10 percentage points, 37.68 % of the blueberry bushes were grouped together with more than three bushes.


**Table 2.** Clustering results that are based on point shapefiles of the sites.

Based on the point shapefiles of the blueberry bushes, density maps were generated to see how the bushes were distributed within the map. Figure 6 provides three examples of the orthomosaics B1 to B3. Areas with a high density are marked in red and low densities in dark green. Orthomosaic B1 has one large density spot in the northwestern part of the map, while the southeast direction the density decreases with only single or paired bushes. Orthomosaic B2 shows four density spots. Two smaller ones were located in the northwest, a larger spot is in the middle of the orthomosaic and a final one in the southeast. The space between the middle and southeast spot is covered by blueberry bushes, which was similar to the distribution of B3. In orthomosaic B4, nearly the whole area is covered with green to reddish colours. There are three dense spots in the northwest, two spots in the middle and one in the southeast. In comparison to B2 and B3, the spots are smaller. Orthomosaic B6 covers a larger area than all other orthomosaics, but only three density spots could be identified in the middle of the orthomosaic. There were smaller groups of blueberry bushes along the borders of the orthomosaic, and single ones are distributed close to the groups of bushes.

Another analysis focused on the point analysis to generate a map of hotspot areas in order to analyse the spread of the blueberry species. The point analysis used the manually marked blueberry bushes to identify where the proximity of the bushes was significantly different (hot and cold), and to quantify those that were not identified as significantly

different. In B1, two 90% confidence hotspots were found in the north. 21 clusters were identified to be 90% significantly different from the study area. The hotspots in B2 were concentrated in the south-easternmost part of the orthomosaic. 26 clusters (out of 220) were significantly different to the study area with a confidence interval of 99%. These points contained all of the bushes located in the south-easternmost part of the orthomosaic. In B3 16 out of the 214 clusters fell into the 99% confidence interval, all located in the south-east of the orthomosaic. The same characteristic was found in B4. 23 clusters out of 248 were found to be significant with a 99% confidence and seven points with 90% to 95% confidence. B4 was the only orthomosaic containing two points considered to be cold spots with 90% confidence in the centre of the orthomosaic.

**Figure 6.** Density map for the blueberry species for the sites B1 to B3 (location see Figure 1). Areas of low densities are marked in green and high densities are red coloured. A gradient between green and red represents values of medium density. White points mark the location of the blueberry bushes.

Finally, the persistent homology was performed, as described in Section 2.3.1. The radius was plotted against the fused region, as can be seen in Figure 7. The orthomosaics B2, B3, and B4 show a similar trend, while B1 and B6 also follow a different, but similar, trend to each other. B2 is the first orthomosaic, where 1% of the blueberry bushes were fused with a radius of 386 and B1 needed the largest radius with a value of 497. In all orthomosaics the radius needed to fuse up to 10% of the blueberry bushes is similar with values between 415 and 557. There is a small gap of approximately 150 between B3 (945), B6 (998), B1 (824), B2 (768), and B4 (831), when 50% of the blueberry bushes were fused. The radius needed to fuse 90% for B1 and B6 are 3279 and 3611, while, for B2 to B4, it is 1610 to 1709.

### *3.2. Deep Learning Results*

In this section, we describe the usefulness of deep learning techniques in order to automatically determine the location of blueberry bushes in our data. As stated in Section 2.5, a widely used network (ResNet50) was chosen and two main aspects were studied: how the data balance affects the final classification results and whether using transfer learning resulted in improved results. Several experiments were presented in a previous study [43]; however, we will only focus on the optimal results for this current study.

**Figure 7.** The persistent homology is plotted by radius against the region. Four fused regions were considered: 1%, 10%, 50%, and 90% and all sites are plotted.

Regarding transfer learning, *unfrozen* ImageNet [45] weights were considered to initialise the network. When the model weights are *unfrozen*, all of the layers are normally trained and, thus, all the weights are updated. Regarding the data balance, the blueberry patch loss was weighted eight times that of the soil class, and four times the amount of the other classes. We also performed upsampling of the blueberry class by creating 12 new samples for each patch, as detailed in Section 2.5. Finally, the soil class was downsampled to 50% of its original number of patches.

Because three orthomosaics were available, a leave-one-mosaic-out cross validation approach was applied in order to evaluate the results. One of the orthomosaics was used for training, another for validation and the last one for testing. In order to ensure that all orthomosaics were used at least once for training, validation, and testing, we rotated them accordingly. This section presents the averages results for the TPR and the accuracy results of the three testing stages that correspond to each orthomosaic.

By using the optimal settings that are presented in this section, the model improved from a low TPR value of 63.8% for the blueberry class when no data balancing was applied to a value of 93.39%. In both cases, the overall accuracy for all classes remained similar (98.83% and 98.10%, respectively). Furthermore, it could be observed that unfreezing the weights had a positive effect on the TPR. The best TPR value for the *frozen* weights was 37.12% without data augmentation while maintaining an overall high accuracy value of 98.01%. However, when using high data augmentation with *frozen* weights, the best TPR value of 87.99% was obtained at the expense of a lower overall accuracy value of 75.20%. These results suggest that ImageNet weights can be used for this problem, but they need to be updated (and *unfrozen*) when using data augmentation to focus on blueberry bushes due to the differences in images.

Regarding the refinement step of the algorithm, Figure 8 presents an example of the obtained results. For most of the predicted patches, the segmentation that was obtained with the refined version of the algorithm is much closer to the manual annotation. However, in a few cases, some of the bushes that had originally been detected are missed after the refinement. Table 3 presents the detailed results of these two issues.

**Figure 8.** Orthomosaic B1 is displayed with a combination of the manual annotations (black marked spots), the coarse mask (dark grey) and the refined mask (light grey). The red box was zoomed in to show a detailed view on the image and masks.

> **Table 3.** Numerical evaluation of the refinement step of the DNN. The rows marked "refined" stand for the algorithm after refinement, while the rows marked "Coarse" correspond to the algorithm without refinement for each of the three studied orthomosaics. The Dice coefficient, as well as the ratio of blueberry pixels in the Ground truth covered by each of the two masks, are presented.


On the one hand, the Dice coefficient, which evaluates the overlap and number of non-GT pixels that are contained in the predicted patches, improved significantly with the refinement algorithm from values around 0.2 to values in the 0.5–0.6 range. On the other hand, the ratio of GT pixels that are covered by the mask, which ranges from 0.88 to 0.95 for the non-refined mask was slightly inferior in the refined version (0.78 to 0.87).

### **4. Discussion**

The applied methodology used UAVs to gather information in a restricted access area. Techniques from several research areas were then applied in order to gain knowledge regarding the distribution and properties of the bushes of the invasive blueberry species. In this section, the results that are presented in Section 3 are interpreted in order to assess the stage of the invasion in each of the mosaics.

### *4.1. Difficulties with Data*

Blueberry plants show a characteristic red leaf colour in autumn, which make them easily recognisable and identifiable in comparison to other classes of vegetation. Both a simple identification and segmentation by colour were proposed and applied in one of the first segmentation approaches. However, partly visible soil with reddish tones constrained blueberry identification. This problem was especially critical for small blueberry bushes. In autumn, the leaf colours can vary between red, red with a yellowish tone, and partly black. This caused challenges for the annotations and for the deep learning algorithm, since the number of blueberry images was already low in comparison to the other classes and it made the colour approach not usable for this study. Further complications were given by light conditions during image taking. When the blueberry bushes had brighter red colours due to sun light, it was difficult to distinguish them from the ground. Bushes, which were located in the shadows, especially the ones that had a predominately black colour, were barely recognisable.

The analysis presented some difficulties in the calculations of the height and surface area of the blueberry bushes. The main problems to determine bush height are occlusions, due to nearby trees and difficulties due to dense floor covering vegetation. As the cluster analysis shows, the high density of blueberry bushes in some areas and their proximity increased the possibilities that the bushes were partly covered and the whole bush area was not visible, as already pointed out by [10]. Furthermore, bushes were often located close to trees that have canopies that can cover most of a blueberry bush. The areas calculated for the blueberry bushes, exceeding 4–5 m2, indicated that there has to be more than one bush, which was difficult to identify in the images, as well as for calculating the height. Wetland regions, imaged in the orthomosaics are grassland and covered with dense hassocks. Therefore, the soil is often not visible and the ground annotations often represent the height of the hassocks, which resulted in values of 0 m maximum height and even more bushes showed a value of 0 m, when the median height was calculated for smaller bushes, especially in B4 and B6. Therefore, it is assumed that the hassocks can reduce the real height of the bushes by 30 cm. However, the calculated height values exceeded 3 m, which is unusual for the blueberry species that are studied here. There were errors produced when the annotations contained parts of an overlapping tree canopy, which increased the maximum height. The median height was resistant to outlier values. When ground areas were annotated, the median generally decreased the height values, especially for small bushes. Annotations of the ground need to be set carefully, since the wetland was uneven and depressions could increase the height values of the blueberry bushes. Furthermore, the differences between the max. height and median height can be influenced by the structure of the bush canopy. Because of these difficulties, a correlation between the height and area values of blueberry bushes was not considered, but a comparison of the distribution of these values showed that the distribution was similar and the bush area was larger than the height.

Even though these values are estimations, the applied methodology gave a good overview over a large study area, which cannot otherwise be done by extensive field measurements due to wetland protection regulations. Therefore, despite the difficulties, the achieved results emphasised the following discussion of applications and the qualitative use of the introduced methodology.

### *4.2. Application in Landscape Management*

The collection of high resolution images and the gathered information can help to map and visualise the findings of this study. This information can be used in order to easily establish management measures against the further invasion of alien species into the wetlands, as pointed out in previous studies [31,35].

The area that was occupied by blueberry bushes was low in terms of the studied area (covering 1 to 1.5 %), which is lower than the identified 3 to 5% in [15]. However, when only the living vegetation was considered, the number of blueberry bushes was found to increase from 8.2 up to 21.1%. These percentages can be considered to be very high, due to the fact that the species is invasive and it does not belong to this sensitive ecosystem. B6 has the largest area and it contains the smallest number of blueberry bushes, with the smallest height and area values measured. Therefore, the invasion seems to be in an early stage and it should be easier to manage. Nevertheless, as shown by persistent homology and the high number of single bushes after clustering, bushes were wide spread, which increases the area where measures against the blueberry bushes need to be considered. The hotspot analysis and density map of B6 indicated that there are some bushes, which are concentrated in a dense spot in the middle of the area and distributed from there homogeneously. These findings allowed for determining that the progress of the blueberry bushes into this site is low.

B1 has a similar distribution, while the number of blueberry bushes per hectare is doubled in comparison to B6. The density map showed high densities in the northern part of the orthomosaic and a gradual decrease of blueberry bushes in the southern direction, which was confirmed by the hotspot analysis. The density of the bushes was higher than in B6, but, as indicated by the persistent homology, the spread was greater. This indicates that the blueberries invaded but did not reach every region of the site. Furthermore, the blueberry bushes in B1 have the largest bush area within all studied sites. The identified bushes maturity suggests that the blueberry species has invaded the area a long time ago. Because the density map provided the information that the blueberry bushes are mainly distributed in the north, there must be conditions in the south, which prevented further spread.

In comparison, B2, B3, and B4 showed a high spread in the southern direction, because blueberry bushes of various size were found everywhere and there were higher concentration areas and several spreading centres. These findings can be confirmed with the persistent homology and density map, indicating a high progress of the invasion. The three orthomosaics seem to have a homogeneous distribution and a gradual change to lower numbers in the southern direction. However, the significant differences that were found by the hotspot analysis indicate that there are conditions that influence the distribution of the blueberry bushes, as in site B1. In addition, all three orthomosaics show an area with a small density of bushes, which can probably be explained by the high water content in the soil correlated with unsuitable conditions regarding plant growth. The invasion of the blueberry species was characterised as such and far advanced for B2 to B4. Only B4 has smaller bushes that cover a smaller area, which suggests that the invasion is less advanced than for B2 and B3. The spread will probably increase there in the following years.

Furthermore, the distribution of the blueberry bushes can be connected to the proximity of trees, especially in B1 to B3, where bushes are mainly found around pine trees and shrubby birches, since birds use these as rest areas and mainly distribute seeds where they rest. An exception is B4, where the trees are located next to depressions that are filled with water, which confirms the unsuitable living conditions for blueberry species. In general, it seems that more blueberry bushes occur when the density of the living vegetation is higher, which can be explained by better living conditions and a better distribution through birds.

To sum up the results and interpretations, it was found that B6 showed an early stage of invasion. B1 shows an advanced stage of invasion with limitations in the south, while B2 to B4 show a critical, advanced stage of invasion, since the blueberry bushes can be found in the whole study site. The methodology used here helped to assess the stages of invasion.

Our study shows a method for helping preserve a sustainable and adaptive conservation of natural ecosystems. Together with further expert interpretations, a deeper understanding of wetland ecosystems can be achieved [30]. The calculated properties, height and area, are indices that can be used for plant growth monitoring [46], and, therefore, they provide useful information for the practical management of wetlands.

### *4.3. Contribution to Invasive Blueberry Studies in Wetlands*

Previous studies were conducted to identify areas of blueberry bush invasions. The first studies were done by [9,14] covering up to 12.5 km2 with estimations of blueberry bush coverage areas. Another study focusing on the invasive blueberry species was presented by [15], studying an area of 4.7 ha, which is nine times smaller than the area that was studied in our work. Reference [10] studied an area of 230 ha and characterised the blueberry invasion. The author conducted 11 days of fieldwork in order to capture the information regarding the cover area of blueberry bushes. The study area is less in our work with 63 ha, but, in comparison to the manual fieldwork, image capturing only took half a day. The pre-processing step needed the longest time, about 4 to 5 h per orthomosaic, since manual annotations were done in the beginning of the work. Therefore, the presented workflow of our study reduces the amount of time for gathering information significantly. The characteristics that were provided by [10] are mainly focused on the blueberry bush cover area, which were 16% of the 230 ha. Other characteristics were provided by general statements about average heights and clustering behaviours. Our study provided height and area measurements for each blueberry bush, which has not been done in previous studies. Based on the orthomosaics, further measures, like hot spot analysis, density calculations, and spread measures, could be performed in order to characterise the blueberry invasion in much more detail than previous studies have shown [10,15]. Furthermore, refs. [10,15] stated a high probability of loss of identification during fieldwork due to the density of the bushes in certain areas near former blueberry plantations, which were therefore predominantly investigated. In comparison to the mentioned studies, our work presents an objective grading, a precise cover area and each bush was detected in the captured high-resolution images. This information is essential in order to compare both the invasion and monitoring in different areas.

### *4.4. Automatic Masks Generation*

Deep learning techniques were used in order to assess whether DNN can be used to automatically detect the presence of blueberry bushes. Since the manual annotation of the blueberry bushes is the most time-consuming step of this workflow, this has the potential to greatly extend the range of studies. The results in Section 3.2 show that the ResNet50 network succeeded at the classification tasks associated to our problem and that the best results were obtained by re-training the whole network. In this respect, relying on pre-trained weights from ImageNet to solve our problem after minor re-training of the last layer is suboptimal. A dataset must be large enough to re-train the full networks. In our case, this meant using images that were taken from three orthomosaics covering a total of 33 hectares.

At the same time, the results also quantify how a data imbalance may result in a network that classifies most of the patches correctly, wven if the blueberry bushes were mainly misclassified. To address this problem, data augmentation as well as a loss function that took into account the number of pixels for each class were used to influence the weights which helped to reduce bias of the training towards the correct classification of blueberry bushes.

Wetland image classification was performed by [36,37] identifying four and six classes, reaching an overall accuracy of 94.82% and 82.02%. Transfer learning which was only applied in [36] showed the effectiveness of this technique for natural environment studies, when the dataset is large enough. Interestingly, the best result of [37] was obtained when using multiview images. We will consider this type of images in our future work. A similar approach to detect invasive species was used by [38]. Their approach applied deep learning with data

augmentation and transfer learning. Their highest accuracy was 97.6%, which is slightly lower than our overall accuracies of 98.83% and 98.10%. It is important to consider that, in [38], the dataset was mainly composed of images of the invasive species, while, in our case, only 2.64% of the generated patches contained the invasive species. This illustrates the effectiveness of the techniques used in our approach to overcome the imbalance in our dataset.

The results of the automatic classification were used in order to map the invasive species, which is an important first step, whereby a refined segmentation is essential to effectively determine the exact location of plant invasions [35]. The deep learning applications used classified images, and a refined mask provided the blueberry bush locations. The refinement step was necessary in this application, since most of the blueberry bushes are small. Hence, the refined mask offered a more precise localisation of the blueberry bushes. Even though not all bushes were found and some soil areas were misclassified as blueberry bushes, the refined mask could significantly reduce the time of manual annotations and provide maps of the studied area. Increasing the amount of blueberry data can help to optimize the classification accuracy and the localisation of blueberry bushes. This will be considered in our future research.

The generation of automatic annotation masks, as performed here, will allow for large scale studies with a minimum of disturbances in the studied environment. Therefore, UAVs and image analysis provide accurate and cost-effective surveys that are needed when studying invasive species [31]. The used techniques provided more information than that gathered by previous field surveys in wetland areas [10,14].

### *4.5. Limitation and Future Works*

The presented work provided a methodology to analyse invasive species from UAV images. Nevertheless, we faced limitations regarding the data that are described in Section 4.1. The autumn season seemed to be a good timing for image taking, but the difficulties regarding the varying colour of the blueberry bushes should be taken into account more carefully. Image taking could be done when the weather is cloudy and not windy, as suggested in previous studies [47,48]. The chosen flight height of 50 m resulted in high-resolution images, which created heavy orthomosaics. The resolution was reduced to 5–7 cm/pixel to be able to use image processing software. This led to difficulties in identifying blueberry bushes and their properties. For the future, different flight settings can be tested, as presented in [49,50], especially smaller flight areas and a reduced flight height can help to increase the resolution and precision of the results of blueberry bush properties.

The use of deep learning proved to be effective for this problem, but also challenging. As in all artificial intelligence approaches, a sufficient number of images that represent the variability of the desired target is necessary in order to train a supervised model that is usable in practice (in the current studies, this amounted to images that corresponded to 33 hectares). Furthermore, with the size of the current blueberry bush data in these images, careful use of data augmentation as well as a dedicated balancing approach was necessary. In order to improve the detection of blueberry bushes, a larger training dataset would likely be needed. For instance, because blueberry bushes are often planted for blueberry production, those fields could offer a good opportunity to collect new images.

Therefore, the detailed mapping within this study can further improve the results of the current status of the blueberry species invasion. Repeated data collection in the same area, as planned, will provide a year-to-year comparison, which will allow for the monitoring and analysis of the ongoing spread in the wetlands. Changes will be easily detectable within the wetlands and the studied blueberry species without disturbances of vulnerable plant and animal species and habitats, as already mentioned by [30,35].

#### **5. Conclusions**

In this paper, we introduced a multi-disciplinary methodology to quantitatively evaluate the role of plant species in ecosystems, including invasive species. The use of UAVs makes the approach applicable, even in restricted access areas and it increases the total area that can be studied, greatly exceeding the range of existing field studies. We used this methodology to gather information regarding wetland vegetation. Simple and time-saving methods were applied to classify vegetation and provide information about the properties of the invasive blueberry species found in our study site. The distribution of blueberry bushes was analysed in terms of their density, clustering, and spread. The relative importance of blueberries in the wetland was analysed (number of bushes, bush area, and bush height). This information was transformed into location, density, and hotspot maps to provide advanced visualization tools. Deep learning techniques were used in order to automatically detect and segment blueberry bushes, opening the possibility to further extend the range of similar studies.

**Author Contributions:** S.K., M.C., L.T., J.G., B.B., K.W., M.L.L.C. and Y.D., conceived the conceptualization and methodology, supported the writing—review and editing. S.K., M.C. and Y.D. developed the software, performed the validation, investigation and writing—original draft preparation. S.K., M.C. and Y.D. carried out formal analysis. S.K. was in charge of the visualisations. S.K., and J.G. administrated the data. J.G, B.B., M.L.L.C. and Y.D. supervised the project and provided resources. J.G., B.B. and Y.D. directed the project administration. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Acknowledgments:** The publication of this article was funded by the Open Access Fund of the Leibniz Universität Hannover. We thank Angie Faust for language corrections and Thomas Beuster (ÖSSM - Ecological protection station Steinhuder Meer) for the opportunity to take aerial photography in the study site and for his helpful contributions to the ecology of blueberries in the Lichtenmoor.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


### *Article* **Hierarchical Mission Planning with a GA-Optimizer for Unmanned High Altitude Pseudo-Satellites**

**Jane Jean Kiam 1,\*, Eva Besada-Portas <sup>2</sup> and Axel Schulte <sup>1</sup>**


**Abstract:** Unmanned Aerial Vehicles (UAVs) are gaining preference for mapping and monitoring ground activities, partially due to the cost efficiency and availability of lightweight high-resolution imaging sensors. Recent advances in solar-powered High Altitude Pseudo-Satellites (HAPSs) widen the future use of multiple UAVs of this sort for long-endurance remote sensing, from the lower stratosphere of vast ground areas. However, to increase mission success and safety, the effect of the wind on the platform dynamics and of the cloud coverage on the quality of the images must be considered during mission planning. For this reason, this article presents a new planner that, considering the weather conditions, determines the temporal hierarchical decomposition of the tasks of several HAPSs. This planner is supported by a Multiple Objective Evolutionary Algorithm (MOEA) that determines the best Pareto front of feasible high-level plans according to different objectives carefully defined to consider the uncertainties imposed by the time-varying conditions of the environment. Meanwhile, the feasibility of the plans is assured by integrating constraints handling techniques in the MOEA. Leveraging historical weather data and realistic mission settings, we analyze the performance of the planner for different scenarios and conclude that it is capable of determining overall good solutions under different conditions.

**Keywords:** HAPS; UAV; monitoring; constrained multiple objective optimization; temporal hierarchical task planning

### **1. Introduction**

Regular monitoring of land development (e.g., agricultural activities, big construction sites, essential infrastructure, wildforest, etc.) can be done using either satellites or airplanes. Recently, Unmanned Aerial Vehicles (UAVs) are preferred for a more cost-efficient and flexible deployment. However, UAVs flying at low altitude may not always be a solution, as their missions depend on the possibility of obtaining a permit-to-fly, on weather conditions that can be quite challenging at low altitude, and on the required flight range. In the case of fixed-wing UAVs, the takeoff and landing can also be troublesome for regular deployments or may not even be an option from surroundings with unfavorable topologies.

Solar-powered unmanned High Altitude Pseudo-Satellites (HAPSs) are considered a viable alternative to overcome the challenges arising from using satellites with a fixed orbit, manned airplanes, or UAVs for regular monitoring. As [1] explains, HAPSs are a type of light-weight High Altitude Long Endurance (HALE) aerial platforms that fly at low speed (in order to be energy efficient), in the lower stratosphere (where the airspace is quite calm and little congested), with extremely long endurance (e.g., [2] reports a continuous HAPSs flight of almost 26 days). Moreover, although still at its infancy, the development of HAPSs is promising and is expected to provide multiple benefits. However, given their light-weight build, operating HAPSs can also be challenging. Table 1 summarizes some of

**Citation:** Kiam, J.J.; Besada-Portas, E.; Schulte, A. Hierarchical Mission Planning with a GA-Optimizer for Unmanned High Altitude Pseudo-Satellites. *Sensors* **2021**, *21*, 1630. https://doi.org/ 10.3390/s21051630

Academic Editor: Alfred Colpaert

Received: 4 January 2021 Accepted: 16 February 2021 Published: 26 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

the general benefits (+) and challenges (−) of these platforms, according to the relevant characteristics of the HAPSs that contribute to each of them.



HAPSs operations contain the typical space flight phases, such as planning, processing, departure and flight operations, return and landing, refurbishment, and turnaround [3]. However, these phases present some peculiarities, due to the HAPSs characteristics. For instance, and according to the analysis presented in [4] on the trajectories obtained from a test flight conducted using the Kelleher platform in Arizona in 2018, this HAPS takes around a day to ascend/descend to/from its operating altitude by flying within a safe vertical corridor allocated for takeoff and landing. Subsequently, the platform stays as long as possible in the air and at the operating altitude in the lower stratosphere.

Given these continuous and extremely long operations, increasing HAPSs autonomy is essential. Besides, it is also useful from a safety and pragmatic point of view, as well as to reduce manpower and human error. Finally, and according to [5], the deployment of HALE platforms can be "cost-efficient", since by increasing autonomy and decreasing piloting, operation cost and be further reduced (without compromising safety and efficiency).

Hence, automated mission planning is convenient for the deployment of HAPSs that have to perform monitoring missions. However, although the airspace at this flight level is often relatively calm with mild winds, some rare weather conditions can pose serious safety-critical risks to the HAPSs. Moreover, since these platforms have limited maneuverability, reactive avoidance of risk zones may not always be possible. Therefore, weather risks must be addressed already in the mission planner on the Ground Control Station (GCS) to minimize the need of an onboard replanning or emergency landing. In particular, the following weather conditions must be considered:


Besides, and although HAPSs airspace is little congested (since the airliners fly below), High-level Flight Rules (HFR) also apply to unmanned flights above Flight Level (FL) 600 [8]. This implies that the airspace regulations must also be considered in the mission

planner (in order to avoid collisions with other stratospheric aircrafts), that it is recommendable to systematically organize and dynamically allocate the airspace [9], and that the flight plans must be communicated to the authorities before the execution of each mission. However, since HAPSs are long-endurance platforms intended to remain airborne, planning must also be performed during flight, before each monitoring mission starts.

Taking into account the previous considerations, this work focuses on increasing the autonomy and efficiency in mission planning that takes place on the GCS during flight operations but before the execution of the mission-related tasks. Our main goal is to optimize the mission success rate of monitoring the requested sites (i.e., to improve the chances of providing images of the sites with sufficient coverage and at the requested time windows), while reducing the risk of replanning by considering, at the planning phase, the predicted time-varying environment and the platform constraints. Furthermore, we assume the presence of one or several human operators in the mission-planning loop. Although their decision-making process is not considered in this work, our mission planner is developed to be part of a decision-support system that is responsible for automatically generating a group of feasible optimal plans and for presenting them as "suggestions" to the operators, who have to perform the selection of the final plan.

The work presented in this paper is closely related with the approach described in [10], which presented the preliminary version of our planner. The current version is improved, by (1) adopting a Multi-Objective Evolutionary Algorithm (MOEA) for constrained problems to optimize the mission plans and (2) by considering the uncertainty associated to the wind variability in the constraints. Besides, this paper presents new scenarios and analyzes the results of the new planner over a wider set of circumstances. Finally, it is worth noting that the relationships of other works with ours will be discussed later in Section 6, after the readers are acquainted with the main characteristics of our planner described through Sections 3 and 4.

The organization of this work is the following. Section 2 presents the problem at an abstract level and describes its main elements. Section 3 provides a more formal description of the problem, including the objective functions as well as the different components that conform with the constraint criteria. Subsequently, the implementation of the MOEA that supports the optimization process of the planner is described in Section 4, providing details on the encoding of the plan, on the hierarchical task decomposition process and on the particularities of our MOEA. Finally, results are illustrated and analyzed in Section 5, while a discussion on related work is provided in Section 6, followed by the conclusion and future work drawn in Section 7.

### **2. Problem Description**

This work focuses on the task planning for multiple HAPSs, equipped with electrooptical (EO) mission cameras and contracted to monitor repeatedly areas on the ground at specific time windows.

This section presents the main elements of the problem, describing the monitoring scenario that will be considered in this paper, introducing how the mission plan is defined, characterizing the payload of the HAPSs, reporting the mission requirements and constraints, and finally, explaining how the weather conditions to be considered at the planning phase are extracted.

### *2.1. Monitoring Scenarios*

The operation is assumed to take place in an organized airspace consisting of different types of dynamically allocated operation areas, in order to reduce congestion in the lower stratosphere. In particular, and as shown in Figure 1, the HAPSs will be able to operate in Mission Areas (MAs, represented in blue), Corridors (Cs, in gray), and Waiting Areas (WAs, in yellow).

**Figure 1.** Mission scenario (plotted on © OpenStreetMap) for monitoring multiple Locations Of Interest (LOI#) on the ground. The operation airspace is organized using dynamically allocated mission elements of Corridors (C#), Waiting Areas (WA#), and Mission Areas (MA#) that encompass LOIs.

Besides, the Locations of Interest (LOIs, in green) are the projection of the ground areas to be monitored within the time windows and at the frequency requested by the clients. LOIs of the same client with the same set of mission requirements are grouped in a MA, which defines the airspace (at the operating altitude for the HAPSs), allocated to allow a HAPS to monitor the encompassed group of LOIs. Additionally, the WAs are airspace made available for the HAPSs to loiter freely (for example upon sunset) or to exploit as a "corridor" to reach another connected MA. HAPSs are allowed to move between MAs only through the designated Cs or through WAs. This also implies that MAs are not to be used as "corridors", i.e., a HAPS entering a MA has to monitor its corresponding LOIs before departing through a connected corridor.

Appendix A includes further details of the HAPSs considered in this work and of the mission scenario represented in Figure 1. In particular, the numerical information on the model of the HAPSs is adapted from [11] and summarized in Table A1, while the dimensions of the mission elements are presented in Table A2.

### *2.2. Hierarchical Task Plan*

Execution of tasks for multiple HAPSs can be structured conveniently in a hierarchical manner, since the order of task execution depends substantially on the organisation of the airspace and on airspace-related constraints and requirements, which can be expressed at different levels of spatial resolution. In particular, we consider the following levels, ordered from the highest to the lowest level, according to the spatial resolution:


Figure 2 illustrates the representation of a hierarchical task plan considered in this work. In particular, on the left side, Figure 2a shows, over a portion of the mission scenario, the execution of the task plan for a HAPS that, after the first task in WA1, continues monitoring first the unique LOI in MA6 and afterwards the two LOIs in MA7. This figure also shows the waypoints followed by the HAPS to move within MA1 and MA7. On the right side, Figure 2b shows the hierarchical structure of the task plan of the HAPS represented in Figure 2a and of a second HAPS (not depicted in Figure 2a). That is, it shows how the plans that govern the two HAPS to monitor the LOIs within the horizon [*T*start, *T*end] are initially decomposed into the tasks expressed at the MA-level (represented at the two top timelines, one for each HAPS), followed by the tasks expressed at the LOIlevel (represented at the two intermediate timelines) and finally by the actions presented at the WP-level (shown at the two bottom-most timelines). At the lowest level, vertical color bars without text annotation represent instantaneous tasks, for example, to turn on or off the mission camera.

the airspace. (**b**) Task execution organized in a hierarchical structure.

**Figure 2.** Hierarchical task execution for HAPS.

### *2.3. Mission Payload*

The HAPSs are equipped with light-weight electro-optical mission cameras. The example camera considered in this work is inspired by the one described by Delauré et al. in [12], specially designed for unmanned HALE platforms. In particular, it is a light-weight (∼2.6 kg) and energy-efficient (<50 W) camera with two custom CMOS image sensors and with resistance to low pressure (down to 60 mbar) and to a wide range of temperature (from −70 °C to 60 °C). Its pixel counts for the width *wI* and height *hI* of the image are 1200 px × 10,000 px. With a ground sampling distance of 30 cm, an image taken from an altitude of 18 km at Nadir position records an area of 360 m × 3000 m of the ground.

With this mission camera and a gimbal that performs a cross-track sweep scan within 10 s from −45◦ to 45◦, a HAPS can record images covering a total width of more than 30 km, while advancing forward. Figure 2a illustrates the scanned footprint, which is a superposition of images taken during the scan. Even in the presence of a tailwind of 5 m/s, the HAPS, flying at the airspeed of 30 m/s (that is considered in Table A1) will not advance more than 360 m within a cross-track sweep, assuring some overlapping of the images between two periodic sweeps. Therefore, and as Figure 2a shows, we adopt a lawn mower scan pattern to monitor each LOI, with the distance between two consecutive tracks set at 30 km.

Finally, the cloud layers between the HAPS and the ground must be considered during the monitoring scans, as they reduce the coverage of the area recorded with the EO mission camera. For this reason, if the stitched image of any one of the LOIs of a MA has a coverage of the ground lower than requested, the monitoring of that MA will not be rewarded by the client.

### *2.4. Mission Requirements*

The HAPS team is rewarded by the contracting client if the ensemble of all the monitoring tasks performed on the LOIs within a MA is considered "successful". Therefore, we consider this ensemble of tasks a "mission" unit, which is rewarded according to the amount agreed upon by each client.

In particular, monitoring a mission unit is successful if the following mission requirements (MRs) are fulfilled:


The coverage percentage and reward obtained for monitoring successfully each MA of the scenario presented in Figure 1 is presented in Table A3 of Appendix A. Besides, the rewarding time windows for each MA are directly depicted together with the mission plans obtained by our planner, which are presented in Figures 10–14 of Section 5, since they are required to observe if the MA can be successfully or unsuccessfully monitored. Finally, in the scenarios analyzed in this paper, the time-lapse between the start times of two consecutive successful visits is set to one hour and each MA must not be visited more than three times a day.

### *2.5. Mission Constraints*

While mission requirements decide if a mission is successful, mission constraints (MCs) dictate the "feasibility" of a plan and are defined in the interest of operational safety by enforcing airspace regulation and measures for risk avoidance.

In particular, a plan is infeasible (i.e., it cannot be executed) if any of the following constraints is violated:


#### *2.6. Weather Conditions*

Weather conditions also affect the HAPSs and can make a given mission plan unsuccessful and/or infeasible. To take them into account, high-resolution global weather forecast based on numerical weather prediction models can be used, because this approach is beneficial compared to wide area weather forecast to foresee risk zones and to consider wind effects.

In particular, for this study we use the COSMO-D2 (COnsortium for Small-scale MOdeling) numerical weather data from the German National Meteorological Service (Deutscher Wetterdienst, DWD), which are updated every couple of hours to provide information on the cloud coverage and on the wind vector field with a horizontal resolution of 2.2 km and a temporal resolution of one hour [13].

In order to argue for availability of weather data that fit the underlying framework, we also list here a set of alternative meteorological services that can be used in the mission planner described in this paper, which also provide numerical global weather data such as the Global Forecast System (GFS, Ref. [14]) and the European Center for Medium-Range Weather Forecast (ECMWF, Ref. [15]).

### **3. Formal Problem Statement**

This section defines the problem formally, detailing the variables used to mathematically define a hierarchical plan, as well as the objective and the constraint functions used to evaluate them.

To help the reader understand the relationship of the elements presented in this section and the previous sections, Figure 3 shows how the MRs and MCs described in Sections 2.4 and 2.5 are mapped into the three Objective Functions (*OF*rew, *OF*eff, *OF*div) and the three constraint criteria (*ϕ*saf, *ϕ*coex, *ϕ*con) that are formally stated in this section. Furthermore, Figure 3 also illustrates the role of the operators as human decision makers, i.e., how they select a plan (*π*MA *<sup>k</sup>* , *<sup>π</sup>*LOI *<sup>k</sup>* , *<sup>π</sup>*WP *<sup>k</sup>* ) among the feasible solution plans that form part of the first Pareto front determined by the planner that will be presented in Section 4.

**Figure 3.** Relationships among the mission requirements (MRs) and mission constraints (MCs) described in Section 2 and the objective functions (OF) and constraints (*ϕ*) presented in Section 3.

### *3.1. Formal Definition of the Hierarchical Plan*

The goal of the planning problem is to find a hierarchically structured plan such as the one depicted in Figure 2b that entails the sequence of tasks to be performed by each HAPS, as well as their expected initial time instant and duration.

Formally, the solution is a set of sequences of time-stamped tasks for each hierarchical level and HAPS. More in detail:

• At the MA level, the plan can be represented as the ordered list of tasks *π*MA *<sup>h</sup>* displayed in Equation (1), where *o*MA *<sup>h</sup>*,*<sup>i</sup>* is the *i*-th mission task (i.e., a MA# or WA# of the mission scenario) on the list that will be performed by HAPS *h*, and *t* MA,start *<sup>h</sup>*,*<sup>i</sup>* and *<sup>δ</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* are the start time and the duration of the *i* − *th* task of HAPS *h*.

$$\tau\_h^{\rm MA} = \varsigma\_{h,1} \alpha\_{h,1}^{\rm MA}(t\_{h,1}^{\rm MA,start}, \delta\_{h,1}^{\rm MA}), \sigma\_{h,2}^{\rm MA}(t\_{h,2}^{\rm MA,start}, \delta\_{h,2}^{\rm MA}), \sigma\_{h,3}^{\rm MA}(t\_{h,3}^{\rm MA,start}, \delta\_{h,3}^{\rm MA}), \dots \tag{1}$$

Under this formulation, the high level mission plan of the first HAPS displayed in Figure 2b will be represented as *π*MA <sup>1</sup> =< WA1(*t* MA,start 1,1 , *<sup>δ</sup>*MA 1,1 ), MA6(*t* MA,start 1,2 , *<sup>δ</sup>*MA 1,2 ), MA7(*t* MA,start 1,3 , *<sup>δ</sup>*MA 1,3 )>.

• At the LOI-level, the plan *π*LOI *<sup>h</sup>* can be represented with Equation (2), where *<sup>o</sup>*LOI *<sup>h</sup>*,*<sup>i</sup>* is the *i*-th mission task (i.e., flyWA# or monitorLOI#∈MA#) that will be performed by HAPS *h*, and *t* LOI,start *<sup>h</sup>*,*<sup>i</sup>* and *<sup>δ</sup>*LOI *<sup>h</sup>*,*<sup>i</sup>* are the start time and duration of the *i* − *th* task of HAPS *h*.

$$\tau\_{h}^{\text{LOI}} =  \tag{2}$$

Under this formulation, the middle level mission plan of the first HAPS displayed in Figure 2b will be represented as *π*LOI <sup>1</sup> =<flyWA1(*t* LOI,start 1,1 , *<sup>δ</sup>*LOI 1,1 ), monitorLOI1∈MA6 (*t* LOI,start 1,2 , *<sup>δ</sup>*LOI 1,2 ), monitorLOI1∈MA7(*t* LOI,start 1,3 , *<sup>δ</sup>*LOI 1,3 ), monitorLOI2∈MA7(*t* LOI,start 1,4 , *<sup>δ</sup>*LOI 1,4 )>. Moreover, we can relate the time variables of the MA and LOI level (e.g., *t* LOI,start 1,1 = *t* MA,start 1,1 , *t* LOI,start 1,2 = *t* MA,start 1,2 , *t* LOI,start 1,3 = *t* MA,start 1,3 , or *t* LOI,start 1,4 = *t* MA,start 1,3 + *<sup>δ</sup>*LOI 1,3 ) to signify the decomposition of the higher level task into lower-level tasks.

	- **–** *π*MA, *π*LOI and *π*WP represent the plans of the set of *H* HAPSs (i.e., *π*<sup>∗</sup> =< *π*∗ <sup>1</sup> , *π*<sup>∗</sup> <sup>2</sup> ,..., *π*<sup>∗</sup> *<sup>H</sup>* >, where \* stands either for MA, LOI, or WP).
	- **–** *π*˜ MA *h*,*i*:*j* , *π*˜ LOI *<sup>h</sup>*,*i*:*<sup>j</sup>* and *<sup>π</sup>*˜ WP *<sup>h</sup>*,*i*:*<sup>j</sup>* represent the partial plans between the *i*-th and *j*-th task (i.e., *π*˜ ∗ *<sup>h</sup>*,*i*:*<sup>j</sup>* =< *o*<sup>∗</sup> *h*,*i* (*t* ∗,start *<sup>h</sup>*,*<sup>i</sup>* , *δ*<sup>∗</sup> *h*,*i* ), *o*∗ *<sup>h</sup>*,*i*+1(*t* ∗,start *<sup>h</sup>*,*i*+<sup>1</sup> , *δ*<sup>∗</sup> *<sup>h</sup>*,*i*+1), ... , *o*<sup>∗</sup> *h*,*j* (*t* ∗,start *<sup>h</sup>*,*<sup>j</sup>* , *δ*<sup>∗</sup> *h*,*j* ) >, where \* stands either for MA, LOI, or WP.

At this point, it is necessary to highlight that the time-dependent variables (*t* MA,start *<sup>h</sup>*,*<sup>i</sup>* , *<sup>δ</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* , *t* LOI,start *<sup>h</sup>*,*<sup>i</sup>* , *<sup>δ</sup>*LOI *<sup>h</sup>*,*<sup>i</sup>* , *t* WP,start *<sup>h</sup>*,*<sup>i</sup>* , *<sup>δ</sup>*WP *<sup>h</sup>*,*<sup>i</sup>* ) are probabilistic in our problem, except for the start time of the mission *t*<sup>0</sup> = *t* MA,start *<sup>h</sup>*,1 = *t* LOI,start *<sup>h</sup>*,1 = *t* WP,start *<sup>h</sup>*,1 . The underlying reason of the probabilistic nature of these variables is that, at the lowest spatial resolution, the duration of each task can only be estimated, since neither the trajectory of the HAPS nor the exact wind vector are computed or considered yet.

For this reason, we model the duration *δ*WP *<sup>h</sup>*,*<sup>i</sup>* of a task at the WP level as a random variable *uniformly* distributed over:

$$[\delta\_{\mathbf{h},\mathbf{i}}^{\text{NP,min}}, \delta\_{\mathbf{h},\mathbf{i}}^{\text{NP,max}}] = [l(o\_{\mathbf{h},\mathbf{i}}^{\text{NP}}) / (|v\_{\mathbf{a}}| + \max(|v\_{\mathbf{W}}|)), l(o\_{\mathbf{h},\mathbf{i}}^{\text{NP}}) / (|v\_{\mathbf{a}}| - \max(|v\_{\mathbf{W}}|))],\tag{3}$$

where *l*(*o*WP *<sup>h</sup>*,*<sup>i</sup>* ) is the total linear distance to travel between the waypoints associated to the task oWP *<sup>h</sup>*,*<sup>i</sup>* , |*v*a| is the cruising airspeed of the HAPS (see Table A1), and max(|*v*w|) is the maximum wind magnitude (which is assumed to be 5 m/s in this study to ensure that MC1 is not violated). Hence, when we consider |*v*a| + max(|*v*w|) we assume that the HAPS is flying with tailwind, while by considering |*v*a| − max(|*v*w|), we assume that the HAPS is flying with headwind. Note that since *l*(*o*WP *<sup>h</sup>*,*<sup>i</sup>* ) can only be estimated upon the decomposition down to the WP level, our hierarchical planning approach searches for a plan by adopting a downward-forward decomposition approach, which will be explained in a later section in Algorithm 1.

Assuming that lingering between the tasks at any level is forbidden, task *o*WP *<sup>h</sup>*,*<sup>i</sup>* terminates at *t* WP,end *<sup>h</sup>*,*<sup>i</sup>* <sup>=</sup> *<sup>t</sup>*<sup>0</sup> <sup>+</sup> <sup>Σ</sup>*<sup>i</sup> j*=1*δ*WP *<sup>h</sup>*,*<sup>j</sup>* , where *t*<sup>0</sup> is the deterministic start time of the plan, while Σ*<sup>i</sup> j*=1*δ*WP *<sup>h</sup>*,*<sup>j</sup>* follows the distribution of the sum of *i* nonidentically distributed uniform random variables *δ*WP *<sup>h</sup>*,*<sup>j</sup>* . Therefore, the probability density function *f*(*t* WP,end *<sup>h</sup>*,*<sup>i</sup>* ) of completing the *i* − *th* WP-level task of HAPS *h* at a given time can be calculated with the following expression, as derived in [16]:

$$f(t\_{h,i}^{\text{NP,end}}) = f(t\_{h,i}^{\text{NP,end}} - t\_0) = f\left(\sum\_{j=1}^{i} \delta\_{h,j}^{\text{NP}}\right) = \frac{\sum\_{j} \left(g(\vec{\varepsilon}^k, \delta\_{h,1:i}^{\text{NP}})\right)^{i-1} \cdot \text{a1} \cdot \text{gn}\left(g(\vec{\varepsilon}^k, \delta\_{h,1:i}^{\text{NP}})\right) \prod\_{j=1}^{i} \varepsilon\_j}{(i-1)! \, 2^{i+1} \prod\_{j=1}^{i} u\_{\delta\_{h,j}^{\text{NP}}}} \tag{4}$$

where V*<sup>i</sup>* comprises the set with all 2*<sup>i</sup>* vectors of signs*<sup>k</sup>* = (*<sup>k</sup>* <sup>1</sup>, ··· , *<sup>k</sup> <sup>i</sup>* ) ∈ {−1, 1}*<sup>i</sup>* , *uδ*WP *h*,*i* = (*δ*WP,max *<sup>h</sup>*,*<sup>i</sup>* <sup>−</sup> *<sup>δ</sup>*WP,min *<sup>h</sup>*,*<sup>i</sup>* )/2, *i*! is the factorial of *i*, and *g*(*k*, *δ*WP *h*,1:*i* ) = ∑*<sup>i</sup> <sup>j</sup>*=<sup>1</sup> *δ*WP *<sup>h</sup>*,*<sup>j</sup>* <sup>+</sup> <sup>∑</sup>*<sup>i</sup> <sup>j</sup>*=1(*juδ*WP *h*,*j* − *mδ*WP *h*,*j* ), with *mδ*WP *h*,*i* being the median value of [*δ*WP,min *<sup>h</sup>*,*<sup>i</sup>* , *<sup>δ</sup>*WP,max *<sup>h</sup>*,*<sup>i</sup>* ].

The distribution of the higher levels (LOI and MA) time-dependent random variables can be modelled, given the hierarchical decomposition of the plan and the lack of lingering between tasks, by considering the distribution of the lowest level (WP-level) time-dependent random variables, i.e.,

$$f(t\_{h,i}^{\text{MA,end}}) = f(t\_{h,j}^{\text{LOL,end}}),\tag{5}$$

$$f(t\_{h,j}^{\text{LOL,end}}) = f(t\_{h,k}^{\text{WP;end}}),\tag{6}$$

where *t* MA,end *<sup>h</sup>*,*<sup>i</sup>* is the end time of the *i* − *th* task of the MA level, *t* LOI,end *<sup>h</sup>*,*<sup>j</sup>* is the ending time of the *j* − *th* task of the LOI level that terminates when the *i* − *th* task of the MA level ends, and *t* WP,end *<sup>h</sup>*,*<sup>k</sup>* is the ending time of the *k* − *th* task of the WP level that terminates when the *i* − *th* task of the MA level and the *j* − *th* task of the LOI level end. In other words, the distributions of the higher levels are associated to some of the distributions of the lower ones. Finally, it is worth noting that the estimated end time *t* ∗,end *<sup>h</sup>*,*<sup>i</sup>* of a task (*o*<sup>∗</sup> *h*,*i* ) at any level ∗ is the estimated start time *t* ∗,start *<sup>h</sup>*,*i*+<sup>1</sup> of the following task (*o*<sup>∗</sup> *<sup>h</sup>*,*i*+1) of the same level \*.

To understand better the implications of the previous distributions, we represent in Figure 4 the results of Equation (4) when considering up to five *δ*WP *<sup>h</sup>*,*<sup>i</sup>* random variables with the median *mδ*WP *h*,*i* and half length *uδ*WP *h*,*i* provided at the figure caption. We can observe how the sum of more than two uniform distributed random variables assimilates towards a Gaussian distribution, while the variance grows with the number of random variables involved in the sum. This implies that the distribution becomes more wide-spread, and in our case, that the knowledge on the start or end time of a task further in the future is more "uncertain" than the knowledge on the start or end time of a task in the near future. Besides, it is possible to calculate the minimum and maximum values of *t* ∗,end *<sup>h</sup>*,*<sup>i</sup>* , as the density functions calculated with Equation (4) have a limited support. Finally, it is worth noting that a correct estimation of the probability distribution of the sum of tasks durations implies the correct estimation of the start or end time of the tasks at each level. This is essential, especially at the MA-level, since some of the mission requirements and mission constraints stated in Sections 2.4 and 2.5 depend on the time-varying weather conditions and on the time-dependent requirements, associated to the mission time windows used to decide if the high-level tasks can be rewarded.

#### *3.2. Objectives*

The aim of this work is to present a multi-HAPS planner that optimizes the HAPSs tasks plans, whose joint quality is evaluated by the three objective functions defined in the following subsections, each contributing to a different aspect of the overall operational performance.

### 3.2.1. Expected Cumulative Rewards per Hour

This objective focuses on the reward the team of HAPS can gain with the generated plan. Since the success of a task depends on its execution time (i.e., on its start and end time), which can only be probabilistically estimated using Equation (4), the reward can only be estimated with an expected cumulative reward function.

To do it, we exploit the Time-Dependent Markov Decision Process (TiMDP) of Boyan and Littman [17] to calculate the expected cumulative reward, obtained when applying at state *sh*,*<sup>i</sup>* (which in our case contains, among others, the current location of the HAPS) and at time *ti* the remaining plan *π*˜ MA *<sup>h</sup>*,*i*:*<sup>n</sup>* under the weather *wti* forecasted for *ti*:

$$\begin{array}{c} E(\mathbb{R}|s\_{h,i}, t\_i, \pi\_{h,i:n'}^{\text{MA}} w\_{l\_i}) = \Sigma\_{\mu \in \{\text{succ,fill}\}} L(\mu|s\_{h,i}, t\_i, \sigma\_{h,i}^{\text{MA}}, w\_{l\_i}). \\ \quad \cdot \int\_{\mathbb{R}} f(t\_{h,i}^{\text{MA},\text{end}} = t\_{i+1}) \cdot [R(\mu, \sigma\_{h,i}^{\text{MA}}, t\_{i+1}) + E(\mathbb{R}|s\_{h,i+1}, t\_{i+1}, \pi\_{h,i+1:n'}^{\text{MA}}, w\_{l\_{i+1}})] dt\_{i+1}. \end{array} \tag{7}$$

where *<sup>L</sup>*(*μ*|*sh*,*i*, *ti*, *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* , *wti* ) is the likelihood that action *o*MA *<sup>h</sup>*,*<sup>i</sup>* , performed at time *ti* at state *sh*,*<sup>i</sup>* is successful (*μ* = succ) or not (*μ* = fail) under the weather conditions *wti* at *ti*; *f*(*t* MA,end *<sup>h</sup>*,*<sup>i</sup>* = *ti*+1) is the probability density function of ending *o*MA *<sup>h</sup>*,*<sup>i</sup>* at *ti*+1, and *<sup>R</sup>*(*μ*, *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* , *ti*+1) is the immediate reward obtained when performing *o*MA *<sup>h</sup>*,*<sup>i</sup>* at time *ti*+<sup>1</sup> successfully (*μ* = succ) or unsuccessfully (*μ* = fail). In the latter case, *R*(*μ* = fail, *o*MA *<sup>h</sup>*,*<sup>i</sup>* , *ti*+1) = 0.

Although Equation (7) was originally designed to devise a strategy aiming at optimizing the success rate of arriving in time at a destination using different combinations of means of transport, we do not seek to use TiMDP this way. Rather, we exploit the equation as a model for computing the expected cumulative reward of a complete plan *<sup>E</sup>*(R|*sh*,0, *<sup>t</sup>*0, *<sup>π</sup>*MA *<sup>h</sup>* , *wt*<sup>0</sup> ), which can be done using a backward iteration, since the immediate reward *R*(*μ*, *o*MA *<sup>h</sup>*,*<sup>i</sup>* , *ti*+1) is piecewise constant with respect to *ti*+1. Moreover, due to the piecewise constant weather data, *<sup>E</sup>*(R|*sh*,*i*, *ti*, *<sup>π</sup>*˜ MA *<sup>h</sup>*,*i*:*n*, *wti* ) is piecewise constant too. Therefore, the integration can be performed in piecewise time intervals that are generated using the minimum and maximum of the start time of a task (according to *f*(*t* MA,end *<sup>h</sup>*,*<sup>i</sup>* ), as well as the minimum and maximum bounding times of the piecewise constant coverage.

Exploiting the formulation in Equation (7) for computing the expected cumulative reward also leverages the following:


Finally, since multiple HAPS can be involved and the start time of the plan for each HAPS can be different, we accumulate the expected reward of each HAPS to obtain the Objective Function (OF) of the expected cumulative reward *OF*rew:

$$OF\_{\text{new}}(\pi^{\text{MA}}) = \Sigma\_h E(\mathbf{R}|s\_{h,0}, t\_{h,0}, \pi^{\text{MA}}\_h, w\_{t\_0}).\tag{8}$$

### 3.2.2. Effort

Although the mission rewards are important, they are not the only objective to consider. Global client satisfaction must be taken into account too. That is, to keep the clientele satisfied, the HAPS team is required to perform monitoring tasks for as much of their time in the air as possible. Therefore, we consider the objective function of effort, which is the percentage of time spent on monitoring the LOIs:

$$OF\_{\text{eff}}(\pi^{\text{LOI}}) = \frac{\Sigma\_{\text{h}}\Sigma\_{\text{l}}E(\delta\_{\text{h},\text{J}}^{\text{LOI}}) \ast \text{isMonitor}(o\_{\text{h},\text{J}}^{\text{LOI}})}{T\_{\text{h}}^{\text{max}} - t\_{\text{h},\text{0}}} = \frac{\Sigma\_{\text{h}}\Sigma\_{\text{l}}m\_{\delta\_{\text{h},\text{J}}^{\text{LOI}}} \ast \text{isMonitor}(o\_{\text{h},\text{J}}^{\text{LOI}})}{T\_{\text{h}}^{\text{max}} - t\_{\text{h},\text{0}}},\tag{9}$$

where *T*max *<sup>h</sup>* is the end time of the plan horizon set for HAPS *<sup>h</sup>*, *<sup>E</sup>*(*δ*LOI *<sup>h</sup>*,*<sup>l</sup>* ) is the expected duration of the monitoring task for *o*LOI *<sup>h</sup>*,*<sup>l</sup>* which, given the symmetric distribution of the random variable, is the median duration *mδ*LOI *h*,*l* , and isMonitor(*o*LOI *<sup>h</sup>*,*<sup>l</sup>* ) returns 1 if the action *o*LOI *<sup>h</sup>*,*<sup>l</sup>* consists of monitoring a LOI (i.e., if *<sup>o</sup>*LOI *<sup>h</sup>*,*<sup>l</sup>* equals monitorLOI#∈MA#) and 0 otherwise.

This objective function contributes to preventing the HAPS from trying too hard to reach more rewarding MAs by crossing multiple corridors and WAs.

### 3.2.3. Diversity

In the presence of missions that are much more rewarding than others, the plan computation can be extremely unfavorable for less rewarding missions. This has a longterm negative effect to the HAPS team in regard of "customer service". In order to satisfy a more diverse clientele pool, the diversity objective function *OF*div is devised using the Simpson index [18]:

$$OF\_{\rm div}(\pi^{\rm MA}) = 1 - \frac{\Sigma\_{\varepsilon=1}^{N\_{\rm MA}} n\_{\varepsilon}(n\_{\varepsilon} - 1)}{N(N - 1)},\tag{10}$$

where *N*MA is the total number of MAs (or clients) considered in the mission scenario, *n*<sup>c</sup> is the number of occurrences of MA*<sup>c</sup>* in the task plan, and *N* is the total number of MAs within the task plan. Note that the function only considers what happens with the mission areas, ignoring what is occurring in the waiting areas.

Optimizing this objective reduces the probability of drawing the same MA when two of them are drawn without replacement from a given plan, preventing therefore the bias towards rewarding missions.

### *3.3. Constraints*

While the missions' requirements presented in Section 2.4 are considered in the evaluation of the objective function *OF*rew, the mission constraints presented in Section 2.5 are evaluated with different constraint criteria.

Besides, while **MC5** and **MC6** are directly encoded in the solutions manipulated by the EA-based planner described in Section 4.1 (and hence, they are never violated), the remaining criteria (**MC1**–**MC4**) are evaluated with the functions described in Sections 3.3.1–3.3.3.

Finally, it is worth noting that our constraint functions consider the number of times that each criterion is violated. Detailed information of this way of proceeding is presented in Section 3.3.4.

### 3.3.1. Safety

The safety constraint criterion comprises **MC1** and **MC2** and is violated if the MA# is a risk zone (due either to substantial obstacle occlusion or strong wind) while HAPS *h* is operating in it. Since the position of a HAPS is probabilistic due to the uncertainty in the task durations (as Equation (4) states), the constraint function associated to the safety violation *ϕ*saf(*π*MA) is incremented if the probability of operating any HAPS *h* in a MA representing a risk zone is greater than a predefined threshold *p*saf:

$$P\left(\left[t\_{h,i}^{\text{MA,start}}, t\_{h,i}^{\text{MA,end}}\right] \cap T\_{\text{risk}}(o\_{h,i}^{\text{MA}}) \neq \bigtriangleup\right) > p\_{\text{saf}}.\tag{11}$$

where *t* MA,start *<sup>h</sup>*,*<sup>i</sup>* and *t* MA,end *<sup>h</sup>*,*<sup>i</sup>* are the start and end time of HAPS *h* performing the monitoring task *o*MA *<sup>h</sup>*,*<sup>i</sup>* , while *<sup>T</sup>*risk(*o*MA *<sup>h</sup>*,*<sup>i</sup>* = MA#) is the set of time windows where the MA# associated to *o*MA *<sup>h</sup>*,*<sup>i</sup>* represents a risk zone. Alternatively, we can compute the same *<sup>ϕ</sup>*saf(*π*MA) by incrementing its value if

$$\exists t \in [\min(t\_{h,i}^{\text{MA}, \text{start}}), \max(t\_{h,i}^{\text{MA}, \text{end}})] \cap T\_{\text{risk}}(o\_{h,i}^{\text{MA}}), \ P(\text{pos}\_h(t) \in o\_{h,i}^{\text{MA}}) > p\_{\text{safe}} \tag{12}$$

where *t* belongs to the intersection of *T*risk(*o*MA *<sup>h</sup>*,*<sup>i</sup>* ) with the maximum time span that the HAPS could be performing task *o*MA *<sup>h</sup>*,*<sup>i</sup>* , pos*h*(*t*) is the position of HAPS *h* at time *t*, and pos*h*(*t*) ∈ *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* indicates that HAPS *h* is positioned within the MA in which the monitoring task *o*MA *<sup>h</sup>*,*<sup>i</sup>* takes place. The probability *<sup>P</sup>*(pos*h*(*t*) ∈ *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>m</sup>* ) in Equation (12) can be further simplified as Equation (13) states, by taking advantage, in the second last step, of the fact

$$\begin{array}{c} \text{that } P(t < t\_{h,i}^{\text{MA,start}} \cap t > t\_{h,i}^{\text{MA,end}}) = P(t < t\_{h,i}^{\text{MA,start}} \cap t > t\_{h,i}^{\text{MA,start}} + \delta\_{h,i}^{\text{MA}}) \stackrel{\delta\_{h,i}^{\text{MA}} > 0}{=} 0. \end{array}$$

$$\begin{array}{lcl}P(\text{pos}\_{h}(t)\in b\_{h,i}^{\text{MA}})&=\\P(t\_{h,i}^{\text{MA},\text{start}}t\_{h,i}^{\text{MA},\text{start}})+P(tt\_{h,i}^{\text{MA},\text{start}}\cup tt\_{h,i}^{\text{MA},\text{start}})+[1-P(t>t\_{h,i}^{\text{MA},\text{sind}})]-[1-P(tt\_{h,i}^{\text{MA},\text{cind}})]\\&=P(t>t\_{h,i}^{\text{MA},\text{sart}})-P(t>t\_{h,i}^{\text{MA},\text{sind}})\\&=\begin{cases}0, & \text{if }t<\min\{t\_{h,i}^{\text{MA},\text{sart}}\}\\\int\_{\min\{t\_{h,i}^{\text{MA},\text{sart}}\}}^{t}f(\frac{\text{MA},\text{sart}}{t\_{h,i}^{\text{MA},\text{sart}}}=t)dt, & \text{if }t<\min\{\frac{\text{MA},\text{sind}}{\text{MA},\text{cind}}\}\\\end{cases}\\&=\begin{cases}0, & \text{if }t<\min\{\frac{\text{MA},\text{sart}}{\text{MA},\text{sart}}\}f(\frac{\text{MA},\text{sart}}{t\_{h,i}^{\text{MA},\text{sart}}})-t|dt-\int\_{\min\{\frac{\text{MA},\text{sind}}{\text{MA},\text{sind}}\}}^{t}f(\frac{\text{MA},\text{sind}}{t\_{h,i}^{\text{MA},\text{sind}}})-t)$$

### 3.3.2. Coexistence

**MC3** is violated and its corresponding violation index *ϕ*coex(*π*MA) is incremented by 1, if the probability of two HAPSs (*h* and *h* , with *h* = *h* ) operating at the same time *t* in the same MA# (i.e., *o*MA *<sup>h</sup>*,*<sup>i</sup>* = *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>j</sup>* = MA#) is greater than an imposed threshold *p*coex. That is, if

$$\exists t \in [\min(t\_{h,i}^{\text{MA}, \text{start}}), \max(t\_{h,i}^{\text{MA}, \text{end}})], \ P(\text{pos}\_h(t) \in o\_{h,i}^{\text{MA}} \cap \text{pos}\_{h'}(t) \in o\_{h',i}^{\text{MA}}) > p\_{\text{cover}}.\tag{14}$$

The probability in the previous expression can be expressed as the product of two probabilities, each of them computable using Equation (13):

$$P(\text{pos}\_h(t) \in o\_{h,i}^{\text{MA}} \cap \text{pos}\_{h'}(t) \in o\_{h',j}^{\text{MA}}) = P(\text{pos}\_h(t) \in o\_{h,i}^{\text{MA}}) \cdot P(\text{pos}\_{h'}(t) \in o\_{h',j}^{\text{MA}}).\tag{15}$$

To illustrate how the coexistence constraint is evaluated, Figure 5 shows the probabilistic evaluation of the existence of two HAPSs in MA6 for a particular *π*MA. Given the probability density functions of the start and end time of each HAPS in MA6, obtained with Equations (4) and (5), and represented in Figure 5a,b, the probabilities of the presence of each HAPS in MA6 (i.e., *<sup>P</sup>*(pos*h*(*t*) ∈ *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* ) and *<sup>P</sup>*(pos*h*(*t*) ∈ *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>j</sup>* ) with *o*MA *<sup>h</sup>*,*<sup>i</sup>* = *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>j</sup>* = MA6) are estimated with Equation (13) and displayed in Figure 5c. Besides, for clarity of the representation, the time limits of Figure 5c are marked with vertical dashed lines in Figure 5a,b. The constraint function *ϕ*coex(*π*MA) associated to the coexistence of both HAPSs in the MA will be incremented if the product of the two probabilities represented in Figure 5c (i.e., *<sup>P</sup>*(pos*h*(*t*) ∈ *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* )· *<sup>P</sup>*(pos*h*(*t*) ∈ *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>j</sup>* )) exceeds the threshold *p*coex.

(**a**) Probability density function of start (dark blue) and end (light blue) time of HAPS1 in MA6 (i.e., *o*MA 1,*<sup>i</sup>* = MA6)

(**b**) Probability density function of start (dark orange) and end (light orange) time of HAPS2 in MA6 (i.e., *o*MA 2,*<sup>i</sup>* = MA6)

(**c**) Probability of HAPS1 (blue) and of HAPS2 (orange) operating in MA6 (i.e., *o*MA 1,*<sup>i</sup>* <sup>=</sup> *<sup>o</sup>*MA 2,*<sup>i</sup>* = MA6) between 15,200 s and 17,900 s

**Figure 5.** Probabilistic evaluation of the start and end time of the operation of HAPS in a MA and probability of their operation in the MA within the duration marked by the vertical dash lines.

### 3.3.3. Connection

This constraint considers the connectivity of mission elements of a plan represented at the MA-level. The mission elements (i.e., either MA# or WA#) are connected if and only if there is a corridor connecting two consecutive elements in the MA-level plan. Each lack of connection increments the constraint criterion *ϕ*con(*π*MA) by 1.

### 3.3.4. Overall Constraint Violation

As each constraint violation increments *ϕ*criteria(*π*MA) of its corresponding criteria; a non-null *ϕ*criteria(*π*MA) implies the infeasibility of the plan. For that reason, the overall constraint function of a given plan *π*MA is simply the sum of all the constraint criteria:

$$
\varphi(\pi^{\text{MA}}) = \varphi\_{\text{safe}}(\pi^{\text{MA}}) + \varphi\_{\text{core}}(\pi^{\text{MA}}) + \varphi\_{\text{con}}(\pi^{\text{MA}}).\tag{16}
$$

Finally, note that in order to determine during the evaluation of *ϕ*saf(*π*MA) and *ϕ*coex(*π*MA) if there is a *t* where Equations (12) and (14) hold, the time variable *t* is discretized, within the corresponding intervals given in those equations, into equally spaced time instances.

### **4. Implementation of a GA-Guided Hierarchical Task Planner**

The purpose of the planner presented in this section is to perform the task planning for a group of HAPSs that maximize the objective functions presented in Section 3.2 (i.e., reward, effort, and diversity), while ensuring that it is feasible according to the constraint criteria introduced in Section 3.3 (i.e., safety, coexistence, and connection).

To achieve it, we use the Genetic Algorithm (GA) based planner described in Section 4.3 that manipulates the codification of the solutions presented in Section 4.1, which encodes the sequence of MA-level tasks that determines the (sub)optimal temporal hierarchical decomposition of tasks governed by the approach presented in Section 4.2.

### *4.1. Plan Codification*

The solutions that the planner must provide are hierarchical plans (*π*MA, *π*LOI, *π*WP) to be presented as suggestions to the HAPS operator during the monitoring mission. Each plan *π*∗ in the hierarchy, as its formal description in Section 3.1 shows, consists of a list of tasks and their start times and durations. However, as the latter are affected by the weather conditions, we decide to code only the tasks in the optimizer and estimate their timing, when required, in the evaluation of the objective and constraint functions.

Besides, a hierarchical plan decomposes the tasks at a given level into a set of tasks of a lower level, until the set of primitive tasks, or rather "actions", are obtained. In our case, the decomposition into tasks at the intermediate (LOI) level and at the lower (WP) level are given by a fixed set of rules. In particular, a *o*MA *<sup>h</sup>*,*<sup>i</sup>* = WA# task is directly converted into *o*LOI *<sup>h</sup>*,*<sup>j</sup>* =flyWA#, while a *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* = MA# is decomposed into the sequence of *<sup>o</sup>*LOI *<sup>h</sup>*,*<sup>j</sup>* = monitorLOI#∈MA# tasks that implies the sequential monitoring of all the LOIs (without revisit) in the MA before departing. As the number of possible sequences of LOIs in a MA is the number of their permutations, we fix the order in which the LOIs are visited, starting with the LOI closest to the HAPS entry point in the MA and following the order that minimizes the distance of the HAPS within the MA. This way of proceeding ensures the shortest travel distance within a MA, simplifies the optimization problem and accelerates the computation of the plans, as we can precalculate all the orders for a given MA, since we know beforehand all its possible entry and exit points. Besides, it is justified by the fact that the weather conditions do not vary much within a MA. Finally, the decomposition of LOI actions in waypoint actions is usually straightforward and the only possible choices are also fixed. As Figure 2a depicts, this can be done by connecting the entry and exit points of a corridor, the entry point at a MA to the start point of the scan, followed by the points that mark the start and end of a scan track and finally, the exit point of the MA.

Taking into account the previous ideas, the remaining effort to determine the (sub)optimal solution lies in the search for the optimal lists of high level tasks (i.e., MA# and WA#) that each HAPS must perform. As the number of possible WA and MA is finite, the elements of the lists can also be encoded with a finite alphabet of labels. Hence, for the GA-based planner, the solution will be encoded as an array of as many elements as HAPSs, where each element contains the list of the high level tasks (MA# and WA#) of each HAPS. Finally, to distinguish this encoding from the corresponding hierarchical plan (*π*MA, *π*LOI, *π*WP), we represent *sol<sup>k</sup>* as the *k*-th possible solution of the planner, *solk*[*h*] as the part of the solutions for HAPS *h*, and *solk*[*h*][*i*] as the *i* − *th* task at the MA-level (i.e., MA# or WA#) to be performed by HAPS *h* of the *k*-th solution of the planner.

The next section explains how to obtain a hierarchical plan (*π*MA *<sup>h</sup>* , *<sup>π</sup>*LOI *<sup>h</sup>* , *<sup>π</sup>*WP *<sup>h</sup>* ) from a given *solk*[*h*].

### *4.2. Temporal Hierarchical Task Decomposition*

Since the coding of the solutions manipulated by the GA that supports the search of (sub)optimal solutions in our planner is only a sequence of MA# and WA# actions, and the objective and constraint functions used to evaluate them require a hierarchical plan and the estimated end time of the tasks at different levels, in this section we detail, with the help of the pseudo-code presented in Algorithm 1, how the conversion from *solk*[*h*] to *π<sup>h</sup>* is carried out.

The algorithm inputs are the solution plan *solk*[*h*] that encodes only the tasks at the MA level and the start mission time *t* 0 *<sup>h</sup>* for HAPS *h*, and its output is the hierarchical plan *π<sup>h</sup>* = (*π*MA *<sup>h</sup>* , *<sup>π</sup>*LOI *<sup>h</sup>* , *<sup>π</sup>*WP *<sup>h</sup>* ). To start with, Line 1 initializes the hierarchical plan as empty sequences, while Line 2 initializes an empty list for the limits of the duration of each WP task (which will be used later to estimate the density functions of the end time of the tasks at WP level) and Line 3 initializes the Boolean flag *b*finish (which is meant to keep track of the temporal plan length and ignore the tasks that start after the maximum plan horizon *T*max *<sup>h</sup>* has been reached).

After the initialization steps, three nested loops are implemented, in order to be able to decompose tasks at MA-level into primitive tasks at WP-level and to determine the probability distributions of the end time of the tasks of the highest levels from the primitive tasks of the lowest. As the number of nested loops depends on the depth of the decomposition, in our case, three loops are necessary, since the primitive tasks (at WP-level) lie two levels below the MA-level at which the initial *solk*[*h*] is given.

The particular behavior implemented in the three loops is the following. At Line 7 the current MA task in *solk*[*h*][*i*] is selected to be decomposed into an ordered list of LOI tasks at Line 8. Next, at Line 9, temporary partial plans of the lower level tasks (*π*˜ LOI *<sup>h</sup>* and *π*˜ WP *<sup>h</sup>* ) are initialized as empty lists to be able to temporarily store the sequences of tasks obtained after the decomposition of the selected task *o*MA current at the MA-level into the lists of tasks at the LOI-level or WP-level. This lowest level decomposition into primitive tasks happens at Lines 13 and 14, where the current LOI task *o*LOI current is selected and decomposed into the corresponding list of WP tasks. Next, we start processing sequentially each of the primitive tasks *o*WP current of our hierarchical task plan. For this, at Line 19 we determine (using Equation (3)) the limits (minimum and maximum) of the duration needed for the task and append them in Line 20 to the list of limits *list*\_*limits*. Next, at Line 21 the minimum temporal plan length up to the current *o*WP current is checked to see if the plan horizon *T*max *<sup>h</sup>* is exceeded. If that is the case, the decomposition must stop and all lower-level partial plans should not be accounted for. If *T*max *<sup>h</sup>* is not exceeded, at Line 26 the probability distribution on the end time of task *o*WP current is determined with Equation (4), and at Line 27 the current *o*WP current task and the determined probability distribution is appended to the temporal plan *π*˜ WP *<sup>h</sup>* . Next and after looping over all the tasks at the WP-level (if the finishing time has not been reached) the probability distribution *f*(*t* end,LOI current ) of the end time of the current LOI-level task is assigned the probability distribution *f*(*t* end,WP current ) of the end time of the last WP-level task, and *o*LOI current and *f*(*t* end,WP current ) are appended to the temporal plan *π*˜ LOI *<sup>h</sup>* . Next, a similar process is repeated to obtain at Line 38 the probability of the end time of the current action of the MA level plan from the probability distribution of the end time of the last LOI level action and to update at Line 39 the MA plan *π*MA *<sup>h</sup>* . Finally, at Lines 40 and 41 the partial plans, *π*˜ LOI *<sup>h</sup>* and *<sup>π</sup>*˜ WP *<sup>h</sup>* are appended to their corresponding plans *<sup>π</sup>*LOI *<sup>h</sup>* and *<sup>π</sup>*WP *<sup>h</sup>* . **Algorithm 1:** Temporal hierarchical task decomposition **Input:** *solk*[*h*], the encoded MA-level task sequence for HAPS *h* **Input:** *t* 0 *<sup>h</sup>*, the initial mission time for HAPS *h* **Result:** {*π*MA *<sup>h</sup>* , *<sup>π</sup>*LOI *<sup>h</sup>* , *<sup>π</sup>*WP *<sup>h</sup>* }, the hierarchical task plan for HAPS *h* **<sup>1</sup>** *π*MA *<sup>h</sup>* <sup>=</sup><>; *<sup>π</sup>*LOI *<sup>h</sup>* <sup>=</sup><>; *<sup>π</sup>*WP *<sup>h</sup>* =<>; // Initialize the hierarchical plan as an empty plan **<sup>2</sup>** *list*\_*limits* =<>; // Initialize an empty list for the limits of the duration of each WP task **<sup>3</sup>** *b*finish = false; // Boolean variable that finishes the algorithm because the plan has reached the maximum time **<sup>4</sup>** *i* = 1; // Index to iterate that actions of the MA level **<sup>5</sup> while** *(i<=length(solk*[*h*]*)&b*finish = false*)* // Loop over the MA level **<sup>6</sup> do <sup>7</sup>** *o*MA current = *solk*[*h*][*i*]; // Current action at the MA level **<sup>8</sup>** *list*\_LOI = *decompose*(*o*MA current); // Decompose *o*MA current in its corresponding actions list at LOI level **<sup>9</sup>** *π*˜ LOI *<sup>h</sup>* <sup>=</sup><>; *<sup>π</sup>*˜ WP *<sup>h</sup>* <sup>=</sup><>; // Initialize temporary lists of lower-level tasks for *<sup>o</sup>*MA current **<sup>10</sup>** *j* = 1; // Index to iterate the list of LOI **<sup>11</sup> while** *(j<=length(list\_*LOI*)&b*finish = false*)* // Loop over the decomposition at the LOI level **<sup>12</sup> do <sup>13</sup>** *o*LOI current = *list*\_LOI[*j*]; // Current action at the LOI level **<sup>14</sup>** *list*\_WP = *decompose*(*o*LOI current); // Decompose *<sup>o</sup>*LOI current in its corresponding actions list at WP level **<sup>15</sup>** *k* = 1; // Index to iterate the list of WP **<sup>16</sup> while** *(k<=length(list\_*WP*)&b*finish = false*)* // Loop over the decomposition at the WP level **<sup>17</sup> do <sup>18</sup>** *o*WP current = *list*\_WP[*k*] // Current action at the WP level **<sup>19</sup>** *limit* = [*l*(*o*WP current)/(|*v*a<sup>|</sup> <sup>+</sup> max(|*v*w|)), *<sup>l</sup>*(*o*WP current)/(|*v*a| − max(|*v*w|)] **<sup>20</sup>** *list*\_*limits*.*add*(*limit*); // Add the duration limits of *o*WP current to the list of WP durations **<sup>21</sup> if** *(th*,0 + *list\_limits*.*sum\_min*() > *T*max *<sup>h</sup> )* // Does the task ends after the maximum allowed time? **<sup>22</sup> then <sup>23</sup>** *b*finish = true; **<sup>24</sup> end <sup>25</sup> else <sup>26</sup>** *f*(*t* end,WP current ) = *compute*(*th*,0, *list*\_*limits*) // Obtain, with Equation (4), the distribution for *t* end,WP current **<sup>27</sup>** *π*˜ WP *<sup>h</sup>* .*add*(*o*WP current, *f*(*t* end,WP current )) // Add current task and its distribution to the WP plan **<sup>28</sup>** *k* = *k* + 1; **<sup>29</sup> end <sup>30</sup> end <sup>31</sup> if** *b*finish = false **then <sup>32</sup>** *f*(*t* end,LOI current ) = *f*(*t* end,WP current ) // Use the same distribution, as *t* end,WP current for last WP equals *t* end,LOI current **<sup>33</sup>** *π*˜ LOI *<sup>h</sup>* .*add*(*o*LOI current, *f*(*t* end,LOI current )) // Add current task and its estimated distribution to the LOI plan **<sup>34</sup>** *j* = *j* + 1; **<sup>35</sup> end <sup>36</sup> end <sup>37</sup> if** *b*finish = false **then <sup>38</sup>** *f*(*t* end,MA current ) = *f*(*t* end,LOI current ) // Use the same distribution, as *t* end,LOI current for last LOI equals *t* end,MA current **<sup>39</sup>** *π*MA *<sup>h</sup>* .*add*(*o*MA current, *f*(*t* end,MA current )) // Add current task and its estimated distribution to the MA plan **<sup>40</sup>** *π*LOI *<sup>h</sup>* .*add*(*π*˜ LOI *<sup>h</sup>* ) // Add the partial hierarhical plan at the LOI-level **<sup>41</sup>** *π*WP *<sup>h</sup>* .*add*(*π*˜ WP *<sup>h</sup>* ) // Add the partial hierarhical plan at the MP-level **<sup>42</sup>** *i* = *i* + 1; **<sup>43</sup> end <sup>44</sup> end**

### *4.3. GA-Guided Search of the Best Plans*

Algorithm 1 determines the hierarchical plan associated to a given HAPS *h* and list of MA-level tasks *solk*[*h*]. However, determining the best list of MA-level tasks *sol<sup>k</sup>* for all the HAPSs in the mission is extremely complex, as we are facing a probabilistic timedependent multiple-vehicle routing problem, where multiple objective functions (*OF*rew, *OF*div, and *OF*div) and constraint criteria (*ϕ*saf, *ϕ*coex and *ϕ*con) have to be considered. To tackle it, we develop a mission planner that exploits the Non-dominated Sorting Genetic Algorithm (NSGA-II, Ref. [19]) to look for the optimal *solk*. For the clarity and completeness of the paper, our implementation of NSGA-II is recapitulated in Algorithm 2, along with the specifics relevant to this work.

**Algorithm 2:** NSGA-II-guided search of nondominated solutions of hierarchical plans

```
Input: NP, Population size
 Input: Imax, Number of Iterations
 Input: pcrossover, Probability of crossover
 Input: pmut, Probability of mutation
 Input: ktournament, Tournament arity
 Result: FirstFront, information (MA-level plan, task decomposition, and evaluation criteria) of all the solutions
       in the first pareto front.
1 Population = ∅ // Start an empty population set
2 k = 1
3 while k ≤ NP // Population Initialization and evaluation loop
4 do
5 solk = InitializeSolution() // Initialize the lists of MA-level actions for all the HAPSs by
     ensuring connectivity among mission elements
6 [πk, evalk] = Decomponse&Evaluate(solk) // Obtain the hierarchical plan and evaluate it
7 Population.add({solk, πk, evalk}) // Add solution to population
8 k = k + 1
9 end
10 i = 1; // Initialize the iteration/generation counter
11 while i < Imax // While the stop condition is not met
12 do
13 Children = ∅ // Start an empty children set
14 k = 1
15 while Children.size() < NP // While not enough children have been created
16 do
17 [park, park+1] = TournamentSelection(Population, ktournament) // Select pair of parents
18 [solk, solk+1] = Crossover(park, park+1, pcrossover) // Create children solk and solk+1
19 for l=k:k+1 do
20 [soll] = Mutate(soll, pmut) // Mutate child soll
21 [πl, evall] = Decompose&Evaluate(soll) // Decompose and evaluate child soll
22 Children.add({soll, πl, evall}) // Add child soll to children set
23 end
24 k = k + 2
25 end
26 Population = Recombine(Population, Children, NP) // Determine the new population based on the
     old population and on the children, using the nondominated sorting of NSGA-II.
     The solutions of the new population will be sorted in Pareto fronts. Duplicates will be
     discarded.
27 i = i + 1;
28 end
29 FirstFront = Population.FirstFront() // Get information of the first Pareto front of the final
  population
```
Between Lines 1 and 9, Algorithm 2 performs the initialization steps, consisting of generating *Np* solutions of high-level lists of actions (*solk*), performing their decompositions into hierarchical plans (*π*), and evaluating their objective functions and constraint criteria (*eval<sup>k</sup>* = [*OF*rew,*OF*div,*OF*div, *ϕ*saf, *ϕ*coex, *ϕ*con]). To do it, on the one hand, *InitializeSolution*() generates a population of solution plans that fulfill the connectivity

constraint, by appending to *solk*[*h*][*i* + 1] a mission element (MA# or WA#) randomly selected (according to a uniform distribution) among the mission elements connected to the last one *solk*[*h*][*i*]. Besides, a predetermined minimum duration, used to decide how long the initialization of *sol<sup>k</sup>* should take for each mission, is calculated based on the maximum ground speed (*v*<sup>a</sup> + | max(*v*w)|) and on a travel distance that lower-bounds all realistic travel distances for the mission element derived from the tasks at the WPlevel (i.e., the shortest diagonal distance of the mission element). On the other hand, *Decompose*&*Evaluate*(*solk*) performs the decomposition into hierarchical plans of all the lists of high level actions *solk*[*h*] using Algorithm 1 and evaluates the obtained plans *π<sup>k</sup>* using the objective functions and constraint criteria. Finally, the *Population* is formed by the high-level list of actions *solk*, their corresponding hierarchical plan decomposition *πk*, and their corresponding evaluation *evalk*.

Next, the generation loop of the algorithm is performed, between Lines 11 and 28, until reaching the stop condition, consistent on testing if a predefined number of iterations is met. In each generation (algorithm iteration), the new set of solutions, named *Children* in Algorithm 2, are created by selecting from *Population* pairs of solution plans expressed at the MA-level (named *par<sup>k</sup>* and *park*+1), which will undergo crossover, mutation, decomposition, and evaluation (see Lines 18 to 21). Afterwards, the old and the new population are combined in Line 26 to determine the new population of the following generation.

In particular, the pairs of parents selection is performed with *TournamentSelection* (*Population*, *k*tournament) that implements the *k*-tournament operator proposed in [19] for constrained multiobjective problems. That is, for each parent, it selects randomly, according to the uniform distribution, *k*tournament solutions of *Population*, and among them it selects the best one, preferring infeasible solutions with smaller *ϕ*(*π*) to infeasible solutions with a bigger *ϕ*(*π*), feasible solutions (i.e., those with *ϕ*(*π*) = 0) to infeasible ones (i.e., those with *ϕ*(*π*) > 0) and the Pareto dominating feasible solutions to the dominated ones.

Next, the crossover of the two parents (*par<sup>k</sup>* and *park*+1) is performed with *Crossover* (*park*, *park*<sup>+</sup>1, *pcrossover*), that implements a single-point crossover that takes into consideration the expected ending time of the MA-level tasks of each parent. That is, unlike the typical genetic operators for crossover (which select the gene where the crossover should be performed in both parents), we select randomly, according to the uniform distribution and as shown in Figure 6a, the crossover time *t*crossover. Each parent is then divided into a head and tail component at the start time of a task (at the MA-level) closest to *tcrossover* (as marked in the red ellipses), and afterwards the head of one parent and the tail of the other (and vice versa) are concatenated to build the new list of solutions of each child, as shown in Figure 6b. Besides, the probability of crossover *pcrossover* is used to decide, for each pair of parents, if they should undergo the crossover process or if they should be directly copied as new possible solutions.

After crossover, each child is mutated in Line 20 with *Mutate*(*soll*, *p*mut), which uses the probability of mutation *p*mut to determine, according to the uniform distribution, if each of the MA-tasks in *sol<sup>l</sup>* has to mutate and be changed by any other MA# or WA# task randomly selected at the MA-level.

After mutation, each of the children is decomposed and evaluated with *Decompose*& *Evaluate*(*solk*). Moreover, as crossover preserves the head actions of already decomposed plans (as Figure 6b shows) and mutation does not influence in the timing (duration) of the parts of the plan that are previous to the mutation point, we can use the corresponding invariant decomposed plans of the parents to perform more efficiently the decomposition of the new children. Besides, it is worth noting that the connection constraint *ϕ*con(*πMA*) can be violated after a crossover or a mutation. Hence, in order to make the decomposition quicker, the sequence of high level tasks (< *o*MA *<sup>h</sup>*,*i*+1, ... , *<sup>o</sup>*MA *<sup>h</sup>*,*end* >) after the last connected one (*o*MA *<sup>h</sup>*,*<sup>i</sup>* ) are not decomposed into tasks of lower levels, neither will the density distribution for the ending time of their tasks be determined. Finally, we prefer to use crossover and mutation operators that allow to create unconnected high-level (MA) plans to allow them to be reconnected afterwards, eventually, after other crossovers and mutations. By doing so, the planner can sometimes create invalid solutions that are used by the search process to transverse infeasible regions of the search space while moving from one side of the feasible search space to the other. The planner configurations under analysis in the following section will show the importance of this fact.

Once the children population has been completely created, the *Population* and their *Children* are first compared to discard the duplicate solution. Afterwards they are sorted together into nondominated fronts by using the same criteria as in the tournament selection (i.e., their objective functions and constraint criteria are taken into account to prefer feasible to infeasible, solutions that are closer to be feasible to those that are farther to be feasible). Finally, the sorted population is truncated to contain only the best *N*<sup>P</sup> solutions, using the crowding distance, as described in [19], to pick the surviving solutions that belong to the last front that can be admitted into the new population.

At the end of the algorithm, once the generation loop has finished, the planner returns the set of solutions that belong to the first front of the last *Population*. In this front, it is expected to find solutions that fulfill constraints and that are equally good, from the Pareto comparison perspective, regarding the objective functions.

Finally, it is worth noting that although it is not stated in Algorithm 2 for simplicity, all the *Population* of all iterations obtained by the planner (in the initialization and during the generation loop) are also stored to be able to analyze the performance of the planner, over different scenarios, in the following section.

### **5. Results and Analysis**

This section analyzes the performance of the GA-based planner described in this paper for determining the hierarchical task decomposition of a set of HAPSs that carry out realistic monitoring missions in complex time-varying environments with a highlyorganized airspace structure. This planner combines the algorithms described in Section 4 as well as the evaluation functions and constraint criteria formally elaborated in Section 3 in order to take into account the realistic mission requirements and constraints described in Section 2.

To highlight the benefits of the planner, different scenarios are used during the performance tests. The subsequent subsections will first introduce the chosen scenarios, followed by a description of the different variants of the planner that are tested (to determine which configurations are better for each scenario), by an interpretation of the graphical representation of the results, and finally, by their in-depth analysis.

The algorithms are implemented in Matlab and tested on a 4-core i7 processor at 1.80 GHz. On average, an iteration takes 15 s and can go up to 30 s under challenging weather conditions or when more HAPSs are involved, due to the constraint evaluations. The computation time is acceptable for the mission at hand, as the planning is meant to

be performed prior to the execution (as opposed to real-time planning), and therefore more generous planning time is allowed. Besides, the planner can also be implemented as an "anytime" planner, as the algorithm provides a Pareto front with feasible solutions at each iteration. To accelerate the code in a future release, the evaluation functions could be implemented in C.

### *5.1. Scenarios*

The three scenarios considered in the performance tests of this paper share the mission map depicted in Figure 1 and the HAPSs and mission parameters presented in Appendix A. The scenarios differ in the weather data and/or the number of HAPSs involved. The following paragraphs briefly introduce the settings of each scenario, while Table 2 provides an overview of all of them.

*Nominal scenario*. In the first scenario, historical weather data of COSMO-DE (predecessor of COSMO-D2) taken from a relatively calm day in April 2018 is used. The weather conditions are considered moderate, with some strong wind before noon time and some cloudy hours. Besides, the mission is performed by two HAPSs, placed initially at WA2 and WA4, that have to monitor the LOIs depicted in Figure 1 taking into account the rewards and coverage information provided in Table A3. This scenario is useful to see how the planner works under good (nominal) weather conditions.

*Challenging weather scenario*. In this scenario, we use a weather data of the same format as the real weather data considered in the first scenario but synthetically increase the wind to make it stronger. In particular, in some mission areas or corridors, strong wind can occur during more than half of the time of the day. This synthetic scenario is created in order to demonstrate the performance of the planner under challenging weather conditions. Finally, similar to the first scenario, only two HAPSs, placed again initially at WA2 and WA4, are considered in this scenario.

*Three HAPSs scenario*. To demonstrate the scalability of the planner regarding the number of HAPSs, in this scenario the monitoring mission is performed by three HAPSs, placed initially at WA2, WA4, and WA5. The weather conditions are identical to those of the first scenario.

Finally, it is worth noting that the first scenario (labelled as SC1 hereafter) will be considered the basis to compare against the other two, as the second scenario (SC2) is similar to the first but with worsened weather conditions, while the third scenario (SC3) is the first with an additional HAPS.


**Table 2.** Scenarios considered for performance tests.

#### *5.2. Planner Configurations*

The general input parameters of Algorithm 2 are presented in Table 3. They have been selected after analyzing the behavior of the planner under different combinations of parameters over the presented scenarios.

Besides, several configurations of the GA are considered to optimize the hierarchical task plan and to analyze the performance and benefits of each one for the different scenarios. The three Planner Configurations (PC1, PC2, and PC3) analyzed in the paper are implemented in general according to Algorithm 2 and two of them (PC2 and PC3) contain some slight variations injected into parts of the code to support the following behaviors:

*Planner Configuration 1 (PC1).* The constraint-handling technique proposed by [19] is used for select the pair of parents in the *k*-tournament selection (at Line 17 of Algorithm 2) and for recombining the old and new populations (at Line 26). In other words, solutions that fulfill or are closer to fulfilling the constraints are preferred to solutions that do not fulfill or are further to fulfilling them, and among solutions that are equally good regarding the constraints, solutions that Pareto dominate the others regarding the objective functions are preferred to solutions that are Pareto dominated. As this configuration implements the standard constraint-handling techniques of NSGA-II [19], it is also the one described in Section 4.

*Planner Configuration 2 (PC2).* The constraint-handling criteria are only applied to select the pair of parents in the *k*-tournament selection in Line 17 of Algorithm 2 and ignored during the recombination of old and new populations. That is, during the recombination step in Line 26 of Algorithm 2, the solutions are sorted by only taking into account the ordering imposed by the Pareto comparison of the objective functions. The motivation of this variation is to have a planner configuration that is less "stringent" with the hierarchical plans that violate the constraints (i.e., that have *ϕ*(*π*MA) > 0), and to give them more chances to be selected for the next generation (or even be selected as parents for the generation of children solutions of the next iteration).

*Planner Configuration 3 (PC3).* The diversity objective function (*OF*div) is ignored both during the parents selection and recombination steps. This configuration has been set up to put forth the benefit of considering the diversity (and not only the expected reward or the effort) for planning.

**Table 3.** Planner configuration parameters.


For readers familiar with the stochastic ranking mechanism for constrained evolutionary optimization presented in [20], it is interesting to highlight that PC1 and PC2 represent the two extreme cases that are obtained when the probability of ignoring the constraints is respectively set to 0 (for PC1) or to 1 (for PC2). That is, during the recombination step, in PC1 the constraints are never ignored while in PC2 the constraints are always ignored. Comparing the behavior of the extreme cases will facilitate the understanding of the effects of taking into account (or ignoring) the constraints in the recombination step.

Finally, for the computation of the expected reward using Equation (7), we assume *<sup>L</sup>*(*<sup>μ</sup>* = success|*sh*,*i*, *ti*, *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* , *wti* ) = 0.8, if the cloud coverage of *wti* is smaller than the image coverage required by the mission, as shown in Table A3. Otherwise, *<sup>L</sup>*(*<sup>μ</sup>* = success|*sh*,*i*, *ti*, *<sup>o</sup>*MA *<sup>h</sup>*,*<sup>i</sup>* , *wti* ) = 0.2.

### *5.3. Results Representation*

In order to provide an overview on the the weather (wind and cloud coverage) conditions of each scenario, on the time windows where each mission area can be visited, as well as on a representative solution obtained by the planner, we use the graphics displayed in Figures 10–14, whose vertical axes represent, from the bottom to the top, the mission areas (MA#) and waiting areas (WA#), while the horizontal axes represent the hour of the day. Further, the graphics also contain the above-mentioned information on the weather, mission, and plan, which is represented by the following items:


Taking into account the previous information, the sequence of mission elements (i.e., MA# and WA#) traversed by each HAPS in a representative solution can be observed, along with the weather conditions and the mission time window of each scenario. For example, in Figure 10a we can observe, following the red line, that HAPS1 moves from WA2 (the starting location of HAPS1, which is not represented in the graphic) to MA1, WA1, WA2, MA2, WA2, WA1, MA3, WA1, and MA3. Besides, MA1 is visited when not requested, while MA2 and the two visits to MA3 are within the correct mission time windows. Besides, MA2 is partially visited under cloudy conditions, which can reduce the expected reward obtained by HAPS1, while MA3 are visited under good weather conditions, which provides HAPS1 two times the total reward of MA3.

Besides, in order to analyze the performance of the different configurations of the planner in different scenarios, we store for each scenario-planner configuration pair and for each iteration of the GA, the values of the objective functions (*OF*rew, *OF*eff and *OF*div ) and constraint criteria (*ϕ*saf, *ϕ*coex and *ϕ*con) of all the feasible solutions (i.e., *ϕ*(*π*) = 0) of the best Pareto front obtained during the execution of the algorithm. With that information, we represent the following graphs:


Finally, we also use two additional types of graphs in order to analyze further certain scenarios:


### *5.4. Comparative Analysis*

In the following sections, the results obtained from each scenario and planner configurations, characterized using the types of graphical representation explained in Section 5.3, are analyzed.

### 5.4.1. Analysis for Scenario 1 (SC1)

For SC1, all the configurations of the planner (PC1-PC3) are tested and their GA are set to run for 100 iterations to illustrate better the convergence behavior of the planner.

Figure 7 summarizes, at three selected iteration counts (in particular at the 1st, 40th, 100th iteration), the evolution of the OF of the population and of the best Pareto front, obtained using PC1. The graphics show: (1) how the number of solutions belonging to the best Pareto front in the population increases as the iteration number grows and (2) how all the solutions move towards the Pareto optimal front, along the axis in the direction of increasing values of the three objective functions. This is the expected behavior of NSGA-II, which is the optimizer that supports the search of the sequence of high-level tasks in our planner. To avoid increasing unnecessarily the length of the paper, it does not include more graphics of this type for the other configurations (PC2 and PC3) or for the remaining scenarios (SC2 and SC3), as they present similar behaviors. Besides, the evolution graphs, used in the rest of the paper are more suitable to provide further insights on the behavior of the planner configurations.

Figure 8a–c show the evolution of the mean and standard deviation of the values of the objective functions (*OF*rew, *OF*eff and *OF*div) of the feasible solutions that belong to the best Pareto front and that are found using the three planner configurations, while Figure 8d–f show the maximum value of each OF. According to Figure 8a–c, the standard deviations on the values of *OF*rew, *OF*eff and *OF*div obtained using PC2 are substantially wider than those obtained with PC1 and PC3. This is due to the fact that the constraint criteria are used only for the parents selection, allowing the MOEA to have a bigger "exploring" capability. Besides, we can also observe that the GA search converges earlier, around iteration 60. Additionally, to show the importance of considering *OF*div, in SC1 we also test PC3, where *OF*div is neglected deliberately during the parents selection and recombination steps. Figure 8c,f show how the diversity criterion evolution is worst for the configuration where *OF*div is neglected (that is, for PC3), while PC1 and PC2 reach similar values (in particular the maximum value of *OF*div for PC1 is not observed as it is equal to the maximum value of *OF*div for PC2). This implies that the mission plans obtained with PC3 suffer from having a low diversity, resulting in a more challenging selection process to be performed by the human operator who is responsible of choosing a "well-balanced" plan among the feasible plans of the best Pareto front returned as plan suggestions by the planner. Besides, if we compare PC1 and PC2, we can conclude that for SC1, PC2 produces overall better solutions regarding *OF*rew, while PC1 produces overall better solutions with respect to *OF*eff. However, the values of *OF*eff of the solutions found using PC1 is only marginally better than the values of *OF*eff of the solutions obtained by PC2, while the values of *OF*rew of the solutions obtained with PC2 is significantly bigger than the values of of

*OF*rew of the solutions obtained with PC1. Therefore, we conclude that PC2, with its better exploring capability, is more suitable for the first scenario.

**Figure 7.** Values of the objective functions of the best Pareto front (marked in red) vs. values of the OFs of the remaining population (marked in black) for SC1 and PC1. The top row of graphics represent in 3D the values of the three OFs, while the second row only shows the values of two of them.

(**d**) Max evolution for *OF*rew (**e**) Max evolution for *OF*eff (**f**) Max evolution for *OF*div

**Figure 8.** Evolution graphics of the OFs of the feasible solutions of the best Pareto front for SC1. The top row of graphics shows the evolution of the mean and standard deviation (M&SD) of each OF, while the bottom row shows the evolution of the best (Max) value of each OF.

Figure 9a shows that the number of infeasible individuals in the population is also higher in PC2 than in PC1, which practically excludes all infeasible individuals after eight iterations. This demonstrates the difficulty of PC1 to explore new regions of the space that can be reached with the help of some infeasible solutions obtained after random crossover or mutation operations.

Figure 10 shows two representative plans of SC1, which have been selected among the feasible plans of the best Pareto front found using PC2 (in the upper figure) and PC3 (in the lower figure). In particular, we have decided to display the plans that have the maximum *OF*rew. Comparing the plans of both figures, we can observe that the plan found with PC2 (displayed in Figure 10a) has fewer repetitions of the visited MA than the plan found using PC3 (represented in Figure 10b), where HAPS2 stays monitoring only MA10. This happens because *OF*div is neglected in PC3 during the search of the solutions. Additionally, both graphics show how both HAPSs try to accommodate their visit to the MAs to the requested time windows and clear sky weather conditions in order to increment the overall obtained reward. Lastly, by analyzing Figure 8d we can observe that the plan returned by the planner with maximum *OF*rew in PC2 has a higher value of *OF*rew than the one obtained with PC3, because by including the diversity objective function and by ignoring the constraints in the recombination step, the GA configuration used in PC2 is able to explore the search space more efficiently, jumping to search regions that contain solutions of higher rewards.

Finally, it is worth noting that for the remaining scenarios we do not test against PC3, in order to focus the analysis on the comparison of PC1 and PC2, i.e., the variants of planner configurations that use and ignore the constraint criteria during the recombination of the old and new populations.

### 5.4.2. Analysis for Scenario 2 (SC2)

In the second scenario, weather data of the same format as the real weather data in SC1 are used but with synthetically increased strong wind. We also set the maximum iterations to 60, which was the iteration number in which the GA converges for SC1.

The M&S and Max evolution graphs of the values of the OFs of the feasible solutions belonging to the first Pareto front are shown in Figure 11. The graphics show that the results obtained with respect to the evolution of the objective functions over iterations are comparable in terms of order of magnitude using PC1 and PC2. However, the results obtained with PC2 fluctuate much more than PC1. In fact, the behavior of PC2 is predictable since in this configuration the best Pareto front can obtain both feasible and infeasible solutions, and the feasible solutions, which are the only ones considered for plotting the M&S and Max evolution graphs, can be overtaken by infeasible solutions whose objective function values dominate the objective function values of the feasible solutions.

(**a**)**.** MA-level task sequencing of the feasible plan with the largest *OF*rew obtained using PC2.

(**b**)**.** MA-level task sequencing of the feasible plan with the largest *OF*rew obtained using PC3.

**Figure 10.** Illustrative examples of feasible plans obtained by PC2 and PC3 for the first scenario (SC1).

**Figure 11.** Evolution graphics of the OFs of the feasible solutions of the best Pareto front for SC2. The top row of graphics shows the evolution of the mean and standard deviation (M&SD) of each OF, while the bottom row shows the evolution of the best (Max) value of each OF.

Although PC2 works well under nominal weather conditions, this planner configuration can be "unstable" under challenging weather conditions, which can facilitate a more frequent violation of the constraints criteria. This behavior can be better explained using Figure 9, where the number of infeasible solutions of the population obtained with PC1 and PC2 for SC1 and SC2 are displayed side-by-side. As already described in the previous subsection, the graphics show how PC1 is much "stricter" against infeasible solutions, as the constraint criteria are used in the recombination step, resulting in a reduction of the "survivability" of infeasible individuals and of the exploring capability of the planner. However, PC2's higher exploring capability appears to be too "lenient" with the infeasible solutions for SC2, allowing an excessive number of them predominate the population in the final iterations. This behavior, which appears in the more constrained scenario imposed by the stronger winds of SC2, is prone to end up having too few feasible individuals remaining in the best Pareto front of the last iteration, thereby losing the best feasible ones identified along the iterations of the algorithm. Therefore, under more challenging weather conditions, PC1 should be the preferred configuration.

Figure 12 depicts the plan that has the highest expected reward among the feasible plans of the best Pareto front found using PC1 over SC2. The figure shows that the mission elements are affected by strong wind (which occupies more than 20% of the time) more often than in SC1. Besides, HAPS1 monitors MA1 and HAPS2 monitors MA6 at time windows that are not requested by the clients, in order to be able to reach other more promising MAs (and due to the fact that a MA cannot be transversed without monitoring its LOIs).

### 5.4.3. Analysis for Scenario 3 (SC3)

Three HAPSs are used in the third scenario to analyze the scalability of the planner. However, since the search space of the possible solutions has grown (due to the additional HAPS), more iterations of the GA are necessarily. For this reason, we set the maximum

iterations of the stop condition to 100, which is also observed to be necessary, as the search takes more iterations to converge according to the evolution graphics of *OF*rew presented in Figure 13. Besides, since the nominal weather setting for the environment is used in this scenario (as in SC1), Figure 13 shows how PC2 again exhibits higher variability and better performance than PC1, thanks to its higher "exploring" capability.

The mission plan with the largest *OF*rew among the feasible plans of the first Pareto front found using PC2 is illustrated in Figure 14, along with the operation environment and requirements. With the additional HAPS, more MAs can be monitored, compared to missions where only two HAPS operate (whose illustrative plans are presented in Figures 10 and 12). This fact is also observable comparing the evolution of the Max graphs of *OF*rew of SC1 and SC3, because the expected reward obtained by the plans for SC3 (Figure 13d) is higher than the one obtained for SC1 (Figure 8d). Figure 14 also shows how the coexistence in the same MA of multiple HAPS is tolerated (e.g., the presence of HAPS2 and HAPS3 in MA10), since the constraint criterion *ϕ*coex is probabilistically evaluated and violated when the probability of coexistence exceeds a given *p*coex (which is set to 0.3 in this paper). Changing the value of this parameter, the constraint violation can be "tightened" or "relaxed" as much as desired. This is a novelty of the planner presented in this paper, since the original version presented in [10] implemented a deterministic evaluation of the coexistence criterion where no overlapping of the start and end time range of MA# tasks was allowed.

**Figure 13.** Evolution graphics of the OFs of the feasible solutions of the best Pareto front for SC3. The top row of graphics shows the evolution of the mean and standard deviation (M&SD) of each OF, while the bottom row shows the evolution of the best (Max) value of each OF.

**Figure 14.** Illustrative example of the feasible MA-level plan with largest *OF*rew obtained by PC1 for the third scenario (SC3).

### **6. Related Works**

UAVs have recently become a popular alternative for monitoring ground activities [21], mapping [22] or search and rescue missions [23], since the operation of these platforms is more cost-efficient than using manned aerial vehicles while achieving the same purpose [5]. Furthermore, the deployment of these platforms is also more flexible, because numerous UAV platforms are capable of vertical take-off and landing, enabling the deployment in many missions where vast areas for takeoff and landing are scarce. Lastly, the use of unmanned platforms also allows the immediate deployment in risk zones, without compromising the safety of human pilots nor delaying the operations.

Following the development of battery technologies, light-weight but robust material as well as technologies for optimal harvesting of solar energy, the development of unmanned High Altitude Long Endurance (HALE) aerial vehicles has became the focus of giant aeronautics industries [2,24]. Moreover, new path and mission planning strategies for solar powered UAVs are being continuously developed, intended to (1) improve their trajectory by deriving the most energy-efficient flight patterns [25–29], to (2) determine the optimal path for improving operational efficiency in missions meant for communications [30–32], or (3) to track different types of targets [33]. The planning strategies exploited in these works are based on different types of optimization approaches, ranging from nonlinear optimization strategies [26,28–31] to rapidly-exploring random trees [32], the grasshopper optimization algorithm [33] and particle swarm optimization [27]. Nevertheless, the planners presented in these works either (1) optimize the trajectories without considering any aspect that is relevant to the mission or (2) tackle missions which are significantly different from the one proposed in this paper (and hence they consider a different set of requirements and constraints). Furthermore, the planning methods proposed are different than the GA-based one presented in this paper, although the last two (i.e., [27,33]) are also variants of evolutionary algorithms.

Moreover, the HAPSs considered in this work are special types of HALE platforms aimed to be an alternative to satellites for long-term remote sensing while offering more flexibility in its deployment. With the success stories around Kelleher [2], HAPSs are deemed fit for deployment in the near future at larger scale. However, although HAPSs operations can be beneficial, they can be extremely challenging, given the fragility of the platform under critical weather conditions, their lack of maneuverability, and the requirement for plans for long operations, which oblige the consideration of weather parameters that vary over time within the plan horizon [34]. Specific studies on automated planning for HAPS include [10,35], both aiming to reduce the operators' workload. In particular, for a complex mission scenario as the one depicted in Figure 1, Ref. [35] proposes a sequential task and motion planning framework for a collective operation area, simplifying the constraints of the planning problem, while [10] uses a GA to extend the temporal hierarchical task planner for multiple HAPSs. Moreover, the current work extends the planner presented in [10] by (1) including the evaluation of the safety and coexistence constraints with the new probability based functions presented in Sections 3.3.1 and 3.3.2, by (2) substituting the weighted evaluation function used in [10] for the constrained multiobjective Pareto-front evaluation mechanisms of NSGA-II, and by (3) returning the set of the hierarchical plans that form part of the final best Pareto front. Besides, this paper analyzes the behavior of the new planner with new scenario and the influence of the diversity objective function and of different constraint handling techniques within our planner.

Evolutionary algorithms, including variants for solving multiple-objective problems with powerful constraint-handling techniques (such as [19,36]), have often been used for the mission planning of Satellite and UAV operations. For instance, Refs. [37–39] present different GA-based planner for scheduling the observation tasks of different satellites, while [23,40–43] use multiple-objective evolutionary algorithms to solve task planning problems for multiple UAVs engaged in performing monitoring tasks in dissected areas of interest. Although our planner also uses a GA algorithm to determine the best solution plans for a given scenario, it solves a different type of monitoring task mission problem, involving exogenous time-varying events (i.e., weather) and time-dependent mission requirements. Therefore, its evolutionary encoding has a different interpretation and is customized for HAPSs instead of satellites or other types of UAVs. Besides, similar to other works that take into account the uncertainty associated to weather conditions [44–46] or to other elements of the mission (e.g., the target location and movement in search and rescue missions [47,48] or the probability of target detection and destruction in hostile environments [49,50]), in this work the uncertainties are incorporated into the models used to evaluate how probable is that each HAPS is at a mission area at a given time, which affects the outcome of the objective and constraint values.

Finally, it is worth highlighting that although this work uses NSGA-II for a constrained multiobjective optimization, it is only a part of the temporal hierarchical task planner, in which the search for optimal decomposition into an ordered list of nonprimitive tasks poses a combinatorial search problem. With an appropriate encoding of the problem at the task level at which the combinatorial problem prevails, NSGA-II is used for guiding the decomposition into executable tasks within a temporal hierarchical task network with a nested Time-Dependent Multi-Vehicle Routing Problem (TDMVRP). Note also that Hierarchical Task Planning often refers to an Artificial Intelligence (AI) planning paradigm and that although there are some domain-independent frameworks meant for it [51,52]; they do not yet support a nested TDMVRP.

### **7. Conclusions and Future Work**

This paper presents a new approach for planning the tasks that a group of HAPSs must perform to carry out ground monitoring mission in a structured airspace. The new approach returns a Pareto front of feasible hierarchical plans, whose sequence of higher level tasks is determined using a MOEA that optimizes the expected reward to be received by the HAPSs team for monitoring the different LOIs, the diversity of the LOIs visited by the HAPSs and the time that the HAPSs are actually monitoring (and not traversing the airspace). Besides, it also considers multiple constraints, some encoded in the decomposition method of the hierarchical planner, while others validated by measuring the constraint criteria related to the mission safety, the coexistence of HAPSs in the same MA and the connectivity of the plan. The planner also considers, through the evaluation functions and constraint criteria, the uncertainty that the weather conditions impose on the duration of each task (due to the wind vectors) and on the visibility for the mission camera (due to cloud coverage).

The performance of the different configurations of the planner, carefully set up for increasing/decreasing the "survivability" of the infeasible solutions or to disable the diversity requirement, is tested against several scenarios, with varying number of HAPSs and different weather conditions. The quality of the results is scenario-dependent, although it seems advisable to use the second configuration (PC2) for the planner when the HAPSs operate under mild weather conditions and the first configuration (PC1) for challenging weather conditions, as suggested by the results of the performance tests presented in Section 5. Besides, when planning for two HAPSs the number of iterations required by the planner to converge is smaller than when planning for three HAPSs.

In order to further improve the planner, we will consider several possibilities. Firstly, a "softer" constraint-handling method can be used to improve PC1. For this purpose, we are planning to adopt the approach proposed by [20], in which, with a low probability, some infeasible solutions can be ranked better than feasible solutions. This new planner configuration could avoid, for example, the early convergence of PC1 in SC2, while still managing to maintain the right balance between feasible and infeasible solution plans in the population.

Secondly, while the planner is typically customized for solving HAPS mission planning problems, it can be extended for more generic uses. As a matter of fact, temporal hierarchical task planners (without a nested TDMVRP) are gearing toward general implementation [52]. Hence, with careful considerations of the encoding of the chromosomes for generic planning problems and more generic approaches for tuning the planner

parameters, the approach presented in this work could be implemented in a "domainindependent" fashion.

Finally, although the underlying mission planner reduces the operator's workload, when the planner suggest many solutions, selecting the one to execute can still be challenging for the operator. Therefore, in order to increase usability of the planner, it can be convenient to provide operators with a set of tools that help them to analyze the solutions of the best Pareto front more easily, by (1) taking into account the explicability of the plans with a visualization interface that can highlight the probable constraint violations, the rewarding mission tasks, the diversity of the clientele pool and the effort; or by (2) designing filter mechanisms that accelerate the selection of the plan of the best Pareto front that better fits the operator preference (e.g., the one with the maximum value of an objective function or the one with the best preference weighting [53]). In a similar line, more interactive functions can be integrated to enable "mixed-initiative planning", which can favor quick local replanning performed by the operator whenever necessary, due to unexpected weather change or to take into account the operator's preferences.

**Author Contributions:** Conceptualization, J.J.K., E.B.-P., and A.S.; methodology, J.J.K., E.B.-P.; software, J.J.K.; validation, J.J.K.; formal analysis, J.J.K.; writing—original draft preparation, J.J.K.; writing—review and editing, E.B.-P.; visualization, J.J.K.; supervision, E.B.-P., A.S.; project administration, J.J.K., A.S.; funding acquisition, A.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Project StraVARIA, a Ludwig Bolköw Campus project. Eva Besada Portas' contributions are funded by the the Spanish National Challenge Grant RTI2018- 098962-B-C21.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors thank partners from the German Aerospace Center and from ADS GmbH for their support in providing information on HAPS, as well as on methodology to be adopted for validation purposes.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **Abbreviations**

The following abbreviations are used in this manuscript:



### **Appendix A. Numerical Details on the HAPS Model and on the Mission Parameters**

Table A1 details numerical information on the HAPS that are assumed in this work.

**Table A1.** Model of the HAPS considered in this work: build, performance, and mission payload.


The dimensions of the mission elements depicted in Figure 1 in form of their longest diagonals in kilometers are given in Table A2.


**Table A2.** Dimensions of the mission elements depicted in Figure 1.

The rewards obtained for the successful monitoring of all LOI(s) of a MA are listed in Table A3, along with the image coverage of the ground required for the monitoring mission to be considered successful.



### **References**


### *Article* **Monitoring Winter Stress Vulnerability of High-Latitude Understory Vegetation Using Intraspecific Trait Variability and Remote Sensing Approaches**

### **Elmar Ritz 1, Jarle W. Bjerke <sup>2</sup> and Hans Tømmervik 2,\***


Received: 4 March 2020; Accepted: 6 April 2020; Published: 8 April 2020

**Abstract:** In this study, we focused on three species that have proven to be vulnerable to winter stress: *Empetrum nigrum, Vaccinium vitis-idaea* and *Hylocomium splendens.* Our objective was to determine plant traits suitable for monitoring plant stress as well as trait shifts during spring. To this end, we used a combination of active and passive handheld normalized difference vegetation index (NDVI) sensors, RGB indices derived from ordinary cameras, an optical chlorophyll and flavonol sensor (Dualex), and common plant traits that are sensitive to winter stress, i.e. height, specific leaf area (SLA). Our results indicate that NDVI is a good predictor for plant stress, as it correlates well with height (*r* = 0.70, *p* < 0.001) and chlorophyll content (*r* = 0.63, *p* < 0.001). NDVI is also related to soil depth (*r* = 0.45, *p* < 0.001) as well as to plant stress levels based on observations in the field (*r* = −0.60, *p* < 0.001). Flavonol content and SLA remained relatively stable during spring. Our results confirm a multi-method approach using NDVI data from the Sentinel-2 satellite and active near-remote sensing devices to determine the contribution of understory vegetation to the total ecosystem greenness. We identified low soil depth to be the major stressor for understory vegetation in the studied plots. The RGB indices were good proxies to detect plant stress (e.g. Channel G%: *r* = −0.77, *p* < 0.001) and showed high correlation with NDVI (*r* = 0.75, *p* < 0.001). Ordinary cameras and modified cameras with the infrared filter removed were found to perform equally well.

**Keywords:** climate change; evergreen plants; extreme events; flavonol and chlorophyll sensor (Dualex); greenness indices; mosses; near-remote sensing active and passive NDVI sensors; Sentinel-2; subarctic vegetation damage

### **1. Introduction**

Global warming will affect arctic and subarctic regions more than any other area in the world [1]. It is expected to increase the productivity of subarctic and arctic ecosystems [2–4]. Increasing productivity and biomass is generally known as 'greening' [5]. Major drivers are a longer growing season and increasing summer warming [6]. However, negative trends in productivity and biomass, known as 'browning', have also been reported [6]. For the Arctic as a whole, trends are complex, as Myers-Smith et al. state: "Figures vary from 42% greening and 2.5% browning from 1982 to 2014 in the GIMMS3g AVHRR dataset to 20% greening and 4% browning from 2000 to 2016 in Landsat data, and to estimates of 13% greening and 1% browning for the MODIS trends calculated for 1,000 random points in the tundra polygon from 2000 to 2018." ([7], p. 107).

In the subarctic region of Scandinavia, i.e. Norway, Finland, and Sweden north of the Arctic Circle, the main drivers of browning are winter warming events and pest outbreaks [8]. Winter warming events can melt the insulating snow cover that normally protects photosynthetic short-statured organisms overwintering with aboveground tissue (e.g. prostrate shrubs, cushion plants, bryophytes, and lichens) from the harsh ambient winter weather conditions. After a few thaw days, ground vegetation becomes exposed to ambient air and hibernation is interrupted, thus reducing the protection of photosynthetic organisms against frost, which may easily lead to freezing damage upon return of normal winter weather [9]. Soil communities, including both micro-arthropods and bacteria, can also be severely affected [10]. Overall, a warmer winter climate changes species compositions and reduces carbon cycling [11]. In subarctic and arctic regions evergreen plants in particular are sensitive to changing winter climate and reduced snow cover [12]. This includes the widespread dwarf shrubs *Empetrum nigrum* L., *Vaccinium vitis-idaea* L., *Cassiope tetragona* (L.) D.Don, and *Calluna vulgaris* (L.) Hull, as well as the tall coniferous shrub *Juniperus communis* L. [6,11,13]. Bryophytes, such as the widespread feathermoss *Hylocomium splendens* (Hedw.) Schimp., deciduous shrubs, such as *Vaccinium myrtillus* L., evergreen horsetails (*Equisetum* spp.), as well as small cushion plants show reduced growth following exposure to winter warming [9,11,13,14]. The other major factor causing browning are pest outbreaks. Recently, increasing frequency and intensity of outbreaks of leaf-defoliating geometrid moths led to massive canopy defoliation of their preferred host tree *Betula pubescens* Ehrh. and understory plants [13,15,16]. Overall, multiple stress events are main drivers of browning. Given the high focus on climate change-induced changes in northern primary productivity, it is important to develop easy and reliable methods for assessment of plant vitality.

For a long time, satellites have monitored the global vegetation status [2,3,17]. Spectral sensors operated near the target vegetation are increasingly applied for assessing the plant status [18,19]. However, near-remote time series of the plant status are still uncommon, which is partly due to the need for expensive equipment, for example spectroradiometers [20]. In recent years, several new and low-cost active and passive proximal sensors were developed. This includes sensors measuring the normalized difference vegetation index (NDVI). NDVI is a radiometric measure of the amount of radiation (≈∼400–700 nm) absorbed by vegetation during photosynthesis. It is calculated from contrasting reflectance at near-infrared (NIR) and red bands [21,22].

NDVI has been widely used in studies of phenology, productivity, biomass, and disturbance monitoring, as it has proven to be a good proxy of the vegetation's photosynthetic activity [19,23]. NDVI works well for subarctic ecosystem monitoring and is widely used on different scales and as a vegetation marker [24–26].

Previously, modified cameras—with the infrared filter removed—were found to be good NDVI surrogates. In such cameras, the NDVI proxy is commonly calculated by using the enhanced red channel and the blue channel (BNDVI) [27]. However, a combination of the enhanced red channel and the green channel might also be of interest due to a strong linear correlation with the chlorophyll content (GNDVI) [28]. Additionally, ordinary cameras were increasingly applied for vegetation analysis and phenology studies in recent years. Greenness indices based on ordinary RGB images from such cameras are promising NDVI substitutes [28,29], even for high-arctic vegetation [30].

In subarctic forests, the contribution of understory vegetation (i.e. dwarf shrubs, herbs, graminoids, bryophytes and lichens) to the total ecosystem productivity is similar to that of trees [31]. Moreover, biodiversity of vascular plants at high latitudes is relatively low, which makes research into dominant species and their vulnerability to environmental change even more important [32]. We hypothesized that in situ estimates of plant damage would be correlated to optical measurements of plant greenness, but that greenness indices would vary in their explanatory power. Our second hypothesis was that plant stress would vary over short distances in a rolling subarctic landscape and that this would be detectable both by near-remote sensing measurements and by Sentinel. To this end, we combined near-remote sensing approaches with classical determination of plant traits of understory vegetation to address the following research questions:


To answer these questions, we made analyses in a widespread subarctic heath ecosystem, focusing on vegetation plots dominated by two evergreen dwarf shrubs and a mat-forming moss.

### **2. Materials and Methods**

### *2.1. Site Description*

We selected plots in wind-exposed areas where snow cover generally is shallow and plants are more susceptible to winter frost-thaw stress. Eighteen plots (1 m × 1 m) were assessed within a total area of approx. 1 km<sup>2</sup> in Tromsø, Troms County, northern Norway (Figure S1, Table S1, Figure S2b,c). Satellite upscaling was conducted in homogeneous plots within a wider area in Tromsø (Table S2), from 8–12 June 2017, corresponding to days of year (DOY) 159–163. The three areas (a total of ca. 0.5 ha) for the upscaling approach were not congruent to the eighteen field plots (a total of 18 m2). We chose three separate areas in order to avoid trails, snow patches, unvegetated ground and rocky steep slopes (as exemplified in Figure S2a). Unvegetated ground was estimated to be around 10% in the area shown in Figure S2a. The other two areas had around 5% of unvegetated ground. Satellite upscaling was performed when snow patches became sparse, but before budburst of the deciduous trees in the heath. The study focuses on the evergreen dwarf shrub species *Empetrum nigrum* and *Vaccinium vitis-idaea* and the mat-forming moss *Hylocomium splendens*. These species are abundant, co-occur in boreal ecosystems, and are linked to browning [9,14,33]. Our continuous monitoring of plant vitality in the study area shows that these species have not been exposed to severely stressful events since 2012, as reported in Bjerke et al. [15]. Minor damage rates were recorded in more recent years, then mostly restricted to wind-exposed sites with little snow accumulation (unpublished observations). Thus, plant traits in the study area were expected to vary naturally along microclimatic gradients. Plots with different stress levels (or health states) were established for each species to monitor natural intraspecific trait variabilities. The studied plots were dominated by (with number of plots in parentheses): *V. vitis-idaea* (two), *H. splendens* (three), *E. nigrum* (nine), and mixed plots of *E. nigrum* with a lower layer of *H. splendens* (four).

### *2.2. Data Collection*

Five greenness measurements were made in each plot (*n* = 87) from DOY 130 to DOY 180. Weather conditions varied between days of measurements. For greenness measurements we used four handheld spectral devices (Table 1). The Mapir NDVI camera was only accessible in the last two sampling cycles (*n* = 35). All passive spectral devices were applied 1.5 to 2.0 m above the plot for photographing. We avoided photographing direct light reflectance in the calibration target, and also avoided overexposure. The active Greenseeker sensor measurements were acquired 60 cm above ground.

Measurements of epidermal chlorophyll and flavonol content were conducted with the optical Dualex 4 scientific instrument (Force-A, Orsay, France). The readings of this instrument show a linear relationship to chlorophyll concentrations calculated from extractions. Readings are given in μg cm<sup>−</sup>2, and the measuring wavelength for chlorophyll is a ratio of transmittance at 710 and 850 nm [34]. Within each plot, we sampled at five different spots. The sampling dates were the same as for the greenness measurements. Measurements were made on the newest, fully developed segment of *H. splendens* and on shoot tips of *E. nigrum*. Eight *V. vitis-idaea* leaves of one plant were measured per sample, starting with the upper (= newer) leaves. Hence, we measured physiological traits in different health states on three plots per species (nine in total). All measurements were performed with the adaxial setting of the device. High correlations with the abaxial side are found [35]. Measurements of *V. vitis-idaea* leaves showed

reliable results. However, for the shoots of *H. splendens*, stable and reproducible results could only be achieved when three shoots were stacked and fixed with a transparent tape (Figure S3b). The same process was used for the shoots of *E. nigrum* (Figure S3a). At least eight chlorophyll measurements were performed on each sample and readings were then averaged and divided by the number of stacks in the tape. This results in 40 measurements per sampling date and plot. Palta [36] identified leaf anatomy, leaf veins, the presence of other pigments, and leaf thickness as main causes of large variations in chlorophyll meter readings. Hence, we decided to take the specific leaf area (SLA) into account. SLA is also of additional value to determine growth and plant stress [37,38]. After the chlorophyll measurements had been performed, a 6 mm circle was punched out of the prepared samples (Figure S3) [39]. First, fresh weight of the samples was measured. Then, the samples were dried for 24 h at 70 ◦C before dry weight was measured. This provided information on moisture content and SLA. Plant height was measured as median height above ground. Plant height of *H. splendens* refers to the thickness of the moss layer. We assessed stress levels for each plot in the first and last sampling periods. The stress estimate is a bare-eye classification of visibly dead or dying leaves versus healthy leaves of evergreens within small plots. Leaves that are dead or dying are brown, while healthy leaves are green. The stress estimate thus ranges from 0 to 100%, and it has turned out to be closely correlated to NDVI [13] and CO2 fluxes [40].

### *2.3. Data Processing*

### 2.3.1. Greenness Indices

We applied four different devices and extracted six different greenness indices from these (Table 1). The calibration methods applied are also listed. The Greenseeker did not need a calibration; internal tests suggest that measurements are not dependent on environmental changes [41].

The spectral properties of the devices we used for NDVI calculation are listed in Table 2. The active Greenseeker device has very similar bandwidths and -peaks to the Sentinel-2 bands 4 and 7. The spectral properties of the passive Mapir camera are closer to the "normal" NDVI calculated with Sentinel-2 bands 4 and 8.


**Table 1.** Background information on the greenness indices used in this study. Market prices refer to price levels in 2019.

<sup>1</sup> According to the manufacturer's documents, even non-chlorophyll-containing surfaces, such as soil, have small NDVI values. Therefore, values below 0.15 are rarely measured. Likewise, normal NDVI, with a range from −1 to 1, shows NDVI values between 0.00 and 0.12 for scarce vegetation or bare soil. <sup>2</sup> Ordinary gray card with (N2, N5, N9.5) with an accuracy of about 5% on the Macbeth color space.


**Table 2.** Comparison of spectral properties of the various NDVI devices listed in Table 1. Numbers in parentheses show the bandwidth in nanometers.

### 2.3.2. Analysis of the Data

For the calibration of the RGB indices as well as of BNDVI and GNDVI, an ordinary gray card was applied. The card is printed on Teslin Synthetic (greywhitebalancecolorcard, Northfleet, UK]. According to the manufacturer, it has an accuracy of 5% on the Macbeth color space. To decide whether a three-step reflectance calibration on black, gray, and white is superior to a normal white balance, the Channel G% index was calculated for both calibrations. For the Mapir camera (Mapir Inc., San Diego, CA, USA) conversion and calibration were performed with the Mapir calibration target V1 and the QGIS software plug-in version 1.1.2. (Mapir Inc.). All other calibrations and NDVI calculations were performed in WINCAM pro 2013a (Regent Instruments Inc., Quebec City, QC, Canada). Further processing was done in EXCEL (Microsoft Corp., Redmond, WA, USA), while statistical analyses were performed with SPSS 25.0. (IBM Corp., Armonk, NY, USA). Additional, logistic curve fitting was analyzed using the Excel add-on Xlfit version 5.3.1.3 (ID Business Solutions Ltd., Guildford, UK). We calculated all correlations with a two-tailed Pearson´s testimony. Percentiles are weighted averages. Satellite NDVI data were retrieved from ESA's Sentinel-2 Open Access Hub [https://scihub.copernicus.eu]. Sentinel-2 satellite images were analyzed using ESA's SNAP software version 5.0.8 with the integrated Sentinel-2 Toolbox (ESA, Common Service Section, Rome, Italy). Atmospherically corrected 2A products from DOY 159 and DOY 163 in 2017 were used. Cloud cover was below 1% and products were analyzed with a resampled spatial resolution of 20 m. Downscaling was done with the "mean method," which calculates the output as mean of every source pixel value. The downscaling was performed to compare the different bands of Sentinel-2 for a pixelwise NDVI comparison, and also to reduce small-scale effects, like imprecise GPS coordinates (up to 3 m). To compare spaceborne and handheld NDVI data, at least four GPS waypoints were taken per area and NDVI values between the waypoints were measured with the Greenseeker handheld sensor (Trimble Inc., Sunnyvale, CA, USA, see Table S2). The sensor can also be used for measurement over a larger area. Then, it calculates an average of the scanned area. The Sentinel-2 image pixels corresponding to the GPS waypoints were identified and values were compared with the Greenseeker data.

### **3. Results**

### *3.1. Descriptive Statistics*

Descriptive statistics of the vegetation indices and plant traits are listed in Table 3. Due to relatively large sampling sizes (*n* > *36*) for all traits on plot level we can assume that data is normally distributed. The combination of three different species in one dataset (Table 3), might lead to less normal distributed data. According a Kolmogorov-Smirnov test, all datasets of Table 1 are normally distributed (*p* > 0.05), except for Channel G%, Plant height, Stress level, Soil depth, SLA and Flav. However, a species-specific normal distribution is achieved for Plant height, NBI, SLA and Flav. For Channel G% the species-specific normal distribution does not hold for *V. vitis-idaea*, which was monitored on only two plots with highly contrasting stress levels. Stress level in total is not expected to be normally distributed due to plot selection by contrasting health states. Moreover, Soil depth is also not expected to be normally distributed.


**Table 3.** Descriptive statistics of the monitored plant traits on plot level.

### *3.2. Comparison of Di*ff*erent Vegetation Indices and Plant Traits*

We assessed the use of the six greenness indices and the different calibration methods. Accuracy was not found to be improved by using a relatively cheap 3-step reflectance target instead of an ordinary gray card for white balance (Table S3). However, it was important to avoid overexposure of the calibration target. Comparing the greenness indices (Table 4): Mapir NDVI and Greenseeker NDVI were significantly correlated (*r* = 0.951, *p* < 0.001; Figure 1a), while ordinary RGB indices showed a much lower correlation, albeit still significant. NDVI to BNDVI (*r* = 0.779, *p* < 0.001) correlated slightly better than Channel G% to NDVI (*r* = 0.749, *p* < 0.001) and NDVI to GRVI (*r* = 0.689, *p* < 0.001). Greenseeker NDVI showed the best correlation with the chlorophyll content (*r* = 0.634; *p* < 0.001), while other indices, such as BNDVI, showed rather low correlations with the chlorophyll content (*r* = 0.433, *p* < 0.01; Table 4). Figure 1 illustrates the results for different species and how they correlate with the other indices. Comparison of BNDVI and Greenseeker NDVI (Figure 1b) reveals several high BNDVI values around 1 and increased deviation of lower values. Moreover, spaceborne NDVI data from the Sentinel-2 satellites based on band 4/7/8 (Figure 1e,f) are highly correlated with ground-sampled NDVI values (*r* = 0.956 and *r* = 0.968, *p* < 0.001, *n* = *14*) obtained using the Greenseeker device. In principle, this allows for a significant upscaling from ground to space.

Joint analyses of all plant species resulted in variable correlations between the greenness indices and other plant traits, i.e. SLA, chlorophyll content, and plant height (Table 4). The nitrogen index (NBI) shows a good correlation with soil depth (*r* = 0.685, *p* < 0.001), indicating that the nutrients may be limited by shallow soil depths. The NDVI and the Channel G% indices allow for an assumption of plant height, as correlations are good (NDVI: *r* = 0.703, *p* < 0.001; Channel G%: *r* = 0.515, *p* < 0.001), even when comparing across functional groups, i.e. by considering mosses and dwarf shrubs together. Species-specific correlations are listed in Tables S5–S8. Specifically, the correlation (Figure 2b) between SLA and chlorophyll (*r* = −0.718, *p* < 0.001) is almost solely driven by the moss *H. splendens,* which shows a strong correlation when analyzed separately (*r* = −0.745, *p* < 0.01), whereas *V. vitis-idaea* and *E. nigrum* showed no significant correlation. A similar case is the SLA to flavonol correlation (*r* = −0.512, *p* = 0.001; Table 4). In this case, *E. nigrum* is the only species showing a significant correlation when analyzed species-wise (*r* = −0.589, *p* < 0.05), while the two other species showed no significant correlation.


**Figure 1.** Species are marked with different symbols (see legend). 95% confidence intervals are grey. All correlations are significant (*p* < 0.001). The panels show the relationships between: (**a**) The active Greenseeker and the passive Mapir NDVI; (**b**) the BNDVI from the modified camera and NDVI; (**c**) Channel G% from the ordinary camera and NDVI; (**d**) chlorophyll content (Chl) and NDVI; (**e**) and (**f**) represent our satellite upscaling; comparing ground-based Greenseeker NDVI and Sentinel-2 NDVI, (note *n* = *14*). NDVI is calculated with the named bands (= B). Equations and RMSE are shown in Table S4.

**Figure 2.** Relationship of two trait correlations from Table 4. Species are marked. 95% confidence intervals are grey. Relation between (**a**) median plant height and NDVI, (**b**) chlorophyll content and SLA. Equations and RMSE are shown in Table S4.

### *3.3. Intraspecific Trait Variablility*

Table 5 illustrates the overall trait variability during the study. Plant traits varied in the course of the study, with species showing contrasting responses (Figure 3). For example, Chl increased in *V. vitis-idaea,* but was stable in *E. nigrum* and *H. splendens* (Figure 3a), while the flavonol content was rather constant over time, but significantly higher in *V. vitis-idaea* than in the other two species (Figure 3b). In early June, however, all studied species showed a sudden decline in chlorophyll content (Figure 3a; DOY 152), (F4,14 = 7.945, *p* < 0.001), which coincided with the temperature dropping almost to freezing point and light snowfall during DOYs 150–151 (Figure S4). The post-hoc Bonferroni test confirms significant differences in chlorophyll content on the sampling dates before (DOY 138) and after (DOY 172) the temperature drop (*p* = 0.020 and *p* = 0.015). NDVI varied considerably within species (long boxes in Figure 3c) and only *V. vitis-idaea* showed a temporal trend in NDVI, which coincided with an increase in chlorophyll content (compare Figure 3a,c). SLA was constant over time, but significantly higher in the moss than in the two dwarf shrubs (Figure 3d).


**Table 5.** Descriptive statistics of the monitored plant traits on species level.

**Figure 3.** Intraspecific trait variability and its changes during the spring season for the species *Empterum nigrum*, *Vaccinium vitis-idaea,* and *Hylocomium splendens*. All species were monitored in varying health states, while each value represents one plot dominated by the named species. (**a**) Chlorophyll content (Chl); (**b**) flavonols (Flav); relative absorbance values; (**c**) Greenseeker NDVI; (**d**) SLA. These results were obtained at a moisture content (percent of wet weight) ranging from 55% to 75% for *H. splendens,* 63 to 68% for *V. vitis-idaea,* and 63 to 71% for *E. nigrum.*

#### *3.4. Suitable Traits for Stress Monitoring*

Our stress level estimate (Table 4) correlated with NDVI (*r* = −0.600, *p* < 0.001) and BNDVI (*r* = −0.654, *p* < 0.001), while the RGB indices performed best; *r* = 0.768, *p* < 0.001 for the Channel G% and *r* = −0.745, *p* < 0.001 for the GRVI, both as linear functions. Allowing a logistic relationship, a higher correlation level is obtained *r* = −0.833, *p* < 0.001 (Figure 4a) and r = −0.650, *p* < 0.001 (Figure 4b). Stress level estimates were also significantly correlated with plant height (*r* = −0.553, *p* = 0.001), as well as with soil depth (*r* = −0.336, *p* < 0.05). For litter, the relation with stress is reasonable, but not significant (*r* = −0.419, *p* = 0.084, *n* = 18). Neither flavonol absorbance nor chlorophyll content or the NBI readings could be related to the stress estimate (Table 4). No correlation was found for SLA and stress level (*n* = *14*).

**Figure 4.** Comparison of stress level to greenness indices. 95% confidence intervals are dashed. Relation between (**a**) stress level and the RGB index Channel G%; (**b**) stress level and Greenseeker NDVI. Equations are shown in Table S4.

#### **4. Discussion**

The range of intraspecific trait variability (1st research question), is attributed well during the study. Some plant traits remained relatively stable during spring (SLA, Plant height, Flav), while others showed more variations during the season and to environmental circumstances (NDVI, ChannelG% and Chl). The small leaves of *E. nigrum* and shoots of *H. splendens* made the SLA measurements challenging. However, the infrequently used method that we decided to apply seemed to work well, as our results are comparable to SLA values retrieved in previous studies [48–50]. To our knowledge, our plots did not suffer from any major stress (browning) events during the last 3 years prior to our measurements, except that *Vaccinium myrtillus* in the area had been partly defoliated by larvae of geometrid moths [51], but this species was rare or absent in our plots. In the early growing season, plants are especially vulnerable to winter-related stress and are showing accumulated stress responses from the previous years [13,14]. Hence, we monitored the natural range of trait variabilities from start of the growing season (DOY 130) onwards. Chlorophyll content was dropping significantly when temperature fell to almost freezing point. However, more research is needed to validate this result. It might be that the slight snowfall, or both parameters jointly, instigated the decline in chlorophyll concentrations.

The second research question was to assess whether any of the studied plant traits are suitable for stress monitoring. Our data show that the stress level differed between plots; we found that plant height was related to soil depth and that soil depth was also related to NBI. Although we did not find any significant correlation between plant height and NBI, we assume that soil depth is a limiting factor for this ecosystem. Lower soil depth affects water and nutrient availability as well as soil temperature [52] and is also associated with areas of low snow accumulation during winter [13]. This is supported by the fact that the stress level decreased with increasing soil depth and that NDVI increased with increasing soil depth.

In general, the flavonol content is associated with plant stress reactions [53]. However, we could not relate the flavonol absorbance to our stress level estimates. As the Dualex device estimates the flavonol content from spectral properties, it might not be able to measure the relevant flavonols in relation to the types of stress occurring in these subarctic plants. Dualex flavonol measurements are performed at the wavelengths 375 nm (UV-A) and 650 nm (red) [34]. This results in screening of mainly kaempferol, quercetin, and myricetin [53]. Our results are in agreement with Lefebvre et al. [54], who concluded that the Dualex device could not accurately predict the flavonol content in the three alpine plants they studied.

Concerning the third research question, our results show that ordinary RGB cameras may be used as NDVI surrogates and that they reflect various plant traits well. They performed equally well as modified cameras (with the infrared filter removed) for near-remote sensing approaches in the subarctic ecosystem. We found that a normal gray card, as used by professional photographers, was sufficient for the calibration process. Based on our findings, we recommend a simple white balance. Even if correlations to NDVI were slightly higher for BNDVI (*r* = 0.779) than for RGB greenness indices (0.689−0.749), one of the main strengths of the RGB cameras is that they are easier to operate than the modified devices. Sonnentag et al. [46] showed that different RGB cameras produce comparable results and that the choice of file format is not that important. Also, Nijland et al. [55] identified band separation and dynamic range as main problems when using converted cameras and therefore recommended the use of true color imaging. Another aspect is that the distribution of RGB cameras via smartphones is enormous and might be valuable for citizen science projects or app development [56]. In general, our greenness measurements are in agreement with existing reports on phenology at higher latitudes [30,57].

Moreover, the Channel G% index performed better than NDVI in characterizing some plant traits. This includes the stress level which showed a stronger correlation to Channel G% (linear: *r* = −0.768 vs. *r* = −0.600; logistic: *r* = −0.833 vs. *r* = −0.651) and NBI which showed a significant correlation (*r* = −0.354) to Channel G%, but not to NDVI. Consequently, the Channel G% index is of additional value for screening plant stress (2nd research question). The significant correlation between RGB indices

and chlorophyll meter readings (*r* = 0.38, *p* < 0.05) also implies that the RGB-based indices could be potential NDVI surrogates (see Table 4). In contrast to previous studies [28,29,58], our GNDVI data did not show any significant correlation with chlorophyll content or other plant traits. Correlations between chlorophyll and NDVI showed reasonable results [29], indicating that chlorophyll measurements are valid in spite of the untypical leaf structures of *H. splendens* and *E. nigrum*.

We found a high correlation between spaceborne NDVI and ground-sampled NDVI measured by the active Greenseeker device (maximum *r* = 0.968 for the Sentinel-2 NDVI calculated with bands 4 and 7). Nevertheless, despite of the strongly significant correlation, it is based only on 14 data points, implying that relationships have to be handled with care. It is a higher correlation than retrieved in previous studies, where near-remotely sensed NDVI data were compared to NDVI from Sentinel-2 and Landsat 8 [59,60]. A likely reason for the very strong correlation is that this study was carried out in a very open subarctic woodland (in parts nearly treeless and then considered as heath) where understory vegetation contributes very much to the NDVI detected by the satellites. We did not find major differences in the correlations, even when spectral properties (bandwidth and wavelength peaks) were not similar. This strongly suggests that active sensors can be used for validation of spaceborne data, for example, from Sentinel-2.

### **5. Conclusions**

The objective of this study was to assess the applicability of common plant traits and near-remote sensing approaches as tools to monitor the health state of dominant understory subarctic vegetation types that previously were shown to be vulnerable to winter climate change and other types of stress. In order to determine intraspecific trait variability, species were monitored in different health states. Due to this screening we are able to better validate the effect size of a browning event on the studied species. As the study was set in an area not recently damaged by stressful events, the different stress levels could be explained by differences in soil depth, which again act as a surrogate for several potential stressful elements, including moisture and nutrient deficits during the growing season and little snow protection during winter. Channel G% was the best RGB-based index in our study, and we recommend the use of this index. Finally, we found promising results by combining spaceborne Sentinel-2 data with the active near-remote sensor for measurements of NDVI. This could be a useful tool for upscaling the role of understory vegetation to the total NDVI measured by satellites in regions where browning occurs. Further research is recommended on the satellite upscaling, but also on the measured chlorophyll drop following a rapid midsummer temperature decrease to freezing point. Finally, we recommend following the same plots after a stressful weather event, to report direct as well as long-term changes in situ.

**Supplementary Materials:** The following are available online at http://www.mdpi.com/1424-8220/20/7/2102/s1, **Figure S1:** Study site of the 18 plots at the northern parts of Tromsøya (Norway). Exact coordinates are given in Table S1. **Figure S2:** Typical vegetation plots within the study area. (**a**) Detail view of one of the satellite upscaling areas. The photo was taken on the day of the upscaling experiment. Note: the snow patch on the left, which made plot selection difficult. The leafless trees show that this is from early spring prior to budbreak. (**b**) RGB image of a healthy plot (1 m2) of *E. nigrum*. (**c**) RGB image of a stressed plot of *E. nigrum*. **Figure S3:** Illustration of the working process for the different species. After measuring at least eight times with the chlorophyll meter (Chl, Flav, NBI), middle parts are hole-punched and cut and stored in sealed, numbered glasses to determine moisture content and SLA. **Figure S4:** Weather statistics for Tromsø from November 2016 to July 2017. The red line shows the mean value of daily temperature. The black line shows the average temperature from 1989-2018. Blue bars indicate the snow depth. Snow depth and long-time temperature data are from the Tromsø weather station (SN90450) located about 5.6 km south of the study area, while daily mean temperature is from the Stakkevollan weather station (SN90495) located about 900 m south of the main cluster of field plots. **Table S1:** Plot descriptions, including coordinates, stress level estimates, plant height, soil depth, litter, slope, and vegetation assessment. **Table S2:** Coordinates of the waypoints for satellite referencing. Waypoint coordinates are different from plot coordinates. **Table S3:** Plot-level relationships between different greenness indices and chlorophyll content for different calibration methods. Wb = white balance, 3-step = 3-step reflectance calibration. **Table S4:** Equations corresponding to the linear regressions in Figures 1 and 2. **Table S5:** Correlation table for plots dominated by *E. nigrum*. Correlations are computed by Pearson's correlation, while significance is two-tailed. **Table S6:** Correlation table for plots dominated by *Vaccinium vitis-idaea*. Correlations are computed by Pearson's correlation, while significance is two-tailed., **Table S7:** Correlation table for plots dominated by *Hylocomium splendens*. Correlations are computed by Pearson's correlation, while significance is two-tailed. **Table S8:** Correlation table for mixed plots of *Empetrum nigrum* and *Hylocomium splendens*. Correlations are computed by Pearson's correlation, while significance is two-tailed.

**Author Contributions:** Conceptualization, E.R., J.W.B. and HT.; methodology, E.R., H.T. and J.W.B.; formal analysis, E.R., J.W.B. and H.T.; writing—original draft preparation, E.R, J.W.B. and H.T.; writing—review and editing, E.R, J.W.B. and H.T.; project administration, J.W.B.; funding acquisition, J.W.B. All authors have read and agree to the published version of the manuscript.

**Funding:** Funding was provided by the Research Council of Norway grant 287402.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## **Assessing the Trend of the Trophic State of Lake Ladoga Based on Multi-Year (1997–2019) CMEMS GlobColour-Merged CHL-OC5 Satellite Observations**

### **Augustine-Moses Gaavwase Gbagir 1,\* and Alfred Colpaert <sup>2</sup>**


Received: 25 September 2020; Accepted: 27 November 2020; Published: 1 December 2020

**Abstract:** The trophic state of Lake Ladoga was studied during the period 1997–2019, using the Copernicus Marine Environmental Monitoring Service (CMEMS) GlobColour-merged chlorophyll-a OC5 algorithm (GlobColour CHL-OC5) satellite observations. Lake Ladoga, in general, is mesotrophic but certain parts of the lake have been eutrophic since the 1960s due to the discharge of wastewater from industrial, urban, and agricultural sources. Since then, many ecological assessments of the Lake's state have been made. These studies have indicated that various changes are taking place in the lake and continuous monitoring of the lake is essential to update the current knowledge of its state. The aim of this study was to assess the long-term trend in chl-a in Lake Ladoga. The results showed a gradual reduction in chl-a concentration, indicating a moderate improvement. Chl-a concentrations (minimum-maximum values) varied spatially. The shallow southern shores did not show any improvement while the situation in the north is much better. The shore areas around the functioning paper mill at Pitkäranta and city of Sortavala still show high chl-a values. These findings provide a general reference on the current trophic state of Lake Ladoga that could contribute to improve policy and management strategies. It is assumed that the present warming trend of surface water may result in phytoplankton growth increase, thus partly offsetting a decrease in nutrient load. Precipitation is thought to be increasing, but the influence on water quality is less clear. Future studies could assess the current chemical composition to determine the state of water quality of Lake Ladoga.

**Keywords:** Lake Ladoga; CMEMS GlobColour CHL-OC5; eutrophication; water quality assessment; pulp and paper mill; climate change; ecological status; remote sensing; phytoplankton and chlorophyll-a; chemical wastewater pollution

### **1. Introduction**

Assessing the trophic state of Lake Ladoga is essential because of its past history of severe water pollution [1,2]. The history of the Great Lake Ladoga begins after the melting of the ice of the last glaciation, when Ladoga became separated from the so-called Yoldia Sea about 10,000 years ago [3]. The earliest drainage of the lake was to the Gulf of Finland in the direction of the region now known as the city of Vyborg through the threshold of Vetokallio near Heinjoki [3]. Post-glacial uplift, which was, and still is, faster in the north, tilted Lake Ladoga causing a transgression of its southern shallow shoreline, leading to the creation of the current Neva River connection about 3300 years ago [3]. For the same reason, about 9500 years ago, a connection was established from Lake Onega to Ladoga via the River Svir [3]. The birth of the Neva caused the surface of Ladoga to drop by about 12 m, producing the present shape of the lake [3]. The Lake was oligotrophic (mostly free from nutrients)

before the 1950s and mesotrophic (moderate nutrients) in the 1970s but some areas of the lake (mostly coastal locations) have become eutrophic due to anthropogenic activities [1,4]. These anthropogenic activities were mainly industrial enterprises (e.g., pulp and paper mills, aluminum) and agriculture [1,5]. Consequently, the discharge of wastewater (containing high amounts of phosphorus and nitrogen) into the lake from these industries resulted in the degradation of water quality [1,2]. The lake has a large catchment (Figure 1), and thus receives huge amounts of water containing anthropogenic nutrients [2,6]. An important part of these waste waters discharge into the shallow Volkhov Bay (southern coastal area) which receives its waters from the Volkhov sub-catchment (West Ladoga), and Syas River (from the South) [5]. Also, extensive agricultural activities resulted in the washing of phosphorus and nitrogen into the lake [1,6]. It is well documented that the lake contains toxic chemical substances, pesticides, other harmful material as well as large amounts of sediment [5,6]. Eventually, Lake Ladoga flows through River Neva near St-Petersburg into the Gulf of Finland [7]. The lake is the main source of clean water not only for local communities, but also for the city of St. Petersburg (over five million inhabitants), thus there is much pressure to maintain good water quality. Also, the importance of the lake for recreational use is steadily increasing [6,8].

Thus, continuous monitoring of the ecological status and water quality of this large lake is essential to understand and assess changes in its environment. Monitoring activities have been on-going for several decades [1,3,5,6]. In this paper, ecological status means the assessment of water eutrophication, through changes in phytoplankton levels. The use of phytoplankton biomass and chlorophyll-a (chl-a) concentration is a common method employed to study ecological status in water bodies [9,10]. This method is successful because chl-a correlates well with phytoplankton biomass abundance [9,11] and is used as an indicator for eutrophication [10]. Currently, low-cost instruments exist for measuring in situ chl-a in water bodies [12–14]. On the other hand, the use of these low-cost instruments still rely on the use of expensive research vessels and time consuming field surveys. However, the development of remote sensing and satellite-based imagery serves a wider audience [10,15]. Although estimates of chl-a based on remote sensing are generally less accurate than laboratory measurements, the approach provides a cost-effective and sufficiently reliable regional picture of a phenomenon that can be used for environmental monitoring and management needs. By using remote sensing data, the amount of chl-a can be calculated using empirical formulas based on the correlation between the violet-green light reflected from the water surface and the measured amounts of chl-a [16]. In addition, remote sensing methods provide historical time series of observations, for example, Landsat satellite data have been available since 1972 and MODIS data since 2000. Pozdnyakov et al. [1] have used SeaWiFS satellite images to study the biochemical properties of Ladoga from 1998–2004. Pozdnyakov et al. [1], were also able to validate the consistency of remote sensing derived chl-a with in situ chl-a measurements in Lake Ladoga. Studies using temporal remote sensing data to study the ecological status of Lake Ladoga are quite few and none have been done after 2010 (e.g., Pozdnyakov et al. [1]). The complex nature of coastal and lake water bodies [excessive colored dissolved organic matter (CDOM) and mineral particles] makes it difficult to use remote sensing to assess chl-a concentrations (so called Case II water bodies), thus many empirical, local, algorithms have been developed [1]. Developing different algorithms to suit each local water type around the globe is a dilemma. Thus, the main goal of this study was to use a global remote sensing data product that has considered the complex nature of different water types. In this study, we utilized the merged chlorophyll-a (chl-a) product derived from SeaWiFS, MERIS, MODIS Aqua, VIIR, and OLCI satellite sensors (1997–2019) to characterize the general trend of chl-a in Lake Ladoga. The novelty in our approach is to use geostatistical analytics of remote sensing data to produce a historical and synoptic picture of the state of Lake Ladoga. We anticipate that the results of this long-term assessment will contribute to the current information on the ecological status of Lake Ladoga. This additional information may be beneficial to on-going policy frameworks and management strategies of Lake Ladoga.

### **2. Materials and Methods**

### *2.1. Study Area*

Lake Ladoga is the largest lake in Europe having a surface area of 17,765 km<sup>2</sup> [17], with an average depth of 48.3 m [17] and maximum depth of 230 m (Figure 1) [6,17]. The lake has a volume of 858 km<sup>3</sup> [17], a water retention time of 12 years, and a coastline of 1570 km [17,18]. The lake has a total catchment area of 282,200 km2, 20% of the area is located in Finland [6] (Figure 1b). The deepest parts of the lake are in the North (60–200 m), and the shallowest parts (10–50 m) in the southern parts of the basin [4]. The northern shores of the lake are rugged and rocky, while the western, southern, and eastern shores are shallow with fine-grained sediments [4]. The East and South contain extensive sandy beaches as well [4]. Water from Finland flows via the River Vuoksi from the west into Lake Ladoga while from the East, the water from Lake Onega drain via the River Svir [4]. From the South, the waters from Lake Ilmen flow into Ladoga via the River Volkhov [4]. Also, the River Syas and dozens of smaller streams and rivers flow into Ladoga as well [5]. Eventually, Lake Ladoga flows through River Neva near St-Petersburg into the Gulf of Finland [7].

Mean air temperature in the area is +3.2 ◦C [18,19]. The coldest month (February) has a mean temperature of −8.8 ◦C, while the warmest month (July) has +16.3 ◦C [19]. The mean annual precipitation in the area is about 475 mm [19].

**Figure 1.** Shows Lake Ladoga, the study area. In (**a**), zoomed in with Ladoga in the center, Lake Onega (upper right), and Gulf of Finland (lower left corner), buffer 1, buffer 2, square are sample sites. In (**b**), Lake Ladoga drainage basin. In (**c**), is the bathymetric map of Lake Ladoga (image adapted and modified from Subetto et al. 1998 [20]).

Along Lake Ladoga's shores there are several cities which include: Sortavala, Priozersk, Shlisselburg, Novaya Ladoga, and Pitkäranta (Figure 1a), with a combined population of about 100,000 people. Also, there is a long history of industrial activity, mainly wood processing industries. Some closed pulp and paper mills were located near Priozersk, and Harlu, Läskelä. Currently, there are still functioning pulp mills in Pitkäranta in the North-East and near the town of Novaya Ladoga in Syasstroy (south). Lake Ladoga is inhabited by several endangered species, as the Ladoga Lake Salmon and the Ladoga Ringed Seal [18].

### *2.2. Data*

We used the merged (4 km × 4 km) global monthly chlorophyll (chl-a) product (1997–2019) from GlobColour [21]. The merged chlorophyll-a OC5 algorithm product (CHL-OC5) is available from September 1997 to present and is updated annually [21]. The global monthly chl-a is a "Cloud Free" level four (L4) product created by merging data from four satellite sensors, the Sea-Viewing

Wide Field-of-View Sensor (SeaWiFS), Medium Resolution Imaging Spectrometer (MERIS), Moderate Resolution Imaging Spectrometer (MODIS), Visible Infrared Imaging Radiometer Suite (VIIRS), and Ocean and Land Color Instrument (OLCI) [21]. The GlobColour products were downloaded from the Copernicus Marine Environmental Monitoring Service (CMEMS) website: http://marine.copernicus. eu/services-portfolio/access-to-products/.

CMEMS CHL-OC5 dataset has been created using the OC5-algorithm which has been shown to work very well for Case II water bodies and has been validated using a large number of global in situ data from sea and lake areas [21,22]. We tested the dataset for lake Ladoga using data from Letanskaya and Protopopova [23], giving a reasonble fit, the CHL-OC5 dataset seemed to overestimate chl-a values in the lower range (below 5 mg m<sup>−</sup>3), but for the higher ranges (over 15 mg m−3) the data was consistent with the in situ values. As our main interst was in the trend of water chlorophyl-a content and not in absolute values, we used the CMEMS CHL-OC5 dataset in our study, with the notion that the values are not exact values, but relative indicators derived from a global dataset.

The datasets are for the months of June, July, August, September, and October. The downloaded chl-a data was processed further using the SeaWiFS Data Analysis System (SeaDAS) software [24]. SeaDAS is a free and opensource software developed and maintained by Ocean Biology Processing Group (OBPG) [24]. In SeaDAS, the land and water mask were created, and the no data layer added. The appropriate vector data was added, and the final images exported and compiled using Inkscape [25].

Additional processing of the chl-a was done with R software (R Core Team, 2020, [26], with raster package [27], to extract the mean, minimum and maximum values of chl-a. The results were then written to Excel format with the R package writexl [28]. Also, for each year (1997–2019), we calculated the (seasonal) mean of means for the five months (June, July, August, September, and October). Hence, in this study we used season to represent the calculated mean of the five months. We calculated the mean over the whole lake as a unit and within three sub-sample areas (buffer 1, buffer 1, and central part of the lake). Buffers 1 and 2 were twelve and twenty kilometers, respectively, from the shoreline (Figure 1). When calculating the season means, we excluded 1997 (only September) in the trend analysis to prevent bias in the result.

### **3. Results**

### *3.1. General Trend of Chl-a Concentration in Lake Ladoga: 1997–2019*

The general trend of seasonal chl-a concentration in Lake Ladoga is negative (Figure 2, wl). The negative trend is gradual (slope <sup>=</sup> <sup>−</sup>0.12416 and R<sup>2</sup> <sup>=</sup> 0.4069). This indicates a moderate improvement (decline in chl-a). In general, 2009 had the highest chl-a concentrations (>10 mg m−3) while the lowest value was measured in 2017 (<5 mg m−3). Similarly, the trend in the sub-sampled locations (s1, s2 and s3), was negative but non-significant ((buffer1: slope = <sup>−</sup>0.2962 and R2 = 0.4212); (buffer2: slope = <sup>−</sup>0.3039 and R2 = 0.3991); (center square: slope = <sup>−</sup>0.2974 and R2 = 0. 433)). Water surface temperature (SST) had no observable trend in the lake during the study period.

**Figure 2.** The average seasonal trend of chlorophyll-a concentration (mg m<sup>−</sup>3) 1998–2019. wl = whole Ladoga lake, buffer1 = 12 km from shore line, buffer2 = 20 km from shore line, and central square = the sample site in the middle of the lake.

### *3.2. Spatial Distribution of Chl-a Concentration Across Lake Ladoga*

Figures 3–7 show the pixel-wise variability of chl-a concentration across the lake. For June, wide distribution of chl-a concentration (moderate to high values) was observed for 1998, 2000, 2002, 2003, 2006 and 2009 (Figure 3). The high values were along the shallow southern shoreline and moderate values were within the central part.

**Figure 3.** The map of chlorophyll-a concentration at Lake Ladoga for the month of June (1998–2019). Partially displayed as reference points are the Gulf of Finland (Southwest) and Lake Onega (Northeast).

**Figure 4.** The map of chlorophyll-a concentration at Lake Ladoga for the month of July (1998–2019). Partially displayed as reference points are the Gulf of Finland (Southwest) and Lake Onega (Northeast).

**Figure 5.** The map of chlorophyll-a concentration at Lake Ladoga for the month of August (1998–2019). Partially displayed as reference points are the Gulf of Finland (Southwest) and Lake Onega (Northeast).

**Figure 6.** The map of chlorophyll-a concentration at Lake Ladoga for the month of September (1997–2019). Partially displayed as reference points are the Gulf of Finland (Southwest) and Lake Onega (Northeast).

**Figure 7.** The map of chlorophyll-a concentration at Lake Ladoga for the month of October (1997–2019). Partially displayed as reference points are the Gulf of Finland (Southwest) and Lake Onega (Northeast).

In July, higher chl-a concentrations spread across the lake (Figure 4). During this month, high concentration values were measured in 2002, 2003, 2006, 2009, 2010. On the other hand, 2012 and 2015, appear almost free from high chl-a values except on the southern coast.

The highest chl-a values in August were measured in 2002 (20.70 mg m<sup>−</sup>3), 2003 (21.61 mg m−3), 2006 (25.34 mg m<sup>−</sup>3), 2007 (18.82 mg m−3), and 2019 (19.76 mg m−3) (Figure 6). The year 2017 had very low values while 2019 had moderate to high values across the entire lake.

September–October marks the beginning of the winter cooling period causing a drop in chl-a concentrations (Figures 6 and 7). The entire lake still had detectable chl-a values in September, except in 2014, 2017 and 2018. The yearly temporal and spatial distribution of water temperature across Lake Ladoga is driven mainly by the thermal bar phenomenon [1,3], which is even noticeable from satellite images [1]. The thermal bar is a common characteristic of large lakes [1]. In Lake Ladoga, the thermal bar is formed in spring and early summer [1,3]. The thermal bar is the column of water formed between the cold (deep central) and warmer (coastal) waters of a lake [17]. It is the location of vigorous water mixing caused by downwelling of water with a temperature of four degrees Celsius [3]. This is the result of the physical property of water, that has its maximum density at +4 ◦C, thus heavier rather than colder or warmer water, causes it to sink. The thermal bar effectively acts as a wall between the coastal and inner water masses of a lake, inhibiting mixing of the two water masses. This, in turn, only allows for near shore cyclonic currents caused by the prevalent winds and the Coriolis effect. Consequently, there is a bloom in phytoplankton biomass (chl-a producer) along the coast. The thermal bar disappears around the end of June and beginning of July when the lake's water temperature is homogeneously above +4 ◦C, usually up to the depth of 50–70 m [1,3]. As the thermal bar disappears, temperature no longer plays a major role in chl-a concentrations in the lake [1,3]. The cyclonic current transports the phytoplankton to the north of Lake Ladoga. It is important to note that during August, the effect of the thermal bar has completely disappeared [1]. Before the disappearance of the thermal bar, the water flow and movement of phytoplankton is cyclonic (counterclockwise) in nature (South-East to North) [3], mainly along the shoreline (Figure 1c). The cyclonic nature of the water movement is driven by the topography, bathymetry, selective westerly winds [1], and the rotation of the earth (i.e., the so-called Coriolis phenomenon). There are also anticyclonic eddy currents in Lake Ladoga [29,30]. In general, anti-cyclonic currents are caused by weak bottom friction and southerly winds [29–31]. Most of the observed low chl-a concentrations are in the north and central parts of the lake while the southern coastal areas remain unchanged. From our results and in-depth literature review, the following factors were identified to be driving the on-going chl-a changes taking place in Lake Ladoga: (i) Temperature, (ii) impact from industry, external load, and littoral settlements. These coastal areas are locations of high industrial activity (e.g., paper mills), settlements and from external sources [1,3].

### **4. Discussion**

As mentioned in Section 3.1 (first paragraph), we observed a gradual and negative trend in chl-a distribution, indicating a moderate improvement of water quality.

### *4.1. E*ff*ect of Temperature and Eutrophication on Chl-a Concentrations*

During the month of June, high chl-a concentrations are observed only along the southern–southwestern coastal areas while the deeper parts of the lake remain clean (Figure 3), due to the presence of the thermal bar. These coastal areas have major influxes of nutrients from rivers like the Syas, Volkov and Svir, carrying municipal, industrial and agricultural waste [1,3].

The action of the southern summer winds causing anti-cyclonic eddies moves and distributes the phytoplankton mass towards the central part of the lake [1]. The temperature along the shallow coastal areas of the lake is now homogeneous [1]. Consequently, eutrophication by nutrients originating from bottom sediments could possibly be a factor contributing to chl-a concentrations [1,3]. These eutrophic sediments are thought to originate from high nutrient fall-out between the 1950s and 1990s, the era of intense eutrophication [32]. However, the present role of eutrophication of bottom sediments is arguably not a major factor [1]. Previous studies analyzing water samples from Lake Ladoga revealed that the coastal areas in the North, Southwest, and South were very eutrophic [33]. Also, the southern part of the basin has high concentrations of dissolved organic matter (DOM), primarily from external water load, excretion from green vegetation and decay of macrophytes [1].

The cooling cycle in Lake Ladoga starts in September, causing reduction in phytoplankton biomass and consequently a decline in chl-a values. An important aspect to consider here is the impact of climate change on Lake Ladoga. In general, the last two decades have been exceptionally warm, in fact, the warmest period in the entire history of climate measurement. Sharov et al. [34], reported a temperature increase of +1.5 degrees Celsius between 1950 and 2010 in Lake Ladoga [34]. Likewise, years 2000 and 2015 were exceptionally warm [18]. Consequently, this has increased the duration of the lake's ice-free period from 210 days to 230 days [34–36]. A recent study by Karetnikov et al. [17], also reported increases in temperature giving rise to shorter ice periods. As a result, the increase in available sunshine (less ice) and higher surface water temperatures promote the growth of algae. A similar observation (rising water surface temperature and fewer ice days) have been reported in Lake Peipsi [37]. The impact of climate change on temperature increase promoting algal growth has also been reported in other regions as well [38,39].

### *4.2. Impact from Industries, Coastal Settlements, and External Load*

The decline in chl-a (an indicator of improved water quality) is largely due to the closing of paper mills that led to less wastewater being discharged into Lake Ladoga [40]. High chl-a areas have been observed around functioning paper mills and coastal cities, these sources discharge a considerable amount of nutrient rich wastewater [5]. The closure of several mills led to a significant reduction in primary phytoplankton (chl-a) production and phosphorus concentrations [5]. The anthropogenic activities of the high-density settlements around Lake Ladoga could also be sources for nutrients driving algal blooms [41]. Also, by analyzing comprehensive water samples from Lake Ladoga, Holopainen et. al. [33], found parts of the lake to be eutrophic. They found high chl-a concentrations near Sortavala Bay (26 mg/m<sup>3</sup> in August) and Pitkäranta (functioning paper mill) in the northern shores. High chl-a values were also in the southwest corner of Ladoga, in front of Zaporozhskoye (Metsäpirtti) (11.9 mg/m3) and in front of the city of Novaya Ladoga (20–25 mg/m3). The paper and pulp mills in Priozersk (Käkisalmi), Harlu and Läskelä were already established during the Finnish period (before 1945) and continued to operate during the Soviet and Russian era until their closure in the late 1980s. The Pitkäranta pulp mill was also established as early as the 1920s, and its production continues today. An active pulp mill is located near the town of Novaya Ladoga, on the south coast of Ladoga. Here a major accident occurred in 1998 when 700,000 cubic meters of toxic sludge spilled into the Syas River, about a kilometer from the shores of Lake Ladoga [5]. Phosphorus and nitrogen emissions from agriculture have also been a major nutrient source, but agriculture has declined since 1990.

External load from the large catchment area is still a major factor keeping chl-a levels elevated [6] (Figure 1b). The effect of the Volkov Bay (southern Lake Ladoga) is also obvious (Figures 3–7), as the Volkov River is the largest external load on Ladoga. For example, high phosphorus concentrations of 210 μg/L were found in the 1980s as compared to 46 μg/L in the 1950s and 1960s [5]. The sources of these phosphorus values were from a large number of industrial plants (594) and agricultural enterprises (680) in the watershed [42,43]. However, due to the closure of factories in the 1990s [5,40] phosphorus concentrations in the Volkhov Bay were found to have dropped to 120 μg/l from 210 μg/l [5]. Especially the closure of paper mills has been a contributing factor to the improved water quality of Lake Ladoga [1–3,5].

A similar trend of water pollution has been reported at the pulp and paper mills located at the shores of Lakes Onega and Imandra [44–46]. Both of these locations (Kondopoga Bay and Imandra Bolshaya Bay), have seen increased nutrient loads, other toxic chemicals and degraded water quality [45]. However, in comparison to Ladoga, Onega and Imandra, are considerably cleaner (Figures 3–7). To put this into perspective, the chl-a values of Lake Onega (1999–2010) in Lahti, Petrozavodsk varied between 1 and 7 mg/L without a clear trend (with an average of 2–3 mg/L), clearly indicating low pollution levels [34]. Also, Onega's burden is only one urban center, the city of Petrozavodsk. On the other hand, recents studies in Onega have indicated significant water browning especially around Petrozavodsk Bay [47].

The pollution and degradation of water quality from wastewater discharged from pulp and paper mills, urban settlements and agricultural activities have been documented in other countries as well. For example, in Finland's pulp and papers mills located along Lake Päijänne polluted the lake's water, thus degrading the water quality [48].

On the other hand, it is well documented that, better waste management practices by paper mills, settlements and agricultural activities have improved the water quality of rivers and lakes [49,50]. For example, the Stora Enso Veitsiluoto Mills at Kemi, northern Finland [51,52] and the Kaukas paper mill on southern Lake Saimaa have been able to improve local water quality through better water management processes [53]. It is worthy to note that the closing of paper mills was a major factor to improved water quality in the absence of better waste management practices, e.g., [51,52,54].

### **5. Conclusions**

This study assessed the trophic state of Lake Ladoga during the last 23 years (1997–2019). Geostatistical tools were used to analyze remote sensing data for this purpose. Our analysis reveals a slight decline in chl-a, suggesting there is a moderate improvement in the state of the lake. This study observed that the southernmost and shallowest part of the water body was the most problematic with high observable concentrations of chl-a. This southern area receives nutrients both from the Syas catchment and from bottom sediments causing high phytoplankton growth. Observable differences were seen in the deeper parts of Lake Ladoga in the northern part of the basin. In this northern part, reduced chl-a concentrations were observed, although there were local differences, especially on the north-eastern shore off the Pitkäranta factory site. Also, on the northern shore of Sortavala, there were exceptions as well which could be due to the discharge of municipal sewage.

It is important to note that reducing the nutrient load in Lake Ladoga due to anthropogenic activities is a slow and complicated process. These processes include effects of climate change, mainly warming and ice reduction, both of which contribute to eutrophication of the lake. The impact of climate change on rainfall is not certain, but it is generally assumed that rainfall will increase, thus, increasing the external nutrient load of Lake Ladoga. Furthermore, deforestation in the catchment area also increases nutrient leaching from the soil, and it is therefore uncertain whether the current good trend in the decrease of chl-a will continue. Despite the slight improve of the state of Lake Ladoga reported in this study, it appears that primary production is still relatively high. Our conclusion is in line with previous studies of the lake [6].

In this study, after the visual assessment of satellite imageries and an in-depth literature review, we are of the opinion that the current nutrient load and chemical waste influx in Lake Ladoga is less now, as compared to 23 years ago. We are of the opinion that to further decrease chl-a concentrations in Lake Ladoga, sewage from municipalities should be treated before entering the lake, agricultural practices have to be adjusted to reduce washing of fertilizers, the nutrient load from fish farming should be reduced and traditional fishing should be increased. These measures could potentially improve the water quality of the lake.

**Author Contributions:** Conceptualization, A.-M.G.G. and A.C.; methodology, A.-M.G.G. and A.C.; software, A.-M.G.G. and A.C.; formal analysis, A.-M.G.G.; investigation, A.-M.G.G. and A.C.; resources, A.C.; data curation, A.-M.G.G. and A.C.; writing—original draft preparation, A.-M.G.G.; writing—review and editing, A.-M.G.G. and A.C.; visualization, A.-M.G.G.; supervision, A.C.; project administration, A.C.; funding acquisition, A.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was part of the Kone foundation funded Green Belt History project.

**Acknowledgments:** We thank the University of Eastern Finland for material assistance, and two anonymous reviewers for their valuable comments.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**

1. Pozdnyakov, D.V.; Korosov, A.A.; Petrova, N.A.; Grassl, H. Multi-year satellite observations of Lake Ladoga's biogeochemical dynamics in relation to the lake's trophic status. *J. Great Lakes Res.* **2013**, *39*, 34–45.


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

### **Generating Time-Series LAI Estimates of Maize Using Combined Methods Based on Multispectral UAV Observations and WOFOST Model**

**Zhiqiang Cheng 1,2,3, Jihua Meng 2,\*, Jiali Shang 4, Jiangui Liu 4, Jianxi Huang 5, Yanyou Qiao 2, Budong Qian 4, Qi Jing 4, Taifeng Dong <sup>4</sup> and Lihong Yu <sup>6</sup>**


Received: 24 August 2020; Accepted: 20 October 2020; Published: 23 October 2020

**Abstract:** Green leaf area index (LAI) is an important variable related to crop growth. Accurate and timely information on LAI is essential for developing suitable field management strategies to mitigate risk and boost yield. Several remote sensing (RS) based methods have been recently developed to estimate LAI at the regional scale. However, the performance of these methods tends to be affected by the quality of RS data, especially when time-series LAI are required. For crop LAI estimation, supplementary growth information from crop model is helpful to address this issue. In this study, we focus on the regional-scale LAI estimations of spring maize for the entire growth season. Using time-series multispectral RS data acquired by an unmanned aerial vehicle (UAV) and the World Food Studies (WOFOST) crop model, three methods were applied at different crop growth stages: empirical method using vegetation index (VI), data assimilation method and hybrid method. The VI-based method and assimilation method were used to generate time-series LAI estimations for the whole crop growth season. Then, a hybrid method specially for the late-stage LAI retrieval was developed by integrating WOFOST model and data assimilation. Using field-collected LAI data in Hongxing Farm in 2014, the performances of these three methods were evaluated. At the early stage, the VI-based method (R2 = 0.63, RMSE = 0.16, *n* = 36) achieved higher accuracy than the assimilation method (R2 = 0.54, RMSE = 0.52, *n* = 36), whereas at the mid stage, the assimilation method (R2 = 0.63, RMSE = 0.46, *n* = 28) showed higher accuracy than the VI-based method (R2 = 0.41, RMSE = 0.51, *n* = 28). At the late stage, the hybrid method yielded the highest accuracy (R2 = 0.63, RMSE = 0.46, *n* = 29), compared with the VI-based method (R2 = 0.19, RMSE = 0.43, *n* = 28) and the assimilation method (R2 = 0.20, RMSE = 0.44, *n* = 29). Based on the results above, we considered a combination of the three methods, i.e., the VI-based method for the early stage, the assimilation method for the mid stage, and the hybrid method for the late stage, as an ideal strategy for spring-maize LAI estimation for the entire growth season of 2014 in Hongxing Farm, and the accuracy of the combined method over the whole growth season is higher than that of any single method.

**Keywords:** crop growth; reflectance saturation; crop model; assimilation; crop growth stage; method combinations

### **1. Introduction**

Leaf area index (LAI) is defined as the one-sided green leaf area per unit ground area in broadleaf canopies and as the projected needle leaf area in coniferous canopies [1,2]. Because green LAI (LAIg) determines light interception and absorption of the crop canopy [3], estimation of crop LAI is critical for understanding the biophysical processes of crop growth that are essential for predicting crop biomass or yield [4,5].

To achieve accurate LAI estimations, a number of remote sensing (RS) based approaches have been developed over the past few decades. These approaches can be grouped into two broad categories: empirical methods and physical models [6]. Empirical methods can be further divided into parametric and non-parametric empirical methods based on whether there is an explicit relationship between selected RS bands and LAI [7]. A series of optical vegetation indices (VIs) have been developed through combinations of reflectance in two or more bands [8,9] for maize LAI estimation. For instance, a normalized difference vegetation index (NDVI) [10] calculated from the reflectance in red and near-infrared bands are commonly used for retrieving canopy biophysical properties of corn [11]. Because NDVI is highly affected by soil reflectance at low LAI and shows asymptotic saturation at high LAI [12], some more VIs were designed, such as the optimized soil adjusted vegetation index (OSAVI) [13] for considering the soil effect and the two-band enhanced vegetation index (EVI2) [14] for tackling reflectance saturation. The building of empirical methods depends on ground LAI observations which were generally made using different instruments [15–17]. Using ground LAI measurements and RS-based VIs, several empirical methods have been developed for maize LAI estimations [7,10,11]. The non-parametric empirical methods usually define the regression function between the RS information and the target variable, e.g., LAI directly [18]. Machine learning techniques, include neural network [19], support vector regression [20] and gaussian processes [21], are typical non-parametric empirical methods to generate LAI products from a variety of RS data products. Although empirical methods are effective in estimating LAI with high accuracy, they are site- and time-specific [22].

Physical models are constructed based on simulations of radiative transfer process in vegetation canopy, and therefore their utilities are not limited to a specific site or a certain time of a year as empirical methods. Physical models can be further divided into geometric optical (GO) model [23], radiative transfer (RT) model [24] and RT-GO model [25]. In these models, LAI is one of the canopy parameters that determine canopy reflectance. With the canopy reflectance acquired from RS data and other canopy parameters known, these models can be inverted to derive LAI. RT models are more suitable for crop LAI retrievals than GO models as a result, the PROSAIL model, a RT model, has been commonly used to simulate LAI for maize using RS data [26,27]. The limitations of physical models include the high cost of parameters calibration and low model running efficiency. Recently, a physical model and a machine learning technique were combined for LAI estimations [6]. For instance, the PROSAIL model and a neural network algorithm were combined to estimate maize LAI with high accuracy [28]. The machine learning algorithm can help improve the efficiency of physical models, but the high cost of parameters calibration cannot be completely avoided.

Because crop growth is a long-term process, continuous LAI information for the entire growth season is critical to calculate corresponding biomass at different growth stages and thus essential to predict the final yield. Since empirical methods and physical models require RS data as inputs for LAI estimation, the quality of RS data will influence the estimation accuracy, especially when time-series RS data are required to generate LAI estimations for different crop growth stages. In the rainy season in northeast China, clouds, shadows, and haze can strongly influence the application of multispectral RS data with high and medium spatial resolutions in time-series LAI estimation. Because plants usually develop fast in the rainy reason, the lack of good quality RS data will cause the LAI estimates to miss the critical growth stage, and thus will consequently influence biomass or yield estimation. At the late-growth stage, the appearance of more senescing leaves will limit the RS application of LAI estimations [29]. The senescing leaves are still attached to plants and can be captured by in-situ

LAI instruments. However, these leaves have reduced photosynthetic capacity and therefore present quite difference spectral behaviors from that of green leaves. The presence of senescing leaves can be at different extents at the late stage [14,30], which is a great source of uncertainty for canopy LAI estimation from RS data. The saturation of reflectance and VIs is another significant limitation for the late-growth stage. The EVI has been designed to provide improved sensitivity at high LAI [31]; however, it is still not sufficient for the spring maize.

As mentioned above, the quality issues of RS data may cause failures of directly applying RS-based empirical and physical models for LAI estimation in rainy seasons or at the late-growth stage. Hence, supplementary information should be referred to address this issue. Unlike other vegetation types, crops usually own a more regular growth period which can be useful in building mechanistic models for simulating crop growth. Crop models [32,33] are able to provide comprehensive mathematical descriptions of key crop physical and physiological processes, which cannot be achieved by RS-based methods. Therefore, it is more applicable to use crop models for LAI estimations during rainy- and late-growth seasons.

Since crop models are generally designed to simulate crop growth at the site level, several data assimilation approaches [34–36] have been developed to incorporate crop parameters estimated from RS data into crop model, aiming to extrapolate the simulations of crop models from a single site to the regional scale. Among them, the Ensemble Kalman Filter (EnKF) [37–39], particle filtering (PF) [40], and four-dimensional variational data assimilation (4DVar) [41,42] are common methods used to link crop models with RS data. Unlike RS-based LAI estimation methods, assimilation methods can tolerate time-series RS data withlong time intervals, making it possible to retrieve accurate LAI estiamtions in rainy season. Therefore, it is feasible to use crop models to fill data gaps where high-quality continuous RS data are not available. Because assimimation methods still require a small amount of RS data, it cannot address the quality issues of RS data at thelate-growth stage. Hence, further studies are needed to improve the multi-phase LAI estimates by integrating RS data and crop models.

The objective of this study is to address the quality issues of RS data for time-series LAI generation. Using the World Food Studies (WOFOST) crop model and RS data acquired by an unmanned aerial vehicle (UAV) in 2014, three methods were applied to estimate LAI for spring maize in Hongxing Farm. Based on the performances of these methods, a method combination was developed to generate LAI for the entire growth season, which includes: (1) a VI-based empirical method for early-stage LAI estimation before rainy season; (2) the EnKF assimilation method for rainy-season LAI generation; (3) the hybrid method by integrating EnKF method and WOFOST model for late-stage LAI retrieval. Complete details of the applied methods and their performances were presented in the following sections.

### **2. Materials and Methods**

### *2.1. Study Area and Field Campaign*

The study was conducted in an experimental field (48◦08 N, 126◦57 E, WGS84) of Hongxing Farm, located in Heilongjiang province, Northeast China. Hongxing Farm is a large state-owned farm that lies within the temperate monsoon climate zone characterized by an average annual precipitation of 548.8 mm (2014) and an average annual cumulative temperature (>10 ◦C) of 2293.0 ◦C (2014). Spring maize accounts for nearly 50% of the planting area of Hongxing Farm. The spring maize growing season extends from the beginning of May till mid-October. Another important crop in this region is soybean, which is usually rotated with spring maize. All the fields in Hongxing Farm have a unique identifier (ID). The ID of the experimental field selected for this study is 5-1-2. Plot 5-1-2 covers 11 hectares (ha), and seeded with spring maize on 18 May 2015. Figure 1 shows the location of the study area and the spatial distribution of field observation sites.

**Figure 1.** Locations of the study area and the field observation sites.

A total of six field campaigns were conducted in 2015 to collect data for algorithm development. During the field campaigns, LAI, basic soil available nutrient (SAN) contents, yield, and growth period were observed. For LAI, three sequential field campaigns were conducted on 29 June, 30 July and 25 August. The number of quadrats involved in these campaigns were 36, 28, and 29 respectively. The isometric sampling method was used to establish these quadrats. The distance of each quadrat is more than 100 m and the locations for 29 June are shown in Figure 1. The area of each quadrat was 4 m × 6 m. The LAI-2000 Plant Canopy Analyzer (LI-COR) [15,43] was used to measure LAI of each quadrat. Using an optical sensor, the LAI-2000 can measure the effective LAI, under the assumption of random leaf spatial distribution. Besides LAI, the instrument can also calculate the mean foliage inclination and fraction of the sky visible from beneath the canopy. In this study, a one-up-seven-down scheme was used to measure the LAI, i.e., in each quadrat, we obtained one measurement of sky light above the canopy and then seven measurements of diffuse light below the canopy. The final LAI was calculated using an LAI-2000 analyzer. Visually recognizable brown leaves in the target quadrats were eliminated before measurements so that in-situ LAI measurements only accounts for green leaves and thus can be comparable with LAI from optical RS data.

Another field campaign was conducted to collect basic SAN contents at the same locations and with the same quadrat size as the LAI campaigns, on 10 May 2015 before the application of basic fertilizer. In each quadrat, three sampling points along the diagonal were located, from which soil samples were collected. The collected samples were used to measure the available nitrogen (N), phosphorus (P), and potassium (K) contents in the lab. The mean value of the three sampling points was calculated and used as the basic SAN content of each quadrat. For crop yield, we collected the total grain weight of the plot after harvesting. The total grain weight divided by experimental plot area is used as the measured yield. The percentage impurity and water content were also recorded, and the final yield was calculated by removing the impurity and adjusting for the water content at 25%.

The key growth stages were also observed and detailed information on LAI, SAN, yield and growth stage experiments, including the sampling strategies, soil sampling procedures, and testing methods, can be found in [44,45].

### *2.2. Remote Sensing Data Acquired by the Unmanned Aerial System (UAS)*

The RS data used in this study were acquired by a UAS. The UAS consists of three parts: an 8-rotor UAV, an MCA (Multispectral Camera Array) system, and a ground system, as shown in Figure 2. The MCA camera acquires five-band image with a resolution of 1280 × 1024 (1.3 M) pixels. The central wavelengths of the five bands are 490 nm (blue), 550 nm (green), 680 nm (red), 720 nm (red edge) and 800 nm (near-infrared). The flight height was set to 100 m above the ground level, and the overlap for the flight lines was 60% longitudinally and 40% laterally.

**Figure 2.** The components of the UAV platform (**a**) and geometric correction accuracy (**b**).

The image pre-processing includes reflectance calculation, image stitching, and geometric correction. Among the 6 MCA channels, one of them was connected to an electronic component (an e-ILS sensor) to receive incoming radiation. Using the incident solar radiation, the reflectance for the other five channels was obtained, thus the reflectance can be called at-sensor-reflectance. Because the UAV flight was conducted under clear weather condition and the fight height is 100 m, the at-sensor-reflectance can be used as the reflectance of crop canopy without the need for atmospheric correction in this study. Two software, the Agisoft PhotoScan and PIE-UAV (Pixel Information Expert), were used to generate the ALS-similar point clouds and to stitch the images using information on the flight height, camera attitude, and GPS. For geometric correction, 18 ground control points were collected within the experimental field using a hand-held Trimble GeoXH differential GPS with a mean estimated error of 0.10 m. Twelve points were used to make the geometric correction and six points were used to calculate the mean deviation error (MDE) of the corrected UAV images. The results show that the MDE was less than two pixels (0.1 m) for the June and July images and three pixels (0.15 m) for the August image. After reflectance calculation, image stitching and geometric correction, six images with a spatial resolution of 0.054 m × 0.054 m were obtained.

### *2.3. Estimation of LAI*

The quality issues of RS data will cause failures of common RS-based empirical methods and physical models for continuous LAI estimations for the entire growth season. Hence, crop models were considered in this study to avoid the use of low quality RS data. By integrating the time-series RS data acquired by the UAS and WOFOST crop growth model, the LAI was estimated using combined methods and the processes of LAI estimation for the entire growth is shown in Figure 3.

**Figure 3.** Processes of LAI estimation for the entire growth season using combined methods.

In Figure 3, the whole growth season was divided into three stages: the early stage, the mid stage, and the late stage. The phase from the emergence to the end of the elongation stage was considered as the early stage, and the beginning of the late stage is the tasseling period in this study. The mid stage was defined as the phase between the early and the late stage. In general, the early stage includes the period with high-quality RS time series, the mid stage can also be considered as the rainy season in this study before leaf senescence, and the late stage are characterized by reflectance saturation and leaf senescence. Based on field observations, we divided the specific time spans for the early stage (from the beginning of the growing season to 14 July) and late stage (from 12 August to the end of the growth season). The 14 July and 12 August dates were determined by calculating the middle dates of the field campaigns (30 June, 29 July and 25 August). Three methods, including the VI-based method, the assimilation method, and the hybrid method integrating crop model and the assimilation method, were applied to estimate LAI for the three stages, respectively. The details of these methods are in Sections 2.3.1–2.3.3.

### 2.3.1. VI-Based Method

Our previous study [44] showed that the empirical model can provide LAI estimates with higher accuracy than that of the physically based model (e.g., via PROSAIL simulation). In this study, the relationship between different VIs and ground LAI were analyzed, and the VI with the highest accuracy was selected to build a linear empirical model to estimate LAI.

To ensure the accuracy of LAI estimation, five VIs were calculated from the UAV data. Besides the commonly used *NDVI* and ratio vegetation index (*RVI*) [8], *OSAVI* was also selected to reduce the effect of soil at low LAI, and *EVI2* and modified triangular vegetation index (*MTVI2*) [9] to lower the influence of reflectance saturation. The selected VIs were calculated using the following equations:

$$NDVI = \frac{NIR - RED}{NIR + RED} \tag{1}$$

$$RVI = \frac{NIR}{RED} \tag{2}$$

$$OSAVI = \frac{1.16 \times (NIR - RED)}{NIR + RED + 0.16} \tag{3}$$

$$EVI2 = \frac{2.5 \times (NIR - RED)}{NIR + 2.4 \times RED + 1} \tag{4}$$

$$MTVI2 = \frac{1.5 \times \{1.2 \ast (NIR - GREEN) - 2.5 \times (RED - GREEN)\}}{\sqrt{\left(2 \times NIR + 1\right)^2 - \left(6 \ast NIR - 5 \times \sqrt{RED}\right) - 0.5}}\tag{5}$$

where *NIR*, *RED* and *GREEN* denote the reflectance of the near-infrared band, red band and green band, respectively. Based on the calculated VIs and ground LAI, linear functions can be built. The VI yielding the highest accuracy of LAI estimation was selected as the optimal method for the early-stage LAI estimation.

### 2.3.2. Data Assimilation Method

As shown in Figure 3, the WOFOST model and EnKF method was used to apply RS assimilation for mid-stage LAI simulation. As a primary member of the Wageningen crop models [46] and a core component of the Crop Growth Monitoring System (CGMS) [47], the WOFOST model can simulate daily crop physiological and ecological processes. There were two reasons to select the WOFOST model. Firstly, it can predict daily LAI by simulating CO2 assimilation, respiration, leaf growth and dry matter formation. Secondly, following model modifications referred in [44], WOFOST can simulate LAI under nutrient-limited conditions. Compared with the water-limited LAI, the accuracy of estimated nutrient-limited LAI can be improved by eliminating the influence of SAN on crop growth.

Before WOFOST can be used for LAI simulation, its main parameters must be calibrated. In this study, three methods were used to acquire the input parameters: the SAN estimation method, field campaigns, and the FSEOPT optimization software. The method proposed in [44] was used to estimate the SAN content. Field data was also used to calibrate crop and soil water parameters. Details of the field campaign method can be found in [45]. However, there are still some core parameters cannot be calibrated using the above two methods. In this study, the FSEOPT optimization software [48] was used to calibrate the uncalibrated parameters based on field LAI observations. We selected LAI because it is the research subject of this study. The mean field LAI value of the experimental plot was calculated as the output variable. Then the calculated LAI was used to conduct optimization in the FSEOPT software. The optimization was conducted once using LAI data collected on each date of 30 June, 29 July and 25 August, respectively. Three values for each parameter were acquired after three-time FSEOPT performances and the mean value was calculated as the final calibrated parameter. The calibrated WOFOST model was used to simulate daily crop growth and output LAI results, which is necessary for the assimilation method.

In this study, the EnKF method [37] was adopted to assimilate the LAI estimated from the VI-based method into the WOFOST model to correct for daily simulated LAI results, and LAI values after data assimilation were taken as the output results for the mid stage. The EnKF method, which is based on the Monte Carlo simulation, performs a model forecast in which the state variables are propagated forward in time based on the modeled dynamics and updated using probability distribution and available observations [49]. EnKF is a major assimilation method that can be easily applied in the WOFOST model [50]. In the recursive algorithm of the Kalman Filter, the assimilation process is divided into two steps [41]: forecast and update. In the forecast step, the covariance matrix (*Ac*) of the state variables is calculated. If ensemble of the state variables is defined as *At*,

$$A\_l = (\mathbf{x}\_1, \mathbf{x}\_2, \mathbf{x}\_3, \dots, \mathbf{x}\_N) \tag{6}$$

then *Ac* can be calculated as follow:

$$A\_{\mathcal{E}} = \left(A\_{\mathcal{E}} - \overline{A}\right) \* \left(A\_{\mathcal{E}} - \overline{A}\right)^{T} / \left(N - 1\right)^{T}$$

where *A* is the mean value of *At*, *T* represents the transpose of a vector, *N* is the element number in *At*.

In the update step, the state variables were updated using the observation (RS based LAI). The simulated ensemble of state variables was calculated by a nonlinear equation:

$$\mathbf{x}\_{i}(t) = M[\mathbf{x}\_{i}(t-1)]\tag{7}$$

where *t* is WOFOST simulation step (Day of year), *M* is the forecast equation.

If the observed ensemble (RS based LAI) was defined as *Dt*, the covariance matrix (*Dc*) of the observed ensemble can also be calculated by Equation (6). Then the standard analysis equation can be built to calculate the updated WOFOST simulated ensemble (*Aa*):

$$A\_a = A\_f + \mathcal{K}(D\_t - HA) \tag{8}$$

where *Aa* is the optimal estimated ensemble, *Af* is the forecasted ensemble calculated using Equation (7), *K* is the Kalman gain matrix to weigh the difference between observation (RS-based LAI) and prior simulation of the model's state (WOFOST-based LAI), which was calculated by the following equation:

$$K = A\_{\mathbb{C}} H^T \left( H A\_{\mathbb{C}} H^T + D\_{\mathbb{C}} \right)^{-1} \tag{9}$$

where *H* is the parameter of observation operator. The *Aa* was used to replace *At* to realize the assimilation process.

Filter divergence was observed in our study. It represents a tendency of the standard EnKF to reject observations in favor of ensemble forecasting in subsequent stages, leading to incremental deviations of LAI from actual measurements. To address this issue, an expansion parameter (*E*) was used:

$$E = R \ast N \ast \frac{\sigma\_1^2}{D \ast \sigma\_2^2} \tag{10}$$

where *R* is a random number that is less than 1, *D* is the total number of days for the assimilation stage, *N* is the number of days (from 1 to *D*) for the current WOFOST's simulation step, σ<sup>2</sup> <sup>1</sup> is the variance of RS based LAI of *N*, and σ<sup>2</sup> <sup>2</sup> is the variance of WOFOST-based LAI of *<sup>N</sup>*. When <sup>σ</sup><sup>2</sup> 1/σ<sup>2</sup> <sup>2</sup> is larger than 4 and *E* is equal or greater than 1, E will be used to enlarge *K* to eliminate filter divergence in this study.

### 2.3.3. The Hybrid Method

As mentioned in the introduction, both VI-based and assimilation methods cannot address the quality issues of RS data at the late-growth stage caused by senescing leaves and reflectance saturation. Hence, we designed a new strategy (the hybrid method) for the late-stage LAI estimation. In the hybrid method, the WOFOST model with and without UAV data assimilation were combined to improve the late-stage LAI estimation. The UAV data were assimilated into the WOFOST model to generate daily crop growth from the beginning of the growth season until the begin of late stage. Then the data assimilation was halted, and WOFOST was only used to simulate LAI until the end of the growth season. The processes of LAI estimation for the late stage is shown in Figure 4.

**Figure 4.** Processes of LAI estimation for the late stage using hybrid method (DVS: the development stage of the crop, DVS = 1 indicates the beginning of tasseling period for maize in this study).

Figure 4 indicates that the hybrid method can be considered as a specific crop model method in which the WOFOST model without using RS data was run from the end of the mid stage (DVS = 1). To enable the simulation of WOFOST, the parameters of crop growth status with DVS = 1 and RS-based SAN contents were required as initial inputs. Then, the WOFOST model was self-operated and daily LAI values were output as the late-stage results. This method was named as the hybrid method to distinguish it from the WOFOST model used in the assimilation method.

Finally, three methods were combined to generate LAI time series for the entire growth season (shown in Figure 3). Detailed LAI estimation processes are described as follows:


### *2.4. Evaluating the Accuracies of LAI Estimations*

The performance of the three LAI estimation methods was evaluated using field LAI data. The correlation coefficient (*R*), coefficient of determination (*R*2) and *RMSE* were selected as the indices to analyze the relationship between the estimated and field-measured LAI values. The *R*<sup>2</sup> and *RMSE* were calculated as follows:

$$L\_{\text{mean}} = \frac{1}{n} \sum\_{k=1}^{n} L\_{\text{obs},k} \tag{11}$$

$$L\_f = a \times L + b \tag{12}$$

$$R^2 = \frac{\sum\_{k=1}^{n} \left(L\_{f,k} - L\_{\text{mean}}\right)^2}{\sum\_{k=1}^{n} \left(L\_{\text{obs},k} - L\_{\text{mean}}\right)^2} \tag{13}$$

$$RMSE = \sqrt{\frac{\sum\_{k=1}^{n} \left(L\_k - L\_{\text{obs},k}\right)^2}{n}} \tag{14}$$

where *Lobs* is the measured LAI values during the first three field campaigns introduced above, *n* is the number of quadrats, *Lmean* is the mean value of *Lobs*, *L* is the RS-based VIs (vegetation index method) or simulated LAI (WOFOST model and assimilation method), *Lf* is the linear fitting formula between *L* (independent variable) and *Lobs* (dependent variable), *a* and *b* are the coefficients.

The *R*<sup>2</sup> and *RMSE* were both used to evaluate the performance of LAI estimation methods. Generally, the LAI estimation method with a higher *R*<sup>2</sup> corresponding to a lower *RMSE* was selected as the ideal method combinations for time-series LAI estimation.

Furthermore, the variable coefficient (*CV*) was selected to examine the spatial heterogeneity of late-stage LAI estimations. The *CV* was calculated as follows:

$$CV = \frac{SD}{Mean} \ast 100\% \tag{15}$$

where *SD* is the standard deviation and *Mean* is the mean value of the research data (LAI and vegetation indices in this study).

### **3. Results and Discussion**

### *3.1. LAI Estimation Using the VI-Based Method*

Using the linear regression model, the R values between RS-based VIs and field LAI were derived. To show the significant correlation level, a significance test was also conducted by calculating the *p*-value. *p*-value < 0.01 indicates highly significant correlation level and *p*-value < 0.05 for significant correlation level. The R values and significance levels are listed in Table 1. The results show that RVI provided the best LAI estimations with the highest R among all the indices on 30 June (the early-growth stage) and 29 July (the mid-growth stage). EVI2 provided LAI estimate with the highest R on 25 August (the late-growth stage). The R values on 30 June (RVI) and 29 July (RVI) reach highly significant level, reaches significant level on 25 August (EVI2).


**Table 1.** The R values between vegetation indices and field LAI.

\*\* indicates highly significant correlation, \* indicates significant correlation.

Subsequently, RVI and EVI2 were selected to build the RS-based statistical models for LAI estimation. The R<sup>2</sup> and RMSE values of the field LAI and estimated LAI were also calculated. The statistical models and the accuracy analysis results are shown in Figure 5.

**Figure 5.** Established empirical vegetation index models (**A**) 30 June, (**B**) 29 July, (**C**) 25 August.

From Figure 5, we can observe that the estimation accuracy on 25 August was considerably lower than that on 30 June and 29 July. The VI saturation that occurs during the late-growth season is an important reason for the significantly reduced accuracy on 25 August. In Table 2, the CV indices of the five VIs on 30 June, 29 July and 25 August were calculated to represent the levels of VI saturation on different dates. The results (listed in Table 2) show that the CV values on 29 July and 25 August were lower than that on 30 June, which means that there is a saturation trend toward the mid and late season.

**Table 2.** CV values of the vegetation indices for the experimental field (%).


To compare the levels of VI saturation at different stages, LAI estimations using the VI-based method were shown in Figure 6. The figure reveals that the relative dynamic range (the dynamic range ratio of simulated and observed LAI) was reduced on 25 August respective to 29 July and 30 June. This result suggests that although the saturation has already appeared on 30 July, it is more obvious on 25 August. Hence, the saturation is an important reason for the poor performance of VI-based empirical method for LAI estimation at the late stage.

**Figure 6.** The accuracy of LAI estimations using the vegetation index method (**A**) 30 June, (**B**) 29 July, (**C**) 25 August.

Besides VI saturation, leaf senesce may also have contributed to the lower estimation accuracy on 25 August. At the late growth stage of spring maize, the primary receiver of dry matter from photosynthesis will be shifted to storge organs. The WOFOST' parameters FOTB (fraction of above ground dry matter to storage organs) and FLTB (fraction of above ground dry matter to leaves) can be used to demonstrate this phenomenon. The FOTB on 25 August is higher than that on 29 July, while FLTB shows the opposite pattern (25 August: FOTB = 0.69, FLTB = 0.12; 29 July FOTB = 0.20, FLTB = 0.18). Less dry matter supply will decrease leaf bioactivity and thus generate more inactive

leaves. Chemical changes in inactive leaves can hardly be detected by optical RS data. In general, saturation and senescence jointly decreased the correlation between LAI and RS-based VIs.

From Figure 6A, we can find that the accuracy of VI-based method for the early-stage LAI estimation is higher than the assimilation method. Hence, the statistical model shown in Figure 5A was selected as the ideal method to provide early-stage LAI results (shown in Figure 3). Furthermore, the RS-based LAI was also required by the assimilation method, and the three statistical models shown in Figure 5 were used to generate the RS-based LAI for the entire growth season. Then the inverse distance weighted interpolation method was used to generate daily RS-based LAI for data assimilation.

Although the VI-based statistical models can provide higher accuracy LAI estimations, these models are commonly used for specific sensors and sampling conditions because the lack of physical basis. Thus, any changes in location, crop type and application years can cause failures in the original model application [51,52]. Because radiative transfer model [53] is the core algorithm of the physical models, they reflect the transfer and interaction of radiation inside the canopy; hence, the physical model methods can overcome the limitations of sensors, geographical locations, and application times. A physical model [26] should be considered before applying the LAI estimation method to a larger area with different crops for different years. Meanwhile, detailed radiation transfer and interaction simulations in physical models require more input parameters; consequently, more field experiments should be conducted before applying physical models. Because there is no significant change in environmental conditions, growth stages, and remote sensing data calibration, the VI-based statistical model was selected and re-calibration of the relationship between RS-based VI and field LAI each time is not necessary.

### *3.2. LAI Simulation Using the Assimilation Method*

### 3.2.1. WOFOST Model Calibration

The results of the WOFOST model calibration are presented in this section. The input parameters of the WOFOST model include meteorological, soil, and crop parameters. In addition to the daily meteorological and fertilization data, which were provided by Hongxing Farm, other parameters also need to be calibrated in this study. Based on a sensitivity analysis (discussed in previous work [44]), 18 parameters sensitive to LAI were selected as core parameters to be calibrated using the three previously described calibration methods in this study. The values of the calibrated parameters and their calibration methods are listed in Table 3.


**Table 3.** Core parameter calibration results of the WOFOST model.


**Table 3.** *Cont.*

The accuracy of simulated crop growth by the calibrated WOFOST model was firstly evaluated using emergence time, anthesis time, maturity time and yield. These parameters were simulated for the experimental plot and compared with field observations, and the results are listed in Table 4. Comparing to observations, the crop growth, as indicated by those parameters in Table 4, simulated by the calibrated model is better than the original model.


**Table 4.** Performance of calibrated and un-calibrated WOFOST model.

### 3.2.2. LAI Simulation Using the WOFOST Model

We applied the calibrated WOFOST model to simulate LAI which is required for the assimilation process, and the model performances were also evaluated using field LAI. Results (shown in Figure 7) show that the calibrated model can simulate LAI on 30 June with high accuracy, but yielded LAI estimates with much lower accuracy on 25 July and 25 August. Calibration errors are the potential reason for the lower accuracy at the mid- and late-stages. Some core parameters of the WOFOST model can only be acquired as constant values at the site level; but in reality they may change with time. Thus, errors cannot be completely avoided using either of the three calibration methods. These errors will propagate and accumulate, hence reduce the accuracy of LAI simulation over time. The higher simulation accuracy on 30 June than the other two dates also indicates that reducing the simulation

time can weaken the calibration error propagation and accumulation and help the WOFOST model provide better LAI simulation.

**Figure 7.** The LAI estimation accuracies for the calibrated WOFOST model (**A**) 30 June, (**B**) 29 July, (**C**) 25 August.

Although the single WOFOST model cannot provide LAI estimates with higher accuracy than the VI-based method and the assimilation method for all the three growth stages (shown in Figures 6–8), WOFOST was still required by the assimilation method and the hybrid method (shown in Figure 3). Errors in the WOFOST-based growth simulation will be reflected in the performance of the assimilation and the hybrid methods. Hence, improve the performance of WOFOST is beneficial for the estimation of LAI time series in this study. Because LAI is simulated and analyzed at the pixel or sampling point level, the parameters with high spatial heterogeneity should also be calibrated at the pixel level to ensure the LAI simulation accuracy. Among input parameters, soil parameters, including soil water and soil available nutrients, are the main variables that change within the experimental field in this study. Considering that soil water contents can be simulated by WOFOST model using daily precipitation data, the SAN values at the pixel level should be acquired before conducting WOFOST-based LAI simulation.

**Figure 8.** The LAI estimation accuracies using assimilation method (**A**) 30 June, (**B**) 29 July, (**C**) 25 August.

The SAN estimation method proposed in [44] was used to provide available N, P and K based on RS data and the WOFOST model. From the accuracy analysis results, we can find that the estimation accuracy for K (R<sup>2</sup> = 0.15, RMSE = 23.56 mg/kg, mean = 168.26 mg/kg) was significantly lower than that of N (R2 = 0.48, RMSE = 18.45 mg/kg, mean value = 296.78 mg/kg) and P (R2 = 0.37, RMSE = 7.05 mg/kg, mean value = 31.63 mg/kg). Optimizing the K estimation would be useful to further improve the LAI simulations. The instability of K and the lower effect of potassium ions in comparison with the other two ions on crop growth, especially on leaf growth, are the possible causes for the lower estimation accuracy using the proposed method. Adjusting the SAN estimation method to consider these factors will be the focus of future studies to further improve the K estimation accuracy.

### 3.2.3. LAI Simulation Using the EnKF Assimilation Method

Using the method presented in Section 3.1, LAI was estimated using time-series UAV data first, and then assimilated into the WOFOST model through the EnKF method. The calibrated model with

data assimilation was used to generate LAI for 30 June, 29 July and 25 August. The R<sup>2</sup> and RMSE were calculated for the three LAI simulations using field LAI. The results (listed in Figure 8) show that the UAV data improved the WOFOST model's LAI simulation performance for 30 June and 29 July. However, the accuracy for 25 August did not improve. Compared with the VI-based empirical method, the assimilation method provides higher simulation accuracy for 29 July and similar accuracies for 30 June and 25 August. From Figure 8A, we see that there are two samples with higher simulated LAI (around 3). Comparing Figures 6B and 8B, we can also find that the assimilation can provide mid-stage LAI estimation with higher accuracy, thus, this method was selected as the ideal method for this stage (shown in Figure 3).

To show the improvements that both calibration and assimilation can bring in LAI simulation, the LAI trajectories were gathered from three methods including the calibrated WOFOST model, the assimilation method, and the VI-based empirical method. We selected the mean LAI value of the experimental plot and LAI's variation range (10–90%) as the indexes and generate three trajectories (shown in Figure 9). The results show that the assimilation can correct WOFOST's LAI simulation to certain extent. Because we didn't conduct UAV flight after 25 August, the trajectories end at that time (Day of year; DOY = 137).

**Figure 9.** The seasonal LAI estimated using three methods: the calibrated WOFOST model, the assimilation method, and the VI-based empirical using RS data.

### *3.3. LAI Estimations Using the Hybrid Method*

From above analyses, we can find that the late-stage LAI estimation accuracies of the VI-based, the WOFOST model and the assimilation methods at the late stage are much lower than those at the early and mid-stages. Model calibration, reflectance and VI saturation, and canopy changes associated with leaf aging, jointly contributed to the reduced performance to certain extents. Hence, the hybrid method was developed to improve the accuracy of AI estimation at the late stage. Using this method, LAI values on 25 August were simulated, and the accuracy was evaluated using field LAI. The analysis result, which is shown in Figure 10, shows that combining the WOFOST model with RS data assimilation provides better LAI simulations than the VI-based empirical method (Figure 5C) and the RS assimilation method (Figure 8C) alone because the problems of leaf aging and error accumulation can be avoided.

**Figure 10.** The LAI estimation accuracies for 25 August using the hybrid method.

The time span delimitation of different crop growth stages is the main limitations for the application of the hybrid method. The time window used for the three crop growth stages in this study influences the application times of the VI-based empirical method, assimilation and crop model without data assimilation. For instance, the beginning of the late-growth stage is the period when reflectance saturation and leaf senescence both appeared; hence, the WOFOST model was used to replace the assimilation method during that period. However, we calculated the start date from the field campaign dates, which may be inaccurate because the goal of the field campaign did not involve measuring reflectance saturation and leaf senescence. More field campaigns should be conducted to acquire the best application time for the three methods.

### **4. Conclusions**

In this study, we applied three methods to conduct time-series LAI estimations for spring maize. Using field-measured LAI over an experimental field, the performances of these methods were investigated. Results show that the VI-based method can be used to provide LAI estimation with high accuracy before the appearance of reflectance saturation and leaf senescence. With the application of the UAV data, the accuracy of the assimilation method was improved, and this method was an ideal choice for the mid-stage LAI simulation. The hybrid method is designed to address reflectance saturation and leaf senescence issues of the VI-based method and error accumulation of the WOFOST model in the late stage, and its LAI estimation accuracy of at the late stage is higher than the other two methods. In general, the accuracy analysis results show that the combined methods, which applied the VI-based method for the early stage, the assimilation method for the mid stage, and the hybrid method for the late stage, can provide highly accurate continuous LAI estimations for the entire growth season. Meanwhile, it's also worth noting that the experiment was only conducted for one year due to the high cost and logistic requirements. The field data gathered in the experiment are insufficient to well evaluated the robustness of the applied methods. A multi-year experiments with additional crop types will be the focus of future studies to further analyze the performance of the combined methods for time-series LAI generation.

**Author Contributions:** J.M., J.S., Y.Q. and Z.C. conceived and designed the experiments; Z.C. applied the combined methods, performed the experiments and analyzed the data; J.L., J.H., B.Q., Q.J., T.D. and L.Y. contributed proposed the comments on method optimization and data analysis. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by National Natural Science Foundation of China (grant number: 41871261), Fujian Provincial Department of Education science and technology project (grant number: JAT190083), and the GF6 Project (grant number: 09-Y20A05-9001-17/18).

**Acknowledgments:** We are grateful to DLO Winand Staring Centre, Wageningen, for providing the original WOFOST code and necessary parameter documentations, to Hongxing Farm for providing their database to calibrate the WOFOST parameters and offering us the experimental field and their laboratory for field campaigns, and to the China Scholarship Council (CSC) for providing a scholarship for Zhiqiang Cheng to pursue studies at Agriculture and Agri-Food Canada as a joint Ph.D. student.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

*Article*

### **Automatic Seamline Determination for Urban Image Mosaicking Based on Road Probability Map from the D-LinkNet Neural Network**

### **Shenggu Yuan 1, Ke Yang 2,\*, Xin Li <sup>3</sup> and Hongyue Cai <sup>3</sup>**


Received: 10 February 2020; Accepted: 24 March 2020; Published: 26 March 2020

**Abstract:** Image mosaicking which is a process of constructing multiple orthoimages into a single seamless composite orthoimage, is one of the key steps for the production of large-scale digital orthophoto maps (DOM). Seamline determination is one of the most difficult technologies in the automatic mosaicking of orthoimages. The seamlines that follow the centerlines of roads where no significant differences exist are beneficial to improve the quality of image mosaicking. Based on this idea, this paper proposes a novel method of seamline determination based on road probability map from the D-LinkNet neural network for urban image mosaicking. This method optimizes the seamlines at both the semantic and pixel level as follows. First, the road probability map is obtained with the D-LinkNet neural network and related post processing. Second, the preferred road areas (*PRAs*) are determined by binarizing the road probability map of the overlapping area in the left and right image. The PRAs are the priority areas in which the seamlines cross. Finally, the final seamlines are determined by Dijkstra's shortest path algorithm implemented with binary min-heap at the pixel level. The experimental results of three group data sets show the advantages of the proposed method. Compared with two previous methods, the seamlines obtained by the proposed method pass through the less obvious objects and mainly follow the roads. In terms of the computational efficiency, the proposed method also has a high efficiency.

**Keywords:** mosaicking; urban image; seamline determination; deep learning; D-LinkNet

### **1. Introduction**

Orthoimages have increasingly become a popular visualization product and planning instrument for integrating the rich information content of images with the geometric properties of maps (ground projection) and can be easily combined with additional information from geographic information systems (GIS) to create an orthoimage map [1]. However, with the development of technology, the orthoimage spatial resolution becomes higher, and the coverage area of an individual orthoimage is typically very small Thus, image mosaicking is a necessary process of constructing multiple images into a large-scale and single seamless composite image. This process has been applied in a wide variety of applications such as environmental monitoring, agricultural monitoring, and disaster management [2,3]. Orthoimages are typically orthorectified by the Digital Terrain Model (DTM) of the same geographical area. The quality of DTM directly affects the accuracy of orthorectification. Objects not contained in the DTM cannot be orthorectified correctly. Those objects would appear at different locations in the overlapping area and cause visual discontinuities in image mosaicking. An

ideal seamline should avoid such objects [1,4–8]. A seamline is the line along which overlapping areas will be mosaicked. Each pixel in the final mosaicking result is represented entirely by only one orthoimage based on which side of the seamline it lies on. Seamline is also helpful when overlapping areas have significant differences in features. When mosaicking orthoimages, seamline determination is one of the most difficult technologies for compositing a single seamless orthoimages. The purpose of seamline determination is to find the seamlines with the minimal intensity and gradient differences in the overlapping area. In this paper, our work focuses on automatic seamline determination for urban image mosaicking.

In order to minimize the transition of the final mosaic image, the ideal seamline should avoid crossing obvious objects as much as possible and go along the objects which have small relief displacement. Differential expression is essential for seamline determination, which is a measure of the difference between left and right image overlap areas [1,7]. The method based on differential expression is the basic method of seamline optimization. The method first measures the difference between the overlap region images to form a difference matrix, and then uses the path search algorithm to obtain the final optimized seamline. According to the differential expression algorithm and the path search algorithm, the recently proposed seamline determination algorithms are as follows.

Milgram [9] defined the "best" seamline point for each line of the overlapping area that minimizes the sum of the gray differences between the left and right images. Afek and Brand [10] integrated global feature matching and local transformation into seamline determination. Soille [11] used the mathematical morphology and marker-controlled segmentation paradigm to determinate the seamlines. The difference (geometric and radiometric discontinuities) can be minimized if the seamlines go along salient image structures.

Kerschner [1] proposed a "two snake" method for seamline determination. The main idea is to design a double snake model, through mutual attraction of the two snake lines, and finally form a snake line to obtain an optimized seamline. The energy of the double snakes is defined based on similarity. The seamlines go along the region of maximum similarity. The criteria for regions of similarity are color similarity (hue and intensity) and texture similarity (orientation and magnitude of gradients). Wang et al. [12] proposed a seamline determination in image mosaicking using improved snakes. The integrated snake model and Bresenham algorithm was presented, which the Bresenham algorithm was used to calculation the photometric. This solves the local optimum problem that exists in the snake model to some extent, but not completely.

Ma and Sun [13] proposed a seamlines optimization for image mosaicking with airborne light detection and ranging (LiDAR). According to the raw laser scanning dataset, the high ground objects of the overlap area were identified as the obstacle. Then, the A\* algorithm was used to determine the final seamlines, and the seamlines were kept away from these obstacles in the registered images. Wang and Wan [14–16] presented a seamline determination with the aid of vector roads for the first time. In this approach, firstly, with the help of the vector roads and their widths, the seamlines will go along the centerlines of roads with a large width as much as possible and avoid crossing the obvious objects. Finally, the shortest path algorithm is applied to determine the final seamline. Chen et al. [17] first used the Digital Surface Model (DSM) and Digital Terrain Model (DTM) to derive an Orthoimage Elevation Synchronous Model (OESM) that accurately reflected the pixel of each digital orthophoto image, and then obtained the final optimized seamline using the Dijkstra algorithm. Wang et al. [2] used vector building maps to determine the seamlines, which guaranteed the seamlines avoiding the crossing of buildings as much as possible. Different from the method of tracking vector roads, the seamlines determined by this method went along the middle line between buildings in order to avoid crossing the obvious objects, especially for high-rise buildings.

Using the normalized difference vegetation index (NDVI) and morphological building index (MBI), Pan et al. [18] introduced ground object classification into the seamline optimization method. In this approach, based on ground object classes, three types of areas, that is, obstacle areas, preferred areas, and general areas, are further formed. Then, each type of region is assigned a different weight to optimize obtained pixel-size costs. Finally, Dijkstra's algorithm is carried out to search the shortest path as the final seamlines based on previously determined pixel-size costs in overlapping areas.

Chon et al. [19] first sought the maximum difference by minimizing the maximum, and then used Dijkstra's algorithm to determine the final seamlines. The method, which is based on minimizing the maximum difference, measures the difference by a local region. It first calculates the Normalized Cross Correlation (*NCC*) of the central pixel in this window, and makes the contrast between the normalized correlation coefficients by exponential stretching of further expansion. Then, using the strategy of minimizing the maximum difference on the basis of the difference matrix, the seamline is prohibited from crossing the region, and finally the final optimized seamline is obtained by the Dijkstra algorithm.

Pan et al. [6] used image segmentation to determine seamlines for orthoimage mosaicking in an urban area for the first time. This method uses segmentation to improve seamline determination. Firstly, the preferred regions were selected according to the spans of objects segmented by the mean shift algorithm. Then, Dijkstra's shortest path algorithm was adopted to determinate the final seamline. Since this method was proposed, several object-based methods have been used to optimize seamlines. After that, Pan et al. [20] introduced the region change rate (RCR) into seamline optimization for orthoimage mosaicking. The change rate of the regions acquired by the mean shift segmentation algorithm were defined by the percentage of the changed pixels. This method determined the seamlines at object-level and pixel-level. Wang et al. [7] adopted watershed segmentation for seamline optimization at both the object and pixel level. Using normalized cross correlation, the obvious objects, such as buildings, were excluded from the preferred objects areas at the object level. Dijkstra's algorithm found the final seamlines in the preferred objects areas at the pixel level.

Li et al. [21,22] first adopted the graph cuts energy minimization framework to find the optimized seamlines. The image color, gradient magnitude and texture were combined in the smooth energy functions in the graph cuts energy minimization framework. The determined seamlines passed through the areas of a smooth texture, such as roads, woodlands, and green lands. Li et al. [23] proposed an automatic seamline optimization based on graph cuts in UVA image mosaicking.

Yuan et al. [24] proposed a seamline optimization based on the disparity image by the semi-global matching (SGM) algorithm. After obtaining the disparity image, the mathematical morphology method was employed to deal with the noises and small holes of the disparity image in order to determine the non-ground area. Finally, an improved greedy snake algorithm was adopted for the final seamlines. Similar to that algorithm, Pang et al. [25] introduced dense matching into seamline determination. Firstly, the SGM was used to estimate the disparity of each pixel. Next, the obstacle and non-obstacle areas were determined by a predefined threshold. Finally, Dijkstra's algorithm was adopted to optimize the final seamlines in avoiding crossing the obstacle area as much as possible.

Based on the integrated deep convolutional neural network (CNN) and graph cuts energy minimization framework, Li [26] proposed a novel algorithm to optimize seamlines for image mosaicking. Different from the previous method [22], this method defined similarity energy terms of the graph cut using the semantic classification classified by the CNN instead of using the color, gradient, or texture.

In our paper, we propose a novel method of seamline determination based on a road probability map which is extracted by the D-LinkNet neural network for urban image mosaicking. This method optimizes the seamlines at both the semantic and pixel level. Firstly, the D-LinkNet neural network is adopted to obtain the road probability map of the overlapping area in the left and right image respectively. Secondly, the preferred road areas (*PRAs*) are determined by binarizing the road probability map of the overlapping area both in the left and right image. The *PRAs* are priority areas which the seamlines cross. Finally, the final seamlines are determined by Dijkstra's shortest path algorithm implemented with binary min-heap at pixel level.

The remainder of this paper is organized as follows: Section 2 describes the proposed seamline determination for urban image mosaicking based on a road probability map from the D-LinkNet neural network, where Section 2.1 introduces road probability map generation by D-LinkNet. Section 2.2 presents the determination of *PRAs,* and Section 2.3 introduces pixel-level seamline determination; Section 3 describes the experimental results and analysis, where Section 3.1 presents the experimental data and platform and Section 3.2 presents the seamline determination results and analysis. Section 4 draws the conclusions.

### **2. Materials and Methods**

The major difficulty issue of seamline determination is to define the differential expression of the overlapping area more accurately. A cost image is generally adopted to express the difference of the overlap image. Most seamline determination methods use the pixel-by-pixel or local regular subimages to define the differential expression, and it is difficult to measure the difference accurately [6,7]. Object recognition is considered to be helpful for differential expression. If object recognition has been solved perfectly, we can set the areas with the high differential expression highest cost, such as buildings. Then, the seamlines can be guaranteed not to go across stand-alone objects such as buildings. However, object recognition is a complicated problem [6]. With the help of a vector roads network, some approaches have used vector roads to optimize the seamlines, in which the seamline follows the centerlines of roads with a large width as much as possible and avoids crossing the obvious objects. Such seamlines are benefited to maintain the integrity of objects and improve the quality of image mosaics [14–16]. Based on this idea, we utilized a road probability map from the D-LinkNet neural network to optimize the seamline determination.

The process flow for the proposed method is shown in Figure 1. The proposed algorithm optimizes the seamlines at the semantic and pixel level. The D-LinkNet neural network is adopted to achieve the road probability map of the overlapping area both in the left and right image. At the semantic level, the *PRAs* are determined by binarizing the road probability map. In this step, most of the roads will be included in the *PRAs*. The *PRAs* are the priority areas which the seamlines cross. At the pixel level, the Dijkstra's shortest path searching algorithm is adopted to find the final seamlines.

**Figure 1.** The process flow for the proposed method.

### *2.1. Road Probability Map Generation by D-LinkNet*

### 2.1.1. Block Road Probability Map Generation by D-Linknet

Road extraction from high-resolution images is a basic application of remote sensing, which has attracted the attention of both academics and industry for a long time [27,28]. With the development of deep learning and contribution of specialized datasets from the remote sensing community, convolutional neural networks (CNNs) have been broadly used as alternatives to traditional methods for visual recognition tasks in remote sensing, including building detection [29], road segmentation [30–32], and topological map generation [33]. In our research, we focus on the main part of road extraction—road probability map generation—and model it with a convolutional neural network.

D-LinkNet [34] is adopted as a basic CNN model of the proposed method due to its excellent performance in 2018's DeepGlobe road extraction challenge and broad use as the baseline in road segmentation tasks [35]. It uses the typical encoder-decoder architecture inherited from LinkNet and adds the delated convolution part to acquire and ensemble multi-scale features to enlarge the receptive field, which is able to handle a road's properties, such as connectivity, complexity, and long span, to some extent [36]. Specifically, ResNet34 [37] is pretrained on ImageNet [38] and used as the encoder part of D-LinkNet, and several dilated convolution layers with skip connections are placed in the center part to enhance the reception ability. The decoder part uses transposed convolution [39] layers to conduct upsampling, restoring the resolution of the feature map from one that is downsampled to the original one. The architecture of the D-LinkNet is presented in Figure 2.

**Figure 2.** D-LinkNet architecture [34].

In practice, considering the large size of satellite and aerial images, clipping is necessary to generate image patches with a proper and fixed size (e.g., 1024 × 1024), to make sure that D-LinkNet implementation works under a constrained computation ability and the output of separated patches has been integrated into the final result. In addition, a small size of overlap between neighboring patches should be considered and part of the redundant output within overlap areas should be discarded, as misclassification often happens at the border pixels of a patch given that the receptive field is constrained by the edge. As for the training step of deep learning, we utilized transfer learning to accelerate the whole train process, and data augmentation of the DeepGlobe road dataset [35] to promote the network's learning ability.

Some road probability map generation by D-LinkNet is shown in Figure 3. Each size of the sample image is 1024 × 1024 pixels. The lighter the gray value of the pixel value in the image, the higher the probability that it will be recognized as a road.

**Figure 3.** Road probability map generation by D-LinkNet: (**a**), (**c**), (**e**), and (**g**) are original images, (**b**), (**d**), (**f**), and (**h**) are road probability maps generation by D-LinkNet of (**a**), (**c**), (**e**), and (**g**).

### 2.1.2. The Post Processing of Road Probability Map Generation

When processing high-resolution remote sensing images, due to the limitations of memory and other factors, block processing is required to extract the roads using D-Link. If the block extraction results are directly stitched together, this may lead to obvious visible transitions. The stitched result is shown in Figure 4a. There are obvious visible transitions in the stitched result. Figure 4a is the result of directly stitching the image according to the size of 1024 × 1024 pixels. In order to eliminate obvious visible transitions, this paper ensures a certain overlap between adjacent blocks during blocking. The overlap provides a foundation for the subsequent elimination of obvious visible transitions. When blocking, the width of the overlapping area is 511 pixels.

**Figure 4.** The stitched result of the road probability map obtained by different methods: (**a**) the stitched result of directly stitching method, (**b**) the stitched result of our stitching method.

After processing according to the blocking principle to obtain the road extraction result, the post processing can be performed to eliminate obvious visible transitions on both sides of the seamline. In this paper, there are only vertical and horizontal seamlines. This kind of processing is performed on the two sides of the stitching point line by line (or row by row) within the artificially specified width range (the width of the range must be smaller than the width of overlap region). The method used is as follows:

$$\begin{array}{l} \text{OI}^{i}\_{\text{L}} = \text{O}^{i}\_{\text{L}} + (\text{O}^{i}\_{\text{L}} - \text{O}^{i}\_{\text{R}})\text{K} \\ \text{OI}^{i}\_{\text{R}} = \text{O}^{i}\_{\text{R}} + (\text{O}^{i}\_{\text{R}} - \text{O}^{i}\_{\text{L}})\text{K} \\ \text{K} = i/\text{W} \quad 0 \le i \le \text{W} - 1 \end{array} \tag{1}$$

where *O<sup>i</sup> <sup>L</sup>* is the gray value of the pixel in the left (top) road probability map, *Oi <sup>R</sup>* is the gray value of the pixel in the right(bottom) road probability map, *OI<sup>i</sup> <sup>L</sup>* is the processed gray value of the pixel in the left(top) road probability map, and *OI<sup>i</sup> <sup>R</sup>* is the processed gray value of the pixel in the right(bottom) road probability map. *W* is the width of the smooth area, and *K* is the weight [40].

In this process, the gray values of the two images to be stitched are weight averaged pixel by pixel to be used as the gray values after stitching. The weights used vary linearly and inversely within the calculation range. This process can basically eliminate the obvious difference near the stitched line, which is shown in Figure 4b. Compared with Figure 4a, there are no obvious visible transitions in Figure 4b.

### *2.2. Preferred Road Areas Determination*

After the generation of the road probability map in the overlapping area in both the left and right image, the accuracy of detecting preferred road areas is determined by the road probability threshold for binarization. In order to estimate the road probability threshold adaptively, we used the Otsu's method [41,42] to estimate the value of the road probability threshold. Otsu's method is used to perform automatic image thresholding. The algorithm returns a single intensity threshold that separates pixels into two classes: foreground and background. This threshold is determined by minimizing the intra-class intensity variance, or equivalently, by maximizing the inter-class variance. We estimated the road probability threshold in the overlapping in the left and right image respectively. The method used is as follows Equations (2)–(4):

$$I\_L(\mathbf{x}, y) \in PRALs \begin{cases} true & P\_L(\mathbf{x}, y) > = T1 \\ false & otherwise \end{cases} \tag{2}$$

$$I\_R(\mathbf{x}, \mathbf{y}) \in \text{PRARs} \begin{cases} true & P\_R(\mathbf{x}, \mathbf{y}) > = T2 \\ false & otherwise \end{cases} \tag{3}$$

$$I(\mathbf{x}, y) \in \text{PRAs} \begin{cases} \text{true} & I\_L(\mathbf{x}, y) \in \text{PRAs} \text{ and } I\_R(\mathbf{x}, y) \in \text{PRAs} \\ \text{false} & \text{otherwise} \end{cases} \tag{4}$$

where *IL*(*x*, *y*) is the pixel in the road probability map of the overlapping area in the left image. *IR*(*x*, *y*) is the pixel in the road probability map of the overlapping area in the right image; *PL*(*x*, *y*) is the value of *IL*(*x*, *y*); *IR*(*x*, *y*) is the value of *IR*(*x*, *y*); *I*(*x*, *y*) is the pixel of the overlapping area; *PRALs* represents the preferred road areas of the overlapping area in the left image; and *PRARs* represents the preferred road areas of the overlapping area in the right image. *PRAs* represents the preferred road areas. *T*1 and *T*2, which are estimated by Otsu's method, are the road probability threshold of the overlapping area in the left image and right image, adaptively.

### *2.3. Pixel-Level Seamline Determination*

### 2.3.1. Pixel-Level Cost Determination

After the determination of the *PRAs,* pixel-level seamline determination is used to determine the final seamlines. The two intersecting pixels of the image borders are confirmed as the start and end points [19].

Similar to Chon et al. [19], the Normalized Cross Correlation (*NCC*) is adopted to quantify the difference between the overlapping area of two images at the pixel level. Considering the efficiency, the quick *NCC* is calculated with Equation (5) using the 5 × 5 windows. *i* and *j* are coordinates in the image coordinate system. *IL*(*i*, *j*) and *IR*(*i*, *j*) are the gray values of the overlapping area in the left and right image at (*i*, *j*), respectively. The cost value is computed in Equation (6), which has a range of 0–1.0.

$$\text{QNCC}(\mathbf{x}, y) = \frac{\sum\_{i=x-2}^{x+2} \sum\_{j=y-2}^{y+2} I\_L(i, j)I\_R(i, j) - \frac{1}{2\mathfrak{S}} \sum\_{i=x-2}^{x+2} \sum\_{j=y-2}^{y+2} I\_L(i, j) \sum\_{i=x-2}^{x+2} \sum\_{j=y-2}^{y+2} I\_R(i, j)}{\sqrt{\left[ \sum\_{i=x-2}^{x+2} \sum\_{j=y-2}^{y+2} I\_L(i, j)^2 - \frac{1}{2\mathfrak{S}} \left( \sum\_{i=x-2}^{x+2} I\_L(i, j) \right) \left[ \sum\_{i=x-2}^{x+2} \sum\_{j=y-2}^{y+2} I\_R(i, j)^2 - \frac{1}{2\mathfrak{S}} \left( \sum\_{i=x-2}^{x+2} \sum\_{j=y-2}^{y+2} I\_R(i, j) \right) \right]} \tag{5}$$
 
$$\text{cost}(\mathbf{x}, y) = 0.5 - 0.5 \times \text{QNCC}(\mathbf{x}, y) \tag{6}$$

The final pixel-level cost of pixel (*x*, *y*) in the overlapping area between images is defined as:

$$DE(\mathbf{x}, y) = \begin{cases} w \times \text{cost}(\mathbf{x}, y) & l(\mathbf{x}, y) \in PRAs \\ \text{cost}(\mathbf{x}, y) & \text{otherwise} \end{cases} \tag{7}$$

where *I*(*x*, *y*) is the pixel of the overlapping area in the left and right image. If *I*(*x*, *y*) belongs to *PRAs*, the cost value should be multiplied by *w*. *w* is the weight for pixels in *PRAs*, which is assigned a value much lower than 1.0. With such weight processing, this makes sure that the difference in the road area can be relatively small, so the seamlines will pass through roads as much as possible.

### 2.3.2. Shortest-Path Searching

After the final pixel-level cost determination, similar to Pan et al. [6], in order to minimize the difference of the seamlines, the proposed method uses the differential cost to calculate the local cost between neighboring pixels when applying Dijkstra's algorithm to search for the shortest path. The differential cost is defined in Equation (8).

$$de\_{mm,pq} = \left| DE(m,n) - DE(p,q) \right| \tag{8}$$

where (*m*, *n*) and (*p*, *q*) are adjacent pixels; *DE*(*m*, *n*) and *DE*(*p*, *q*) are the pixel-level costs of pixels (*m*, *n*) and (*p*, *q*), respectively, which are calculated in Equation (7). Let *near*(*m*, *n*) be the eight neighboring nodes of (*m*, *n*), *DCost*(*m*, *n*), and *DCost*(*p*, *q*) be the global minimum costs from the start pixel to (*m*, *n*) and (*p*, *q*), respectively. Then:

$$D\text{Cost}(m,n) = \min \left| d e\_{m,pq} + D\text{Cost}(p,q); (p,q) \in \text{new}(m,n) \right| \tag{9}$$

Dijkstra's algorithm is a classic global optimization method which solves the single-source shortest-path problem for arbitrary directed graphs *G* = *(V, E)* with unbounded non-negative weights [43,44]. Given a source vertex s in a weighted directed graph *G* = *(V, E)* where all edges are nonnegative, the pseudo-code for Dijkstra's algorithm is presented in Algorithm 1. Dijkstra's algorithm uses a data structure for storing and querying partial solutions sorted by distance from the start. The computational complexity of the original Dijkstra's algorithm is Θ( *V* · *V* + *E* ), if the min-priority queue is implemented by an ordinary linked list. |*V*| is the number of nodes in the graph and |*E*| is the

number of edges in the graph. The computational complexity depends on how to the min-priority queue is implemented. In order to improve the efficiency of Dijkstra's algorithm, similar to Wang et al. [7], the proposed method implements the min-priority queue with a binary min-heap. The pseudo-code for Dijkstra's algorithm with a binary min-heap is presented in Algorithm 2. The computational complexity of the improved Dijkstra's algorithm is Θ(( *V* + *E* ) · lg *V* ) [7].


**Algorithm 2** Dijkstra's algorithm with Binary Min-heap

```
1 Dijkstra_Binary_Min-heap(G, s)
2 dist[s] = 0
3 for each vertex v∈V
4 if v -
           s
5 dist[v] = ∞
6 pre [v] = undefined
7 Q.add_with_min-priority(v, dist[v])
8 S = Ø
9 Q = V
10 while Q -
            Ø do
11 u = extract_min_with_min- priority(Q)
12 S = S∪{u}
13 for each vertex v∈Adj(u) do
14 dist[v] = min(dist[v], dist[u]+w(u, v))
15 pre[v] = u
16 Q.decrease_min-priority(v, dist[v])
```
### **3. Experimental Results and Analysis**

### *3.1. Experimental Data and Platform*

The experiment consists of two parts. The first part uses the D-LinkNet neural network to generate the road probability map. The second part determines the seamline based on this road probability map.

The D-LinkNet neural network was trained and tested with a single NVIDIA GeForce GTX 1080Ti using the TensorFlow library in python in Linux. The training set was composed of 6226 images from the DeepGlobe Road Extraction dataset [35] and 1000 image sets of data for manually marking roads with 0.5-m resolution remote sensing images. The image size was 1024 × 1024. Details for training a D-LinkNet-34 network are as follows: *Batchsize* = 1, *epoch* = 200, *train\_best\_loss* = 50, *learning\_rate* = 2 <sup>×</sup> 10−4, and the loss function defined as a mixture of binary cross entropy loss representing the error between pixels with a dice coefficient loss suitable for the error between batches based on the IOU(intersection over union) as an evaluation index. The output of the neural network is the probability of judging whether the pixel belongs to a road. It took almost 50 h to complete the network training. Once the network training is completed, the network can be applied to other datasets of different areas. After training, the error train loss is 0.18796 and the accuracy rate is 0.81204.

The proposed determination of the seamline method was implemented by C++ programming on a portable computer with four Intel(R) Core(TM) i7-6700HQ CPU at 2.60 GHz, 16.0 GB of internal memory, and a mechanical hard drive with a 1 TB capacity, a 32 MB cache, and a 7200 r/min speed for data processing. The Geospatial Data Abstraction Library (GDAL), which is a widely used open source library, was adopted to read and write a large variety of raster spatial data formats. *w* was the weight for pixels in *PRAs* which is shown in Equation (7). The proposed method was performed with *w* = 0.001. In our practice, the value of w was determined by our experience.

In order to verify the algorithm proposed in this paper, three sets of aerial images were selected for experiments. An overview of the three sets of data is shown in Table 1. Among them, the coverage of Dataset A is the central urban area of a large city; the coverage of Dataset B is the suburb of a big city, and the coverage of Dataset C is the center of a medium-sized city.


**Table 1.** Basic information of the datasets.

### *3.2. Seamline Determination Results and Analysis*

In order to compare the effect and efficiency of the algorithm of our proposed approach, Dijkstra's algorithm [43] and the OrthoVista method (INPHO, 2005) were selected to compare with the proposed algorithm. Dijkstra's algorithm is one of the earliest and simplest algorithms. OrthoVista is one of the most widely used professional mosaicking products in the world, which is a desktop software of INPHO's digital photogrammetric system [45].

Figure 5 illustrates the experiments of *PRAs* determination and the intermediate results for Dataset A. Figure 6 illustrates the experiments of *PRAs* determination and the intermediate results for Dataset B. Figure 7 illustrates the experiments of *PRAs* determination and the intermediate results for Dataset C.

In Figures 5–7, (a) illustrates the overlapping area of the left image; (b) illustrates the overlapping area of the right image; (c) illustrates the road probability map of the overlapping area of the left image; (d) illustrates the road probability map of the overlapping area of the right image; (e) illustrates the *PRAs* of the overlapping area; and (f) illustrates the NCC cost of the overlapping area. Figures 5c, 6c, and 7c illustrate the road probability map of the overlapping area of the left image of Datasets A, B, and C, respectively. The larger the pixel value, the more likely it is to be determined as a road. Figures 5d, 6d, and 7d illustrate the road probability map of the overlapping area of the right image of Datasets A, B, and C, respectively. The larger the pixel value, the more likely it is to be determined as a road. Most of the roads in the orthoimages are extracted. In order to enhance the appearance of the road probability map, the stretch method of histogram equalize is used to adjust the value of the road probability map. Because of the differences between the overlapping area of the left and right image, the road probability maps obtained by them are also not the same, which are illustrated in (c) and (d) of Figures 5–7. With the help of the Otsu method [41,42], the preferred road areas of the left and right image are determined by the road probability threshold for binarization, respectively. The final *PRAs* of the overlapping are defined from the preferred road areas of left and right image by Equation (4). In Figures 5e, 6e, and 7e, the white areas are the *PRAs* of the overlapping areas, which are the priority areas that the seamlines pass. Most roads of the overlapping area are included in the *PRAs*, which meets our requirement. Figures 5f, 6f, and 7f illustrate the NCC cost of the overlapping area of

Datasets A, B, and C, respectively. The greater the brightness value of a pixel, the greater the NCC cost. The NCC cost is an effective method for assessing the difference of the pixel-level.

**Figure 5.** Experiments of preferred road areas (*PRAs*) determination for Dataset A: (**a**) the ovrlapping area of the left image, (**b**) the overlapping area of the right image, (**c**) the road probability map of the overlapping area of the left image, (**d**) the road probability map of the overlapping area of the right image, (**e**) the *PRAs* of the overlapping area, and (**f**) the Normalized Cross Correlation (*NCC*) cost of the overlapping area.

**F G** 

**Figure 6.** *Cont.*

**Figure 6.** Experiments of preferred road areas (*PRAs*) determination for Dataset B: (**a**) the overlapping area of the left image, (**b**) the overlapping area of the right image, (**c**) the road probability map of the overlapping area of the left image, (**d**) the road probability map of the overlapping area of the right image, (**e**) the *PRAs* of the overlapping area, and (**f**) the Normalized Cross Correlation (*NCC*)cost of the overlapping area.

**D E** 

**Figure 7.** *Cont.*

**Figure 7.** Experiments of preferred road areas (*PRAs*) determination for Dataset C: (**a**) the overlapping area of the left image, (b) the overlapping area of the right image, (**c**) the road probability map of the overlapping area of the left image, (**d**) the road probability map of the overlapping area of the right image, (**e**) the *PRAs* of the overlapping area, and (**f**) the Normalized Cross Correlation (*NCC*)cost of the overlapping area.

**H I** 

Figure 8 illustrates the experiments of seamline determination of the three different methods for Dataset A. Figure 9 illustrates the experiments of seamline determination of the three different methods for Dataset B. Figure 10 illustrates the experiments of seamline determination of the three different methods for Dataset C. The three methods were tested without a down-sampling strategy.

In Figures 8–10, (a), (**c**), and (e) illustrate the seamlines determined using Dijkstra's algorithm, OrthoVista and the proposed algorithm, respectively; (b), (d), and (f) illustrate the details of the white boxes in (a), (c), and (e), respectively. The ideal seamline should avoid crossing obvious objects, such as buildings, as much as possible and go along the objects which have small relief displacement, such as a road, river, grass, or bare land [6]. From the seamline detection results of the three data sets and

especially the details of the white boxes, compared with the two previous methods, the seamlines determined by the proposed method mainly go along the roads where no significant differences exist.

**D E** 

**H I** 

**Figure 8.** Experiments of seamline determination for Dataset A: (**a**) seamline determined by Dijkstra's algorithm, (**b**) details of the white box in (**a**), (**c**) seamline determined by OrthoVista, (**d**) details of the white box in (**c**), (**e**) seamline determined by the proposed method, and (**f**) details of the white box in (**e**).

*Sensors* **2020**, *20*, 1832

**Figure 9.** Experiments of seamline determination for Dataset B: (**a**) seamline determined by Dijkstra's algorithm, (**b**) details of the white box in (**a**), (**c**) seamline determined by OrthoVista, (**d**) details of the white box in (**c**), (**e**) seamline determined by the proposed method, and (**f**) details of the white box in (**e**).

**H I** 

**F G** 

**Figure 10.** *Cont.*

**Figure 10.** Experiments of seamline determination for Dataset C: (**a**) seamline determined by Dijkstra's algorithm, (**b**) details of the white box in (**a**), (**c**) seamline determined by OrthoVista, (**d**) details of the white box in (**c**), (**e**) seamline determined by the proposed method, and (**f**) details of the white box in (**e**).

In many related studies, it is difficult to find a general method for automated quantitative assessment of the seamline quality. Therefore, similar to the evaluation method in other relevant studies [2,6,7,25], the quantitative index applied in the proposed method is the number of times that seamlines cross obvious objects. The seamlines which have a smaller number of times are considered ideal seamlines. In order to compare the efficiency of the different algorithms fairly, all the algorithms were implemented without a down-sampling strategy and parallel computing strategy. The comparison results of the three different methods in the three groups of test data are shown in Table 2.


**Table 2.** Comparison of previous methods with the proposed method.

The coverage area of Dataset A is located in the central urban area of large cities, for which the spatial resolution is 0.5 m. There are many high-rise buildings, overpasses, residential areas, and other buildings in the image. Such objects are the objects which the ideal seamlines should bypass. The seamline determination by Dijkstra's algorithm crossed five obvious objects, which is shown in Figure 8a. It crossed four bridges and an overpass, which is illustrated in Figure 8b. The seamline determination by the OrthoVista algorithm crossed nine obvious objects, which is shown in Figure 8c. Figure 8d shows that the seamline crossed several bridges and a buildings. Seamline determination by the proposed algorithm crossed one building, which is shown in Figure 8e. The details of the one crossed building are illustrated in Figure 8f. As shown in Figure 5e, during the optimization of the seamline at the semantic level, the proposed algorithms almost included the roads in *PRAs*. By giving the *PRAs* area a smaller weight, which is shown in Equation (7), the seamline mainly passes through the *PRAs*, and the final seamline is mainly along the roads, bypassing most of the areas with large relief displacement.

The coverage area of Dataset B is located in the suburbs area of large cities, for which the spatial resolution is 0.2 m. There is much agricultural land, woodland, and bare land in the image. There are some original villages and factories in it. A main road runs from north to south. The seamline determination by Dijkstra's algorithm crossed six obvious objects, which is shown in Figure 9a. The details of the crossed buildings are illustrated in Figure 9b. The seamline determination by the OrthoVista algorithm crossed six obvious objects, which is shown in Figure 9c. The details of the crossed buildings are illustrated in Figure 9d. Seamline determination by the proposed algorithm crossed no obvious objects and was along the north–south main road, which is shown in Figure 9e. The details of the crossed buildings are illustrated in Figure 9f. As shown in Figure 6e, during the optimization of the seamline at the semantic level, the proposed algorithms almost included the roads in *PRAs*. By giving the *PRAs* area a smaller weight, which is shown in Equation (7), the seamline mainly passes through the *PRAs*, and the final seamline is mainly along the roads, bypassing most of the areas with large relief displacement.

The coverage area of Dataset C is located in the central urban area of a medium-sized city, for which the spatial resolution is 0.5 m. In the image, in addition to buildings and residential areas, there is much agricultural land and woodland. The seamline determination by Dijkstra's algorithm crossed two obvious objects, which is shown in Figure 10a. It crossed two buildings, which is illustrated in Figure 10b. The seamline determination by the OrthoVista algorithm crossed 11 obvious objects, which is shown in Figure 10c. Figure 10d shows that the seamline crossed several buildings. Seamline determination by the proposed algorithm crossed two obvious objects, which is shown in Figure 10e. The details of the two crossed buildings are illustrated in Figure 10f. As shown in Figure 7e, during optimization of the seamline at the semantic level, the proposed algorithms almost included the roads in *PRAs*. By giving the *PRAs* area a smaller weight, which is shown in Equation (7), the seamline mainly passes through the *PRAs*, and the final seamline is mainly along the roads, bypassing most of the areas with large relief displacement.

In summary, due to the use of D-LinkNet, the roads were almost extracted and the *PRAs* were determined, which are shown in Figures 5e, 6e, and 7e. The seamline obtained by our method had the best result and passed through the less obvious objects and mainly went along the roads.

In practice, efficiency has to be taken into consideration. The processing time are recorded in the fourth column of Table 2. This shows that the seamlines determined by Dijkstra's algorithm took around 644.025 s on average. OrthoVista's method was better. It took around 14.333 s on average. The seamlines determined by the proposed method took around 22.784 +Δs on average. This processing time for the proposed method consists of two parts. The first part is the time required for extracting the road probability map using D-LinkNet, and the second part is the time required for seamline optimization. Δ represents the time required for training the network and extracting the road probability map. It took around 50 h. Regardless Δ, the proposed and OrthoVista methods are at the same level. Compared with Dijkstra's and OrthoVista's methods, Δ of the proposed method includes the network training time and road probability map extraction time. The network training time accounted for most of Δ. Although network training takes some time, once processed completely, it can be used to extract the road probability map for other data.

Figure 11 shows the experiments of seamline determination for Dataset A with different values of *w*. A quantitative comparison of seamlines determined by the proposed method for Dataset A with different values of *w* was conducted, as shown in Table 3. *w* is the weight for pixels in *PRAs,* which is shown in Equation (7). With such weight processing, this makes sure that the difference in the road area can be relatively small, so the seamlines will pass through roads as much as possible. In our paper, we suggested setting *w* to 0.001. According to Table 3, an acceptable seamlines determination result can be obtained by setting *w* to 0.001 for Dataset A.

**Figure 11.** Experiments of seamline determination for Dataset A with different values of *w*: (**a**) *w* is 0.001 (red line) and *w* is 0.1 (cyan line), (**b**) *w* is 0.001 (red line) and *w* is 0.01 (cyan line), (**c**) w is 0.001 (red line) and *w* is 0.0001 (cyan line), and (**d**) *w* is 0.001 (red line) and *w* is 0.00001 (cyan line).

**Table 3.** Comparison of the proposed method with different values of *w* for Dataset A.


### **4. Conclusions**

In this paper, an automatic seamline determination method was presented for urban image mosaicking based on road probability map from the D-LinkNet neural network. The road probability map was used to improve the seamline determination. This method optimizes the seamlines at both the semantic and pixel level. At the semantic level, the *PRAs* are determined by binarizing the road probability map. In this step, most of the roads are included in the *PRAs*. At the pixel level, Dijkstra's algorithm is adopted to find the final seamlines. To improve the efficiency, the minimum heap is adopted to store the graph in the form of adjacency lists and extract the minimum efficiently [7]. Three group data sets of aerial orthoimages with different ground resolutions located in different cities were used to test and validate the proposed method in this paper. The comparative experimental results show the advantages of the proposed method. Compared with two previous methods, the seamline obtained by the proposed method had the best result in that it passed through the less obvious objects and mainly followed the roads. In terms of the computational efficiency, the proposed method also has a high efficiency. Moreover, the proposed method can easily be applied to the seamlines network determination framework easily [3,46,47].

Nevertheless, the proposed method may be improved in the future as follows: (1) Road probability map generation by D-LinkNet may have a significant influence on the final seamline determination. Therefore, more training samples should be made to better train the D-LinkNet neural network. (2) The proposed algorithm can be applied to the seamline network determination framework [3,46,47] to construct a single seamless composite image automatically.

**Author Contributions:** Conceptualization, S.Y.; methodology, S.Y. and K.Y.; software, S.Y., K.Y., and X.L.; writing—original draft preparation, S.Y., K.Y., X.L., and H.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the National Natural Science Foundation of China, grant number 41901388 and China Transport Telecommunications and Information Center Reserve Project, grant number 2018CB07. The APC was funded by the National Natural Science Foundation of China, grant number 41901388.

**Conflicts of Interest:** The authors declare no conflicts of interest.

### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

### **Satellite Laser Ranging for Retrieval of the Local Values of the Love** *h***<sup>2</sup> and Shida** *l***<sup>2</sup> Numbers for the Australian ILRS Stations**

### **Marcin Jagoda 1,\*, Miłosława Rutkowska 1, Paweł Lejba 2, Jacek Katzer 3, Romuald Obuchovski <sup>4</sup> and Dominykas Šlikas <sup>4</sup>**


Received: 26 October 2020; Accepted: 28 November 2020; Published: 30 November 2020

**Abstract:** This paper deals with the analysis of local Love and Shida numbers (parameters *h*<sup>2</sup> and *l*2) values of the Australian Yarragadee and Mount Stromlo satellite laser ranging (SLR) stations. The research was conducted based on data from the Medium Earth Orbit (MEO) satellites, LAGEOS-1 and LAGEOS-2, and Low Earth Orbit (LEO) satellites, STELLA and STARLETTE. Data from a 60-month time interval, from 01.01.2014 to 01.01.2019, was used. In the first research stage, the Love and Shida numbers values were determined separately from observations of each satellite; the obtained values of *h*2, *l*<sup>2</sup> exhibit a high degree of compliance, and the differences do not exceed formal error values. At this stage, we found that it was not possible to determine *l*<sup>2</sup> from the data of STELLA and STARLETTE. In the second research stage, we combined the satellite observations of MEO (LAGEOS-1+LAGEOS-2) and LEO (STELLA+STARLETTE) and redefined the *h*2,*l*<sup>2</sup> parameters. The final valueswere adopted, and further analysesweremade based on the values obtained from the combined observations. For the Yarragadee station, local *h*<sup>2</sup> = 0.5756 ± 0.0005 and *l*<sup>2</sup> = 0.0751 ± 0.0002 values were obtained from LAGEOS-1 + LAGEOS-2 and *h*<sup>2</sup> = 0.5742 ± 0.0015 were obtained from STELLA+STARLETTE data. For the Mount Stromlo station, we obtained the local *h*<sup>2</sup> = 0.5601 ± 0.0006 and *l*<sup>2</sup> = 0.0637 ± 0.0003 values from LAGEOS-1+LAGEOS-2 and *h*<sup>2</sup> = 0.5618 ± 0.0017 from STELLA + STARLETTE. We found discrepancies between the local parameters determined for the Yarragadee and Mount Stromlo stations and the commonly used values of the *h*2, *l*<sup>2</sup> parameters averaged for the whole Earth (so-called global nominal parameters). The sequential equalization method was used for the analysis, which allowed to determine the minimum time interval necessary to obtain stable *h*2, *l*<sup>2</sup> values. It turned out to be about 50 months. Additionally, we investigated the impact of the use of local values of the Love/Shida numbers on the determination of the Yarragadee and Mount Stromlo station coordinates. We proposed to determine the stations (*X*, *Y*, *Z*) coordinates in International Terrestrial Reference Frame 2014 (ITRF2014) in two computational versions: using global nominal *h*2, *l*<sup>2</sup> values and local *h*2, *l*<sup>2</sup> values calculated during this research. We found that the use of the local values of the *h*2, *l*<sup>2</sup> parameters in the process of determining the stations coordinates influences the result.

**Keywords:** Love/Shida numbers; satellite laser ranging (SLR); Yarragadee station; Mount Stromlo station; LAGEOS; STELLA; STARLETTE satellites; SLR stations coordinates; ITRF2014

### **1. Introduction**

There are different kinds of external forces acting on the Earth which cause its gradual changes; for this reason, our planet needs constant monitoring. One of these forces is tidal forces, which are reflected, among other things, in the displacement of Earth's masses and, consequently, in changes in the position of points on the Earth's surface. To describe the flexible reaction of the Earth to tidal stresses, the concept of so-called Love (*h*, *k*) and Shida (*l*) numbers were introduced. These are tidal parameters whose detailed description was presented in the fundamental works of A.E.H. Love "Some problems of geodynamics" [1] and T. Shida and M. Matsoyama "Note of Hecker's observations" [2].

The Earth's dynamics is currently studied using satellite measurement techniques, including the satellite laser ranging (SLR) technique [3]. In our earlier research programme, e.g., [4–6], we have successfully demonstrated that the SLR technique makes it possible to determine tidal parameters with very high accuracy, it was also indicated in [7]. Other satellite measuring techniques can also be used for such purposes, e.g., the VLBI technique [8,9]; satellite altimetry [10,11]. All these publications are focused on determining the global values of tidal parameters averaged over the whole Earth.

Due to the heterogeneous structure of our planet, it is reasonable that the reaction to tidal stresses is not the same for the whole Earth. With this in mind, we have launched a research programme to analyze the local tidal parameters. The research carried out so far focused on the Baltic Sea region [12,13], where local tidal parameters for the SLR stations from Poland and Latvia were analyzed based on data from the LAGEOS-1 and LAGEOS-2 satellites. In this study, we made an attempt to determine and analyze local values of tidal parameters for two Australian SLR stations: Yarragadee (no. 70900513, approx. 29◦ S, 115◦ E) and Mount Stromlo (no. 78259001, approx. 35◦ S, 149◦ E). In addition, we assessed the impact of their use on the determination of the coordinates of these stations. These tasks constitute the main research objective of this work. Estimation of the minimum time interval ensuring the stability of the determination and the assessment of the possibility of determining local tidal parameters from the data of the LEO satellites STELLA and STARLETTE constitute the intermediary purpose of this study.

The data provided by the Australian Yarragadee and Mount Stromlo stations are extremely important for geodynamic research. The global SLR network consists of 38 stations, of which only eight are located in the Southern Hemisphere; two of them on the Australian continent. Their location is shown in Figure 1.

**Figure 1.** Location of the Australian satellite laser ranging (SLR) stations.

The Yarragadee station is located in Western Australia, near the city of Dongara. The Mount Stromlo satellite laser ranging observatory is located in the south-eastern part of the continent, near the city of Geraldton. These stations are part of the Western Pacific Laser Tracking Network and contribute to the International Laser Ranging Service (ILRS). These are some of the best stations in terms of accuracy and number of observations. The data they collect plays a very important role, in synergy with other geodetic techniques, in defining International Terrestrial Reference Frame (ITRF) and determining Earth Orientation Parameters (EOP).

The basis of the satellite laser ranging technique (SLR) is the measurement of two-way time of light pulses flight between a station and a satellite fitted with retroreflectors. The distance measured to the satellite must be adjusted to accommodate the effects of a speed of light decrease and the difference between the straight and curved paths of ray. Furthermore, it must take into consideration the distance from the retroreflector to the satellite mass center, influence of the satellite motion, the Earth rotation and relativistic effects [14]. In its simplified form, the equation of laser observation is as follows [14]:

$$
\rho = \frac{\mathbf{C} \Delta t}{2} \tag{1}
$$

where ρ is the distance between a station and a satellite, Δ*t* is the two way time interval of light pulses flight between a station and a satellite, and *C* is the speed of light.

Typical, geodetic SLR satellites are sphere-shaped, covered with retroreflectors and can be divided into two main groups: Medium Earth Orbit (MEO) satellites; e.g., LAGEOS-1 (Perige = 5860 km), LAGEOS-2 (Perige = 5620 km); and Low Earth Orbit (LEO) satellites, e.g., STELLA (Perige = 804 km) and STARLETTE (H = 812 km). The data from these satellites is widely used in geodynamic research; e.g., to determine stations coordinates [15–17], to study the gravitational field of the Earth [18], to determine Earth Orientation Parameters [19–22], or to study the tidal phenomenon [23,24]. In this work, we used the data of the LAGEOS-1, LAGEOS-2, STELLA, and STARLETTE satellites to determine the local values of the tidal parameters and coordinates of the Australian SLR Yarragadee and Mount Stromlo stations. A detailed description of the SLR technique can be found in the works [25,26], while a wide range of applications of laser satellites in geodynamic research has been presented in [27–29].

The gravitational impact of the Moon, the Sun, and the Solar System planets on the Earth's surface results in the creation of earth and ocean tides. The tidal forces cause the displacement of earth and ocean masses. A detailed description of the tide phenomenon and its mathematical basis can be found in fundamental work "The tides of the planet Earth" by P. Melchior [30]. These changes in the distribution of the Earth's masses related to tides are expressed by movements of observation stations, as described by Equation (2) given in [31]:

$$\begin{aligned} \Delta X &= \sum\_{j=2}^{3} \Biggl[ \frac{\text{GM}\_{j}}{\text{GM}\_{\text{eff}}} \frac{d}{d\_{j}^{3}} \Bigg] \Bigg[ \left[ 3l\_{2} \left( \hat{\mathsf{R}}\_{j} \hat{\mathsf{r}}\_{\text{sta}} \right) \right] \overline{X}\_{j} + \left[ 3 \left( \frac{h\_{2}}{2} - l\_{2} \right) \left( \hat{\mathsf{R}}\_{j} \hat{\mathsf{r}}\_{\text{sta}} \right)^{2} - \frac{h\_{2}}{2} \right] \overline{X}\_{\text{sta}} \Bigg] \\ \Delta Y &= \sum\_{j=2}^{3} \left[ \frac{\text{GM}\_{j}}{\text{GM}\_{\text{eff}}} \frac{d}{d\_{j}^{3}} \right] \Bigg[ \Bigg[ \left[ 3l\_{2} \left( \hat{\mathsf{R}}\_{j} \hat{\mathsf{r}}\_{\text{sta}} \right) \right] \overline{Y}\_{j} + \left[ 3 \left( \frac{h\_{2}}{2} - l\_{2} \right) \left( \hat{\mathsf{R}}\_{j} \hat{\mathsf{r}}\_{\text{sta}} \right)^{2} - \frac{h\_{2}}{2} \right] \overline{Y}\_{\text{sta}} \Bigg] \\ \Delta Z &= \sum\_{j=2}^{3} \Biggl[ \frac{\text{GM}\_{j}}{\text{GM}\_{\text{eff}}} \frac{d}{d\_{j}^{3}} \Bigg] \Bigg[ \Bigg{[ \left[ 3l\_{2} \left( \hat{\mathsf{R}}\_{j} \hat{\mathsf{r}}\_{\text{sta}} \right] \right] \overline{Z}\_{j} + \left[ 3 \left( \frac{h\_{2}}{2} - l\_{2} \right) \left( \hat{\mathsf{R}}\_{j} \hat{\mathsf{r}}\_{\text{sta}} \right)$$

where

*GMj*—gravitational parameter for the Moon (*j* = 2) or the Sun (*j* = 3), *GME*—gravitational parameter for the Earth,

*ae*—equatorial radius,

*dj*—distance to the Moon (*j* = 2) or Sun (*j* = 3),

*R*ˆ *<sup>j</sup>*—the unit vector from the geocenter to the Moon (*j* = 2) or Sun (*j* = 3),

*r*ˆ*sta*—the unit vector from the geocenter to the station,

(*Xj*,*Yj*,*Zj*)—the Cartesian components of the unit vector *R*ˆ *<sup>j</sup>*, (*Xsta*,*Ysta*, *Zsta*)—the Cartesian components of the unit vector *r*ˆ, *h*2, *l*2—second degree of Love and Shida numbers.

In Equation (2), there are the tidal parameters *h*<sup>2</sup> and *l*<sup>2</sup> (Love and Shida numbers for the second-degree tides). The former refers to the radial tidal displacement of the station, the latter to the horizontal displacement, it is described in [32]. The tidal parameters are a measure of the flexible Earth's response to stresses created by tidal forces. If we assume that the Earth is a rigid body, then no elastic deformation takes place, and *h*, *l* are both 0. If we assume another extreme case in which the Earth is not just elastic but rather a liquid body, then the Love and Shida numbers are both equal to 1. Thus, for a rigid Earth *h* = 0, *l* = 0, for a liquid Earth *h* = 1, *l* = 1 and for an elastic Earth they take intermediate values: 0 < *h* < 1, 0 < *l* < 1. According to International Earth Rotation and Reference Systems Service (IERS) Conventions (IERS Technical Note No. 36) [33], the Earth's global (so called nominal) averaged values of the Love and Shida numbers for the second degree tides are *h*<sup>2</sup> = 0.6078, *l*<sup>2</sup> = 0.0847.

### **2. Materials and Methods**

To determine the local tidal parameters of the Australian Yarragadee and Mount Stromlo stations, we used observation data in the form of normal points of these stations collected for the LAGEOS-1, LAGEOS-2, STELLA, and STARLETTE satellites for the 5-year interval from 01.01.2014 to 01.01.2019. For the Yarragadee station, these were respectively: 57,299 LAGEOS-1 normal points, 58,133 LAGEOS-2 normal points, 38,031 STELLA normal points, 90,953 STARLETTE normal points; for the Mount Stromlo station: 25,249 LAGEOS-1 normal points, 25,962 LAGEOS-2 normal points, 21,654 STELLA normal points, 50,172 STARLETTE normal points. The method of creating normal points from SLR measurements is described in [34]. The data from the analyzed period were used to create 7-day orbital arcs. In total, 260 orbital arcs were obtained for each of the satellites. Satellite orbits were determined using the Cowell Numerical Integration method as described in detail in [31], using standard procedures, force models and constants recommended by the International Earth Rotation and Reference Systems Service (IERS) [33] and International Laser Ranging Service (ILRS) [35]. RMS values of the post-fit residuals, calculated from formula (3), were used as the satellites orbits accuracy determination [5]:

$$\text{RMS of the post-fit residuals} = \sqrt{\frac{\sum\_{i=1}^{n} \left(O\_i - C\_i\right)^2}{n-1}} \tag{3}$$

where *i* denotes successive number of normal points, (*Oi* − *Ci*) is the SLR observation minus the computed distance from the station to the satellite. The following values were obtained: RMS of the post-fit residuals: RMS(LAGEOS-1) = 1.02 cm, RMS(LAGEOS-2) = 1.01 cm, RMS(STELLA) = 1.98 cm, RMS(STARLETTE) = 1.87 cm.

To determine the local tidal parameters *h*2, *l*<sup>2</sup> values of the Yarragadee and Mount Stromlo stations and their coordinates, an observation Equation (4) was formulated and solved using the Bayesian least square method, a detailed description of this procedure is given in [31]. The local tidal parameters and coordinates were determined independently for both of the analyzed stations.

$$\left(\left(O\_i - \mathbb{C}\_i\right) = -\left\{\sum\_{j=1}^n \frac{\partial \mathbb{C}\_i}{\partial \varepsilon\_j} d\varepsilon\_j + \frac{\partial \mathbb{C}\_i}{\partial h\_2} dh\_2 + \frac{\partial \mathbb{C}\_i}{\partial l\_2} dl\_2\right\} + dO\_i \tag{4}$$

where

*j*—number of adjusted parameters (satellite position and velocity, empirical accelerations, and the station position),

*d*ε*j*—corrections to the *j*-th parameter,

*dh*2, *dl*2—corrections for Love number *h*<sup>2</sup> and for Shida number *l*2,

*dOi*—error of observation associated with the *i*-th measurement.

Given in Equation (4) the <sup>∂</sup>*Ci* ∂*h*<sup>2</sup> , ∂*Ci* <sup>∂</sup>*l*<sup>2</sup> quantities are calculated by differentiating Equation (2) and are expressed as follows [31]:

$$\frac{\partial \mathbf{C}\_{i}}{\partial \mathbf{l}\_{2}} = \frac{\partial \mathbf{C}\_{i}}{\partial X\_{\text{sta}}} \frac{\partial X\_{\text{sta}}}{\partial \mathbf{l}\_{2}} + \frac{\partial \mathbf{C}\_{i}}{\partial Y\_{\text{sta}}} \frac{\partial Y\_{\text{sta}}}{\partial \mathbf{l}\_{2}} + \frac{\partial \mathbf{C}\_{i}}{\partial Z\_{\text{sta}}} \frac{\partial Z\_{\text{sta}}}{\partial \mathbf{l}\_{2}} \tag{5}$$

$$\frac{\partial \mathbf{C}\_{i}}{\partial l\_{2}} = \frac{\partial \mathbf{C}\_{i}}{\partial X\_{\text{sta}}} \frac{\partial X\_{\text{sta}}}{\partial l\_{2}} + \frac{\partial \mathbf{C}\_{i}}{\partial Y\_{\text{sta}}} \frac{\partial Y\_{\text{sta}}}{\partial l\_{2}} + \frac{\partial \mathbf{C}\_{i}}{\partial Z\_{\text{sta}}} \frac{\partial Z\_{\text{sta}}}{\partial l\_{2}} \tag{6}$$

where

∂*Xsta* <sup>∂</sup>*h*<sup>2</sup> = 3 *j*=2 *GMj GME a*4 *e d*3 *j* 3 2*ae* (*R*ˆ *jr*ˆ*st*) <sup>2</sup> <sup>−</sup> <sup>1</sup> 3 *Xsta*, ∂*Ysta* <sup>∂</sup>*h*<sup>2</sup> = 3 *j*=2 *GMj GME a*4 *e d*3 *j* 3 2*ae* (*R*ˆ *jr*ˆ*st*) <sup>2</sup> <sup>−</sup> <sup>1</sup> 3 *Ysta*, ∂*Zsta* <sup>∂</sup>*h*<sup>2</sup> = 3 *j*=2 *GMj GME a*4 *e d*3 *j* 3 2*ae* (*R*ˆ *jr*ˆ*st*) <sup>2</sup> <sup>−</sup> <sup>1</sup> 3 *Zsta*, (7) ∂*Xsta* <sup>∂</sup>*l*<sup>2</sup> = <sup>3</sup> 3 *j*=2 *GMj GME a*4 *e d*3 *j* <sup>3</sup>(*r*ˆ*stR*<sup>ˆ</sup> *<sup>j</sup>*)*Xj* <sup>−</sup> <sup>1</sup> *ae* (*R*ˆ *jr*ˆ*st*) 2 *Xsta* , ∂*Ysta* <sup>∂</sup>*l*<sup>2</sup> = <sup>3</sup> 3 *j*=2 *GMj GME a*4 *e d*3 *j* <sup>3</sup>(*r*ˆ*stR*<sup>ˆ</sup> *<sup>j</sup>*)*Yj* <sup>−</sup> <sup>1</sup> *ae* (*R*ˆ *jr*ˆ*st*) 2 *Ysta* , ∂*Zsta* <sup>∂</sup>*l*<sup>2</sup> = <sup>3</sup> 3 *j*=2 *GMj GME a*4 *e d*3 *j* <sup>3</sup>(*r*ˆ*stR*<sup>ˆ</sup> *<sup>j</sup>*)*Zj* <sup>−</sup> <sup>1</sup> *ae* (*R*ˆ *jr*ˆ*st*) 2 *Zsta* . (8)

The sequential method was used to determine local tidal parameters. In the first step, the *h*<sup>2</sup> and *l*<sup>2</sup> parameters were determined separately from each orbital arc (arc1, arc2, arc3, ... , arc260). The following steps consisted in adding subsequent arcs to the calculations, one after another, following the scheme: arc1 + arc2, arc1 + arc2 + arc3, ... , arc1 + arc2 + ... + arc260. In each subsequent step, *h*<sup>2</sup> and *l*<sup>2</sup> parameters were re-computed. The values given in IERS Technical Note No. 36 [33] were taken as priori values (*h*<sup>2</sup> = 0.6078 and *l*<sup>2</sup> = 0.0847). In the first calculation stage, the local tidal parameters were determined separately from LAGEOS-1, LAGEOS-2, STELLA, and STARLETTE data, then data from LAGEOS-1 and LAGEOS-2 and STELLA and STARLETTE were pooled (LAGEOS-1+LAGEOS-2 and STELLA+STARLETTE) and re-computed to increase the accuracy and stability of the solutions. The final values were adopted and further analyses were made based on the values obtained from the combined observations of 260 orbital arcs.

Additionally, the coordinates of the Yarragadee and Mount Stromlo stations were determined in course of the analysis. These coordinates were calculated from the Equation (4). The determination method of the stations' coordinates from the SLR data was set out in detail in [14,36]. The coordinates of the Yarragadee and Mount Stromlo stations were determined from the LAGEOS-1 + LAGEOS-2 data with a presumptive assumption of the stations coordinates in the ITRF2014 reference frame [37]. The adjustment was performed in two calculation versions. As regards the first one, Yarragadee and Mount Stromlo stations coordinates were calculated using of the global nominal values (recommended in the IERS Conventions [33]) of tidal parameters. In the second one, Yarragadee and Mount Stromlo stations coordinates were estimated using the local values of tidal parameters calculated in this present paper. The impact of the application of different values of tidal parameters on the determination of these stations coordinates was then investigated.

The GEODYN II NASA GSFC software [31] was used for all the calculations related to the determination of satellite orbits, local tidal parameters and coordinates of Yarragadee and Mount Stromlo SLR stations.

### **3. Results and Discussion**

In this paper, we present the results of the determination of the local values of tidal parameters *h*2, *l*<sup>2</sup> for the Australian SLR stations Yarragadee and Mount Stromlo, and their coordinates in the ITRF2014 reference frame [37]. The first stage of the research included determining the local tidal parameters separately from the data of each of the satellites: LAGEOS-1, LAGEOS-2, STELLA, and STARLETTE. The obtained *h*2, *l*<sup>2</sup> values show a high degree of consistency, and the differences do not exceed formal error values (please refer to Table 1). At this stage, we found that it was not possible to determine *l*<sup>2</sup> from STELLA and STARLETTE data. Then, to increase the accuracy and stability of the solutions, we pooled the data of the individual satellite groups, LAGEOS-1+LAGEOS-2 and STELLA+STARLETTE, and re-computed the local tidal parameters. We did not determine the *l*<sup>2</sup> parameter from STELLA+STARLETTE data. The values obtained in this way were assumed final and subjected to further analysis. The final estimated values of the local tidal parameters for the Yarragadee and Mount Stromlo stations are given in Table 1, whereas the results of the sequential determination method are shown in Figures 2–7. For clarity and readability of the figures, we present results orbital arcs combined in groups of ten (arcs 1–10, 1–20, 1–30, ... , 1–260).

**Table 1.** Local tidal parameters *h*2, *l*<sup>2</sup> for Yarragadee and Mount Stromlo SLR stations.


**Figure 2.** Sequential solution for the Yarragadee local *h*<sup>2</sup> parameter based on LAGEOS-1+LAGEOS-2 data.

**Figure 3.** Sequential solution for the Yarragadee local *h*<sup>2</sup> parameter based on STELLA+STARLETTE data.

**Figure 4.** Sequential solution for the Yarragadee local *l*<sup>2</sup> parameter based on LAGEOS-1+LAGEOS-2 data.

**Figure 5.** Sequential solution for theMount Stromlolocal *h*<sup>2</sup> parameter based on LAGEOS-1+LAGEOS-2 data.

**Figure 6.** Sequential solution for the Mount Stromlo local *h*<sup>2</sup> parameter based on STELLA+STARLETTE data.

**Figure 7.** Sequential solution for theMount Stromlo local *l*<sup>2</sup> parameter based on LAGEOS-1+LAGEOS-2 data.

In the first step of the sequential method, the *h*2, *l*<sup>2</sup> parameters were determined from two orbital arcs. The computed values significantly deviate from the final ones. Adding arcs in weekly cycles (up to 260) allows the observation of a slowly emerging stability approaching the final *h*2, *l*<sup>2</sup> values determined from the 260 arcs. The values of formal errors of the determined parameters also asymptotically approach their final values. The process of achieving stability varies across parameters and stations. For the Yarragadee station, for the *h*<sup>2</sup> parameter, the designation stability (understood as the repeatability of the results obtained for subsequently added arcs down to the level of formal error) for LAGEOS-1+LAGEOS-2 data (Figure 2) emerges at about 200 arcs. The situation is similar for the determination from STELLA+STARLETTE data (Figure 3). The *l*<sup>2</sup> parameter (Figure 4) exhibits a lower degree of determination stability, achieved after about 230 arcs. In turn, for the Mount Stromlo station, the stability of the *h*<sup>2</sup> parameter determination was achieved for about 190 LAGEOS-1+LAGEOS-2 arcs (Figure 5) and 200 STELLA+STARLETTE arcs (Figure 6). The determination stability of the *l*<sup>2</sup> parameter for the Mount Stromlo station is similar to that of Yarragadee station, and was achieved after about 230 arcs (Figure 7). It proves that the number of arcs needed to determine local tidal parameters of these stations is about 200, which corresponds to about a 50-month interval (seven-day orbital arcs). For next added arcs, the estimated parameters values vary less than the formal error value.

Figures 2–4 show the results of the sequential solution for the Yarragadee station local tidal parameters. The values of the *h*<sup>2</sup> and *l*<sup>2</sup> numbers for this station, determined from LAGEOS-1+LAGEOS-2 data are 0.5756 ± 0.0005 and 0.0751±0.0002, respectively, and differ from the global values *h*<sup>2</sup> and *l*<sup>2</sup> given in IERS Technical Note No. 36 [33] by 0.0322 (5%) and 0.0096 (11%), respectively. A similar value of the *h*<sup>2</sup> parameter was obtained from the data of the STELLA+STARLETTE satellites: *h*<sup>2</sup> = 0.5742 ± 0.0015 (the difference with respect to the global value is 0.0336, i.e., about 6%). The *l*<sup>2</sup> parameter was not determined due to an unacceptable value and large error obtained when independently determining from the STELLA and SRTARLETTE data (see Table 1). Jagoda and Rutkowska [5], where global values of tidal parameters determined from LEO satellites data were analyzed from January 2005 to July 2007, present similar findings. The values of horizontal displacement of Earth masses in effect of tidal forces which are described by Shida *l*<sup>2</sup> number are significantly lower and harder to be measured than radial displacements which are expressed by Love *h*<sup>2</sup> number. This can potentially affect a determination of *l*<sup>2</sup> parameter from the LEO satellites data.

In general, the results of determining *h*<sup>2</sup> for the Yarragadee station from STELLA+STARLETTE data are very similar to those from LAGEOS-1+LAGEOS-2, with the difference being 0.0014, i.e., in the range of formal error.

In turn, the formal error in *h*<sup>2</sup> designation is three times greater for STELLA+STARLETTE, which is due to the impaired orbit designation of these satellites. The LEO satellites STELLA and STARLETTE move in the lower, dense layers of the atmosphere (at an altitude of about 800 km), and therefore their orbits are determined with greater errors than those of LAGEOS satellites. For the STELLA and STARLETTE satellites in [38] authors obtained mean RMS values of the post-fit residuals from 1.30 cm to 1.87 cm depending on the Earth gravity field model used. In another paper [39], mean RMS values of the post-fit STELLA/STARLETTE were given from 1.87 cm to 2.90 cm depending on the frequency of estimation of empirical acceleration parameters. In [40] these were 3.11 cm for STELLA and 2.40 cm for STARLETTE. In this paper, the mean RMS values of the post-fit STELLA and STARLETTE were 1.98 cm and 1.87 cm, respectively. In turn, the LAGEOS satellite orbits at an altitude of about 6000 km are determined with an accuracy of about 1 cm, and RMS values of the post-fit of this order were obtained, e.g., in [6,39]. The mean RMS values of the post-fit residuals for LAGEOS-1 and LAGEOS-2 obtained in this analysis are 1.02 cm and 1.01 cm, respectively.

Figures 5–7 depict the results of sequential solution for Mount Stromlo station local tidal parameters. The values of the tidal parameters for this station determined from the LAGEOS-1+LAGEOS-2 data are *h*<sup>2</sup> = 0.5601 ± 0.0006, *l*<sup>2</sup> = 0.0637 ± 0.0003. The differences with respect to global values are 0.0477 (8%) for *h*<sup>2</sup> and 0.021 (25%) for *l*2. The value of the *h*<sup>2</sup> parameter for this station determined from the STELLA+STARLETTE data is 0.5618 ± 0.0017, the difference from the nominal value is 0.046, that is about 7%. Similarly, as in the case of the Yarragadee station, the *l*<sup>2</sup> parameter from combined LEO satellites data was not determined. There is a high degree of conformity between the *h*<sup>2</sup> values obtained from LAGEOS-1+LAGEOS-2 and STELLA+STARLETTE data, with the difference being 0.0017 and not exceeding the formal error level. Similar to the Yarragadee station, the formal error of the *h*<sup>2</sup> parameter is higher for the LEO satellites; about three times in this case.

The comparison of Love/Shida numbers for the Yarragadee and Mount Stromlo stations shows that they differ by 0.0155 ± 0.0005 (LAGEOS-1+LAGEOS-2 data) and 0.0124 ± 0.0015 (STELLA+STARLETTE data) for the *h*<sup>2</sup> number and 0.0114 ± 0.0002 (LAGEOS-1+LAGEOS-2 data) for the *l*<sup>2</sup> number. These differences exceed the formal error level. So far, no similar studies have been carried out for SLR stations from Australia, so it is impossible to relate the results obtained to the work of other researchers. However, data for two European SLR stations from the Baltic Sea region are available: Borowiec (no. 78113802) and Riga (no. 18844401). Jagoda and Rutkowska [12] were found that the local tidal parameters for the Borowiec station determined from the LAGEOS satellites data in the 01.01.2009–01.01.2019 interval are *h*<sup>2</sup> = 0.7308 ± 0.0008 and *l*<sup>2</sup> = 0.1226 ± 0.0003. In another paper Jagoda and Rutkowska [13], were obtained *h*<sup>2</sup> = 0.6891 ± 0.0009 and *l*<sup>2</sup> = 0.1043 ± 0.0004 for the Riga station from the LAGEOS satellites data in the 01.01.2004–01.01.2019 interval. Significant differences can be found when comparing the results obtained in this work to the results for the European stations Borowiec and Riga. These are the largest for Mount Stromlo and Borowiec stations: about 23% for the *h*<sup>2</sup> parameter and about 48% for the *l*<sup>2</sup> parameter.

The differences between the global and local tidal parameters may be influenced by the geological structure and physical factors of the observation site, in this case Australia. The Australian continent is located in the eastern part of the Indo–Australian lithosphere plateau. The greater part of Australia is occupied by the Precambrian Craton called the Australian Craton, which is adjacent to the structure of the Flinders Ranges and the Barrier Ranges, and the structure of the Great Dividing Range [41]. The Yarragadee station is located within the Australian Craton on the so-called Perth Basin. The Perth

Basin is filled mainly with continental Permian sediments which lie directly on crystalline rocks. The rocks of the Perth Basin sedimentary cover are mainly sandy-loam and marine sediments of Triassic and Cretaceous periods [42,43]. The eastern part of Australia where the Mount Stromlo station is located, is occupied by the of the Great Dividing Range. The Palaeozoic structures of the Great Dividing Range were created as a result of subduction processes on the border between Panthalassa and the Gondwana Craton, on the fringe of which the Australian continent was situated [42]. The mountain range created as a result of these processes is characterised by a varied structure and geological history. In the western part, the structures formed in the Neoproterozoic era region dominate. In the central part, the main phases of the tectonic movements, magmatism, and metamorphism were at work from the early Ordovician to the lowest Devonian. In the eastern part, on the rocks of the older Palaeozoic there are thick sediments of the Devonian and the lower Carboniferous as well as the Permian. The tectonic movements lasted here from the Carboniferous. They were accompanied by lava outflows, the covers of which amount to many thousands of square kilometers of the area. In numerous places, after the fold movements, tectonic subsidences and grabens were formed [42,43].

In addition we have studied the impact of adjusted local tidal parameters *h*2, *l*<sup>2</sup> values on the determination of the Yarragadee and Mount Stromlo SLR stations coordinates in the ITRF2014 reference frame [37]. The test consisted in determining the *X*, *Y*, *Z* coordinates of Yarragadee and Mount Stromlo stations in ITRF2014 in two computational versions. In the first computational version, the coordinates were determined using the nominal global values *h*<sup>2</sup> = 0.6078, *l*<sup>2</sup> = 0.0847 [33]. The second version consisted in determining the coordinates using the proposed in this analysis local tidal parameters *h*<sup>2</sup> =0.5756, *l*<sup>2</sup> = 0.0751 for the Yarragadee station and *h*<sup>2</sup> = 0.5601, *l*<sup>2</sup> = 0.0637 for the Mount Stromlo station. Table 2 presents the test results.


**Table 2.** The *X*, *Y*, *Z* coordinates of the Yarragadee and Mount Stromlo SLR stations estimated in two calculation versions.

The use of local tidal parameters *h*2, *l*<sup>2</sup> values instead of global nominal *h*2, *l*<sup>2</sup> values affects the result of the coordinate determination. The *Z* coordinate seems to be the most affected one, with the difference between version 1 and 2 being 0.0055 m and −0.0052 m for Mount Stromlo and Yarragadee stations, respectively. The smallest difference was observed for the *X* coordinate: −0.0033 m for Yarragadee and −0.0038 m for Mount Stromlo. The *Y* component differed by 0.0041 m (Yarragadee) and 0.0050 m (Mount Stromlo). In [12], in a similar test performed for the Borowiec station, the same order of differences was obtained (Δ*X* = −0.0035 m, Δ*Y* = 0.0033 m, Δ*Z* = 0.0042 m) as for the Yarragadee and Mount Stromlo stations. However, in [13] describing the Riga station, these discrepancies are larger, namely, Δ*X* = 0.0044 m, Δ*Y* = −0.0047 m, Δ*Z* = 0.0069 m.

Similar results of determining the coordinates of the Yarragadee and Mount Stromlo stations in ITRF2014 system were obtained in the paper [17], where the authors proposed a kinematic method to estimate the coordinates of SLR stations by using the Global Navigation Satellite System (GNSS) technique onboard a low Earth orbiting (LEO) satellite. They applied SLR and GNSS observations of the GRACE-A satellite from January to December 2012. They found that the GRACE-A satellite, as a

connection between the SLR and GNSS techniques, allowed the accurate estimation of SLR stations positions with the high agreement with the ITRF2014 system.

In another paper [22], the author used the STARLETTE, LAGEOS-1 and LAGEOS-2 data over a 14-year period (1993–2007) for determination and analysis of SLR stations coordinates in ITRF2005 system [44]. The author pointed out a good agreement of the estimated coordinates with respect to the values given in ITRF2005. However, in both of these studies the influence of the application of different values of *h*2, *l*<sup>2</sup> parameters on the results of determining the SLR stations coordinates was not investigated.

### **4. Conclusions**

Based on the results obtained in the considered case studies, the following conclusions can be drawn:


**Author Contributions:** Conceptualization, M.J., P.L., and M.R.; methodology, M.J.; software, M.R.; validation, M.J., P.L., and M.R.; formal analysis, M.J., P.L., and M.R.; investigation, M.J., P.L., M.R., D.Š., and R.O.; resources, M.J., R.O., D.Š., and J.K.; data curation, M.J. and M.R.; writing—original draft preparation, M.J., P.L., and J.K.; writing—review and editing, M.J., M.R., P.L., R.O., and D.Š.; visualization, M.J., J.K., D.Š., and R.O.; supervision, M.J.; project administration, M.J.; funding acquisition, M.J. All authors have read and agreed to the published version of the manuscript.

**Funding:** The authors would like to express their gratitude to the National Science Center, Poland (PL—Narodowe Centrum Nauki) for the financial support for this study under Project No: 2019/03/X/ST10/01595.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

### *Article* **Determination of Motion Parameters of Selected Major Tectonic Plates Based on GNSS Station Positions and Velocities in the ITRF2014**

**Marcin Jagoda**

Faculty of Civil Engineering, Environmental and Geodetic Sciences, Koszalin University of Technology, Sniadeckich 2, 75-453 Koszalin, Poland; marcin.jagoda@tu.koszalin.pl ´

**Abstract:** Current knowledge about tectonic plate movement is widely applied in numerous scientific fields; however, questions still remain to be answered. In this study, the focus is on the determination and analysis of the parameters that describe tectonic plate movement, i.e., the position (Φ and Λ) of the rotation pole and angular rotation speed (*ω*). The study was based on observational material, namely the positions and velocities of the GNSS stations in the International Terrestrial Reference Frame 2014 (ITRF2014), and based on these data, the motion parameters of five major tectonic plates were determined. All calculations were performed using software based on a least squares adjustment procedure that was developed by the author. The following results were obtained: for the African plate, Φ = 49.15 ± 0.10◦, Λ = −80.82 ± 0.30◦, and *ω* = 0.267 ± 0.001◦/Ma; for the Australian plate, Φ = 32.94 ± 0.05◦, Λ = 37.70 ± 0.12◦, and *ω* = 0.624 ± 0.001◦/Ma; for the South American plate, Φ = –19.03 ± 0.20◦, Λ = −119.78 ± 0.39◦, and *ω* = 0.117 ± 0.001◦/Ma; for the Pacific plate, Φ = −62.45 ± 0.07◦, Λ = 111.01 ± 0.14◦, and *ω* = 0.667 ± 0.001◦/Ma; and for the Antarctic plate, Φ = 61.54 ± 0.30◦, Λ = −123.01 ± 0.49◦, and *ω* = 0.241 ± 0.003◦/Ma. Then, the results were compared with the geological plate motion model NNR-MORVEL56 and the geodetic model ITRF2014 PMM, with good agreement. In the study, a new approach is proposed for determining plate motion parameters, namely the sequential method. This method allows one to optimize the data by determining the minimum number of stations required for a stable solution and by identifying the stations that negatively affect the quality of the solution and increase the formal errors of the determined parameters. It was found that the stability of the solutions of the Φ, Λ, and *ω* parameters varied depending on the parameters and the individual tectonic plates.

**Keywords:** GNSS stations; tectonic plate motion parameters; ITRF

**1. Introduction**

Since the beginning of the Earth, its surface has been undergoing dynamic changes, which are caused both by external forces as well as those that act inside the planet, and their effects are visible on the surface. One such dynamic change process is tectonic plate movements on the surface of the asthenosphere. Current knowledge of the motion of the plates has been applied in numerous areas of research such as environmental sciences. Processes that result from plate tectonics, such as seismic or volcanic activity, have a direct influence on the environment. An issue that has been recently discussed by McEvoy et al. [1] is the influence of the movement of the lithosphere on the safety of radioactive waste stored underground. In addition, in [2,3], the authors analysed the influence of plate tectonics on climate change. Another research area where knowledge of tectonic plate motion has been applied is geodynamics and geodesy, in particular, related to defining the Earth's reference systems [4,5]. For years, a problem for studies focused on lithospheric deformations, has been the formulation of laws that govern these phenomena, in particular, explaining the driving mechanism of this motion, and thus the dynamics of the lithosphere. A pioneer of the idea of lithospheric motion, based on the adherence of the coastal lines of both

**Citation:** Jagoda, M. Determination of Motion Parameters of Selected Major Tectonic Plates Based on GNSS Station Positions and Velocities in the ITRF2014. *Sensors* **2021**, *21*, 5342. https://doi.org/10.3390/s21165342

Academic Editor: Alfred Colpaert

Received: 6 July 2021 Accepted: 5 August 2021 Published: 7 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Americas, Africa, and Europe, was A. Snider-Pellegrini [6]. However, only the arguments later provided by A. Wegener [7] contributed to the fact that the latter is considered to be the author of the foundation of the lithospheric motion theory. Currently, studies in the literature, for example, studies by [8,9], divide the lithosphere into seven major plates of various sizes (Eurasian, African, North American, South American, Australian, Pacific, and Antarctic), and several smaller ones, referred to as minor plates and microplates, which are parts of the major plates or complement them. Most of the plates are located along the western margin of the Pacific Ocean, as detailed in [10] and more detailed information about tectonic plate movement can be found, for example, in [11,12]. The 1980s was a period that witnessed dynamic development of space measurement techniques, including satellite laser ranging (SLR), doppler orbitography and radiopositioning integrated by satellite (DORIS), very long baseline interferometry (VLBI), and global navigation satellite systems (GNSS). These techniques have started to be used for precise determination of tectonic plate motion, due to the possibility of conducting observations at a global scale. Nowadays, the SLR, DORIS, VLBI, and GNSS techniques are the basis for studies on crustal movements [13], as they allow plate movements to be quantified at a level of submillimetres per year. Data from satellite systems are the basis for creating kinematic models of plate movements, referred to as plate kinematic and crustal deformation models, such as the series of models developed by H. Drewes [14–21]. Then, the obtained results have been compared with the geological models of plate movement that have been developed based on geophysical data. Examples of such models include: AMO-2 [22], NNR-NUVEL-1 [23], NNR-NUVEL-1A [24], PB2002 [25], and the newest one, NNR-MORVEL56 [26]. Conducting studies on a global scale requires a reference system that is uniform in geometrical and physical terms for the whole Earth. The International Terrestrial Reference System (ITRS) is such a reference system, and an International Terrestrial Reference Frame (ITRF) is a realization of the ITRS. To date, the International Earth Rotation and Reference System Service (IERS) [27] has realised more than ten solutions of the ITRF system, among which the currently valid one is the ITRF2014 [28]. The ITRF2014 was generated using the complete observation history of the four space techniques SLR, DORIS, VLBI, and GNSS. The corresponding international services, i.e., the International Laser Ranging Service [29], the International DORIS Service [30], the International VLBI Service [31], and the International GNSS Service [32], provided reprocessed time series of station positions and daily Earth orientation parameters (EOP). The International GNSS Service submitted time series comprising 7714 daily solutions, resulting from the second reprocessing campaign, covering the time period 1994.0–2015.1 [33]. It is worthwhile noting, here, that another realisation of the ITRF, namely the ITRF2020, should be available by the end of 2021. More details regarding specifications of the ITRF solutions can be found in [34,35] or IERS Technical Note 36 [36].

In order to meet the expectations of the users of various geodynamic, geodetic, and environmental applications, the realisations of the ITRF have been accompanied by the publication of a tectonic plate motion model, for example, APKIM2005 [19], ITRF2008 PMM [4], and ITRF2014 PMM [5]. These are kinematic models that have been created based on the position (coordinates) and velocities of the SLR, DORIS, VLBI, and GNSS stations in the given ITRF frame as one common solution (SLR + DORIS + VLBI + GNSS).

Considering the need for continuous analysis of tectonic plate movement due to the demands of contemporary geodynamics, geodesy, and the influence on the environment, this study was conducted to determine and analyse plate motion parameters based on the coordinates and velocities of SLR, DORIS, VLBI, and GNSS stations, separately, for each of these techniques. Such an approach offers the possibility to evaluate the contribution of each of these space techniques to the creation of a model of movement of specific tectonic plates, and to assess the accuracy, of each technique, for determining the plate motion parameters. In this study, a new approach is proposed for determining tectonic plate motion parameters, namely the sequential method, described in Section 2. It is successfully demonstrated that this method can be used to identify, and then to eliminate, from the calculations, the stations which, due to various reasons (e.g., being located on cracked, unstable areas of the given plate, areas of seismic activity, or on a microplate, so their movement is not consistent with the motion of the analysed plate), contribute to an increase in formal errors of the determined parameters and the effects on the determination results. The method also allows for data optimisation, i.e., specifying the minimum number of stations on the given tectonic plate for which the calculated motion parameters are stabilised (i.e., their changes do not exceed the value of formal errors). To date, studies have been carried out on the SLR [37,38], DORIS [38,39], and VLBI [38,40] techniques and, on the GNSS technique, for the Eurasian plate [41]. In this study, the subsequent stage of the conducted studies is presented. The aims of the study are: (i) to determine and analyse the motion parameters of the African, Australian, South American, Pacific, and Antarctic plates based on the coordinates and velocities of the GNSS stations in the ITRF2014; (ii) to identify the stations that increase the value of formal errors in the defined parameters and negatively affect the calculation results; and (iii) to estimate the stability of the solutions of the motion parameters of a given plate.

### **2. Materials and Methods**

The data provided by SLR, DORIS, VLBI, and GNSS are the basis for the determination of the positions *φ* and *λ* of the observation stations of these techniques. The positions of the stations are subject to changes in time as a result of, among others, the movement of the tectonic plates on which they are located. According to the station positions *φ* and *λ* determined at time intervals Δ*t*, the movement of the station in time Δ<sup>→</sup> *x* can be calculated. Then, knowing the value of the displacement Δ<sup>→</sup> *x* (in terms of coordinate shifts Δ*φ* and Δ*λ*) of the station on a given plate, we can determine three parameters that describe the movement of this plate, i.e., the geographical latitude Φ and longitude Λ of the pole of rotation Ω and the angular rotation speed *ω* of the given plate or the specific elements of the pole of rotation *ω*x, *ω*y, and *ω*<sup>z</sup> around the *X*, *Y*, and *Z* axes (Figure 1). The relations between these values are described in Equation (1) [42]. The geometric relations between the plate motion parameters and coordinate shifts are shown in Figure 1, where *X*<sup>0</sup> is the position of the station on the initial epoch *t*0, *X*<sup>1</sup> is the position of the station on the epoch *t*<sup>1</sup> (after plate moving), and point *P* denotes the Earth pole. Equation (1):

$$\begin{aligned} \tan\Phi &= \frac{\omega\_z}{\sqrt{\omega\_x^2 + \omega\_y^2}}\\ \tan\Lambda &= \frac{\omega\_y}{\omega\_x} \\ \omega &= \sqrt{\omega\_x^2 + \omega\_y^2 + \omega\_z^2} \\ \omega\_x &= \omega \cos\Lambda \cos\Phi \\ \omega\_y &= \omega \sin\Lambda \cos\Phi \\ \omega\_z &= \omega \sin\Phi \end{aligned} \tag{1}$$

According to [17], the displacement of the observational station Δ<sup>→</sup> *<sup>x</sup>* = ( <sup>→</sup> <sup>Ω</sup> <sup>×</sup> <sup>→</sup> *x* )Δ*t* expressed as a function of the tectonic plate motion parameters Φ, Λ, *ω* (Δ*ϕ* = *f*(Φ, Λ, *ω*), Δ*λ* = *f*(Φ, Λ, *ω*)) is described by Equation (2):

$$\begin{aligned} \Delta \phi &= \omega \cdot \Delta t \cdot \cos \Phi \cdot \sin(\lambda - \Lambda) \\ \Delta \lambda &= \omega \cdot \Delta t \cdot \left( \sin \Phi - \cos(\lambda - \Lambda) \cdot \tan \varphi \cdot \cos \Phi \right) \end{aligned} \tag{2}$$

Determining the motion parameters of a tectonic plate requires knowledge of the coordinates and velocities of at least two stations on that plate. This allows for the creation of two observational equations, i.e., Equation (3), and for the determination of three plate motion parameters Φ, Λ, and *ω*. In such a case (when the number of observational equations is higher than the number of the determined unknowns) it may be aligned with the use of the least-squares adjustment method. Assuming that more than two stations are located on a given plate allows one to evaluate the influence of the number and location of the stations on the determined motion parameters of the plate and the accuracy of their determination. A description of the least squares adjustment procedure was presented, among others, by McCarthy et al. [43], and its practical applications for the determination of plate motion parameters in [14], and in the earlier study by Jagoda et al. [40].

According to [14], the observational equations for the least squares adjustment procedure, which allow for the determination of plate motion parameters based on shifts in the station coordinates are expressed in Equation (3):

$$\begin{array}{l} \upsilon\_{\varphi} = \left(\frac{\partial \Lambda \rho}{\partial \Phi}\right) \mathrm{d}\Phi + \left(\frac{\partial \Lambda \rho}{\partial \Lambda}\right) \mathrm{d}\Lambda + \left(\frac{\partial \Lambda \rho}{\partial \omega}\right) \mathrm{d}\omega - \left(\Delta \rho^{\mathrm{obs}} - \Delta \rho^{\mathrm{cal}}\right) \\\upsilon\_{\lambda} = \left(\frac{\partial \Delta \lambda}{\partial \Phi}\right) \mathrm{d}\Phi + \left(\frac{\partial \Lambda \lambda}{\partial \Lambda}\right) \mathrm{d}\Lambda + \left(\frac{\partial \Delta \lambda}{\partial \omega}\right) \mathrm{d}\omega - \left(\Delta \lambda^{\mathrm{obs}} - \Delta \lambda^{\mathrm{cal}}\right) \end{array} \tag{3}$$

where *obs* and *cal* mean observed and calculated values, respectively.

The expressions given in Equation (3) are calculated based on the following relations, Equation (4) [14]:

$$\begin{aligned} \frac{\frac{\partial \Delta \rho}{\partial \Phi}}{\frac{\partial \Delta \rho}{\partial \Lambda}} &= -\omega \cdot \Delta t \cdot \sin \Phi \cdot \sin(\lambda - \Lambda) \\ \frac{\frac{\partial \Delta \rho}{\partial \Lambda}}{\frac{\partial \Delta \rho}{\partial \omega}} &= -\omega \cdot \Delta t \cdot \cos \Phi \cdot \cos(\lambda - \Lambda) \end{aligned} \tag{4}$$
 
$$\begin{aligned} \frac{\frac{\partial \Delta \lambda}{\partial \Phi}}{\frac{\partial \Delta \lambda}{\partial \Phi}} &= \omega \cdot \Delta t \cdot \cos \Phi + \omega \cdot \Delta t \cdot \cos(\lambda - \Lambda) \cdot \tan \rho \sin \Phi \\ \frac{\frac{\partial \Delta \lambda}{\partial \Lambda}}{\frac{\partial \Delta \lambda}{\partial \omega}} &= -\omega \cdot \Delta t \cdot \cos \Phi \cdot \sin(\lambda - \Lambda) \cdot \tan \rho \\ \frac{\frac{\partial \Delta \lambda}{\partial \omega}}{\frac{\partial \Delta \lambda}{\partial \omega}} &= \Delta t \cdot (\sin \Phi - \cos(\lambda - \Lambda) \cdot \tan \rho \cos \Phi) \end{aligned} \tag{4}$$

For the purposes of this study, the sequential method was applied to determine the plate motion parameters. It involves several calculation steps. In the first step, the Φ, Λ, and *ω* parameters are determined based on the coordinates and velocities of two stations, here, the GNSS stations (station 1 + station 2) located on a given tectonic plate, adopted from the ITRF2014 [28] and available for users on the website http://itrf.ensg.ign.fr/ITRF\_ solutions/2014/ [44] (accessed on 8 June 2021).

The next steps of the sequential method consist of adding further stations to the calculations, one by one, according to the scheme: station 1 + station 2, station 1 + station 2 + station 3, station 1 + station 2 + station 3 + , ... , station *n*, where *n* is the number of stations on the given plate as adopted in the solution. The number of stations added varies depending on the specific plate. In every subsequent step of the sequential method, the Φ, Λ, and *ω* parameters are calculated again. The application of the sequential method enables the obtainment of an increasingly stable adjustment, which is characterised by a decreasing formal error of the determined parameters. By increasing the number of stations, it arrives at a solution that, for a certain number of stations, is characterised by the high stability of the solution and minimum error values. A further increase in the number of stations results in the variability of the calculated motion parameter values within the limits of formal error, which is discussed in Section 3. The final values of the Φ, Λ, and *ω* parameters that were adopted and used in further analyses were those obtained from *n* stations (i.e., from the last step of the sequential method) for a given tectonic plate. The selection of the stations that are the basis for the determination of the motion parameters of specific tectonic plates should take into consideration the geophysical conditions of their location. Another factor that influences the accuracy of the solution is the geometrical configuration of the station network, which has been explained in [14,17]. Consistency with the previous determination of the Φ, Λ, and *ω* parameters for the SLR [37,38], DORIS [38,39], and VLBI [38,40] techniques were maintained, and the same manner of selecting stations was used. These assumptions were created based on the tests conducted during the realisation of previous studies. Thus, as in the previous studies, for example, [40,41], the stations should be located on a stable, uncracked area of the given plate, outside deformation zones. It is recommended that the stations should be distributed as evenly as possibly on the given tectonic plate. The concentration of a large number of stations in a small area does not significantly improve the accuracy and stability of the solution. The distance between the

first two stations should equal approximately 60% of the distance between plate boundaries, and it should not be shorter than 50 km. The minimum timespan of observations conducted at a given station should be three years. This is required to meet the requirements of the rigid plate motion theory, which was discussed in [4]. The number of GNSS stations used in the calculations differs for specific plates: for the African plate, 25 stations were used; for the Australian plate, 20 stations were used; for the South American plate 29 stations were used; for the Antarctic plate, 13 stations were used; and finally, for the Pacific plate, 19 stations were used. As far as the Eurasian plate is concerned, which was analysed by Jagoda and Rutkowska [41], 120 GNSS stations were used (4 calculation scenarios were applied with 30 stations for each scenario). Figure 2 presents the stations that were used (black dots) and those that were rejected (red dots) from the solution and estimated positions (green stars) of the pole of rotation for particular plates (SOAM, South American; AFRC, African; PACF, Pacific; ANTC, Antarctic; AUST, Australian).

**Figure 1.** Relations between <sup>→</sup> Ω (Φ, Λ, and *ω*) and Δ<sup>→</sup> *x* (Δ*φ* and Δ*λ*).

**Figure 2.** Distribution of the GNSS stations on tectonic plates and estimated positions of the pole of rotation for a given tectonic plate.

All calculations related to the determination of the Φ, Λ, and *ω* parameters were performed with use of the software in FORTRAN90, developed by the author, based on the least squares adjustment procedure. A block diagram of the adjustment of the plate motion parameters used in the author's own software was shown in an earlier study [40] and is not repeated here. The adopted weights of observations were the formal errors in the determination of shifts of individual GNSS stations given in the ITRF2014 [28].

### **3. Results and Discussion**

The positions and velocities of the GNSS stations in the ITRF2014 [28], determined by more than 21 years of GNSS observations that covered the period 1994.0–2015.1 [33], were the basis for the determination and analysis of the values of the Φ, Λ, and *ω* parameters that describe the movement of the five major tectonic plates: African, Australian, South American, Pacific, and Antarctic. A study by Jagoda and Rutkowska, on the largest tectonic plate, i.e., the Eurasian plate, has already been conducted and published in the literature [41]. In addition, the last of the major plates, the North American plate, should be analysed separately due to a very large amount of observational data. The North American plate is covered with a vast number of GNSS stations, which enables a detailed analysis of the area with very high tectonic activity, located along the boundary with the Pacific plate.

The results of each step of the sequential methods for the Φ, Λ, and *ω* parameter values are presented in Figures 3–17 for each plate and parameter, both separately and in Appendices A–E, with the names of the stations used for the calculations. The final values of the Φ, Λ, and *ω* parameters and of formal errors adopted for further analyses correspond to the values from the last step of the sequential method.

**Figure 3.** Results of sequential method for the Φ parameter for the South American plate.

**Figure 4.** Results of the sequential method for the Λ parameter for the South American plate.

**Figure 5.** Results of the sequential method for the *ω* parameter for the South American plate.

Φ Λ

ω

**Figure 6.** Results of the sequential method for the Φ parameter for the Australian plate.

**Figure 7.** Results of the sequential method for the Λ parameter for the Australian plate.

**Figure 8.** Results of the sequential method for the *ω* parameter for the Australian plate.

**Figure 9.** Results of the sequential method for the Φ parameter for the Antarctic plate.

**Figure 10.** Results of the sequential method for the Λ parameter for the Antarctic plate.

**Figure 11.** Results of the sequential method for the *ω* parameter for the Antarctic plate.

**Figure 12.** Results of the sequential method for the Φ parameter for the Pacific plate.

**Figure 13.** Results of the sequential method for the Λ parameter for the Pacific plate.

**Figure 14.** Results of the sequential method for the *ω* parameter for the Pacific plate.

**Figure 15.** Results of the sequential method for the Φ parameter for the African plate.

**Figure 16.** Results of the sequential method for the Λ parameter for the African plate.

The first plate that was analysed was the South American plate, which is bordered by the African and Nazca plates as well as several smaller ones with the oceanic lithosphere. It moves northwest at a rate of approximately 3 cm/year towards the North American plate, moving continuously away from Africa. It consists of three main geological units: the South American craton, which covers the northern, eastern, and central parts of the continent; the Palaeozoic Patagonian platform in the southeast; and the Andes, an alpine mountain range that stretches along the western boundary of the plate. In total, 29 GNSS stations

were used to determine the motion parameters of the South American plate. The motion parameters calculated for all stations in total are: Φ = –19.03 ± 0.20◦, Λ = −119.78 ± 0.39◦, and *ω* = 0.117 ± 0.001◦/Ma, which is presented, respectively, in Figures 3–5 and in Appendix A. The comparison of the results with the values obtained in a previous study for the DORIS technique [39] demonstrated that the consistency of results was similar to the value of formal errors, and the differences were 1.27◦, 1.57◦, and 0.016◦/Ma for the Φ, Λ, and *ω* parameters, respectively. For the other techniques, i.e., SLR and VLBI, the motion parameters of this plate were not calculated due to the lack of the required number of stations.

The solution becomes stable after 10, 11, and 7 steps of the sequential method, respectively, for the Φ, Λ, and *ω* parameters. At that stage, the changes in the values of the calculated parameters after adding subsequent stations to the process do not exceed the formal error. Hence, one may assume that the minimum number of stations required to determine the motion parameters of this plate is approximately 13. Due to high values of formal errors and discrepancies with the final values of the determined Φ, Λ, and *ω* parameters (please refer to Table 1), the following stations were rejected from the solutions: Valparaiso, Antuco, Concepcion, Quito III, Arequipa, and Callao (red dots on Figure 2). These stations accumulate interseismic strain associated with locking of the Peru-Chile megathrust, and therefore are moving east relative to the South American plate [45]. The Quito III station is located near the boundaries of four smaller plates: Cocos, Caribbean, Nazca, and North Andes. Including it in the solution causes a change in the Φ, Λ, and *ω* parameters, respectively, by approximately 3◦, 10◦, and 0.01◦/Ma. The other stations: Callao, Arequipa, Valparaiso, Concepcion, and Antuco are situated along the boundary of the Nazca plate. Among them, the Callao station has the highest influence on the change in motion parameters, namely parameter Φ by approximately 6◦, parameter Λ by approximately 11◦, and parameter *ω* by −0.016◦/Ma.



The motion parameters of the Australian plate were determined based on the position and velocity of 20 stations. Their final values are: Φ = 32.94 ± 0.05◦, Λ = 37.70 ± 0.12◦, and *ω* = 0.624 ± 0.001◦/Ma. The results of the solution for each step of the sequential method are presented in Figures 6–8 and in Appendix B. The solutions become stable for Φ, Λ, and *ω* parameters, respectively, after 10, 11, and 4 steps of the sequential method. Hence, one may assume that the minimum number of stations required to determine the motion parameters of this plate is approximately 12. The Australian plate is the most tectonically stable among all of the analysed plates, it moves northeast towards the Eurasian and Pacific

plate at a rate of approximately 6–7 cm/year. The greater part of the Australian plate is occupied by the Precambrian Craton, called the Australian Craton, which is adjacent to the structure of the Flinders Ranges and the Barrier Ranges, and the structure of the Great Dividing Range [46]. The tectonic stability of this plate is reflected in the formal errors of the determined motion parameters, i.e., their values are lower than those of the remaining plates. Similar findings have been observed in previous studies on the SLR and DORIS techniques [37–39]. The comparison of the results with the values obtained in an earlier study on the SLR, DORIS, and VLBI techniques, the highest compatibility of results was found with the VLBI technique [40], and the lowest one with the SLR technique [37]. The differences in the determined Φ, Λ, and *ω* values for the GNSS and VLBI techniques are, respectively, 0.31◦, 0.29◦, and 0.007◦/Ma, while, for the GNSS and SLR techniques, they are 1.52◦, −1.78◦, and 0.007◦/Ma, respectively, for Φ, Λ, and *ω*. Two stations (Wellington and Coco Island) that disturb the result of calculating the Φ and Λ parameters by approximately 3◦ were found on the Australian plate (please refer to Table 1). Including these stations in the determination of the parameters also increases the formal errors of the parameters multiple times, therefore, these stations were rejected from the solutions. The Wellington station is located at the boundary with the Pacific plate near the so-called Alpine Fault, and the Coco Island station is located near the boundary with the Capricorn and Sunda small plates.

The next plate, the Antarctic plate is bordered by the African, South American, Australian, Pacific, Nazca, and Scotia plates. Within the plate, three main geological units can be distinguished: the Antarctic craton, which covers the eastern part of the continent; the Palaeozoic platform in the western part; and the Alpine fold zone of the Antarctic Peninsula. The Antarctic plate moves in the northeast direction (the part bordering the South American plate) and the southern direction (the part bordering the Australian plate) at a rate of about 1–1.5 cm/year. The following motion parameter values were obtained for the Antarctic plate: Φ = 61.54 ± 0.30◦, Λ = −123.01 ± 0.49◦, and *ω* = 0.241 ± 0.003◦/Ma. The results of the sequential method are depicted in Figures 9–11 and presented in Appendix C. The solution was based on 13 stations, whose positions are shown in Figure 2. The inclusion of the King George Island station located near the boundary with the Shetland microplate and the Scotia plate, in the area of the alpine fold of the Antarctic Peninsula, results in an approximately 10-fold increase in the formal errors and a change in the values of the Φ and Λ parameters by about 6◦ (Table 1); this station was not included in the calculations.

In general, the formal errors of the Φ, Λ, and *ω* parameters for the Antarctic plate were the highest among all of the analysed plates. The comparison of the obtained results to the solutions for the DORIS [39] and VLBI [40] techniques revealed that the compatibility was higher for DORIS. The differences in the determined values between the GNSS and DORIS techniques are 1.26◦ for Φ, 1.89◦ for Λ, and −0.009◦/Ma for *ω*, while the differences between GNSS and VLBI are, respectively, 2.26◦, 4.64◦, and 0.025◦/Ma. The stability of the solution of the Φ, Λ, and *ω* parameters appear after five steps of the sequential method. At that stage, the changes in the values of the calculated parameters, after the addition of another station to the process, do not exceed the formal error. Hence, one may assume that the minimum number of stations required to determine the motion parameters of this plate is approximately 7.

The sequential method for the motion parameters for the Pacific plate are presented in Figures 12–14 and in Appendix D. It is the largest tectonic plate, and, within it, there are areas of very high tectonic activity, the so-called Ring of Fire. The Pacific plate moves northwest towards the Eurasian and Australian plates at a rate of approximately 6–10 cm/year. In the solution, 19 stations were included. Their locations are shown in Figure 2, and their names are listed in Appendix D. The final values of the motion parameters of the Pacific plate are: Φ = −62.45 ± 0.07◦, Λ = 111.01 ± 0.14◦, and *ω* = 0.667 ± 0.001◦/Ma. Similar values were obtained in a previous study for the SLR, DORIS, and VLBI techniques [38]. The smallest differences were noted in the comparison with the SLR technique: 0.05◦ for the Φ parameter, 2.50◦ for the Λ parameter, and the value of the *ω* parameter was the same.

The stability of the solution of the Φ, Λ, and *ω* parameters appear, respectively, after 12, 15, and 5 steps of the sequential method. Adding further stations, up to 19 stations, resulted in a change of the Φ, Λ, and *ω* parameters by values that did not exceed formal errors. Hence, one may assume that the minimum number of stations required to determine the motion parameters of this plate is approximately 17. The solution took into consideration the Point Reyes Lig. station located on the North American continent that, on the one hand, its movement was compatible with that of the Pacific plate and it did not have a negative effect on the solution of the Φ, Λ, and *ω* parameters (please refer to Appendix D). On the other hand, the Nuku Alofa station was rejected, as it caused a multi-fold increase in the values of formal errors for the Φ, Λ, and *ω* parameters (please refer to Table 1) and changed their values by approximately 5◦ (Φ and Λ) and by 0.020◦/Ma (*ω*).

Figures 15–17 depict the results of the sequential method for the motion parameters of the African plate. It moves northeast towards the Eurasian and Arabian plates. The main geological unit is the Precambrian craton (the so–called African Megacraton) with a system of tectonic grabens, forming the East African Rift system. The sequential method was based on 25 GNSS stations, which are listed in Appendix E, and their locations are presented in Figure 2. The final values of the Φ, Λ, and *ω* parameters equal: Φ = 49.15 ± 0.10◦, Λ = −80.82 ± 0.30◦, and *ω* = 0.267 ± 0.001◦/Ma. The obtained values of plate motion parameters in the first few steps of calculations were significantly divergent from the final results, as shown in Appendix E. Adding further stations in the calculation process led to the stabilisation of the results, until final values were reached based on all 25 stations. The stability of the results for the Φ, Λ, and *ω* parameters was noted, respectively, for 10, 7, and 7 stations used in the solution. Hence, the minimum number of stations required to ensure a stable solution is approximately 12 stations. The results were rather compatible with the values obtained in previous studies for the SLR [37] and DORIS [39] techniques, and the differences are, respectively, −1.63◦ and −0.38◦ for Φ, −4.00◦ and −1.70◦ for Λ, and −0.016◦/Ma and 0.017◦/Ma for *ω*. The Awra and Marion Island stations were rejected from the solution. The Awra station (located on the African continent near the boundary with the Arabian plate) contributed to a change in the Φ parameter by approximately 2.5◦, Λ by approximately 2◦, and *ω* by approximately 0.02◦/Ma, while the Marion Island (located on an island near the boundary with the Antarctic plate) station changed the values by approximately 1.5◦ for Φ, by approximately 3.5◦ for Λ, and by approximately 0.01◦/Ma for *ω* (please refer to Table 1). These stations also caused an approximately six-fold increase in the value of the formal errors in the determined parameters. The following stations are included in the solution: Addis Ababa, Mbarara, Tanzania CGPS, and Richardsbay. Although they are located on the Somalia and Lwandle small plates, they do not have a negative effect on the results of the solution (Appendix E).

### *Comparison with Geological Model NNR-MORVEL56 and Geodetic ITRF2014 Plate Motion Model (PMM)*

Geological models are developed based on geophysical observations, such as sea floor spreading rates, earthquake slip vectors, and transform fault azimuths. The most commonly used geological model is NNR-NUVEL1A [24] that originates from NNR-NUVEL1 [23]. It has recently been replaced with a new model, namely NNR-MORVEL56 [26], which is considered to be better than NNR-NUVEL1A. The NNR-MORVEL56 model was determined from more and higher quality spreading rates and azimuths. Moreover, it excluded circum-Pacific data (earthquake slip vectors and Pacific North America spreading rates) that were biased measures of relative plate velocity, addressed in [26]. The comparison of the results of the solutions of motion parameters of the specific tectonic plates with the NNR-MORVEL56 geological model should be approached with caution. The observations that are used for creating geological models are limited to plate boundaries, where local deformations occur quite often, and therefore the obtained results are not always representative of the movement of the whole plate. Moreover, geological models provide an average movement of individual plates across a very long period of time, which may range from hundreds of thousands to millions of years, and due to this, they may not reflect the

potential current speeding up or slowing down of specific tectonic plates, which, however, are recorded by very precise space techniques: SLR, DORIS, VLBI, and GNSS. Nevertheless, the ITRF2000 was defined based on the tectonic plate movement obtained from the NNR-NUVEL1A geological model. Subsequent solutions of the ITRF were adopted to the previous ones: ITRF2005 to ITRF2000, ITRF2008 to ITRF2005, and ITRF2014 to ITRF2008, but they are still indirectly linked to NNR-NUVEL1A. The question about the use of the new geological model, i.e., the NNR-MORVEL56, in future solutions of the ITRF, is still important and was discussed in [28]. To date, it has been found that the angular velocities in NNR-MORVEL56 differ significantly from those in NNR-NUVEL1A for all plates [5,26]. A detailed comparative analysis of these models was conducted by Argus et al. [26] and it will not be discussed here. However, the comparison of the motion parameters of individual plates that were determined in this study with the NNR-MORVEL56 model is presented in Table 2.


**Table 2.** Comparison with the NNR-MORVEL56 model.

The calculated differences between the values of the Φ, Λ, and *ω* parameters for the NNR-MORVEL 56 model and the results of the solution presented in this study revealed that the most similar values were obtained for the Australian plate: the Φ parameter differed by approximately 1◦, the Λ parameter by approximately 0.2◦, and the *ω* parameter by 0.008◦/Ma, which corresponded to approximately 1 mm/year. A difference of approximately 1◦ for the Φ parameter was also obtained for the Pacific plate, although for the Λ parameter, the difference was higher and approximately 4◦, while the angular rotation speed, *ω*, differed by approximately −0.02◦/Ma, which corresponded to approximately 2 mm/year. Regarding the South American and Antarctic plates, the difference in the values of the Φ parameter was approximately 4◦, and for the Λ parameter, approximately 7◦ (the South American plate) and approximately 5◦ (the Antarctic plate). The angular rotation speed *ω* differed by −0.008◦/Ma for the South American plate and by 0.009◦/Ma for the Antarctic plate, which corresponded to approximately 1 mm/year. The NNR-MORVEL56 model does not present the results for the African plate, which distinguishes two plates within it, namely Nubia and Somalia; hence, the inability to compare the obtained results. Nubia covers approximately 95% of the surface area of the African continent and the area to the West, towards the boundary of the South American plate, while the Somalia plate covers about 5% of the continent (the Eastern part) and the area to the East to the boundary with the Australian plate and the smaller tectonic plate, i.e., the Indian plate.

The ITRF2014 PMM [5] is the geodetic model describing the motion of 11 tectonic plates (the major plates and several selected small plates). It is dedicated to the currently in force ITRF2014 frame [28]. In developing it, horizontal velocities of a subset of the ITRF2014 stations of all space techniques in a combined solution (SLR + DORIS + VLBI + GNSS) were used, localized away from plate boundaries and deforming zones. For the South American plate, it was a total of 30 stations apart from the GNSS stations also 2 DORIS ones); for the Australian plate, it was 36 stations (including five DORIS, five VLBI, and three SLR stations); for the Pacific plate, it was 18 stations (including four DORIS, three VLBI, and one SLR); for the Antarctic plate, it was seven stations (GNSS only). Similar to the NNR-MORVEL 56 model, no parameters were determined for the African plate. In the ITRF2014 PMM, the plate motion was described providing specific elements of the pole of rotation: *ω*x, *ω*y, *ω*z, and the angular rotation speed *ω*. In order to compare the results, the values of geographical latitude (Φ) and longitude (Λ) of the pole of rotation determined in this study for each plate were calculated with the use of the formulas (Equation (1)) into *ω*x, *ω*y, and *ω*z. The comparison is presented in Table 3. As it can be seen from that table, the largest differences are noted for the South American plate, whereas small differences can be noted for the Australian plate. From the analysis of separate components of the pole of rotation (*ω*x, *ω*y, and *ω*z), it becomes evident that the largest differences for the *ω*<sup>x</sup> component occur for the South American plate (−0.082 mas/year) and the Antarctic plate (−0.023 mas/year); for the *ω*<sup>y</sup> component, the largest differences occur for the South American plate (0.050 mas/year) and the Australian plate (0.029 mas/year); while for the *ω*<sup>z</sup> component, the largest differences occur for the Antarctic plate (−0.088 mas/year) and the Pacific plate (−0.057 mas/year). For the rotation angular velocity (*ω*), the biggest difference is in the case of the Antarctic plate (−0.022◦/Ma) and the Pacific plate (0.012◦/Ma), which corresponds to approximately 2 mm/year and 1 mm/year, respectively. For the remaining plates the differences do not exceed 1 mm/year.


**Table 3.** Comparison with the ITRF2014 PMM.

In general, it can be stated that the agreement with the NNR-MORVEL56 and ITRF2014 PMM models is good for all plates.

### **4. Conclusions**

In this study, the motion parameters, i.e., the latitude (Φ) and longitude (Λ) of the rotation pole, and the angular rotation speed (*ω*) of five major lithospheric plates (South American, African, Australian, Antarctic, and Pacific plates) were determined. The observational material includes the positions and velocities of 106 highly quality GNSS stations in the ITRF2014. As many as 29 of these stations are located on the South American plate, 13 stations on the Antarctic plate, 20 stations on the Australian plate, 19 stations on the

Pacific plate, and 25 stations on the African plate. The most accurate solutions of the Φ, Λ, and *ω* parameters were determined for the Australian and Pacific plates, while the least accurate were determined for the Antarctic plate. The comparison of the obtained results to previously conducted research on the SLR [37,38], DORIS [38,39], and VLBI [38,40] techniques revealed high compatibility. The most similar results were obtained for the comparison with the VLBI technique for the Australian plate, the SLR technique for the Pacific plate, and the DORIS technique for the African, Antarctic, and South American plates. The most accurate solutions for the Φ, Λ, and *ω* parameters were obtained with the GNSS technique as compared with the other techniques. Because of the dense coverage of the Earth with GNSS stations, this is the only space technique that offers the possibility to determine the motion parameters of all the major lithospheric plates. The applied sequential method allowed us to define the minimum number of stations that ensured a stable solution and to indicate the stations that negatively affected the result of the solution. The minimum number of stations that should be used to determine the Φ, Λ, and *ω* parameters to guarantee a stable solution differs depending on the individual plates. For the South American and Australian plates, this number is approximately 13 stations; 7 stations for the Antarctic plate; 17 stations for the Pacific plate; and 12 stations for the African plate. Including a larger number of stations in the calculations does not have a significant influence on the values of the determined Φ, Λ, and *ω* parameters and formal errors. A total number of 12 stations that disturbed the solutions of the Φ, Λ, and *ω* parameters and increased the values of formal errors were found; these stations were rejected from the solution. Six of these stations are located on the South American plate (Valparaiso, Antuco, Concepcion, Quito III, Arequipa, and Callao), two stations on the Australian (Wellington and Coco Island) and African (Awra and Marion Island) plates, and one on the Antarctic (King George Island) and Pacific (Nuku Alofa) plates. Including them in the solution leads to a change in the Φ and Λ parameters in the range of 2–11◦, and of the *ω* parameter in the range from −0.016 to 0.058◦/Ma.

The values that were the most similar to the current geological model NNR-MORVEL56 [26] were obtained for the Australian plate (the differences do not exceed 1◦ for the Φ and the Λ parameters, and 0.008◦/Ma for the *ω* parameter). The differences for the other plates are higher, ranging from approximately 1 to 7◦ for the Φ and the Λ parameters, and from −0.016 to 0.008◦/Ma for the *ω* parameter. These differences may indicate a current slowing down or speeding up of the movement of certain tectonic plates, which are detected by the very precise GNSS space technique. Comparing the results with the geodetic model ITRF2014 PMM [5], one can find a high agreement between them, the largest differences are found for the South American plate (*ω*<sup>x</sup> and *ω*<sup>y</sup> components) and the Antarctic plate (*ω*<sup>z</sup> component). The rotation angular velocity (*ω*) differs within the range from −0.022◦/Ma (the Antarctic plate) to 0.012◦/Ma (the Pacific plate).

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The author declares no conflict of interest.

### **Appendix A**

Determined plate motion parameters and their formal errors of the South American plate for the GNSS network using the sequential method.


### **Appendix B**

Determined plate motion parameters and their formal errors of the Australian plate for the GNSS network using the sequential method.


### **Appendix C**

Determined plate motion parameters and their formal errors of the Antarctic plate for the GNSS network using the sequential method.


### **Appendix D**

Determined plate motion parameters and their formal errors of the African plate for the GNSS network using the sequential method.


### **Appendix E**

Determined plate motion parameters and their formal errors of the African plate for the GNSS network using the sequential method.



### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Sensors* Editorial Office E-mail: sensors@mdpi.com www.mdpi.com/journal/sensors

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel: +41 61 683 77 34

www.mdpi.com

ISBN 978-3-0365-5362-7