Unsupervised Characterization of Water Composition with UAV-Based Hyperspectral Imaging and Generative Topographic Mapping

Abstract

Unmanned aerial vehicles equipped with hyperspectral imagers have emerged as an essential technology for the characterization of inland water bodies. The high spectral and spatial resolutions of these systems enable the retrieval of a plethora of optically active water quality parameters via band ratio algorithms and machine learning methods. However, fitting and validating these models requires access to sufficient quantities of in situ reference data which are time-consuming and expensive to obtain. In this study, we demonstrate how Generative Topographic Mapping (GTM), a probabilistic realization of the self-organizing map, can be used to visualize high-dimensional hyperspectral imagery and extract spectral signatures corresponding to unique endmembers present in the water. Using data collected across a North Texas pond, we first apply GTM to visualize the distribution of captured reflectance spectra, revealing the small-scale spatial variability of the water composition. Next, we demonstrate how the nodes of the fitted GTM can be interpreted as unique spectral endmembers. Using extracted endmembers together with the normalized spectral similarity score, we are able to efficiently map the abundance of nearshore algae, as well as the evolution of a rhodamine tracer dye used to simulate water contamination by a localized source.

1. Introduction

Inland water bodies present a unique challenge to characterization by remote sensing imagery due to their complex spectral characteristics and small-scale spatial variability. The broad bands of multispectral imagers coupled with the irregular shape of lakes and rivers result in pixels with highly mixed signals that are easily dominated by reflectance from shore and nearshore vegetation sources [1,2]. Recently, the combination of hyperspectral imaging with unmanned aerial vehicles (UAVs), such as drones, has emerged as a powerful approach to simultaneously address the spectral, spatial, and temporal limitations of traditional high-altitude and satellite-based collection [3,4]. UAVs are significantly less expensive to deploy than their satellite- or aircraft-based remote sensing counterparts, and low-altitude flights enable centimeter-scale sampling while limiting the need for complicated atmospheric corrections [5]. However, the significant increase in the data volume generated by these systems presents a new challenge, namely, how to efficiently extract water quality parameters of interest from intricate pixel spectra.

Significant research efforts have focused on the development of techniques and algorithms to retrieve water quality parameters from UAV-captured hyperspectral images (HSI). Onboard computers installed alongside hyperspectral imagers enable the rapid evaluation of spectral indices from HSI band ratios [6]. These band ratios and polynomial combinations of bands have been used to successfully invert optically active water quality parameters such as turbidity directly from UAV-acquired imagery [7,8]. Supervised machine learning techniques such as tree-based models, support vector machines, and neural networks have also been used to estimate a wide range of parameters, such as colored dissolved organic matter, chlorophyll A, blue–green algae, and suspended sediment concentrations [9,10]. The calibration and evaluation of these data-driven models require a significant volume of coincident in situ data. This can be streamlined by coordinating UAV flights with reference data collection using autonomous robotic boats [11,12]. New proximal sensors for water quality parameters are continually being developed [13,14]. However, this approach relies on the prior knowledge of expected sources in order to select appropriate reference instruments for model validation. The presence of unanticipated contaminants cannot be directly identified in this sensing paradigm.

Extending the capabilities of UAV-based hyperspectral imaging to enable water quality monitoring in real-time scenarios where contaminant sources may not be known in advance requires two additional capabilities: dimensionality reduction techniques to permit the visual comparison of HSI and endmember extraction techniques which can identify spectral signatures corresponding to unique sources within the imaging scene. In remote sensing where reference data are typically sparse, many approaches have been explored. For example, principal component analysis (PCA) and t-distributed stochastic neighbor embedding (tSNE) are dimensionality reduction methods commonly used to reduce HSI to two or three dimensions for visualization [15,16]. Similarly for endmember extraction, there are a variety of established approaches including geometric methods like vertex component analysis, statistical methods like k-nearest neighbors and non-negative matrix factorization (NMF), and deep learning methods based on autoencoder architectures [17,18,19,20,21,22,23]. Methods based on linear mappings like PCA and NMF are often too restrictive for HSI, where the assumption of linear mixing is easily broken. However, the increased complexity of nonlinear methods like tSNE and autoencoders often lead to significant increases in computation time. An ideal approach should enable both visualization and nonlinear identification of relevant spectral endmembers.

The self-organizing map (SOM) developed by Teuvo Kohonen is an unsupervised machine learning method which nonlinearly maps high-dimensional data to the nodes of a two-dimensional grid [24]. By preserving the topological relationship between nodes during training, the SOM ensures that similar spectral signatures are mapped together such that related HSI pixels naturally cluster together. This presents an attractive compromise by enabling the simultaneous visualization of HSI data and endmember extraction via the weight vector associated with each SOM node [25,26,27]. When reference data are available, the SOM can be utilized to provide semisupervised labeling of HSI spectra [28]. Furthermore, Danielsen et al. demonstrated that the dimensionality reduction offered by the SOM can even be used for onboard data compression of HSIs acquired by a CubeSat [29]. Despite these clear capabilities, the SOM relies on a heuristic training algorithm with hyperparameters that can be challenging to tune and offers no direct probabilistic interpretation. To address these limitations, Bishop et al. developed generative topographic mapping (GTM), a probabilistic latent-variable model inspired by the SOM [30]. GTM has been utilized in a variety of domains, including drug design and chemical data visualization, but has yet to see adoption for the analysis of hyperspectral imagery [31,32,33].

In this paper, we explore the application of GTM to UAV-acquired HSI for the characterization water quality. Using data collected at a pond in Montague, North Texas, we first train a GTM using water-only pixels to produce a low-dimensional representation of the collected HSI. We use this mapping to explore the highly detailed small-scale variability within the pond and discuss how this can be used to guide reference data collection. Next, we demonstrate how GTM can be utilized to identify relevant spectral endmembers from a combined dataset including ground pixels, nearshore filamentous blue–green algae, water, and a simulated contaminant plume using a rhodamine tracer dye. Once identified, these endmembers can be used to rapidly map the abundance of spectral features within the pond. We demonstrate this capability by using GTM to map the abundance of filamentous blue–green algae near the shore, as well as the dispersion of a rhodamine dye plume.

2. Materials and Methods

In this study, we explore the use of GTM as a tool for dimensionality reduction and unsupervised endmember extraction of hyperspectral imagery. To this end, a dataset of HSI was collected at a pond in Montague, North Texas, on 23 November 2020 using a UAV-mounted hyperspectral imager configured as described in [11,12]. The pond spans an area <0.1 ${\mathrm{km}}^2$ and has a maximum depth of 3 m. Across multiple flights, many data cubes like the example shown in panel a of Figure 1 were collected. Additionally, rhodamine dye was released and subsequently imaged to simulate a localized contaminant source. Using the HSI shown in Figure 1, a set of exemplar spectra were visually identified corresponding to nearshore filamentous blue–green algae, water, rhodamine dye, and the ground (a combination of soil and dry grass). The location of these samples is shown in panel b and their corresponding reflectance spectra are visualized in panel c. In the remainder of this section, we first provide a detailed overview of the GTM algorithm. Next, we describe the UAV platform used for HSI collection and the various steps employed to process captured HSI. Finally, we describe the two case studies presented in this paper for the utilization of GTM as an HSI visualization tool and as an endmember extraction technique.

2.1. Generative Topographic Mapping

GTM is a probabilistic latent-variable model inspired by the SOM for visualizing and clustering high-dimensional data. Like the SOM, GTM assumes vectors $\mathbf{x}$ in the d-dimensional data space (here representing reflectance spectra) are constrained to a low-dimensional embedded manifold. The SOM describes this manifold using a regular grid of nodes, each having an associated weight vector defining the mapping from the manifold to the data space. The position of each data record $\mathbf{x}$ in the manifold is assigned to the position of the node whose weight vector has the minimum Euclidean distance to $\mathbf{x}$. An iterative training procedure updates the weight vectors of each node to fit the manifold to the data such that nodes near each other in the SOM grid correspond to similar records in the data space.

GTM mimics the grid of the SOM by assuming data are generated from latent variables $\xi$ that are constrained to the K-many nodes of a regular grid. This assumption corresponds to establishing a prior distribution on the latent space of the form

$$p\left(\xi \right)=\frac{1}{K}\sum _k^K\delta (\xi -{\xi }_k)$$

(1)

where $\delta (·)$ is the Dirac delta function.

Points $\xi$ in this latent space are mapped to the embedded manifold by a nonlinear function $\psi$ parameterized by weights W. However, since real data are rarely noise-free, this embedded manifold will not be perfectly thin. To account for this, the points $\xi$ are described in the data space by a radially symmetric Gaussian distribution, $\mathcal{N}(\psi \left(\xi \right),{\beta }^{-1})$, with mean $\psi \left(\xi \right)$ and precision $\beta$. For a dataset ${\left\{{\mathbf{x}}_n\right\}}_{n=1}^N$ consisting of N-many records, this choice yields a log-likelihood function of the form

$$\mathcal{L}(W,\beta )=\sum _n^{N}ln\left({\displaystyle \frac{1}{K}}\sum _k^{K}p({\mathbf{x}}_n\mid {\xi }_k,W,\beta )\right)$$

(2)

which can be maximized to obtain optimal W and ${\beta }^{-1}$.

The nonlinear mapping $\psi$ is typically chosen to be given by the sum of M-many radial basis functions (RBF) evenly distributed in the latent space such that $\psi \left(\xi \right)=W\varphi \left(\xi \right)$ with centers ${\mu }_m$ and width $\sigma$ so that

$${\varphi }_m\left(\xi \right)=exp\left(-{\displaystyle \frac{{\parallel \xi -{\mu }_m\parallel }^2}{2{\sigma }^2}}\right).$$

(3)

The width $\sigma$ is taken to be the distance between neighboring RBF centers multiplied by a scale factor, s. Together, the number of RBFs, M, and s are hyperparameters for the model which governs the smoothness of the resulting manifold in the data space. A visual representation of the GTM is shown in Figure 2. Additionally, sparsity of W can be enforced by introducing an additional hyperparameter $\alpha$ corresponding to a prior distribution over the weights given by

$$p(W\mid \alpha )={\left(\frac{\alpha }{2\pi }\right)}^{MD/2}exp\left(-\frac{\alpha }{2}{∥W∥}_F^2\right).$$

(4)

The occurrence of the sum within the natural logarithm in Equation (2) prevents an analytic solution for the maximum of $\mathcal{L}$. However, the form of $\psi$ enables an Expectation–Maximization routine which optimizes $\mathcal{L}$ by iteratively increasing a lower bound given by the expected complete-data log-likelihood [34]. During the expectation step of the fitting process, the posterior probability $p({\xi }_k\mid {\mathbf{x}}_n,W,\beta )$, i.e., the responsibility of the kth node for explaining the nth record, is computed as

$$R_{kn}=p({\xi }_k\mid {\mathbf{x}}_n,W,\beta )={\displaystyle \frac{p({\mathbf{x}}_n\mid {\xi }_k,W,\beta )}{{\displaystyle \sum _{k^{\prime }}^K}p({\mathbf{x}}_n\mid {\xi }_{k^{\prime }},W,\beta )}}.$$

(5)

Together, these responsibilities form a matrix with entries $R_{kn}$ which are kept fixed during the maximization step. The maximization step is then performed by updating the weights W and variance ${\beta }^{-1}$ according to

$$\begin{array}{cc}\hfill W_{\mathrm{new}}& ={\left({\mathsf{\Phi }}^{T}G\mathsf{\Phi }+{\displaystyle \frac{\alpha }{\beta }}I\right)}^{-1}{\mathsf{\Phi }}^{T}RX\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill \frac{1}{{\beta }_{\mathrm{new}}}& =\frac{1}{ND}\sum _n^N\sum _k^K{R}_{kn}{∥{\psi }_k-{\mathbf{x}}_n∥}^2\hfill \end{array}$$

(7)

where ${\mathsf{\Phi }}_{km}={\varphi }_m\left({\xi }_k\right)$, X is the data matrix formed by concatenating the records ${\mathbf{x}}_n$, and G is a diagonal matrix with $G_{kk}={\displaystyle \sum _n^N}{R}_{kn}$. This process is repeated until the log-likelihood converges to a predetermined tolerance level.

Slices $R_{[:,n]}$ of the matrix R define the responsibility of each latent node ${\xi }_k$ for the nth data record ${\mathbf{x}}_n$. Therefore, the final responsibility matrix after GTM training can be used to represent each record in the latent space by the following mean:

$${\widehat{\xi }}_n=\sum _k^K{R}_{kn}{\xi }_k.$$

(8)

A freely available implementation of the GTM algorithm was developed for this study and is accessible at [35]. The code is written in the Julia programming language and complies with the Machine Learning in Julia (MLJ) common interface [36,37].

2.2. UAV-Based Hyperspectral Imaging

A Freefly Alta-X autonomous quadcopter was used as the UAV platform in this study. This UAV was equipped with a Resonon Pika XC2 visible+near-infrared (VNIR) hyperspectral imager to capture HSI with 462 wavelengths per pixel ranging from 391 to 1011 nm. This imager is in a pushbroom configuration so that HSI are captured one scan-line at a time resulting in data cubes consisting of 1000 scan-lines with 1600 pixels each. Additionally, the camera includes an embedded GPS/INS unit to enable georectification of collected HSI. An upward-facing Ocean Optics UV-Vis NIR spectrometer with a cosine corrector was also included to provide measurements of the incident solar irradiance spectrum. The configuration of the UAV with the attached hyperspectral imager is shown in Figure 3. Data collection and processing were controlled by an attached Intel NUC small-form-factor computer. A detailed description of the UAV platform can be found in [11].

To account for the variability of incident light, raw HSI are converted to units of reflectance using the downwelling irradiance spectrum simultaneously captured with each HSI. With the hyperspectral imager oriented to nadir, the reflectance is given by

$$\rho \left(\lambda \right)=\pi L\left(\lambda \right)/E_d\left(\lambda \right)$$

(9)

where L is the spectral radiance, $E_d$ is the downwelling irradiance, and a factor of $\pi$ steradians results from assuming Lambertian (diffuse) upwelling radiance [38]. HSI collection was performed near solar noon to maximize the amount of incident sunlight illuminating the water. For the site in North Texas, this corresponded to an average solar zenith angle of ${54.9}^{\circ }$ for 23 November 2020. During flights, the hyperspectral imager was oriented to nadir, resulting in HSI with negligible sunglint effects.

After conversion to reflectance, each HSI must also be georectified to assign geographic coordinates to each pixel. The UAV flights were performed at an approximate altitude of 50 m above the water so that the imaging surface can be considered to be flat. Consequently, HSI were rapidly georectified to a 10 cm resolution using position and orientation data from the embedded GPS/INS, as outlined in refs. [39,40,41]. An example hyperspectral data cube is illustrated in Figure 1a where the log10-reflectance values are plotted along the z-axis for each pixel in the scene, and a psuedocolor image at the top of the data cube illustrates how the water would appear to the human eye from the perspective of the UAV.

As a final processing step before training each GTM, we limit the wavelengths of each HSI to $\lambda \le 900$ nm, as wavelengths above 900 nm showed significant noise. Additionally, each spectrum was rescaled to a peak value of $1.0$ to account for incident light variability between HSI.

2.3. GTM Case Studies

To explore the ability of the GTM to segment HSI pixels and aid in source identification, we first consider a dataset of water-only spectra consisting of >36,000 records identified from collected HSI by a normalized difference water index (NDWI) greater than $0.25$, where the NDWI is defined as

$$\mathrm{NDWI}={\displaystyle \frac{\rho \left(550\right)-\rho \left(860\right)}{\rho \left(550\right)+\rho \left(860\right)}}$$

(10)

where $\lambda =550$ and $\lambda =860$ nm were chosen to represent the green and NIR bands as described in ref. [42]. We then apply the trained GTM to all water-only pixels to visualize the distribution learned by the GTM and examine spectra associated with a subset of nodes in the latent space in order to assess the small-scale spatial variability within the pond.

Next, we consider a dataset of combined HSI pixels including both water and land. To simulate the dispersion of a potential contaminant source, a rhodamine tracer dye was released into the western portion of the pond and two additional UAV flights were used to collect HSI capturing the evolution of the resulting plume. From these HSI, a collection of >145,000 pixels were sampled for model training. To demonstrate the ability to extract relevant spectral endmembers from the trained GTM, we utilize the exemplar spectra corresponding to open water, ground, nearshore algae, and rhodamine dye visually identified in Figure 1. The responsibility matrix obtained by the trained GTM provides a probability distribution over the latent space nodes for every data record. We identify relevant endmembers for each reference spectrum via the set of nodes with nonzero responsibility. We then utilize the mapping $\psi$ to obtain the spectral representation of these nodes. Since the GTM is unsupervised, these extracted endmembers do not correspond to individual records but rather spectral features learned by the GTM.

Values for the hyperparameters m, s, and $\alpha$ are determined by training multiple GTM models and selecting values which minimize the Bayesian Information Criterion (BIC) given by

$$\mathrm{BIC}=Pln\left(N\right)-2\mathcal{L}$$

(11)

where P is the total number of model parameters, N is the number of records in the dataset, and $\mathcal{L}$ is the log-likelihood defined in Equation (2).

The normalized spectral similarity score (NS3) introduced by Nidamanuri and Zbell combines the root mean square (RMS) difference together with the spectral angle to provide a spectral distance function [43]. For two spectra ${\rho }_1\left(\lambda \right)$ and ${\rho }_2\left(\lambda \right)$, it is defined by

$$\mathrm{NS}3({\rho }_1,{\rho }_2)=\sqrt{RMS{({\rho }_1,{\rho }_2)}^2+{(1-cos\theta )}^2}$$

(12)

where

$$\begin{array}{cc}\hfill \mathrm{RMS}({\rho }_1,{\rho }_2)& =\sqrt{\frac{1}{N-1}\sum _{\lambda }{\left({\rho }_1\left(\lambda \right)-{\rho }_2\left(\lambda \right)\right)}^2}\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill cos\theta & =\frac{\langle {\rho }_1,{\rho }_2\rangle }{\parallel {\rho }_1\parallel \parallel {\rho }_2\parallel }.\hfill \end{array}$$

(14)

We use the NS3 together with extracted GTM endmembers to map the abundance of filamentous blue–green algae near the shore, as well as the evolution of the rhodamine dye plume. To do this, the GTM node with maximum responsibility is selected from the endmembers identified for the algae and rhodamine reference spectra. The distribution of NS3 values between each endmember and all HSI pixels is then used to select a suitable cutoff threshold. The area subsumed by HSI pixels with an NS3 below this threshold is then used to infer the spatial extent of algae and rhodamine in the pond.

3. Results

3.1. Water-Only Pixel Segmentation

A GTM with $K=32\times 32$ nodes was trained on the dataset of water-only pixels described in Section 2.3 in order to explore the distribution of reflectance spectra captured by the HSI. The resulting GTM is visualized in Figure 4. First, we note that the GTM has utilized most of the latent space as illustrated Figure 4b where the position corresponding to the mean responsibility of each data point ${\widehat{\xi }}_n$ has been plotted. For this water-only GTM, the spectra appear to be largely clustered toward the left edge, bottom, and right edge of the latent space. Spectral signatures corresponding to GTM nodes from the four corners and center of the latent space are shown in Figure 4c, illustrating the spectral variability represented across the latent space.

To visualize the distribution of HSI learned by the GTM, we can associate a color with each dimension of the latent space. In Figure 4, we used the red channel to represent ${\xi }_1$ and the blue channel to represent ${\xi }_2$. Applying the trained GTM to compute mean node responsibilities for all water pixels in collected HSI allows us to illustrate the spatial distribution of the spectra on a map, as shown in Figure 4c. Here, we observe a clear distinction between water near the shore and water in the middle of the pond. Additionally, the eastern alcove of the pond is significantly more blue than the rest of the water. We note that flow through this region is restricted by the small entrance near the cyan square shown in panel a. However, despite its similar depth and surface characteristics, the GTM distinguished this region from the rest of the water.

The close proximity of highly dissimilar GTM classes illustrated by sharp color gradients in the map reflects the small-scale spatial variability typical of inland water bodies like this pond. For example, water near the cyan diamond in panel a of Figure 4 includes both blue and red pixels in close proximity. As shown in Figure 4b, these colors correspond to the top-left and bottom-right corners of the GTM latent space. As the GTM model maps neighboring nodes to similar spectra, this region includes maximally dissimilar pixels. Furthermore, this point is away from the shore where the depth is approximately uniform. The precision parameter $\beta$ learned by the GTM enables the model to account for spectral variability so that effects from localized disturbances such as surface ripples are constrained to small displacements in the latent space. With these considerations in mind, contrast in this region of the pond therefore reflects real variability of water composition, surface sediments, and vegetation.

3.2. Endmember Extraction

A second GTM was trained on the combined dataset of HSI with pixels covering shore, water, and the rhodamine tracer dye release. The resulting latent space distribution is shown in Figure 5, where each sampleis plotted at the position of the mean node responsibility and colored by the normalized difference vegetation index (NDVI) computed from the original reflectance spectrum. The NDVI is a spectral index sensitive to variations in vegetation health with negative values corresponding to water, small positive values corresponding to sparse vegetation, and large values near 1 corresponding to dense vegetation [44]. From this, we see that the GTM can clearly separate spectra by vegetation content with high values concentrated to the left of the latent space and negative, water-based pixels concentrated to the right. Additionally, the position of exemplar spectra for algae, rhodamine, water, and ground points are included as color-filled circles, further illustrating the separation of distinct sources obtained by the GTM. We note that a portion of the nearshore algae was observed to float at the surface of the water near the shore. Consequently, the position of the exemplar spectrum in Figure 5 used for algae occurs at a non-negative NDVI. The accumulation of records with negative NDVI values in close proximity (in the latent space) to the selected algae spectrum reflects the presence of algae within the water.

As outlined in Section 2.1, there are three hyperparameters that need to be chosen to fit a GTM model: the number of RBF centers M = m² with m the number along each axis, the scale factor s which controls the RBF overlap, and the regularization parameter $\alpha$. To choose appropriate values, we performed a grid search for m values between 2 and 20, s values between $0.1$ and $3.0$, and $\alpha$ values between $0.001$ and $10.0$. The best values were determined as those which minimized the BIC, with $m=14$, $s=1.0$, and $\alpha =0.1$, respectively. Heatmaps comparing the BIC for different hyperparameter values are provided in Figure 6, and a table of the top 25 models is given in Appendix A. Since the matrix of RBF activations $\mathsf{\Phi }$ need only be computed once at the GTM initialization step, the number of latent nodes given by $K=k^2$ can be chosen to be large enough to ensure a smooth mapping. We found that a value of $k=32$ provided a sufficient number of GTM nodes without significantly impacting training time.

Examining the latent space distribution learned by the GTM provides an unsupervised method to extract endmember spectra corresponding to unique sources within the imaging scene. To demonstrate this clearly, we consider the exemplar spectra visually identified in Figure 1. Using the trained GTM, we can compute spectral signatures for each node in the latent space via the nonlinear mapping $\psi$. The responsibilities $R_{kn}$ therefore correspond to the contributions of the $kth$ node ${\xi }_k$ to the nth sample spectrum in the dataset. Endmembers are identified by nodes with nonzero responsibility values for each of the selected exemplar spectra and are plotted in Figure 7. We note that these extracted endmembers are able to accurately capture the shape of each exemplar spectrum including thin reflectance peaks. The GTM has also managed to smoothly interpolate through noisy wavelengths, as shown by the water and algae endmembers for $\lambda >800$ nm.

3.3. Abundance Mapping with NS3

Spectral endmembers extracted from the GTM can be used to map the abundance of water constituents in HSI by identifying pixels with a NS3 value below a given threshold. In Figure 8, we demonstrate this by using the extracted algae endmember spectrum to map the abundance of filamentous blue–green algae near the western shore of the pond. The distribution of NS3 values was multimodal with a majority of pixels above $0.4$ belonging to dissimilar spectra. Consequently, a cutoff threshold of $0.4275$ was chosen to distinguish algae values corresponding to the 20th-quantile of the NS3 distribution. The estimated area subsumed by algae is shown in Figure 8 and calculated to be $587.3$ m². Similarly, we are able to track the evolution of the rhodamine dye release into the western portion of the pond by computing the NS3 for HSI collected across multiple flights. Here, the same cutoff threshold was used corresponding to the 33rd-quantile of the NS3 distribution. In the left panel of Figure 9, we see that the dye plume initially encompasses an area of roughly $330.7$ m². A second flight performed 15 min later reveals the dye plume to have increased to an area of $1495.6$ m². The increase in NS3 values between flights reflects the dilution of dye as it diffused into the water.

4. Discussion

The application of UAV-based hyperspectral imaging (HSI) for water quality assessment is gaining significant traction due to its ability to efficiently capture detailed spectral data at high spatial resolutions. Most studies using UAVs have employed supervised techniques that map the captured spectra to specific water quality parameters of interest. For example, Lu et al. evaluated a variety of machine learning methods for the extraction of chlorophyll-a and suspended sediment concentrations from HSI using data from UAV flights over 33 sampling locations [10]. However, this approach relies on the collection of paired in situ data, which can be challenging to obtain in sufficient quantities to facilitate model training. Additionally, supervised methods require prior knowledge of expected sources in order to select and calibrate suitable reference instruments. Using models purpose-built for specific water quality parameters like chlorophyll-a or turbidity discards potential information contained in HSI, which can reveal unanticipated sources. Therefore, unsupervised methods which aid in the visualization of HSI and enable the identification of spectral endmembers within the imaging scene are needed to complement these supervised approaches.

Recently, some researchers have begun to explore endmember extraction using UAV-based imaging where the increased spatial resolution provided by UAV platforms is hypothesised to yield more pure pixels than their satellite counterparts. For instance, Alvarez et al. used multispectral UAV imagery to extract endmembers which were then applied to unmix remote sensing data for plant abundance estimation across a broad region of France [45]. Similarly, Gu et al. explored UAV-to-satellite hyperspectral unmixing by applying endmembers extracted from UAV-based HSI to satellite observations [46]. These studies underscore the potential of combining endmember extraction techniques with UAV-based HSI, which has yet to see widespread adoption for water quality analysis.

In this study, we explored the GTM as a unsupervised approach to simultaneously enable visualization of HSI and perform endmember extraction of spectral signatures. First, we showed how the representation of data in the latent space of the GTM given by the mean responsibility can be used to visualize the small-scale structures within inland water bodies as evidenced in Figure 4. In particular, we note that the sharp gradients in colors found when visualizing the spatial distribution of GTM nodes across the pond reflects significant spectral variability at the submeter scale. This has important consequences for water quality assessment where the location of in situ data collection will have a strong impact on the ability of any model to predict water quality parameters from HSI. Visualizing the distribution of spectra from collected HSI is therefore highly relevant to guide in situ data collection, as shifting the sampling location by as little as a meter can lead to significant differences in measured values.

Based on these observations, one clear application of the GTM for real-time water quality assessment is for intelligently guided reference data collection. In our previous work, we showed that coordinating UAV-based hyperspectral imaging with in situ data collection by an autonomous boat can dramatically improve the inversion of water quality parameters from HSI pixels [12]. However, the spatial distributions of parameters such as chlorophyll-a, blue–green algae, and temperature are often highly dissimilar posing a challenge for optimal route planning. Since GTM estimates the full distribution of reflectance spectra and not a single water quality parameter, one could construct a prize collecting travelling salesman problem (PC-TSP) which seeks to find the optimal route maximizing the area explored in the GTM latent space [47]. Similar approaches have been used to guide data collection with autonomous vehicles to optimize data quality subject to resource constraints [48].

The second application of the GTM presented in this study is for the unsupervised extraction of endmembers corresponding to unique sources observed in the HSI. Here, the GTM is an attractive choice, as it does not rely on the assumptions of linear mixing and the presence of pure pixels which are easily broken by the effects of multiple scattering in realistic scenarios [49]. Additionally, because the GTM is a probabilistic model, the values of model hyperparameters can be selected objectively using information criteria like the BIC. This is a clear advantage over similar methods like the SOM, on which the GTM is based. Endmembers identified for exemplar spectra corresponding to rhodamine, algae, ground, and water pixels demonstrate that the method can successfully identify spectral signatures corresponding to diverse sources. If a set of labeled spectra for known sources are available, their representation in the GTM latent space could be used to perform a semisupervised classification, similar to the SOM approach developed by Riese et al. [28]. For example, in clear waters where variation in sediments and vegetation at the pond floor will contribute to observed spectral variability, the distribution learned by the GTM can be paired with in situ data to classify the surface type.

Once endmembers are identified, estimating their abundance using the NS3 provides a quick method to map the distribution of sources in water. We note that the spatial distribution of algae mapped in Figure 8 realistically reflects clustering of algae near the shore. Additionally, the ability for the UAV to quickly survey the same area in rapid succession is highly advantageous for tracking the diffusion of point sources, as demonstrated in Figure 9 for the rhodmaine dye release.

The primary limitation of the GTM is that the number of nodes increases exponentially with the dimension of the latent space. However, when constrained to two dimensions, in order to enable visualization of the resulting map, the number of latent nodes has a negligible impact compared with the size of the dataset on which it is trained. Additionally, the GTM considers individual HSI pixels and does not exploit spatial structure like other methods, such as convolutional autoencoders and non-negative matrix factorization using superpixels [19,23]. Extensions to GTM have been proposed to enable batch training for large datasets, as well as manifold-aligned noise models which replace the fixed precision parameter $\beta$ with a full covariance matrix [50].

Finally, we note that the representation obtained by GTM can be used for nonlinear feature extraction to improve supervised models. Traditionally, PCA is used as a common preprocessing technique to reduce high-dimensional HSI by keeping the first r-many principal components. For example, Uddin et al. report improved classification of HSI by using PCA to extract features for a support vector machine [51]. GTM can similarly be used to provide a sparse representation of the input data via the latent node responsibilities $R_{kn}$ obtained for each record.

5. Conclusions

In this study, we present GTM as a useful unsupervised method for the visualization of UAV-based hyperspectral imagery and associated extraction of spectral endmembers. Using data collected at a North Texas pond, we demonstrate how the latent space of the GTM can be used to visualize the distribution of observed reflectance spectra revealing the small-scale spatial variability of water composition. Spectral signatures extracted from GTM nodes are used to successfully map the abundance of algae near the shore and to track the evolution of a rhodamine tracer dye plume. These examples illustrate the power of combining unsupervised learning with UAV-based hyperspectral imaging for the characterization of water composition. Future work will further develop the GTM as a tool to guide in situ data collection and enable contaminant localization for real-time applications.