Advancements in Remote Compressive Hyperspectral Imaging: Adaptive Sampling with Low-Rank Tensor Image Reconstruction

Abstract

We advanced the practical development of compressive hyperspectral cameras for remote sensing scenarios with a design that simultaneously compresses and captures high-quality spectral information of a scene via configurable measurements. We built a prototype imaging system that is compatible with light-modulation devices that encode the incoming spectrum. The sensing approach enables a substantial reduction in the volume of data collected and transmitted, facilitating large-scale remote hyperspectral imaging. A main advantage of our sensing design is that it allows for adaptive sampling. When prior information of a survey region is available or gained, the modulation patterns can be re-programmed to efficiently sample and detect desired endmembers. Given target spectral signatures, we propose an optimization scheme that guides the encoding process. The approach severely reduces the number of required sampling patterns, with the ability to achieve image segmentation and correct distortions. Additionally, to decode the modulated data, we considered a novel reconstruction algorithm suited for large-scale images. The computational methodology leverages the multidimensional structure and redundant representation of hyperspectral images via the canonical polyadic decomposition of multiway arrays. Under realistic remote sensing scenarios, we demonstrated the efficiency of our approach with several data sets collected by our prototype camera and reconstructed by our low-rank tensor decoder.

1. Introduction

Hyperspectral (HS) cameras are advanced imaging devices that capture a wide range of information across the electromagnetic spectrum. Unlike traditional cameras that detect only three primary colors (red, green, and blue), hyperspectral cameras can capture hundreds or even thousands of narrow spectral bands, providing highly detailed and precise data about the composition and properties of objects. Hyperspectral cameras excel at distinguishing subtle differences in the spectral signatures of objects. This enables the identification and characterization of materials, even those with similar visual appearances. It is particularly useful in areas such as remote sensing, environmental monitoring [1,2], and agriculture, where precise material identification and image segmentation are crucial.

However, the enhanced spectral information leads to a substantial rise in the volume of raw data captured. In situations where both high spatial and spectral resolution are necessary, the required bandwidths might pose constraints on effectively transmitting images from remote survey sites. This limitation becomes particularly critical in compact devices like CubeSats [3], where the availability of on-payload computing hardware for storage and transmission is restricted.

To overcome such constraints, the practice of compressive hyperspectral imaging was proposed to reduce the volume of collected data, drawing inspiration from the field of compressive sensing [4,5,6,7,8,9,10]. Unlike conventional data acquisition methods, these approaches simultaneously sample and compress HS images by modulating (or encoding) the incoming spectrum. The primary advantage of these novel methods is a significant reduction in the acquisition time and the volume of raw data collected. In this compressive HS imaging framework, dense on-site data acquisition can be alleviated by shifting the focus to large-scale optimization as a post-acquisition step to achieve data reconstruction (i.e., decoding). This acquisition regime is ideal for remote sensing situations, where hardware and transmission requisites are reduced and computational imaging can be achieved at a ground station (as illustrated in Figure 1).

While introducing a distinct sensing paradigm with the potential to miniaturize HS cameras, further efforts are necessary to advance compressive HS imaging for remote sensing missions. In particular, previous studies predominantly focused on numerical aspects. Most experiments in the literature consider pre-processed data captured by cameras whose hardware is not consistent with compressive HS imaging or remote settings with limited chassis space. To comprehensively understand the limitations of compressive sensing-based approaches, it is imperative to capture and experiment with raw HS footage compatible with light-modulation-based sensing.

Furthermore, previous computational approaches for decoding HS images do not scale to the high resolution demanded by modern remote sensing missions. Most studies in the literature conduct tests on small benchmark datasets, which typically have spatial resolutions of 300 × 300 pixels or less (e.g., the reduced AVIRIS image of Cuprite [9,10]). While such benchmark datasets are crucial for a fair comparison of an algorithm’s performance, reconstruction experiments on images of practical resolution would be most informative to advance the technology.

Finally, while the main goal of compressive HS cameras is to reduce the volume of acquired data, another advantage of these designs is the ability to re-configure the sensing patterns on the fly. With prior information of a scene or signatures of interest, the encoding procedure can be adapted to more efficiently extract targeted information in a way that minimizes pattern transmission requisites. Such capabilities are rarely considered in the literature for improved spectrum reconstruction or image segmentation, but are crucial in remote acquisition scenarios.

This study attempted to address these gaps in order to advance the practice of remote compressive hyperspectral imaging. Our contributions can be summarized as follows:

• We designed a prototype imaging system and captured high-resolution field data. The sampling technique we considered is similar to sensing methods previously proposed in the literature [4,5,6,7,8,9,10,11] that encode the spectral signature at each pixel. Our primary contribution lay in the construction of a camera to gather real data and evaluate the effectiveness of this sensing paradigm in practical settings.
• We explored adaptive sampling to minimize the encoding complexity and transmission requisites. Our methodology capitalized on remote sensing scenarios where spectral information was gained through periodic surveys. We used estimated endmembers to design encoding patterns that re-configured light modulation hardware (e.g., digital micromirror devices [12]) for subsequent acquisitions. By incorporating spectra of interest, we propose an optimization technique to determine minimal encoding patterns that reduce data transmission while achieving efficient sampling and segmentation.
• We validated a recently proposed image reconstruction technique. This methodology built upon our initial work [13] by extending the concept of compressive sensing [14] to encompass multiway arrays that adhere to a low-rank tensor structure rather than a sparse signal model. Through experiments on footage collected by our prototype camera, we demonstrated robust imaging that reduced the sampling requisites by a factor of 20.

Our focus was on satellite-based remote sensing missions that conducted multiple surveys of an area. In such situations, encoding patterns (known as the codebook) must be transmitted to the payload in real time to re-program the onboard hardware for subsequent surveys. An illustration of such a sampling workflow is shown in Figure 1. Our adaptive sampling methodology is particularly well suited for this scenario, and was designed to minimize the bandwidth requisites by significantly reducing the number of sampling patterns that efficiently extract the desired spectral information.

The remainder of this paper proceeds as follows: The remainder of Section 1 discusses related works in the literature. Section 2 elaborates on the spectral sensing scheme and specifies the design of our prototype camera. Section 2.3 showcases our acquired field data. Section 2.4 considers the codebook design, including our adaptive sampling methodology. Section 2.5 discusses the tensor-based image reconstruction (decoding) approach, where Section 2.5.1 and Section 2.5.2 provide more details on tensor decompositions and pseudocode for our program. Section 3 provides numerical experiments, presenting reconstructed images captured by our prototype HS camera and comparing our approach to compatible algorithms in the literature. Finally, we conclude with ending remarks and future work in Section 4.

Related Works

The field of compressive hyperspectral imaging has been active since the advent of compressive sensing [15,16]. Most studies focused on sampling and reconstruction approaches [4,5,6,7,8,10,17,18,19,20,21,22,23]. Many approaches require auxiliary data (e.g., training data, RGB images, or a spectral library) [24,25,26,27] or apply sensing that is not compatible with our spectral-only encoding scheme [28,29,30]. Furthermore, fewer authors focused on hardware development to collect encoded hyperspectral measurements. We provide a brief overview of previous approaches most related to our work in order to highlight the novelties of our paper.

In terms of algorithms, to reconstruct HS data from encoded measurements, most approaches in our literature review utilized vector- or matrix-based processing. In other words, these works processed 3D HS volumes by sequentially considering matrix slices or vector fibers of the tensor. Notably, sparse signal reconstruction continues to be a popular approach [31,32,33,34]. However, the approaches in [19,21,22,23] utilized the 3D structure of HS cubes to improve image reconstruction via the tensor ring and Tucker decompositions. Our work also exploited the high dimensional structure of HS data, but we considered the canonical polyadic (CP) decomposition. The CP decomposition was the first method proposed to extend the singular value decomposition to higher-order arrays [35,36]. Compression capabilities of the CP decomposition when applied to HS images are demonstrated in the literature [37,38,39,40,41,42]. Despite its popularity, this factorization approach was not considered for compressive hyperspectral imaging until recently [13]. In this work, we showed that this relatively simple tensor representation was sufficient to provide reliable compressive imaging on real data and also outperformed several lower-dimensional alternatives.

Hardware development in the compressive HS imaging is more rare. Our design was built upon our prior work [11,43], where a DMD-based spectrometer was converted into an HS camera. This previous prototype modified a DLP NIRScan Nano EVM [44] that covered near-infrared bands (900–1400 nm) to provide an imager capable of measuring the visible spectral range of 400–700 nm. Recently, the authors in [30] developed a hyperspectral video imaging system based on a single pixel detector. Their approach applied a spatial–spectral encoding scheme, and severely compressed measurements were obtained after a sparse selection from the full dense measurements is applied. Such a sensing scheme achieved a high compressive ratio, but required additional on board computing. In contrast to many camera design studies (e.g., [28]), our work focused on minimizing such computing hardware for remote sensing missions. In particular, our end goal was to further advance our system for compact CubeSat deployment.

2. Materials and Methods

In this section, we elaborate on our compressive HS sensing scheme and specify our camera design. Our sampling methodology is very similar to other approaches in the literature [4,5,6,7,8,9,10,11], but we focus on the setting of multidimensional arrays. With this in mind, it is best to briefly introduce tensor notation before proceeding.

Notation: to remain consistent with prior work on tensor decompositions, we adopt notation that largely agrees with the literature [45]. Vectors are denoted by lowercase bold letters and matrices are denoted by boldface capital letters, e.g., $\mathbf{a}$ and $\mathbf{A}$, respectively. The transpose of a matrix is written ${\mathbf{A}}^T$ and higher-dimensional arrays (i.e., with three or more dimensions) are denoted by boldface Euler script letters, e.g., $\mathcal{X}$. Linear maps that take arrays as an input are denoted by calligraphic letters, e.g., $\mathcal{D}:{\mathbb{R}}^{I_1\times \cdots \times I_K}\mapsto {\mathbb{R}}^{I_1\times \cdots \times I_K}$.

The i-th entry of a vector $\mathbf{a}$ is denoted by $a_i$, the $(i,j)$ element of a matrix $\mathbf{A}$ is denoted as $a_{ij}$, and the elements $(i_1,i_2,\cdots ,i_K)$ of a K-th-order tensor $\mathcal{X}$ are denoted as $x_{i_1{i}_2\cdots i_K}$. The norm of an array is the square root of the sum of the squares of all its elements, i.e.,

$$\parallel \mathcal{X}\parallel ={\left(\sum _{i_1=1}^{I_1}\sum _{i_2=1}^{I_2}\cdots \sum _{i_K=1}^{I_K}{\left|x_{i_1{i}_2\cdots i_K}\right|}^2\right)}^{1/2}.$$

2.1. Sensing Scheme

We focused on sensing via digital micromirror device (DMD) light-modulation hardware [12]. Let $\mathcal{X}\in {\mathbb{R}}^{I_x\times I_y\times I_s}$ denote our HS image of interest. The first two dimensions $I_x$ and $I_y$ are the number of pixels in each spatial axis, while $I_s$ is the number of spectral bands for each pixel. Our collected data will be denoted as $\mathcal{Y}\in {\mathbb{R}}^{I_x\times I_y\times M}$, where M is the number of encoding patterns per pixel. For a fixed pixel $(i,j)\in \{1,2,\cdots ,I_x\}\times \{1,2,\cdots ,I_y\}$, the k-th encoded measurement, where $k\in \{1,2,\cdots ,M\}$, is given as

$$y_{ijk}=\sum _{𝓁=1}^{I_s}{d}_{ij𝓁k}·x_{ij𝓁}+n_{ijk}$$

where $\mathcal{D}\in {[0,1]}^{I_x\times I_y\times I_s\times M}$ encapsulates our modulation patterns (known as the codebook) and $\mathcal{N}\in {\mathbb{R}}^{I_x\times I_y\times M}$ models unwanted additive noise. The codebook entries are restricted to the interval $[0,1]$ due to the DMD hardware, which allows for grayscale attenuation of the incident light.

The entire sampling scheme is represented by a linear map $\mathcal{D}:{\mathbb{R}}^{I_x\times I_y\times I_s}\mapsto {\mathbb{R}}^{I_x\times I_y\times M}$ defined as

$$\mathcal{D}\left(\mathcal{X}\right)=\sum _{𝓁=1}^{I_s}{d}_{ij𝓁k}·x_{ij𝓁}.$$

Notice that $\mathcal{D}$ depends implicitly on $\mathcal{D}$ and we obtain the samples

$$\mathcal{Y}=\mathcal{D}\left(\mathcal{X}\right)+\mathcal{N}.$$

(1)

Therefore, $\mathcal{Y}$ gives our encoded noisy measurements according to the programmed codebook $\mathcal{D}$. Notice that with $M<I_s$, we reduced the amount of collected data (relative to the ambient dimensions of $\mathcal{X}$), thereby entering the regime of compressive sensing.

2.2. Camera Design

Our system uses a 260R Plane Ruled Diffraction Grating from the Richardson Grating Lab (Rochester, NY, USA) [46]. It has a nominal blaze wavelength of 500 nm, nominal blaze angle of 8.6°, and 600 grooves per mm. It is suitable for our desired spectral range of 400–800 nm, but our system will collect data that covers the 343–827 nm range. The incoming light collected through the imaging lens will propagate through a 10 $\mathsf{\mu }$m slit before reaching the diffraction grating to separate the light into different spectral components. The system Zemax optical model is shown in the top-left and middle images of Figure 2, while the bench setup is shown in the top-right image.

Once diffracted, a 10:90 beam splitter will direct the light to two different paths: $90\%$ of the light will reach Camera 1 and $10\%$ of the light will reach Camera 2 (see Figure 2). The input into Camera 1 was not used for this study. Our intended finalized system is not complete due to further optical modifications required for Camera 1, which includes a CEL5500 light-modulation device manufactured by Digital Light Innovations (Austin, TX, USA). The final design is meant to utilize Camera 2 to obtain the image input to the DMD in order to understand the post-modulation distortions of data collected by Camera 1.

To advance the research while the system is finalized, the work presented here only utilized images collected by Camera 2. As a consequence, the data were not modulated via hardware and we instead simulated the encoding process numerically (as in Section 2.1, see Section 3). Camera 2 is a Kiralux Monochrome CMOS Camera (CS235MU) manufactured by Thorlabs (Newton, NJ, USA). This camera has a 1920 × 1080 resolution with a pixel size of 5.86 $\mathsf{\mu }$m. The peak quantum efficiency of the camera is 78% at 500 nm [47].

During the field test, a mirror was placed in front of the imaging lens of the bench setup. A Zaber linear stage was used to rotate the mirror to horizontally scan through the field of view, as shown in Figure 2 (bottom image). The spectral sensitivity was evaluated in the lab using three laser diodes at 405 nm, 520 nm, and 650 nm. The normalized response is shown in the top plot of Figure 3. The FWHMs (full widths at half maximum) for the three wavelengths ranged from 2 nm to 2.66 nm.

2.3. Field Data

In this study, we evaluated our prototype system without hardware-based modulation capabilities (i.e., Camera 1 was not implemented). Instead, a hardware-in-the-loop simulation was employed to use the data captured with Camera 2 as the input. To study compressive imaging capabilities, we modeled our encoding scheme numerically using the images acquired by Camera 2 (see Section 3).

The system captured scenes at two locations on the Harbor Branch Oceanographic Institute campus (HBOI), as indicated in Figure 4 (top image). The target in Scene 1 consisted of six different color panels placed 10 m from the system (bottom image in Figure 4). The panels had dimensions 0.3 m × 0.6 m. In Scene 2, the system was mounted on the balcony of the second floor of the building to capture the channel. For this study, we only considered images collected from Scene 1.

Figure 5 and Figure 6 showcase two collected images, labeled as HBOI1 and HBOI2, respectively. The figures also illustrate sample spectral signatures from a grass pixel, the speed limit sign, the green panel, and the blue panel. Notice that the HBOI2 green and blue panel spectra suffer from oversaturation, i.e., the pixels exhibit blown out endmembers that were missing information of the peak wavelengths. This corruption was introduced as a mistake. During the initial data collection, HBOI1 was acquired at dawn with a properly determined shutter speed for the ambient light conditions. However, the authors neglected to readjust the shutter speed later in the day when HBOI2 was recorded with increased sunlight levels.

As a surprise to the authors, our results in Section 3 demonstrate that we could mitigate this type of corruption using our low-rank tensor decoder and adaptive compressive sampling methodology. Saturation is a common type of corruption for cameras and sensors [48]. However, as the authors in [49] noted, approaches for restoring hyperspectral images that suffer from spectral clipping are rare. Additionally, our scenario dealt with encoded measurements, which further complicated the data infilling process. Therefore, the ability of our methodology to provide some level of saturation correction is arguably novel and merits further study. We postpone such research for future work, but present our preliminary findings in Section 3.

2.4. Codebook Design and Adaptive Sampling

This section discusses some choices for the codebook design, i.e., how to generate $\mathcal{D}$ from Section 2.1 to program the light-modulation device. When a user has no information of a scene, Section 2.4.1 focuses on initial random patterns that are efficient for imaging in data-oblivious settings. Section 2.4.2 then considers adaptive sampling schemes that are meant to minimize the codebook complexity and detect specific signatures when prior knowledge of the spectral content is available.

Our sensing framework is tailored for remote sensing situations that periodically survey a desired area, e.g., satellite imaging as in Figure 1. Random mirror patterns considered in Section 2.4.1 are adequate to conduct initial compressive probing of an area and provide preliminary spectral signatures. Once this encoded information is relayed to a ground station, an HS image can be reconstructed and the estimated endmembers can be applied to the methodology of Section 2.4.2 for subsequent acquisitions. The efficiency of this entire process is numerically demonstrated in Section 3.1.

2.4.1. Initial Codebook Design

To initially probe the HS content of a survey area, sampling patterns with universal guarantees are most adequate [50]. Intuitively, the term universal refers to the property that such sampling schemes can capture a significant amount of information uniformly for all signals of interest. Universal sensing properties are ideal in situations where practitioners are oblivious to the data’s structure. Many results in the compressive sensing literature are available to provide such guarantees for random sensing ensembles, e.g., via mutual coherence and the restricted isometry property [14].

In particular, a sampling basis with orthogonal elements possesses many benefits for signal acquisition and reconstruction [16]. In our context, an orthogonal sampling scheme would require $\mathcal{D}$ to be composed of mutually orthogonal patterns. In other words, for a fixed pixel $(i,j)\in \{1,\cdots ,I_x\}\times \{1,\cdots ,I_y\}$, distinct patterns $k\ne \tilde{k}$ probing the spectrum at that pixel should obey

$$\sum _{𝓁=1}^{I_s}{d}_{ij𝓁k}{d}_{ij𝓁\tilde{k}}=0.$$

Such sensing elements are arguably the most efficient at maximizing the amount of information acquired.

In order to design orthogonal patterns that are compatible with DMD-type hardware, notice that with a non-negative codebook $\mathcal{D}\in {[0,1]}^{I_x\times I_y\times I_s\times M}$, it is necessary that the M patterns have disjoint support. This means that for a fixed pixel $(i,j)\in \{1,\cdots ,I_x\}\times \{1,\cdots ,I_y\}$ and spectral band index $𝓁\in \{1,\cdots ,I_s\}$, the codebook must satisfy

$$d_{ij𝓁k}{d}_{ij𝓁\tilde{k}}=0$$

when $k\ne \tilde{k}$. In words, per pixel, only one pattern can include samples from a given spectral band. We stress that this constraint is only necessary because we focus on DMD-based hardware.

To satisfy this, M fair Bernoulli patterns are generated for each pixel so that each codebook entry $d_{ij𝓁k}$ is 0 or 1 with equal probability. Then, a partition is generated that groups the spectral band indices $\{1,2,\cdots ,I_s\}$ into M disjoint sets. Each set in the partition is uniquely assigned to one of the M patterns of a given pixel, and each pattern is modified to be zero on the indices not contained in its assigned set of the partition. The process is repeated independently for each pixel, i.e., each pixel is assigned distinct partitions of the spectral bands. For our experiments, we generated the partitions uniformly at random and of equal (or nearly equal) size. Figure 7 provides an example of $M=6$ sampling patterns generated for a single pixel.

Independently generated Bernoulli patterns and partitions per pixel are crucial to ensure a rich sampling scheme. This sensing approach is based on the principle that a survey region sampled at a high spatial resolution will contain many pixels with similar endmembers. The success of our probing patterns is due to the assumption that with high probability, each spectral signature is densely sampled in a cumulative manner over the entire image. This approach pairs well with our tensor-based reconstruction program, which is designed to process the entire HS cube in order to optimally exploit this assumption (see Section 2.5).

This is a simple procedure but highly efficient, as seen in our numerical results of Section 3. One benefit of this construction is that the resulting codebook is a binary and sparse array $\mathcal{D}\in {\{0,1\}}^{I_x\times I_y\times I_s\times M}$. This observation allows for memory-efficient storage and transmission of the encoding patterns [51]. We would like to further stress that in our experience, orthogonal encoding patterns have the additional benefit of expedited and stable numerical convergence. In other words, when orthogonal codebooks are applied, we empirically observe that iterative reconstruction approaches converge faster and avoid overfitting as the iterations proceed.

2.4.2. Adaptive Codebook Design

In this section, we assume that prior spectral information of a scene is available to guide the codebook design. Let $\widehat{\mathcal{X}}$ be a given HS image whose endmembers are relevant to the target image $\mathcal{X}$. Our approach in this section applies to prior information in any form, but for simplicity, we assume the prior information can be arranged as a cube matching the target image dimensions $\widehat{\mathcal{X}}\in {\mathbb{R}}^{I_x\times I_y\times I_s}$.

Our goal is to provide M patterns that can be uniformly applied to compressively sample the spectrum at all pixels. In other words, given $\widehat{\mathcal{X}}$, our output in this section is a matrix $\mathbf{Q}\in {[0,1]}^{I_s\times M}$ whose columns are the M sensing patterns. The codebook $\mathcal{D}$ is defined in terms of $\mathbf{Q}$ as

$$d_{ij𝓁k}=q_{𝓁k}\text{for} \text{all} \text{pixels}(i,j)\in \{1,\cdots ,I_x\}\times \{1,\cdots ,I_y\}.$$

In contrast to the initial codebook design of Section 2.4.1, notice that the encoding approach here severely reduces the memory requisite and programming complexity of the DMD. This is ideal for remote sensing settings, where an initial codebook for preliminary probing (as in Section 2.4.1) can be pre-programmed on the hardware prior to deployment, and subsequently, the hardware can be re-configured by minimal transmission of the matrix $\mathbf{Q}$.

To elaborate on the adaptive pattern construction, we reshape our HS cubes $\mathcal{X},\widehat{\mathcal{X}}\in {\mathbb{R}}^{I_x\times I_y\times I_s}$ as respective matrices $\mathbf{X},\widehat{\mathbf{X}}\in {\mathbb{R}}^{I_x{I}_y\times I_s}$ whose rows are the $I_x{I}_y$ spectral signatures. Notice that in terms of the adaptive patterns, our measurements can now be written in matrix form as

$$\mathbf{Y}=\mathbf{X}\mathbf{Q}+\mathbf{N}\in {\mathbb{R}}^{I_x{I}_y\times M},$$

(2)

where $\mathbf{Y}$ and $\mathbf{N}$ are analogous reshaped versions of $\mathcal{Y}$ and $\mathcal{N}$, respectively (from Section 2.1). Given prior information $\widehat{\mathbf{X}}$, we defined our output patterns as

$$\mathbf{Q}=\underset{\mathbf{C}\in {[0,1]}^{I_s\times M}}{arg\; min}\parallel \widehat{\mathbf{X}}-\widehat{\mathbf{X}}\mathbf{C}{\mathbf{C}}^T\parallel .$$

(3)

The approach attempts to produce patterns that efficiently capture all the spectral content of the scene in a least squares sense. Intuitively, (3) describes a $\mathbf{Q}$ that acts as an orthogonal projection onto an M-dimensional subspace that best approximates the row space of $\widehat{\mathbf{X}}$, while obeying DMD grayscale constraints. To attempt to solve (3), we used a block-coordinate descent method with iterative projections onto ${[0,1]}^{I_s\times M}$ (see Algorithm 1). We note that steps 2 and 3 of Algorithm 1 can be solved by the least squares method, e.g., using MATLAB’s lsqr solver. Figure 7 provides an example of $M=6$ adaptive patterns generated via (3) using our field data.

Algorithm 1 Adaptive Codebook Construction

Input: prior information $\widehat{\mathbf{X}}\in {\mathbb{R}}^{I_x{I}_y\times I_s}$, number of patterns $M\in \mathbb{N}$, and number of updates $L\in \mathbb{N}$.      Output: adaptive codebook $\mathbf{Q}\in {[0,1]}^{I_s\times M}$.      Define: entry-wise threshold function $${\mathrm{thresh}}_{[0,1]}\left(x\right)=\left\{\begin{array}{cc}x\hfill & \mathrm{if}x\in [0,1]\hfill \\ 0\hfill & \mathrm{if}x<0\hfill \\ 1\hfill & \mathrm{if}x>1.\hfill \end{array}\right.$$      Initialize: $\mathbf{Q}$ with entries independently generated from the uniform distribution on $[0,1]$. 1:  for $𝓁=1,2,\dots ,L$ do 2:      $\tilde{\mathbf{Q}}\leftarrow {arg\; min}_{\mathbf{C}\in {\mathbb{R}}^{I_s\times M}}\parallel \widehat{\mathbf{X}}-\widehat{\mathbf{X}}\mathbf{Q}{\mathbf{C}}^T\parallel$ 3:      $\mathbf{Q}\leftarrow {arg\; min}_{\mathbf{C}\in {\mathbb{R}}^{I_s\times M}}\parallel \widehat{\mathbf{X}}-\widehat{\mathbf{X}}\mathbf{C}{\tilde{\mathbf{Q}}}^T\parallel$ 4:      $\mathbf{Q}\leftarrow$ ${\mathrm{thresh}}_{[0,1]}\left(\mathbf{Q}\right)$ 5:  end for

This approach was shown to achieve accurate HS imaging via a minimized codebook with the ability to combat oversaturation. However, it is important to notice that the approach causes the sensing system to focus on probing endmembers with a certain structure (according to $\widehat{\mathbf{X}}$). Therefore, this approach is most applicable when the prior information encompasses the entire spectrum of the target image or when only certain pixels are of interest. In particular, the latter scenario is adequate for HS image segmentation, where certain signatures can be identified and efficiently reconstructed. Finally, with M patterns applied uniformly to all pixels, numerical reconstruction approaches can be implemented more efficiently using (2) to recast the sensing model (1) in matrix form. See Section 3.1 for the numerical results.

We warn the reader that our adaptive methodology assumes that subsequent surveys are conducted in similar conditions to the initial imaging. This assumption may be violated under weather conditions that cause severe changes in the spectral properties. For example, clouds and shadows have been known to shift peak wavelengths [52]. We do not explore such situations in this work, but they will likely alter our findings from those obtained in this study and we propose to study these cases in future work.

We end this section by noting that several other authors considered the design of encoding patterns for spectral sampling [6,53,54]. However, to the best of our knowledge, these works in the literature aim to output codebooks that are efficient in a universal sense (i.e., not adaptive). In other words, these methodologies do not incorporate input spectral data to guide the encoding process. While such work is very valuable for more efficient initial probing of a scene (as in the setting of Section 2.4.1), our work here focused on the situation where prior information or specific endmembers were targeted. In particular, we focused our scope on remote satellite imaging, where the light-modulation hardware could be re-configured as more spectral information became available.

2.5. HS Image Reconstruction: HSCP-ALS

In contrast to most numerical work in HS compressive sensing, the novelty of our decoding method is that we exploited the multidimensional structure of HS images. In particular, we employed the redundant representation of many HS images via the canonical polyadic (CP) decomposition of tensors [37,38,39,40,41,42]. Inspired by the compression capabilities of the CP decomposition, our chosen approach utilizes this low-dimensional representation for HS image reconstruction that is compatible with previously proposed sensing schemes [11].

The approach discussed in this section is a continuation of the preliminary work in our conference paper [13]. Here, we further validated this approach by testing its reconstruction capabilities on real footage and other codebook designs (e.g., adaptive patterns from Section 2.4.2). We also compared our decoding approach to other techniques proposed in the literature (see Section 3.3). In the current section, we summarize and expand on the approach proposed in our prior work. Before proceeding to the reconstruction program in Section 2.5.2, we briefly discuss the canonical polyadic decomposition of tensors.

2.5.1. Canonical Polyadic Decomposition

The canonical polyadic (CP) decomposition is one approach to generalize the singular value decomposition of matrices to higher-dimensional arrays (also known as the CANDECOMP/PARAFAC decomposition). This representation decomposes a tensor into a sum of “simple” rank-one tensors. Focusing on the three-dimensional case, a rank-one tensor $\mathcal{T}$ can be written as the outer product of three vectors:

$$\mathcal{T}=\mathbf{a}\circ \mathbf{b}\circ \mathbf{c},$$

where “∘” denotes the vector outer product. In other words, each entry of a rank-one tensor $\mathcal{T}$ is given as $t_{ij𝓁}=a_i{b}_j{c}_{𝓁}$. This is the simplest form a tensor can have under a CP decomposition, and any other tensor can be written as a sum of rank-one components. A rank-R CP decomposition of a three-dimensional tensor $\mathcal{X}\in {\mathbb{R}}^{I_x\times I_y\times I_s}$ is of the form

$$\mathcal{X}=\sum _{k=1}^R{\mathbf{a}}_k\circ {\mathbf{b}}_k\circ {\mathbf{c}}_k,$$

(4)

where ${\mathbf{a}}_k\in {\mathbb{R}}^{I_x}$, ${\mathbf{b}}_k\in {\mathbb{R}}^{I_y}$, and ${\mathbf{c}}_k\in {\mathbb{R}}^{I_s}$ for each $k\in \{1,2,\cdots ,R\}$. The smallest integer R that allows $\mathcal{X}$ to be written as (4) is called the rank of $\mathcal{X}$. The CP decomposition (4) is concisely expressed as $\mathcal{X}=〚\mathbf{A},\mathbf{B},\mathbf{C}〛$ if ${\mathbf{a}}_k$, ${\mathbf{b}}_k$, and ${\mathbf{c}}_k$ represent the k-th columns of $\mathbf{A},\mathbf{B},$ and $\mathbf{C}$, respectively (known as the tensor factors). Equivalently, in terms of the factors, this notation signifies that each entry of $\mathcal{X}$ is given as

$$x_{ij𝓁}=\sum _{k=1}^R{a}_{ik}{b}_{jk}{c}_{𝓁k}.$$

Compatibility with the linear mixing model: the CP decomposition of HS images is suitable for the linear mixing model (LMM) [55] used to decompose the spectral content of a scene. The LMM assumes that each pixel contains a linear combination of pure spectra (i.e., endmembers). The distribution of the endmembers throughout the image is determined by the abundance, which specifies the fraction a material occupies in each pixel. The CP decomposition (4) also allows for such an interpretation. The factor $\mathbf{C}\in {\mathbb{R}}^{I_s\times R}$ encompasses the endmembers, where each column contains independent components of spectral signatures throughout the scene. The outer product of the respective columns of $\mathbf{A}\in {\mathbb{R}}^{I_x\times R}$ and $\mathbf{B}\in {\mathbb{R}}^{I_y\times R}$ specify the spatial abundance of each corresponding endmember in $\mathbf{C}$. See our references for further discussion [37,38,39,40,41,42], where these observations were used for spectral unmixing.

2.5.2. HS Image Reconstruction via CP-ALS

To decode, i.e., estimate the HS image $\mathcal{X}$ given $\mathcal{Y}$ and $\mathcal{D}$ (encoded image and codebook), we propose to solve a CP decomposition constrained least squares problem. Specifically, we obtain a HS image estimate $\tilde{\mathcal{X}}\approx \mathcal{X}$ using

$$\tilde{\mathcal{X}}=\underset{\mathcal{T}\in {\mathbb{R}}^{I_x\times I_y\times I_s}}{arg\; min}\parallel \mathcal{D}\left(\mathcal{T}\right)-\mathcal{Y}\parallel \text{subject} \text{to}\mathcal{T}=〚\mathbf{A},\mathbf{B},\mathbf{C}〛,$$

(5)

where $\mathcal{D}$ is defined in Section 2.1 to apply the sensing scheme. Program (5) ensures that our output estimate $\tilde{\mathcal{X}}$ fits the encoded measurements in a least squares sense while obeying the CP decomposition with factors $\mathbf{A}\in {\mathbb{R}}^{I_x\times R}$, $\mathbf{B}\in {\mathbb{R}}^{I_y\times R}$, and $\mathbf{C}\in {\mathbb{R}}^{I_s\times R}$ for some choice of R. Notice that this methodology implicitly depends on R, which will be referred to as the rank parameter. This parameter must be chosen by a practitioner to accurately approximate $\mathcal{X}$. Intuitively, the rank parameter captures the low-dimensional structure of the HS image. An accurate representation of $\mathcal{X}$ via a CP decomposition with rank parameter R will reduce the storage of $I_x{I}_y{I}_s$ free variables to $R(I_x+I_y+I_s)$. In our experiments discussed in Section 3, the rank was chosen to be as large as possible according to our GPU memory limitations while not exceeding the largest array dimension. We empirically observed that reconstructed HS images can be stored in CP format with less than $1\%$ of the uncompressed memory requisite.

To solve (5), we adopted a modified version of the popular CP-ALS program that applied our sensing scheme. The CP-ALS algorithm is elaborated in [45] (see Section 3.4 therein) when no linear measurement operator is applied (i.e., $\mathcal{D}$ in our scenario). For brevity, we refer the reader to [45] for details and only discuss here the minor changes required to solve (5) in the case of three dimensions.

The alternating least squares (ALS) method fixes two of the tensor factors while updating the remaining factor, then alternates to update each factor iteratively. In this manner, the approach attempts to solve the overall non-convex program via a sequence of least squares sub-problems. The procedure requires the user to input the rank parameter R and initial estimates $\mathbf{B}$ and $\mathbf{C}$. In this work, the initial factors were chosen with entries identically and independently distributed from the uniform distribution on $[0,1]$.

With $\mathbf{B}$ and $\mathbf{C}$ fixed (as initial estimates), $\mathbf{A}$ was updated via

$$\mathbf{A}\leftarrow \underset{\mathbf{T}\in {\mathbb{R}}^{I_x\times R}}{arg\; min}\parallel \mathcal{D}(〚\mathbf{T},\mathbf{B},\mathbf{C}〛)-\mathcal{Y}\parallel .$$

(6)

To simplify, we introduce the operator ${\mathcal{D}}_{\mathbf{B},\mathbf{C}}:{\mathbb{R}}^{I_x\times R}\mapsto {\mathbb{R}}^{I_x\times I_y\times M}$ defined as

$${\mathcal{D}}_{\mathbf{B},\mathbf{C}}\left(\mathbf{A}\right)=\mathcal{D}(〚\mathbf{A},\mathbf{B},\mathbf{C}〛).$$

Using observations in [45], the action of this operator can be described via the Khatri–Rao product $\mathbf{B}\odot \mathbf{C}$, which yields an $I_y{I}_s\times R$ matrix. The transpose of this product is multiplied with $\mathbf{A}$, giving $\mathbf{A}{(\mathbf{B}\odot \mathbf{C})}^T\in {\mathbb{R}}^{I_x\times I_y{I}_s}$ to input as the argument of the encoding operator $\mathcal{D}$ after properly reshaping $\mathbf{A}{(\mathbf{B}\odot \mathbf{C})}^T$ as an $I_x\times I_y\times I_s$ tensor. Notice that ${\mathcal{D}}_{\mathbf{B},\mathbf{C}}$ is linear, and therefore, our update is given in closed form as

$$\mathbf{A}\leftarrow {\mathcal{D}}_{\mathbf{B},\mathbf{C}}^{†}\left(\mathcal{Y}\right)$$

where ^† denotes the pseudoinverse. Equivalently, with multiway arrays properly reshaped as matrices (as in Section 2.4.2), (6) can be recast as a standard least squares problem in vector form for which iterative solvers are widely available. Updates for $\mathbf{B}$ and $\mathbf{C}$ can be achieved via the operators ${\mathcal{D}}_{\mathbf{C},\mathbf{A}}$ and ${\mathcal{D}}_{\mathbf{A},\mathbf{B}}$, respectively, which are analogously defined.

This modified CP-ALS procedure is known as HSCP-ALS. See Algorithm 2 for the pseudocode of the entire HSCP-ALS algorithm to solve (5). As described here, the number of factor updates K must be specified by the user but we note that stopping criteria for similar block-coordinate descent methods were proposed to halt factors updates once the overall estimate is deemed to converge [56]. In our experiments of Section 3, we used MATLAB’s lsqr solver limited to 10 iterations to approximate each pseudoinverse (i.e., to achieve steps 2, 3, and 4 in Algorithm 2).

Algorithm 2 HSCP-ALS

Input: encoded image $\mathcal{Y}\in {\mathbb{R}}^{I_x\times I_y\times M}$, codebook $\mathcal{D}\in {[0,1]}^{I_x\times I_y\times I_s\times M}$, rank $R\in \mathbb{N}$, and number of updates $K\in \mathbb{N}$.      Output: estimate HS image $\tilde{\mathcal{X}}\in {\mathbb{R}}^{I_x\times I_y\times I_s}$.      Initialize: tensor factors $\mathbf{B}\in {\mathbb{R}}^{I_y\times R}$ and $\mathbf{C}\in {\mathbb{R}}^{I_s\times R}$ with entries independently generated from the uniform distribution on $[0,1]$. 1:  for $k=1,2,\dots ,K$ do 2:      $\mathbf{A}\leftarrow {\mathcal{D}}_{\mathbf{B},\mathbf{C}}^{†}\left(\mathcal{Y}\right)$ 3:      $\mathbf{B}\leftarrow {\mathcal{D}}_{\mathbf{C},\mathbf{A}}^{†}\left(\mathcal{Y}\right)$ 4:      $\mathbf{C}\leftarrow {\mathcal{D}}_{\mathbf{A},\mathbf{B}}^{†}\left(\mathcal{Y}\right)$ 5:  end for 6:  $\tilde{\mathcal{X}}=〚\mathbf{A},\mathbf{B},\mathbf{C}〛$

3. Results

We present the imaging experiment results to demonstrate the efficiency of our reconstruction program and the capabilities of our proposed adaptive sampling techniques. Section 3.1 showcases the image reconstruction on footage captured by our prototype system, which applied the initial and adaptive codebook designs of Section 2.4. Section 3.2 uses selected spectral signatures to design sampling patterns and identify specific endmembers from a scene, i.e., image segmentation. Finally, Section 3.3 compares our HSCP-ALS methodology to other decoding approaches from the literature.

HSCP-ALS implementation: we used gpuArray from MATLAB (R2023b) to implement Algorithm 2 on an NVIDIA RTX A6000 GPU with 48 GB of memory. In other words, variables $\mathcal{Y},\mathcal{D},\mathbf{A},\mathbf{B}$, and $\mathbf{C}$ were stored on the GPU to allow for distributed processing. This achieved expedited image reconstruction but imposed a memory limitation. In our experiments, this varied our choice of the rank parameter. As a rule, we chose R as the largest multiple of 50 that allowed for GPU storage while not exceeding $max\{I_x,I_y,I_s\}$. The number of updates was always fixed to $K=10$.

3.1. Reconstruction Results: HBOI Field Data

As discussed in Section 2.3, the preliminary data collected in this work did not incorporate light-modulation hardware. To gain instructive insight of our ongoing compressive sensing design, this study numerically simulated DMD-based spectral encoding to the collected noisy footage. HSCP-ALS was then applied to decode the compressed measurements and produced an estimate image.

Our aim was to demonstrate the entirety of our proposed remote sensing scheme from Section 2.4, as in Figure 1 with initial probing of a scene followed by adaptive sampling in subsequent surveys. To elaborate, we considered HS images from Section 2.3 labeled as HBOI1 and HBOI2 (see Figure 5 and Figure 6). These are images of Scene 1 in Figure 4, which were captured at different times of the day. Our strategy applied an initial random codebook from Section 2.4.1 to HBOI1, followed by HSCP-ALS (5) to produce an image estimate $\widehat{\mathcal{X}}$. Using the reconstructed data, we applied the adaptive codebook design of Section 2.4.2 to produce $\mathbf{Q}$ as the output of (3) and apply the respective encoding to HBOI2. Finally, an estimate $\tilde{\mathcal{X}}$ of HBOI2 was reconstructed by HSCP-ALS.

We truncated HBOI1 and HBOI2 to have a spatial resolution of $1700\times 1700$ pixels with 120 spectral bands (ranging from 343 nm to 827 nm). This was done to remove the field of view obstructed by the camera chassis and to reduce the image size for the GPU distributed processing. Our experiments achieved imaging with $5\%$ sampling relative to the target 120 wavelengths.

For the initial probing of HBOI1, we generated $M=6$ binary patterns independently per pixel, as in Section 2.4.1, with the partitions chosen uniformly at random. The estimate $\widehat{\mathcal{X}}\in {\mathbb{R}}^{1700\times 1700\times 120}$ was then produced as the output of Algorithm 2 with rank $R=450$ and $K=10$ updates. This choice of rank outputs estimated $\widehat{\mathcal{X}}$ in a CP format that occupied $.46\%$ of the bytes relative to the uncompressed HS image. See the left plot of Figure 7 for an example of the generated initial codebook, Figure 8 for the HBOI1 results, and Figure 9 for a close-up of the reconstructed image details.

To achieve adaptive sampling, we used the estimate $\widehat{\mathcal{X}}$ of HBOI1 to generate $\mathbf{Q}\in {[0,1]}^{120\times 6}$ as the output of Algorithm 1 with $L=30$ updates. These sampling patterns were applied uniformly to all pixels of HBOI2, again reducing the amount of collected data by a factor of 20 ($5\%$ sampling). The adaptive estimate $\tilde{\mathcal{X}}\in {\mathbb{R}}^{1700\times 1700\times 120}$ was produced as the output of Algorithm 2, with rank $R=700$ and $K=10$ updates. The severely reduced codebook size allowed this larger choice of rank parameter for GPU processing. The output HS estimate in a CP format that occupied $.71\%$ of the uncompressed bytes. The adaptive patterns are shown in the right plot of Figure 7, the final output is illustrated in Figure 10, and Figure 9 provides a close-up of the image details.

As noted in Section 2.3, image HBOI2 suffered from oversaturation in several pixels. The image reconstruction result of Figure 10 demonstrates that our adaptive sampling technique allowed HSCP-ALS to mitigate the saturation by providing some information of the peak wavelengths. Simultaneously, notice that we also reliably estimated non-saturated signatures (e.g., grass and speed limit sign spectra). Therefore, using reliable signatures as prior information, our adaptive codebook design could output sampling patterns that helped to estimate the missing data while not affecting the distortion-free pixels. This source of corruption was unknown to the authors during data collection. These unexpected results merit further research, with the potential to correct other kinds of unwanted distortions. We propose to focus on these aspects of our methodology in future work.

Figure 9 shows the reconstruction of the speed limit sign found in the HBOI images. These results explore the ability of our approaches to recover spatial details. We found that the initial probing scheme (i.e., results on HBOI1) seemed to be better at picking up spectral details in comparison to the adaptive patterns. This is evident in the plots of Figure 8 and Figure 10, where the HBOI1 reconstruction provided an endmember with more curvature details, while the adaptive HBOI2 signature gave a smoother estimate. In terms of spatial details, in Figure 9, it is clear that the adaptive result of HBOI2 provided a sharper image of the speed limit sign compared with the blurry output of HBOI1. These observations imply an interesting tradeoff between adaptive and non-adaptive techniques to reconstruct spatial vs. spectral details, which is informative for practitioners. However, we note that adaptive samples were designed to minimize the amount of patterns that needed to be transmitted. This limitation was due to our focus on remote sensing scenarios, and thus, the adaptive results may be improved by allowing for more spatially varied patterns.

3.2. Hardware-Based Segmentation

This section showcases the image segmentation capabilities of light-modulation-based sensing. The main novelty of our approach is that with prior information available, segmentation and endmember identification can be implemented into the hardware and directly achieved from the compressed samples. This allows one to focus computational resources to specific pixels and improve the estimation quality for signatures of interest.

To demonstrate this, we considered the scenes of our captured footage in Section 2.3 and of a river mouth region surveyed by NASA’s HICO missions [1]. In both scenarios, we used the initial probing techniques of Section 2.4.1 to obtain preliminary HS information. From these estimate images, we selected several pixels with spectral signatures of interest and used these endmembers to specify the sampling patterns for subsequent surveys. The goal of this methodology was to specialize the sensing scheme to identify and efficiently sample regions whose spectrum is similar to the chosen endmembers. We note that the initial probing step is not necessary if a practitioner already has prior information. We focus on this specific workflow that fits our remote sensing scheme illustrated in Figure 1.

HBOI field data: in our captured HS footage (see Section 2.3), we attempted to identify the pixels of the red, green, and blue panels. We used the reconstructed HBOI1 image from Section 3.1 as our initial data estimate; see Figure 8. We manually selected several pixels of the target panels, then averaged and normalized them to produce exemplar signatures $\mathbf{r},\mathbf{g},\mathbf{b}\in {[0,1]}^{120}$ (red, green, and blue, respectively). We constructed our adaptive codebook $\mathbf{Q}\in {[0,1]}^{120\times 4}$ by setting $\mathbf{r}$, $\mathbf{g}$, and $\mathbf{b}$ as its first three columns and the “all-ones” vector $\mathbf{1}\in {\left\{1\right\}}^{120}$ as the last column. The all-ones sensing pattern provides normalization coefficients in order to achieve segmentation in compressed format (i.e., directly from our observed measurements $\mathcal{Y}\in {\mathbb{R}}^{I_x\times I_y\times 4}$).

Let $\tilde{\mathcal{Y}}\in {[0,1]}^{I_x\times I_y\times 3}$ be defined in each pixel $(i,j)\in \{1,\cdots ,I_x\}\times \{1,\cdots ,I_y\}$ as

$${\tilde{y}}_{ijk}=\frac{y_{ijk}}{y_{ij4}}\forall k\in \{1,2,3\},$$

providing normalized sensing via $\mathbf{r},\mathbf{g}$, and $\mathbf{b}$. Then, simple segmentation techniques can be applied to identify outliers of $\tilde{\mathcal{Y}}$ in each channel and extract pixels with signatures most similar to the chosen endmembers. In this work, we used MATLAB’s grayconnected function to achieve this segmentation step. Using this information, we applied HSCP-ALS only to a $242\times 266$ region of identified pixels with the choice $R=266$ (largest array dimension).

The segmentation result is shown in Figure 11, providing accurate detection of the three bins of interest. Notice that this approach also mitigates HBOI2’s oversaturated pixels (in the green and blue panel), while providing accurate estimation of non-distorted signatures (i.e., the red panel pixels). In contrast to the previous adaptive sampling result (Figure 10), the result here arguably improved the ability to infill information. The estimated spectral plots in Figure 11 involve more structure with identifiable peak wavelengths. These improved capabilities are likely due to the additional tailoring of the modulation patterns for panel identification. However, as a trade off, the selected patterns are probably not adequate to address the deformations of unrelated endmembers. Further investigation and discussion of the distortion-correcting abilities of adaptive samples is left as future work.

NASA HICO data: the same procedure was used to segment water-related pixels in the HICO satellite data. We considered two HS images labeled as H2011251110401 and H2010261105802, which we refer to as HICO401 and HICO802, respectively. These were images of the same river mouth region surveyed on different days. We truncated these cubes to be of size $508\times 1997\times 120$, removed corrupt pixels, and focused on the wavelengths 369.7–1051.3 nm. We used an HSCP-ALS reconstructed estimate of HICO401 as prior data (with a GPU-allowed rank $R=1800$) that were synthetically encoded via $M=6$ patterns ($5\%$ sampling) per pixel as in Section 2.4.1. We then selected several pixels from ocean and water regions of the image to generate the average and normalized signature $\mathbf{w}\in {[0,1]}^{120}$ and codebook $\mathbf{Q}\in {[0,1]}^{120\times 2}$ as before (incorporating the all-ones pattern for normalization). $\mathbf{Q}$-based sensing was applied to HICO802 ($1.67\%$ sampling), followed by segmentation of the encoded image (as was done for the HBOI data). Finally, an HSCP-ALS estimate of the HICO802’s water pixels was produced with rank $R=1997=max\{I_x,I_y,I_s\}$. The segmentation and reconstruction results are shown in Figure 12.

3.3. Comparison of HSCP-ALS to Other Methods: Benchmark Dataset

To further validate our image reconstruction program (HSCP-ALS), we compared it to several approaches from the literature. Our previous work [13] compared our methodology to SpeCA (spectral compressive acquisition) [4] and showed that it performed significantly better. Here, we additionally compared it to CPPCA (compressive projection principal component analysis) [16] and sparsity-based reconstruction schemes. Specifically, we considered ${𝓁}_1$-norm minimization [31] and a distributed counterpart [25,57].

We used the benchmark Cuprite image [9,10], with $190\times 250$ spatial resolution and 188 bands. With the ground truth provided, we used the signal-to-noise ratio (SNR) as the metric of the reconstruction quality:

$$SNR\left(\mathcal{X},\tilde{\mathcal{X}}\right)=10{log}_{10}\left(\frac{{\parallel \mathcal{X}\parallel }^2}{{\parallel \mathcal{X}-\tilde{\mathcal{X}}\parallel }^2}\right),$$

where $\mathcal{X}$ is the HS image of interest and $\tilde{\mathcal{X}}$ is the provided estimate.

We conducted image reconstruction experiments with varied numbers of measurements and noise levels. We considered sampling percentages of ∼5%, ∼10%, and ∼15% relative to the 188 spectral bands. In other words, three surveys with $M=10,19$, and 23 sampling patterns were generated from our initial probing scheme of Section 2.4.1. Our noise model generated random additive noise according to chosen hardware specifications. We incorporated the wavelength-wise grating prism efficiency included in the ZEISS product manual to generate structured noise. This noise was distorted via normally distributed random variables with variance chosen according to a desired SNR and added to the encoded measurements (i.e., additive noise). We added corruption from this synthetic noise model with SNR levels of 10 dB, 15 dB, and 20 dB.

The competitive approaches had several other parameters that needed to be chosen, including the number of “lines” to reconstruct simultaneously (i.e., number of image rows) and number of singular vectors to estimate (for CPPCA). To produce the best result, we exhaustively cycled through all possible variations of these parameters and only showcase the best results. Our methodology (5) requires the rank parameter and number of factor updates, which we limited to $K=10$ updates and chose $R=250=max\{I_x,I_y,I_s\}$ according to our rule. The comparisons in

Table 1. Comparison of HSCP-ALS with other reconstruction methods. These results were for Cuprite data with $5\%$ sampling and various measurement corruption levels from our noise model.

Input SNR	CPPCA	${𝓁}_1$-Min	Distributed ${𝓁}_1$-Min	HSCP-ALS
10 dB	12.8795 dB	5.5652 dB	15.9992 dB	17.9423 dB
15 dB	13.6119 dB	10.3080 dB	17.8556 dB	21.8641 dB
20 dB	13.8958 dB	12.4858 dB	18.7570 dB	23.3095 dB

,

Table 2. Comparison of HSCP-ALS with other reconstruction methods. These results were for Cuprite data with $10\%$ sampling and various measurement corruption levels from our noise model.

Input SNR	CPPCA	${𝓁}_1$-Min	Distributed ${𝓁}_1$-Min	CP-ALS
10 dB	14.1017 dB	5.3540 dB	16.5761 dB	21.3264 dB
15 dB	14.6541 dB	10.2561 dB	17.8350 dB	23.6715 dB
20 dB	14.8048 dB	12.7200 dB	18.5365 dB	24.2014 dB

and

Table 3. Comparison of HSCP-ALS with other reconstruction methods. These results were for Cuprite data with $15\%$ sampling and various measurement corruption levels from our noise model.

Input SNR	CPPCA	${𝓁}_1$-Min	Distributed ${𝓁}_1$-Min	CP-ALS
10 dB	14.3713 dB	10.2561 dB	16.5031 dB	21.6496 dB
15 dB	14.9681 dB	10.1624 dB	17.6900 dB	23.9046 dB
20 dB	15.2057 dB	12.8459 dB	18.2990 dB	24.5190 dB

show that HSCP-ALS outperformed these other reconstruction programs in every scenario.

4. Conclusions

This work advanced remote compressive hyperspectral imaging by designing and testing a prototype camera. We showcased our initial sensing system by capturing real hyperspectral volumes of complex scenes through utilizing hardware that was compatible with light-modulation devices. The collected images were used to validate our proposed tensor reconstruction algorithm that decoded spectrally modulated measurements. We further developed an approach to output adaptive samples when prior information was available or gained through subsequent surveys. This sampling workflow was validated on footage collected by our imaging system. The overall approach allowed us to capture, segment, and transmit hyperspectral images from a severely reduced number of samples while producing high quality output images.

To further explore the capabilities and limitations of compressive cameras in remote scenarios, more hardware development is needed. Our camera was designed to be compatible with light-modulation devices, but the work here did not include such components. Instead, our current work numerically simulated this encoding process on densely sampled volumes. Future work will utilize our findings in this study to achieve adequate optical modifications and additionally collect hardware encoded data from Camera 1 (see Section 2). This will allow for a comprehensive study of compressive hyperspectral cameras suitable for remote missions, and further validate our proposed reconstruction algorithm and sampling methodologies on true hardware-modulated data. The work presented here provided instructive findings that will guide our next-generation camera design.

Our numerical experiments demonstrated that our low-rank tensor methodology for image reconstruction was robust and outperformed other lower dimensional approaches in the literature. The main novelty is that our method processes the entire volume in order to fully exploit the multidimensional structure of the data. However, other numerical methods exist that also exploit tensor decompositions and likewise demonstrate their superiority to vector- and matrix-based analytics. It would be of interest to compare and explore the tradeoffs of different tensor decomposition choices for the purpose of compressive hyperspectral imaging.

Unknown to the authors, the collected HBOI data exhibited oversaturation with a loss of peak wavelength structure. Our results demonstrated that the proposed adaptive sampling methodology was able to mitigate this source of corruption and estimate the missing information. Such distortion correction was not the focus of this paper and came as a surprise to the authors. As future work, it would be of interest to systematically explore the ability of adaptive samples to combat oversaturation and other types of image corruption. For example, surveys may be conducted under varying ambient conditions (e.g., clouds and shadows). Such situations will alter spectral signatures, and thus, it is important to identify the limitations of our adaptive approach in this scenario or explore its ability to mitigate such distortions.