Collection of a Hyperspectral Atmospheric Cloud Dataset and Enhancing Pixel Classification through Patch-Origin Embedding

Abstract

Hyperspectral cameras collect detailed spectral information at each image pixel, contributing to the identification of image features. The rich spectral content of hyperspectral imagery has led to its application in diverse fields of study. This study focused on cloud classification using a dataset of hyperspectral sky images captured by a Resonon PIKA XC2 camera. The camera records images using 462 spectral bands, ranging from 400 to 1000 nm, with a spectral resolution of 1.9 nm. Our preliminary/unlabeled dataset comprised 33 parent hyperspectral images (HSI), each a substantial unlabeled image measuring 4402-by-1600 pixels. With the meteorological expertise within our team, we manually labeled pixels by extracting 10 to 20 sample patches from each parent image, each patch consisting of a 50-by-50 pixel field. This process yielded a collection of 444 patches, each categorically labeled into one of seven cloud and sky condition categories. To embed the inherent data structure while classifying individual pixels, we introduced an innovative technique to boost classification accuracy by incorporating patch-specific information into each pixel’s feature vector. The posterior probabilities generated by these classifiers, which capture the unique attributes of each patch, were subsequently concatenated with the pixel’s original spectral data to form an augmented feature vector. We then applied a final classifier to map the augmented vectors to the seven cloud/sky categories. The results compared favorably to the baseline model devoid of patch-origin embedding, showing that incorporating the spatial context along with the spectral information inherent in hyperspectral images enhances the classification accuracy in hyperspectral cloud classification. The dataset is available on IEEE DataPort.

1. Introduction

Hyperspectral imagery (HSI) has applications in diverse areas, including agriculture, forestry, detection of mineral resources, land use classification for land management, and meteorology [1,2,3,4,5]. Airborne and spaceborne HSI have increasingly become powerful tools in atmospheric sciences for air quality analysis, measuring concentrations of water vapor and other atmospheric gases, and characterizing weather systems and clouds [4]. Ground-based hyperspectral sky imaging has also been used in many applications, including the retrieval of cloud microphysical and optical properties in both thin water clouds and cirrus clouds [6,7]. The use of ground-based hyperspectral sky imaging for the classification of clouds is an emerging field. Clouds of various types form at different levels in the Earth’s atmosphere and have varying degrees of opacity to solar and terrestrial radiation. As a result, cloud type is an important factor in determining the flux of solar radiation at both the Earth’s surface and the top of the atmosphere. The influence of cloud type on the Earth’s radiation budget has been found to be as important as that of cloud amount. Therefore, the regional and global climates are closely tied to the frequency of occurrence of different cloud types [8]. Major cloud genera are based on cloud height, morphology, and microphysical properties. Some of the major cloud genera, such as nimbostratus, infrequently occur at the location where our hyperspectral camera captures cloud/sky images to create the dataset for the study. The dataset we created emphasizes the small-scale ($50\times 50$ pixels) cloud/sky features that can be directly observed by a ground-based camera and related to the microphysical and radiative properties of clouds and a clear sky. It may be possible to deduce other cloud features related to cloud morphology from the analysis of the larger-scale images from which we took the small-scale samples, making it possible to identify cloud genera.

This study focused on the classification of regions within hyperspectral images into seven distinct cloud/sky categories. While each pixel in a hyperspectral image contains a wealth of spectral information, including textural details can improve the determination of membership in a particular cloud/sky category. Texture and the spatial extent of cloud elements are important factors in traditional methods for cloud classification. One approach to include texture would be to input a group of neighboring pixels into a classifier, such as a neural network. However, that approach does not necessarily optimize single-pixel classifier performance or feature extraction. A better method for feature extraction involves utilizing the contextual information contained in neighboring pixels within the images in a manner similar to positional embedding techniques recently popularized by transformers [9].

We propose a novel approach to pixel classification that includes the original spectral features but also enriches the data by incorporating contextual/textural information derived through the proposed augmentation algorithm. This algorithm, which we term “patch-origin embedding”, augments the pixel feature vector with the posterior probabilities from several classifiers that identify the patch-origin of a pixel. Since each patch may represent a distinct texture, features that distinguish these textures or highlight their similarities can be beneficial. This enriched dataset is designed to enhance the distinctiveness of the spectral features, thereby improving the classifier’s ability to discriminate between different cloud/sky types.

By comparing the classification performance on the original dataset against the augmented (enriched) dataset, this study aimed to demonstrate the potential benefits of our proposed method in enhancing the accuracy and reliability of hyperspectral cloud classification. The dataset is available on IEEE DataPort [10].

2. Collection of the Hyperspectral Dataset

This section describes the collection and post-processing of hyperspectral images and the sampling of pixels from those images to produce the dataset used in this study.

2.1. Image Data Acquisition

The Resonon PIKA XC2 hyperspectral camera used in this study operates as a scanning spectrometer. The camera first captures a small ribbon-like image of the sky within its integration field of view, and then it rotates clockwise around a vertical axis to capture an image of the abutting ribbon and appends it to the preceding image. It continuously rotates, captures, and appends until an entire image is accumulated. The result is an image of $4402\times 1600$ pixels, with each pixel containing 462 bands of spectral data. The working principle of the Resonon camera is illustrated in Figure 1.

The camera is mounted on an Oben VH-R2 tilt head attached to a rotational scanning stage [12]. The tilt head maintains the vertical orientation of the camera at a stable and specific elevation angle during the scanning process. Figure 2 shows how the camera is set up. To seamlessly capture a high-quality image without blurring, the scan rate (SR), the number of degrees the stage pans per second,  the frame rate (FR), and the number of individual exposures the camera captures per second must be synchronized. For our camera, the scan rate and frame rate were related by the following:

$$SR=0.02035FR$$

(1)

The camera captures spectral data across 462 channels, spanning wavelengths from 400 nm to 1000 nm, with measurements taken at intervals of approximately 1.31 nm and a spectral resolution of 1.9 nm FWHM (full width at half maximum) [13]. This configuration ensures a detailed spectral resolution across the visible to near-infrared spectrum. Thus, our HSI dataset has each pixel containing 462 channels, numbered from 0 to 461. The channels are organized in ascending order by wavelength, which supports a more intuitive analysis and data processing. The camera is equipped with an objective lens with a 17 mm focal length, giving a field of view of 30.8 degrees distributed over 1600 spatial pixels. The large number of spatial pixels and the small integration field of view of 0.71 mrad give the camera a spatial resolution high enough to reveal fine details in the cloud structure.

2.2. Image Data Processing Using Radiance Calibration and Normalization

Raw images (e.g., Figure 3) captured by the PIKA XC2 were calibrated in a post-processing step to obtain images for which the digital number of each wavelength channel is equal to monochromatic radiance in units of $\mu$flicks. The calibration was performed using Resonon’s Spectronon software (version 3.5.5) and calibration data from Resonon.

After the calibration, to address variations in total radiance, which can arise from factors like atmospheric conditions and sensor sensitivity, the spectra were normalized using a method that involves dividing each value of monochromatic radiance by the radiance at a specific wavelength (586 nm), which is proportional to the spectrally averaged radiance under all sky conditions. This approach utilizes the radiance at 586 nm as a proxy for the spectrally averaged radiance, as it simplifies the process, and is effective for our purpose. The normalization formula is given by the following:

$$R^{\prime }\left(\lambda \right)={\displaystyle \frac{R\left(\lambda \right)}{R\left(586\mathrm{nm}\right)}}$$

(2)

where $R^{\prime }\left(\lambda \right)$ represents the normalized radiance at wavelength $\lambda$, and $R\left(\lambda \right)$ is the monochromatic radiance at wavelength $\lambda$.

Normalization emphasizes the chromatic content of the data. The sky radiance in a clear sky regions is dominated by short-wavelength light originating from Rayleigh scattering by air molecules. In contrast, the sky radiance in cloudy regions is dominated by Mie scattering by cloud particles, which occurs more uniformly with wavelength [14,15,16,17,18]. By emphasizing the chromatic content of the images, the chromatic differences between dense clouds, thin or semi-transparent clouds, and clear sky are also highlighted. Figure 4 shows exemplary patches for each cloud/sky category. Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10 show the contribution of the normalization to the comparability of spectral signatures by standardizing variations in total radiance. Although normalization using the L2 (or L1) norm could be considered for multispectral imaging due to its ability to account for variations in total radiance across multiple wavelengths, we decided against using it. This decision was based on its computational complexity and the greater benefits derived from focusing on specific critical wavelengths tailored to potential multispectral applications. The L2 norm method would be represented as follows:

$$R^{\prime }\left(\lambda \right)={\displaystyle \frac{R\left(\lambda \right)}{\sqrt{{\sum }_{\lambda }R{\left(\lambda \right)}^2}}}$$

(3)

However, instead of the costly integration over the full set of wavelengths in the denominator, by identifying 586 nm as a key wavelength and using it in Equation (2), we reduce computational requirements and enhance our ability to generalize our approach to more cost-effective multispectral studies in the future.

2.3. Dataset Preparation

A total of 33 images, referred to as parent images, were selected for preparing the dataset for this research project. Using Resonon’s Spectronon software, we extracted 10 to 20 sample patches from each parent image, each with a $50\times 50$ pixel field. Based on our experiments and observations, we decided to use a sample patch size of $50\times 50$ pixels to exclude many unwanted pixels surrounding the cloud of study in the parent image. As a result, we obtained cleaner pixel data for each cloud/sky category used for this study. Figure 3 presents a parent image with 10 selected sample patches, highlighted by red squares, indicating their respective locations.

Each sample patch was also saved as an image, referred to as a patch image. This process yielded a collection of 444 patches. These patches were manually labeled into seven cloud/sky categories based on the meteorological expertise within our team.

Table 1. The cloud/sky categories and the distribution of our 444-patch dataset among these seven categories.

CategoryID	Category Description	Total Patches
c01	Dense Dark Cumuliform Cloud	54
c02	Dense Bright Cumuliform Cloud	79
c03	Semi-transparent Cumuliform Cloud	76
c04	Dense Cirroform Cloud	46
c05	Semi-transparent Cirroform Cloud	56
c06	Clear Sky—Low Aerosol Scattering (Dark)	68
c07	Clear Sky—Moderate to High Aerosol Scattering (Bright)	65
	Total	444

lists the seven cloud/sky categories used in this work and the number of patches in each category. The panels (a)–(g) of Figure 4 display exemplary patches from all seven sky/cloud categories in context.

Although using the texture in the entire patch is the simplest approach, the advantage of HSI cameras lies in the richness of individual pixels, which is why we focused on pixel classification in this study. Once individual pixels can be classified most effectively, a patch-based classification study can follow, potentially achieving even higher accuracies. Our study leveraged spatial information through “patch-origin embedding”, which enhances pixel representation by capitalizing on each pixel’s patch membership. This approach, detailed in Section 3, utilizes predictor features indicative of the patch ID to enhance pixel representation without explicitly analyzing the entire $50\times 50$ pixel field for texture. By concentrating on hyperspectral pixel classification, we leverage the detailed spectral information provided by HSI data. This method, which falls under the umbrella of transfer learning, is enabled by creating auxiliary classes using contextual guidance (spatial relations among pixels) and inspired by our past studies [19,20]. It can be extended to other applications within the field of remote sensing.

It is expected that each category of the labeled dataset has its signature. Figure 5,  Figure 7 and Figure 9 display the spectral signatures of the three main categories: cumuliform (c01), cirroform (c04), and clear sky (c07). Each waveform represents the spectral data from a single pixel extracted from the central location (25,25) of a $50\times 50$ patch for each category. Figure 6, Figure 8 and Figure 10 display the normalized spectral data for each category (as described in Equation (2)). In these six figures, the red dashed-line plots represent the average spectrum for each class, while the other colors illustrate three exemplary individual pixels. For instance, Figure 6 simplifies the signature so that it is more effectively processed by the downstream classifier, in contrast to the unnormalized spectra shown in Figure 5. Each plot in other colors represents the spectrum from a single pixel, with three such pixels provided as examples for each class.

We established a structured naming convention for our image files to ensure systematic organization and straightforward interpretation of our images. Although this dataset, including HSI image files encoded with extensive metadata, is available at [10], we provide a detailed explanation of our naming convention here. This clarification aims to make this paper a useful guide for users of the dataset and to facilitate its ease of use. The filenames for the parent image files are structured as follows:

$$\mathtt{SCAN}\_<\mathtt{date}>\_<\mathtt{time}>\_<\mathtt{azimuth}>\_<\mathtt{elevation}>\_<\mathtt{location}>\_<\mathtt{calibration}>$$

where each component in the filename represents the following:

• <date>: The date the image was captured, formatted as MM-DD-YYYY.
• <time>: The time the image was captured, formatted as HHMM.
• <azimuth>: The azimuth angle of the camera at the beginning of the scan is in degrees, formatted as AZ###. AZ is the horizontal angle measured clockwise from North to the direction the camera was aimed at when the scan began.
• <elevation>: The elevation angle of the camera at the time of capture, formatted as EL##. EL is the vertical angle between the local horizon and the direction the camera was aiming at, measured in degrees. The elevation angle was constant during a given scan.
• <location>: The location of the camera during image capture, either Ground (G) or roof of the AUM library (L).
• <calibration>: The indicator of whether the image has undergone radiance calibration, denoted as calibrated to spectral raDiance (D) or raW (W).

Extensions for these files are .bip for the data and .bip.hdr for the header. The .bip extension indicates a binary file with interleaving by pixel. For each patch image file, we append the following suffix before the file extension to provide specific information about the patch:

$$\_<\mathtt{category}>\_<\mathtt{x}\_\mathtt{coord}><\mathtt{y}\_\mathtt{coord}><\mathtt{width}><\mathtt{height}>$$

This suffix includes the following:

• <category>: The classification of the patch, formatted as c## (c01 through c07 corresponding to our seven categories).
• <x_coord>, <y_coord>: The x and y coordinates indicate the position of the patch within the parent image, each formatted as four digits.
• <width>, <height>: The dimensions of the patch in pixels, formatted as two digits each corresponding to the number of samples and the number of lines in the patch, respectively.

When displayed as a so-called waterfall image, the left-hand side of an image corresponds to those pixels with the highest elevation, i.e., the upward direction from the horizon is to the left of the displayed image. Pixels located at the top of the waterfall image have an azimuth equal to that at the beginning of the scan, i.e., the downward direction on the displayed image is counterclockwise from the beginning azimuth. The origin of the x, y coordinate system, used to identify pixel position, is located in the upper left-hand corner of the waterfall image. The x-coordinate increases to the right and is also referred to as the sample number since each pixel along a horizontal line was imaged during the same exposure of the integration field of view. The y-coordinate increases toward the bottom of the waterfall image and is also referred to as the line number since it numbers the exposures of the integration field of view sequentially. Figure 11 shows the coordinate system displayed with increasing elevation toward the top of the image and increasing azimuth to the right, i.e., the waterfall image is rotated ninety degrees clockwise. The suffix appended to the name of each patch image indicates the x and y coordinates of the upper left corner of the patch within the parent image when the image is viewed as a waterfall image.

Figure 12 shows an example of a parent image file name demonstrating the naming convention. Figure 13 shows the extension added to the name of a parent image to produce the name of a patch selected from the parent image.

Although the whole fully-processed dataset is also available, to enhance data accessibility and facilitate straightforward spreadsheet-based predictive analysis, 100 pixels are randomly selected from each sample patch. This process yields a data matrix with dimensions of $\mathrm{44,400}\times 462$. In addition to the spectral channel data, each pixel (each row of the data matrix) also includes the category of sky/cloud to which it is classified, the ID of the parent image, and the ID of the patch image. These features enhance the granularity and contextual content of the dataset instances. All 444 patch images, with their corresponding appended information, are recorded in a CSV file and made available on IEEE DataPort [10]. The patch and parent images as HSI files (.bip and .hdr) are also available upon request. Each parent image contains a 6.36 GB .bip file and a 5 KB .hdr file. Each sample patch contains a 2.25 MB .bip file and a 5 KB .hdr file. As the parent files are very large image files, we also made 30 smaller unlabeled samples, each approximately $800\times 800$ pixels in size, available on IEEE DataPort [21].

3. Proposed Classification Method with Patch-Origin Embedding

The proposed method aims to enhance classification accuracy by integrating patch-specific information into each pixel’s feature vector. This method generates posterior probabilities for each pixel from classifiers trained on these patches, which are combined with the original spectral data to create an enriched feature vector. These vectors are subsequently merged and then classified into the seven categories using a final classifier. We tested logistic regression (LR) and convolutional neural networks (CNNs) to enrich the dataset. The enrichment was achieved by LR and CNN using the posterior probabilities as features and deriving features from the hidden layers of CNNs that are trained to predict those posteriors. There are alternative methods for generating such patch-embedding features that we experimented with in this study. These features make explicit the discriminative information in a pixel’s spectrum. We called adding these features to the original dataset “patch-origin embedding”. The rationale for relying on such features is rooted in transfer learning in deep learning and positional embedding in transformers. The models trained on the base task to predict the original patch from which each pixel is derived serve as the base task in the parlance of transfer learning [20,22,23]. Once they are learned, they enrich the original dataset, and together, they are used to train-test the target task [20,23,24], which in our application is the supervised 7-class cloud/sky classification task. In this section, we review LR and the computation for the posterior probabilities of patch-origins and describe how the feature set is enriched. We also test CNNs for the same task of modeling patch-origin prediction, as detailed in the Experimental Results. After the enrichment, a Random Forest (RF) classifier is used as one of the target task classifiers. RF is known to have high performance and simplicity [25,26,27,28]. As detailed in the Results section, LR and RF are applied to both the original and enriched datasets, and we compared their performances. The RF classifier has proven to be a robust ensemble of decision trees [28,29].

In its most basic form, the proposed classification method with patch-origin embedding employs LR for linear classification. LR, in its application to binary classification (i.e., $K=2$), models the probability that a given input belongs to the class labeled as “1” (as opposed to class “0”). This is achieved by using a logistic function to map the predicted values to probabilities. The logistic function, also known as the sigmoid function, is defined as follows:

$$\sigma \left(z\right)={\displaystyle \frac{1}{1+e^{-z}}},$$

(4)

where z is the linear combination of the input features ($\mathbf{x}$) weighted by the coefficients (${\mathit{\beta }}_{\mathit{i}}$) from the model, expressed as $z={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_n{x}_n$. In summary, LR calculates a boundary line between the two classes for binary classification. This boundary line’s probability is 0.5, which exemplifies an equal likelihood of the data belonging to either class. If p > 0.5, it resides on the positive side of the boundary line for class 1, symbolizing a greater probability of being related to class 1. For p < 0.5, it is more likely to belong to class 0. Thus, the probability of the negative residing data value to class 0 would be the complement of the probability of residing in class 1 [27].

The logistic regression model, thus, predicts the probability p that an observation belongs to the class 1 as follows:

$$p=\sigma ({\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_n{x}_n).$$

(5)

This probability p can be interpreted as the likelihood that a given pixel belongs to a cloud/sky category based on its spectral characteristics. Logistic regression is particularly advantageous in scenarios where a clear probabilistic interpretation is required, making it suitable for classifying pixels in hyperspectral images where each pixel’s class membership (such as cloud/sky category) needs to be probabilistically quantified.

For scenarios involving multiple classes (i.e., $K>2$), the binary logistic regression model is extended to multinomial logistic regression, also known as softmax regression. This model handles multiple categories by modeling the probability that a given input belongs to each class. The decision rule for multinomial logistic regression involves computing a set of probabilities for each class using the softmax function, defined as

$$P(y=k|\mathbf{x})={\displaystyle \frac{e^{{\mathbf{x}}^{\top }{\beta }_k}}{{\sum }_{j=1}^K{e}^{{\mathbf{x}}^{\top }{\beta }_j}}},$$

(6)

where $P(y=k|\mathbf{x})$ is the probability that the observation $\mathbf{x}$ belongs to class k, ${\beta }_k$ is the coefficient vector for class k, ${\mathbf{x}}^{\top }{\beta }_k$ is the dot product of the transpose of the input pixel feature vector $\mathbf{x}$ and the coefficient vector ${\beta }_k$, and K is the total number of classes.

The predicted class label $\widehat{y}$ for an observation is then given by the class that has the highest probability:

$$\widehat{y}=arg\underset{k}{max}P(y=k|\mathbf{x}),$$

(7)

where $arg{max}_k$ denotes the argument of the maximum, which identifies the class k that maximizes the probability $P(y=k|\mathbf{x})$.

Rather than applying $arg{max}_k$ to the logistic regression output for obtaining a nominal value for decision-making, our study utilizes the calculated probabilities as posterior probabilities (each pixel’s likelihood of belonging to each one of the K patches). We employ both a single-shot version, with K set to the number of all patches in the training set, and a combination of multiple runs with smaller K values and, thus, fusing posterior probabilities from multiple such K-class classifications. Specifically, we perform a procedure where K classes are randomly selected, and we compute the merged posterior probabilities over these classes. This process is repeated N times to ensure generalization in the predictions. For our study, we set $K=30$ and $N=20$, allowing us to capture a broad spectrum of scenarios and reduce the influence of any single random selection of classes. The specific details of incorporation of the contextual information can be varied; to that end, we also performed another alternative in which we used the shattering in VC-dimension: we randomly picked a set of patches as class-1 and the rest as class-0 and repeated such random splits N times to ensure there is rich enough contextual information embedded into the feature vector.

The enhancement process involves stacking the posterior probabilities $p_{i,k}$ for each pixel i across multiple classes k and potentially across multiple model runs (iterations). For the multi-selection approach where N random selections of K classes are considered, the stacked vector for each pixel i is constructed as follows:

$${\mathbf{v}}_i=\left[{\mathbf{x}}_i,{\mathbf{p}}_i^{\left(1\right)},{\mathbf{p}}_i^{\left(2\right)},\dots ,{\mathbf{p}}_i^{\left(N\right)}\right],$$

(8)

where

• ${\mathbf{x}}_i$ is the original spectral feature vector of pixel i.
• ${\mathbf{p}}_i^{\left(n\right)}$ represents the output of a softmax (or hidden) layer of the network for pixel i obtained from the n-th run. This vector contains the probabilities for each class k, which are computed by the network. Specifically,

  $${\mathbf{p}}_i^{\left(n\right)}=\left[p_{i,1}^{\left(n\right)},p_{i,2}^{\left(n\right)},\dots ,p_{i,K}^{\left(n\right)}\right],$$

  where $p_{i,k}^{\left(n\right)}$ is the probability of pixel i belonging to class k in the n-th run, and K is the total number of classes.
• ${\mathbf{v}}_i$ is the enhanced feature vector for pixel i, created by concatenating the original feature vector ${\mathbf{x}}_i$ with the hidden layer outputs from all N runs.

These enhanced vectors ${\mathbf{v}}_i$ are then utilized as the input for the final classifier, such as a Random Forest. The RF utilizes decision trees in its evaluation to produce a probability of belonging. At each level of the decision tree, the data are tested and branch onto the next corresponding node that matches the evaluation outcome. This process continues until it ends at a leaf node, where further testing will guarantee the same results. It is crucial to note that decision trees do not assume a linear relationship and constantly adapt the discriminate tests accordingly. The combined results of multiple decision trees, the most frequent result or majority vote, produce a probability of belonging [27,30]. This method ensures that each classifier not only utilizes the inherent spectral characteristics of each pixel but also leverages the learned probabilistic distinctions between patches, thereby capturing both the spectral and contextual information crucial for accurate cloud classification.

4. Results

This section presents a concise analysis of the experimental results obtained by systematically varying the feature sets used to train our classifiers, thereby assessing the efficacy of different feature enrichment techniques in hyperspectral image classification. For training/testing the classifier models and for computing the embeddings, the Group Shuffle Split (GSS) technique was used to prevent class-label leakage between the training and test sets. GSS effectively ensures that all pixels from the same patch are allocated exclusively to either the training set or the test set, rather than being dispersed across both. We applied GSS to our dataset of 444 patches, designating 20% as the test size. This approach resulted in 355 patches being used for training in each experimental run and 89 patches for testing. Results were averaged over ten runs to enhance the reliability and robustness of our findings.

We start this section by demonstrating the challenging nature of the problem. Using an 80-20 train–test split with a decision tree on the raw (unnormalized) data for the initial evaluation yields low classification results with an accuracy of 71.90% out of 8900 test examples. The results summarized in

Table 2. Detailed accuracy by class for the initial analysis using a simple decision tree classifier, without radiance normalization, and without contextually guided feature augmentation.

Class	TP Rate	FP Rate	Precision	F-Measure	MCC
c01	0.672	0.047	0.511	0.580	0.551
c02	0.802	0.036	0.848	0.824	0.782
c03	0.684	0.072	0.674	0.679	0.608
c04	0.672	0.029	0.723	0.697	0.664
c05	0.364	0.054	0.366	0.365	0.311
c06	0.847	0.061	0.737	0.788	0.744
c07	0.738	0.028	0.868	0.798	0.755
Weighted Avg.	0.683	0.047	0.675	0.676	0.631

and

Table 3. Confusion matrix on the sample test set for the initial analysis using a simple decision tree, without radiance normalization, and without contextually guided feature augmentation.

	Predicted Class
Actual Class	c01	c02	c03	c04	c05	c06	c07
c01 = Cumuliform-1	403	28	96	57	16	0	0
c02 = Cumuliform-2	156	1443	191	2	7	0	1
c03 = Cumuliform-3	175	125	1094	50	152	2	2
c04 = Cirroform-1	20	1	20	605	254	0	0
c05 = Cirroform-2	35	4	193	123	255	87	3
c06 = Clear-Sky-1	0	0	21	0	12	1271	196
c07 = Clear-Sky-2	0	100	7	0	0	365	1328

indicate the difficulty of the classification problem. The imbalance of class labels and similarity among different classes (such as cumuliform classes c01/c02/c03 and cirroform classes c04/c05) contribute to the challenge, as evidenced by the varying precision, recall, F-measure, and Matthews Correlation Coefficient (MCC) values for each class. The results presented in Table 2 and Table 3 benefit from additional examples from class c04, which helps improve its classification accuracy. However, distinguishing between the closely related cirroform classes c04 and c05 remains challenging, indicating the need for further refinement in future dataset updates [10].

In addition to the pixel classification study, we explored the effectiveness of RGB renders for classification by employing both custom-built and pre-trained deep learning models. We used these models to classify $50\times 50$ patches from the dataset into seven classes, splitting the 444 image patches into training and test sets (80:20 ratio). Models trained from scratch achieved only about 40% accuracy. Testing several pre-trained models, including VGG16, EfficientNetV2, ResNetV250, InceptionV3, MobileNetV2, and DenseNet121, with dataset-specific fine-tuning improved accuracy but still fell short compared to our proposed pixel classification method. These experiments show that differentiating subclasses was particularly challenging. We reclassified the seven original classes into three broader categories: merging the first three cumuliform classes into one, combining the next two cirroform classes into another, and grouping the last two clear-sky classes into a final category. Using this revised three-class classification scheme, the custom-built CNN model (see Figure 14) achieved an average validation accuracy of 76% across ten data splits. Similarly, pre-trained models yielded validation accuracies between 74% and 76%, comparable to the custom-built model. The model that achieved the highest accuracy was structured as follows.

The proposed techniques, utilizing normalization and feature enrichment via contextual guidance, significantly improve these poor initial results as follows.

Table 4. Overview of feature sets and their dimensionalities.

Feature Set	Description	Dimensions	Total Features
Original	Raw hyperspectral data	462	462
CNN Hidden	Original + CNN Hidden Layer Features	462 + $20\times 32$	1102
CNN Posterior	Original + CNN Softmax Outputs	462 + $20\times 30$	1062
LR Posterior	Original + Incremental LR Posteriors	462 + $20\times 30$	1062

provides a detailed overview of these feature sets along with their respective dimensions. The original dataset comprises 462 dimensions. In addition to this, three other feature sets were utilized: CNN Hidden, CNN Posterior, and LR Posterior. These three are designed for incremental learning, which is advantageous for managing and processing large datasets effectively with $K=30$ and $N=20$ as outlined in Equation (8).

For our proposed incremental approach, for each one of the $N=20$ classifiers, we randomly selected $K=30$ classes from a total of 355 training patches at a time and trained a CNN to compute posterior probabilities for these classes. As Table 4 shows, the resulting $30\times 20$ features are appended to the original 462-dimensional pixel feature vector, creating the “CNN Posterior” feature set with a total of $462+20\times 30$ dimensions for each pixel. Alternatively, the “CNN Hidden” feature set utilizes features from the intermediate layers of the CNNs to enrich the original spectral data. This method incorporates spatial-contextual features extracted from hidden layers, enhancing the base 462 spectral features with additional data from 20 convolutional filters in a key hidden layer. Each filter produces 32 feature maps, resulting in a total of 1102 features. The CNN significantly augments the dataset with informative features crucial for precise classification by capturing and encoding complex spatial relationships within the hyperspectral images.

Both the CNN Hidden and CNN Posterior methods utilize features extracted by the CNN architecture illustrated in Figure 15. In this architecture, the final layer employs a softmax activation function to generate posterior probabilities of patch memberships for each input pixel, as utilized in Equation (8) for the CNN Posterior method. Additionally, we implemented the CNN Hidden approach, which extracts features from the hidden layers of the CNN. This approach is a well-established practice in transfer learning and deep learning, recognized for its effectiveness in feature extraction and serving as an excellent initialization point for downstream classification tasks [20,22,31]. Our CNN architecture in Figure 15 begins with an input layer designed to process hyperspectral images, which are input as 1D arrays (n_channels = 462) with dimensions corresponding to the number of spectral channels (it is designed to process a single pixel as input).

• First Convolutional Layer: This layer uses 32 filters with a kernel size of 25 and a stride of 4. The use of a relatively large kernel size in the initial layer is intended to capture broader spectral features early in the network.
• Pooling and Dropout: Following the first convolution, a max pooling layer reduces the output’s spatial dimensions, which helps reduce computation and control overfitting. This is accompanied by a dropout layer with a rate of 0.2 to further regularize the model.
• Subsequent Convolutional Layers: Additional convolutional layers with smaller kernels enhance the network’s ability to learn more refined features. Each convolutional stage is followed by dropout and max pooling layers.
• Global Max Pooling and Dense Layers: The final stages of the network flatten the output and transition to dense layers, culminating in a softmax output layer that classifies each pixel into one of the K classes.

After the patch-origin embeddings are learned from the base task using the 355 training patches, we initiate the target (downstream) task, which involves classifying pixels into seven cloud/sky classes. For this, pixels from the 20% of patches reserved for testing are processed through the K-class classifiers to compute their resemblances to the training patches. This process aims to augment the test set in a manner analogous to the training set preparation, ensuring consistency in feature representation across both sets. These enhanced training and test sets are then input into a final classifier, either Logistic Regression (LR) or Random Forest (RF), chosen for their simplicity. This choice allows us to more effectively evaluate the enhancement’s contributions without retraining a complex deep network for the seven-class classification task. The RF classifier operates on a majority voting principle among a number of decision trees, and we used the default configuration in the scikit-learn implementation that employs 100 trees [29,30].

Table 5. Classifier performances without the radiance normalization.

Classifier	Feature Set	Accuracy (%)	MCC (%)
RF	Original	$78.12\pm 2.88$	$74.51\pm 3.09$
RF	LR Posterior	$79.19\pm 2.50$	$75.75\pm 2.71$
LR	Original	$74.91\pm 3.81$	$70.73\pm 4.35$
LR	LR Posterior	$75.11\pm 3.36$	$70.95\pm 3.79$

demonstrates that the classification accuracy suffers without the normalization process outlined in Equation (2). As shown in

Table 6. Classifier performances with the radiance normalization on all feature sets.

Classifier	Feature Set	Accuracy (%)	MCC (%)
RF	Original	80.60	77.30
RF	CNN Hidden	82.90	79.96
RF	CNN Posterior	81.79	78.81
RF	LR Posterior	81.80	78.69
LR	Original	77.94	74.27
LR	CNN Hidden	80.95	77.65
LR	CNN Posterior	79.28	75.70
LR	LR Posterior	78.14	74.48

, the normalization addresses variations in total radiance and emphasizes chromatic content, leading to better differentiation between the signatures of cloud types and clear sky. The results presented in Table 6 evaluate and compare the performance of the classifiers with different feature sets based on the accuracy that measures the proportion of correctly classified instances and on Matthews Correlation Coefficient (MCC) that provides a more informative metric than accuracy, especially with imbalanced datasets. MCC balances for the bias created by classes with higher prior probabilities. An MCC of 1 indicates a perfect prediction, while an MCC of 0 indicates a prediction no better than random guessing based on the probability distribution of the priors [32].

Figure 16 shows four sample images from [21], which contains 30 large images (approximately $800\times 800$ pixels) not used in the training set of this study. The first column displays the original image. For the middle and rightmost columns, we applied the trained model to classify each pixel individually. The middle column presents the classification results using the original features with a Random Forest classifier. The rightmost column illustrates the results using CNN Hidden features combined with a final Random Forest classifier. The middle column offers a more readily interpretable segmentation. However, the rightmost column reveals additional details concerning brightness that are missed in the middle column. In some cases (e.g., the third row), this column appears to blur the segmentation by adding details not discernible in the original render, although these details may be present in the richer data.

5. Discussion

Our study was supported by a rigorously collected, calibrated, and normalized dataset, enhanced with metadata such as azimuth, elevation, date–time, and patch-membership. Each image in our dataset underwent processing to ensure uniform radiance levels, with bounding boxes designating patches that were subsequently class-labeled for supervised learning in a seven-class classification problem. Now made publicly available, this dataset effectively facilitates the analysis of cloud/sky conditions, serving as a valuable resource for climate analysis and related HSI research.

The analysis of the dataset collected clearly demonstrates that both Random Forest (RF) and Logistic Regression (LR) classifiers benefit from the integration of enhanced feature sets, with improvements in accuracy, Matthews Correlation Coefficient (MCC), and visually more plausible classification results on the new large test images. The experimental results provide a comprehensive evaluation of two classifiers, RF and LR, where RF consistently outperforms LR in terms of both mean accuracy and MCC on the original feature set. This outcome is expected because of the nonlinear capabilities of RF [25]. Additionally, as the comparison between Table 5 and Table 6 shows, the normalization process outlined in Equation (2) improves the classification accuracy of all classifiers on all features sets.

Overall, our results highlight that CNNs work favorably in extracting informative features, and their contextual enrichment significantly enhances pixel classification performance. Moreover, our contextual guidance approach, which was successful in this study, can be extended to other applications beyond remote sensing.

Our experimental results demonstrate that enriching the dataset with additional examples from undersampled classes improves classification accuracy. Therefore, as future work, we plan to enhance the dataset hosted in repository [10]. Future updates will include adding more classes and increasing the number of patches to provide a more comprehensive representation of cloud and sky conditions. These updates will be implemented within the same repository and made available as Open Access to ensure broader academic use in hyperspectral image analysis and cloud classification.