Symmetric U-Net Model Tuned by FOX Metaheuristic Algorithm for Global Prediction of High Aerosol Concentrations

Abstract

In this study, the idea of using a fully symmetric U-Net deep learning model for forecasting a segmented image of high global aerosol concentrations is implemented. As the forecast relies on historical data, the model used a sequence of the last eight segmented images to make the prediction. For this, the classic U-Net model was modified to use ConvLSTM2D layers with MaxPooling3D and UpSampling3D layers. In order to achieve complete symmetry, the output data are given in the form of a series of eight segmented images shifted by one image in the time sequence so that the last image actually represents the forecast of the next image of high aerosol concentrations. The proposed model structure was tuned by the new FOX metaheuristic algorithm. Based on our analysis, we found that this algorithm is suitable for tuning deep learning models considering their stochastic nature. It was also found that this algorithm spends the most time in areas close to the optimal value where there is a weaker linear correlation with the required metric and vice versa. Taking into account the characteristics of the used database, we concluded that the model is capable of generating adequate data and finding patterns in the time domain based on the ddc and dtc criteria. By comparing the achieved results of this model using the AUC-PR metric with the previous results of the ResNet3D-101 model with transfer learning, we concluded that the proposed symmetric U-Net model generates data better and is more capable of finding patterns in the time domain.

1. Introduction

The concept of symmetry belongs to the concepts of the highest order in terms of generality. These terms are called categories, and they cannot be determined simply, but rather by relating them to a series of other terms: part, whole, structure, element, and law. That is why symmetry as a category is defined differently in various sources. There are usually two types of approaches: symmetry represents harmonious, aesthetic proportionality, and harmony of parts such that it reflects beauty and perfection; and the second approach is more precise and presents symmetry as a well-defined concept of balance that can be shown and proven in accordance with the rules of the formal system such as geometry, mathematics, physics, and chemistry. Different types of symmetries occur in various scientific fields. For example, symmetry in geometry is a characteristic of geometric figures and other abstract mathematical elements when there is a correspondence with respect to a center, axis, or plane. In nanotube physics, the distribution of channeled ions is symmetric whether the nanotubes are radially deformed or not or whether the nanotubes are carbon or not. Symmetric distributions of protons channeled through nanotubes are shown in [1,2].

Symmetry has been underutilized in the understanding of the mechanics of neural networks. Symmetry can provide a theoretical framework based on the intrinsic symmetries built into the network architecture that are present for any dataset [3]. Symmetry in deep learning can explore various types of fundamental properties of nonlinearity and the connection to symmetries of weight space [4]. In the study [5], a new method was proposed that automatically detects symmetries of a dataset using a deep learning method.

Aerosols are solid particles or small liquid droplets formed from smoke, dust, fly ash, and condensed gaseous substances that can be found in atmospheric air. Some of these particles are so large that if they are bright, they can be seen as fog or cloud, or if they are dark, they can be seen as smoke or soot. Forecasting the global aerosol optical thickness (AOT) can help a variety of scientific considerations beyond those of global air pollution, such as the influence of the Saharan air layer on Atlantic tropical cyclones [6].

AOT serves as a key metric for understanding the impact of aerosols on human health and the Earth’s climate. Its range from 0 to 1 covers the different aerosol scenarios encountered in our atmosphere. An AOT value of less than 0.1 corresponds to a crystal-clear sky with maximum visibility. A value of 1 indicates very hazy conditions due to the significant presence of aerosols [7].

Based on the attention mechanism, in the study [8], the U-Net model adopts a symmetric encoder–decoder structure, and this symmetric network structure is concise and stable, greatly improving the remote sensing image segmentation effect. The U-Net is a very efficient image segmentation method that has excellent performance in many image segmentation tasks, especially medical image segmentation tasks. The U-Net was developed based on a fully convolutional network architecture and was first applied to biomedical image segmentation in 2015 [9].

Some studies have shown that attention mechanisms can improve classification effects [10,11]. In this study, a model on the basis of the U-Net and attention mechanism is proposed like in [8]. This improvement allows the model to better focus on areas with high aerosol optical thickness (AOT). The attention module serves to learn the useful patterns of the input image and which important features it should focus on to perform its task [8].

In order to optimize the parameters of the deep learning model, authors often use metaheuristics as a method that performs a fine search for optimal values in a multidimensional space. Metaheuristic methods are usually nature-inspired and can be classified as evolutionary-, swarm-, physics-, human-, biology-, system-, math-, or music-based [12].

The structure of this paper is as follows: The methodology and literature review are described in Section 2. Subsequently, the model design is given in Section 3. The results are presented and discussed in Section 4. Finally, Section 5 summarizes the work of this study and future work.

2. Data and Methodology

2.1. Literature Review and Related Work

In keras, there are many state-of-the-art pre-trained models that can be used as the backbone for a new model in image processing [13]. Several papers showed that U-Net, as a certain type of CNN, gives excellent performance in satellite remote sensing image segmentation , in order to detect contrails [8,14]. To explore temporal and spatial spreading of contrails with their climate impact, the accurate and precise detection is needed... Similar AI-based detection method was used to detect sea-ice leads from satellite thermal infrared window data of moderate resolution imaging spectroradiometer (MODIS) [15].

In the area of aerosol image prediction, a deep learning model intended for global time–space prediction showed very good characteristics [16]. This model, which is composed of ConvLSTM2D layers followed by Dropout and Normalization layers, showed that it easily captures changes in the time domain with better characteristics than the ResNet3D model developed in our previous study with applied transfer learning technology [17].

The relatively new FOX metaheuristic algorithm, inspired by the method of hunting a red fox in the snow, turned out to be a very good candidate for machine learning model optimization. This swarm-based method showed better characteristics in certain segments than the classic gray wolf, particle swarm, whale algorithm, and fitness dependent optimizers [18].

In machine learning classification, precision and recall metrics provide very useful information, and a curve obtained by graphically comparing their characteristics is known as area under curve for precision vs. recall (AUC-PR) [19]. This precision is a measure that tells how many instances are correct in relation to all positive answers, defined as:

Precision = TP/(TP + FP)

(1)

where TP is the true-positive rate and FP is the false-positive rate. Recall is a measure of how accurately the model predicted from all positive instances and is defined as:

Recall = TP/(TP + FN)

(2)

where FN is the false negative rate. AUC-PR is a highly valued metric for evaluating model performance, especially in cases of unbalanced databases where positive classes are of particular importance, which is our case.

2.2. Pre-Training Process

The input dataset consisted of snapshots from 18 February 2000 to 17 November 2023 on every 8 days of global satellite images with a total of 1094 images. The images used were selected from NASA’s Terra MODIS satellite with an original resolution of 3600 × 1800 pixels. This resolution provided us with clean information without the effect of averaging neighboring pixels as a side effect of the downscaling method. Unread pixels in images are shown in black, and AOT intensity ranges are shown from white to dark brown. The useful reference range is scaled with decimal values from 0 to 1. The images were created using a combination of the dark target and deep blue methods. The deep blue algorithm was used for all land areas, while dark target was used for oceans and dark soil or vegetation. [7]. The input data were 3D sequences of 8 Terra MODIS images in RGB format resized to 400 × 200 pixels with the third temporal component. The output was one 2D-segmented image with the same resolution of 200 × 400. The original input dataset was converted by masking to, black/white (0/1) values, Binary image segmentation model was trained for classifying each pixel. The unread data and AOT values less than 0.8 (black pixels [0, 0.8]) were masked by 0, while AOT values above 0.8 (white pixels (0.8, 1]) were masked by 1. An image of probabilities range from 0 to 1, as segmentation result, was encoded as bytes ranging from 0 to 255, [14].

To avoid overfitting, care was taken to avoid cross-contamination of the training, validation, and testing datasets. A ratio of 70%/20%/10% was used for training, validation, and testing, respectively, in a given period of time. In order to achieve complete symmetry of the deep learning model, we prepared the database in such a way that the input and output data had the same form; i.e., we decided on sequence-to-sequence mode. With this, we achieved that the last output image represents the prediction of high concentrations of AOT in 8 days.

By examining the database, we found that the distance-domain criterion (ddc) and time-domain criterion (dtc) for the area under the curve for the precision versus recall (AUC-PR) metric were 0.368 and 0.496, respectively [17].

3. Symmetric U-Net Model

The model is written in Python (3.10.13 version) (Jupyter Notebook environment) using tensorFlow/keras as well as other data science libraries. Like in study [8], the proposed architecture was obtained by modifying the original U-Net model maintaining a symmetric structure. We also tried the ConvLSTM2D, Normalization, and Dropout layers proposed in the study [16], as well as the Conv3D and AvgPooling3D layers.

The original U-Net model [9], as well as the model in a similar study [8], is intended for the purpose of segmenting the input images. The model developed in this study is primarily adapted from the original one by using image sequences that provide the model with historical changes within the used image sequence. This time window, which does not exist in the original setup, provides the model with the ability to predict the future segmented image. Therefore, by introducing the third time dimension in our case, ConvLSTM2D was used instead of the Conv2D layer; MaxPooling3D was used instead of MaxPooling, and UpSampling3D was used instead of UpSampling.

During the manual search of effective and useful layers, we found that Conv3D, Attention, Dropout, and AvgPooling3D layers do not have a positive impact on the model results. Also in the mentioned process, we found that the ConvLSTM2D, Normalization, and MaxPooling3D layers have a positive impact on the model. The model development process was divided into two phases: the first was manual, and the second phase was automated metaheuristic screening. The reason for this is that as the required variables increase in metaheuristic search, hardware resources and time increase significantly, a problem better known as the ‘curse of dimensionality’ [20]. The best combination of layers in a symmetric U-Net model that we found is shown in Figure 1. The encoding path is on the left half (side), while the decoding path is on the right half (side). Every block in the encoding path of the U-Net contains a ConvLSTM2D, Normalization, and MaxPooling3D layer. The max pooling reduces the dimensionality of each feature map. Each block in the decoding path contains symmetrically to the encoding path the up-sampling and dimension recovery, the fusion operation, as well as Normalization and ConvLSTM2D layers. Each blue box corresponds to a multi-channel feature map.

4. Results and Discussion

Based on the proposed structure of the model, hyperparameters were tuned, which significantly affected the results of the model. The stochastic nature of deep learning models introduces a certain level of uncertainty. Due to the study of very small individual influences of the required parameters on the results of the model, the learning process was adapted to the deterministic principle.

The stochastic nature of the deep learning model occurred due to the random setting of the initial values of weights and biases, as well as due to the parallelization process of training where the results of individual epochs were not obtained simultaneously [21]. A deterministic learning process can be enabled in the tesorflow/keras environment, but it leads to an increase in training time. We solved the problem of different weights and biases by setting the seed work.

The bias values in all ConvLSTM2D layers were performed by setting the bias_initializer parameter to 0. The weights values were set by defining the set_random_seed parameter, as well as by defining the GlorotUniform and orthogonal initializers to specific seed values. The mentioned two initializers were used for the kernel_initializer and recurrent_initializer parameters of the ConvLSTM2D layers.

We also achieved prevention of overfitting by using the EarlyStopping and ReduceLROnPlateau as regulation methods during learning. Monitors for these methods were tied to metrics achieved on the validation dataset, so the test dataset was used exclusively for testing purposes.

The selected loss function ‘BinaryCrossentropy’ was calculated based on the differences between the prediction and the original images for the entire data sequence; i.e., for a set of eight images, while the metric ‘BinaryAccuracy’ was modified to calculate the difference for only the last output image, which represents the prediction of the image of high aerosol concentrations. In this way, the regulation of the learning rate parameter during the learning process was related to the quality of the last image, i.e., prediction, and the regulation of EarlyStopping was related to the possibility of finding patterns in the time domain of the entire segment, i.e., sequence.

During the metaheuristic process, the required parameters are those that showed potential during the manual search. The mentioned parameters and their range with optimal values are given in

Table 1. Symmetric U-Net model parameters searched by metaheuristics.

Parameters	Type	Range/Values	Optimal Solution
Recurrent dropout	Float	0, 0.5	0
Learning rate	Float	0.001, 0.1	0.044
Number of filters in hidden layers	Integer	1, 6	5
Activation	String	sigmoid, hard sigmoid	hard sigmoid
Recurrent activation	String	sigmoid, hard sigmoid, relu, tanh	hard sigmoid
Threshold	Float	0.1, 0.9	0.1

.

The FOX metaheuristic was initialized to search the above six parameters of different types and ranges with a population size of 10 and 15 epochs. The parameters Recurrent dropout, Number of filters in hidden layers, Activation, and Recurrent activation refer to the ConvLSTM2D layers. The Threshold value represents the limit when classifying pixel values in model output data. The required criterion was finding the maximum value of the AUC-PR metric on the test dataset. The range of values obtained for this metric was from 0.033 to 0.572, where the maximum value of the metric was obtained with the values shown in Table 1 as the optimal solution.

During the search, the FOX algorithm sometimes repeated the entire combination of parameter values, which makes it suitable for working with deep learning models that are stochastic in nature. The distribution of values for the searched parameters in the metaheuristic process of the mentioned algorithm is given in Figure 2.

The candidate values belong to the set of values of the defined range for each required parameter by the FOX metaheuristic algorithm given in the column Range/Values, Table 1. In the cases of a continuous domain search by searching for decimal values, for better visibility of the graphics, rounding was performed to 0.1, 0.05, and 0.01 for Threshold, Recurrent Dropout, and Learning Rate, respectively. The Solution represents the optimal value for each parameter individually according to the FOX metaheuristic algorithm, and these values are given in the Optimal solution column, Table 1.

As can be seen from Figure 2, the FOX algorithm most often searches areas close to optimal values, which is expected for metaheuristic algorithms. The percentage of searches in areas close to Optimal were 95.03%, 91.93%, 77.64%, 56.52%, 27.33%, and 0.62% for Activation, Recurrent activation, Threshold, Recurrent dropout, Number of filters in hidden layers, and Learning rate, respectively. However, with some parameters, a large retention is observed in areas that are not of interest, such as with the search Learning rate and Number of filters in hidden layers.

In order to better understand mutual relations, Pearson correlation testing was performed between all required parameters and the obtained AUC-PR metric,

Table 2. Pearson correlation of hyperparameters and AUC-PR metrics.

	Recurrent dropout	Learning rate	Number of filters in hidden layers	Activation	Recurrent activation	Threshold	AUC-PR
Recurrent dropout		0.319679	0.483386	0.1497	0.201581	0.306453	0.362163
Learning rate	0.319679		0.314502	0.022654	0.126075	0.17775	0.505931
Number of filters in hidden layers	0.483386	0.314502		0.180714	0.199243	0.417116	0.475473
Activation	0.1497	0.022654	0.180714		0.35348	0.210917	0.049967
Recurrent activation	0.201581	0.126075	0.199243	0.35348		0.437246	0.094611
Threshold	0.306453	0.17775	0.417116	0.210917	0.437246		0.388265
AUC-PR	0.362163	0.505931	0.475473	0.049967	0.094611	0.388265

.

From Table 2, it can be concluded that the Learning rate parameter has the strongest linear Pearson correlation with the AUC-PR metric with a value of 0.506. The next-strongest correlation with the mentioned metric is Number of filters in hidden layers with 0.475. On the other hand, the Activation function of the ConvLSTM2D layers of the deep learning model has the lowest correlation with the selected metric with 0.05. Heatmap colors given in the Table 2 are arranged from green to red and visually represent lowest to greatest values respectively.

On the basis of the previous two tests, it can be concluded that the time spent in the area close to the optimal value for some required parameter in FOX algorithms is inversely proportional to the intensity of the linear Pearson correlation, viewed relative to all parameters.

Taking into account the fact that the achieved prediction results of the machine learning model are highly dependent on the database and its preparation, we used an objective way of presenting the results through the ddc and dtc criteria, as well as the relative coefficients ddr and dtr [17]. Comparing the best-achieved result of the AUC-PR metric of 0.572 with the ddc and dtc criteria of 0.368 and 0.496, respectively, it can be concluded that the symmetric U-Net model is capable of generating adequate data and finding patterns in the time domain. The calculated relative coefficients ddr and dtr have values of 1.554 and 1.153, respectively. The previously developed ResNet3D-101 model with transfer learning [17] gave an AUC-PR metric for global aerosol prediction of 0.553, achieving values of ddr and dtr relative coefficients of 1.502 and 1.115, respectively. Based on the comparison of the values of the relative coefficients, it can be can concluded that the symmetric U-Net model from this study is capable of generating better data than the developed ResNet3D-101 model from previous study, i.e., better-achieved relative coefficient ddr andmore capable of finding patterns in the time domain based on a better relative coefficient dtr.

5. Conclusions and Future Research

The paper used a metric based on precision and recall rates that describe the success of reproducing positive labels, i.e., high concentrations of aerosols; thus, the AUC-PR metric is adequate for this application. By establishing a deterministic mode when training the model, we ensured the absolute reproducibility of the results during the metaheuristic process, which provided us with an insight into the finest influences of the required parameters on the results.

The model training process was adapted to finding patterns in the time domain via a loss function for the entire sequence of images and through EarlyStopping controller. The quality of the last image in the sequence for aerosol prediction was regulated through the metric modified for only the last image and through the process of reducing the learning rate during training. The Fox metaheuristic has proven to be very effective for this purpose. It has shown adaptability to the stochastic nature of deep learning algorithms, but it would be good if there was a Boolean parameter for deterministic to increase efficiency for such use. Based on the distribution of the required parameters as well as Pearson linear correlation, it can be concluded that the FOX algorithm spends the most time in areas close to the optimal value with the weakest correlation of the parameter with the required metric and vice versa.

The proposed symmetric U-Net model for forecasting high concentrations of aerosols achieved an AUC-PR metric of 0.572. Based on the calculated ddc and dtc criteria, the developed model is capable of generating adequate data and finding patterns in the time domain. Based on the comparison of the achieved relative coefficients ddr and dtr, the proposed symmetric model generates better quality data and is more capable of finding patterns in the time domain than the previously developed ResNet3D-101 model with transfer learning technique [17].

Considering that the concept of the symmetric U-Net model is well proven in this study for predicting high concentrations of aerosols with segmented satellite images, in further research, we plan to also apply it to datasets of a local character with a potential connection to other physical quantities measured from the ground.