SAM-Net: Spatio-Temporal Sequence Typhoon Cloud Image Prediction Net with Self-Attention Memory

Abstract

Cloud image prediction is a spatio-temporal sequence prediction task, similar to video prediction. Spatio-temporal sequence prediction involves learning from historical data and using the learned features to generate future images. In this process, the changes in time and space are crucial for spatio-temporal sequence prediction models. However, most models now rely on stacking convolutional layers to obtain local spatial features. In response to the complex changes in cloud position and shape in cloud images, the prediction module of the model needs to be able to extract both global and local spatial features from the cloud images. In addition, for irregular cloud motion, more attention should be paid to the spatio-temporal sequence features between input cloud image frames in the temporal sequence prediction module, considering the extraction of temporal features with long temporal dependencies, so that the spatio-temporal sequence prediction network can learn cloud motion trends more accurately. To address these issues, we have introduced an innovative model called SAM-Net. The self-attention module of this model aims to extract an inner image frame’s spatial features of global and local dependencies. In addition, a memory mechanism has been added to the self-attention module to extract interframe features with long temporal and spatial dependencies. Our method shows better performance than the PredRNN-v2 model on publicly available datasets such as MovingMNIST and KTH. We achieved the best performance in both the 4-time-step and 10-time-step typhoon cloud image predictions. On a cloud dataset consisting of 10 time steps, we observed a decrease in MSE of 180.58, a decrease in LPIPS of 0.064, an increase in SSIM of 0.351, and a significant improvement in PSNR of 5.56 compared to PredRNN-v2.

1. Introduction

Considering the powerful nonlinear modeling capabilities demonstrated by deep learning in image processing tasks, many studies have attempted to use a recurrent neural network (RNN) or a combination of convolutional neural network (CNN) and RNN to characterize and predict satellite cloud images, in order to provide better visual weather forecasts for end users. In addition, spatio-temporal sequence prediction can also be applied to disaster prevention [1] and unmanned driving [2,3]. However, early model prediction performance was not satisfactory, because satellite cloud images have strong complexity in both the time and spatial dimensions, and dynamic temporal and spatial features are difficult to capture.

At present, spatio-temporal sequence prediction has become a mainstream method. Shi et al. [4] first defined meteorological prediction as a video prediction problem, and a convolutional long short-term memory network (ConvLSTM) was proposed. This model combines CNN and RNN, utilizing their unique advantages in spatio-temporal sequence data extraction to simultaneously obtain temporal relationships and spatial features. It has achieved good results in radar echo extrapolation tasks. In 2017, Shi et al. [5] introduced real-time variable convolution on the basis of ConvLSTM and proposed TrajGRU. This improvement makes the model more capable of capturing synchronous changes such as rotation and scaling, and also performs better in precipitation prediction. In order to improve the non-linear fitting ability of the network to spatio-temporal sequence features, ConvLSTM is often stacked and used. However, for simple stacked structures, spatio-temporal sequence information is transmitted horizontally along time steps and vertically along stacked layers, but the spatio-temporal sequence information transmitted vertically is not effectively utilized. Therefore, Wang et al. [6] proposed the spatio-temporal LSTM (ST-LSTM) neural network. Chen et al. [7] introduced 3D convolution on the basis of ConvLSTM and proposed a hybrid prediction model based on CNN and LSTM to better understand the spatial relationships of typhoon features. Phermphoonphiphat et al. [8] proposed a prediction model based on ConvLSTM and introduced up and down sampling operations to make short- and long-term predictions of the tropospheric circulation in the northern hemisphere, with higher accuracy than other existing models.

However, existing cloud spatio-temporal sequence prediction methods have some problems: relying on stacked convolutional layers to obtain mostly local spatial features, in order to cope with the complex changes in cloud position and shape in the cloud image, the prediction module of the model needs to be able to extract the inner image frame’s global and local spatial features from the cloud image. Secondly, for irregular cloud motion, the prediction module should pay more attention to the temporal sequence features between input cloud image frames and consider extracting interframe temporal features with long temporal dependencies. In this way, the prediction network can more accurately understand the cloud motion trend. Especially in multi-step cloud image prediction, the lack of long temporal features and global spatial features may lead to inaccurate cloud image prediction.

To tackle these challenges, we designed a typhoon cloud image prediction network based on self-attention memory. In order to cope with the irregular motion of cloud images, the self-attention mechanism in the self-attention memory module aggregates inputs from all positions to generate predictive features, solving the problem of not being able to capture global spatial features in stacked convolutional layers. The additional storage unit M in the self-attention memory module can preserve the long temporal and spatial features between cloud images, thereby solving the problem of inaccurate cloud image prediction due to the lack of long temporal and spatial dependencies in multi-step cloud image prediction. In summary, our contributions are as follows:

•

We propose a typhoon cloud image prediction model called SAM-Net, which excels at capturing global spatial features as well as long-range dependent spatio-temporal features in cloud images.

•

In SAM-Net, we solved the problem of complex changes in cloud position and shape, as well as irregular cloud motion in cloud prediction, through the SAM module.

•

We conducted multi-frame predictions using SAM-Net on publicly available datasets, including MovingMNIST and KTH, as well as cloud image datasets, demonstrating improved performance over PredRNN-v2.

2. Related Work

Next, we will introduce related work from traditional methods and deep learningal methods.

2.1. Traditional Methods

The traditional typhoon forecasting methods are mainly divided into numerical forecasting, statistical forecasting, and statistical dynamic forecasting.

Numerical weather prediction, first proposed by Godske et al. [9] in 1957, is one of the most commonly used objective methods for typhoon forecasting. This approach utilizes large-scale meteorological data collected from various observational sources (such as radar and satellites), including meteorological factors, physical factors, and terrain factors, to construct atmospheric dynamical equations. Based on the current state of the atmosphere, and given initial conditions and boundary conditions, these equations are approximated using large-scale computers to achieve numerical forecasts of typhoon tracks and intensities. Sanders et al. proposed the SANBAR [10] and barotropical [11] models, which have been implemented for the prediction of typhoon paths in the sky for the following one to three days, laying the foundation for subsequent research. After decades of research, Qian et al. [12] discussed the impact of different initial fields and lateral boundary conditions on the accuracy of typhoon numerical forecasting. Statistical forecasting refers to realization of the long-term and short-term forecasting of a typhoon through the statistical analysis of a large number of historical meteorological data, mining the linear relationship between the forecast amount and the forecast factors related to the cause of the typhoon, and building a statistical model based on regression equation. Compared with numerical forecasting, statistical forecasting has the advantages of strong flexibility, low consumption of computing resources, and better performance in forecasting typhoon intensity. The climate sustainability method (CLIPER), as an early classic statistical forecasting model, was developed by Neumann [13], who proposed it in 1972. Chand et al. [14] used a Bayesian probability regression model to predict the causes of tropical cyclones in the Fiji region. Kim et al. [15] used a multimodal dynamic statistical model based on the Asia Pacific Climate Center (APCC) to make seasonal forecasts for typhoons in the northwest Pacific. Statistical dynamic forecasting is a forecasting method that combines numerical forecasting with statistical forecasting. Its principle is to use numerical forecasting to solve the atmospheric dynamics equation, and then use statistical forecasting to further correct the prediction results obtained by numerical forecasting, so as to effectively reduce the prediction errors of typhoon track and intensity. This method not only makes up for the shortcomings of numerical forecasting in strength prediction, but also overcomes the shortcomings of statistical forecasting, which relies too much on historical data.

2.2. Deep Learning Methods

There are mainly two types of methods based on deep learning: one is based on RNN, and the other is a hybrid model based on CNN and RNN.

Recurrent neural networks (RNNs) are widely used in video prediction. The commonly used long short-term memory (LSTM) [16] networks for video prediction algorithms and the gated recurrent unit GRU [17] are variants of RNN. In 2014, Ranzato [18] applied RNN to the field of video prediction and proposed a new strong baseline model for unsupervised feature learning. Srivastava [19] introduced LSTM into video forecasting. Using LSTM as the encoder and decoder of the prediction model for video prediction, and highlighting the combination of convolutional neural network (CNN) and LSTM for video prediction, this demonstrated the feasibility of improving prediction accuracy. Chen et al. [1] used an attention mechanism and GRU to accurately predict floods.

Srivastava [19] highlighted the feasibility of improving prediction accuracy through the combination of convolutional neural network (CNN) and LSTM for video prediction. Shi et al. [4] drew inspiration from Srivastava’s ideas and combined CNN with LSTM, proposing ConvLSTM for image prediction, designing an encoding forecasting (EF) structure based on the transmission of information flow in the network, and applied this prediction model to precipitation prediction, achieving good results. Afterwards, Shi et al. [5] further optimized the EF structure and proposed a more lightweight approach compared to ConvLSTM. ConvGRU and TrajectoryGRU were proposed by introducing learnable convolution, which is theoretically more suitable for predicting irregular motion characteristics such as precipitation data. Deng et al. [20] used a refined pyramid grid to increase the receptive field of image extraction. From 2017 to 2019, Wang et al. [6,21,22] integrated 3D convolution, spatio-temporal memory units, and gradual highways and PredRNN [6], PredRNN++ [21], and E3D LSTM [22], making significant contributions to the development of video prediction technology. In 2022, Wang et al. [23] proposed PredRNN-v2, which was based on PredRNN and utilized memory decoupling loss to prevent memory cells from learning redundant features. In the same year, Gao et al. [24] released the SimVP video prediction model, and in 2023, Lian et al. [25] proposed the SCSTque model, which achieved prediction on ordinary cloud images.

In recent years, with the popularity of generative adversarial networks (GANs) [26,27], attention mechanisms [28], and self-attention mechanisms [29,30] in the field of deep learning, Xu et al. [31] and Lin et al. [32] have respectively integrated GANs and self-attention mechanisms into LSTM, and the video prediction achieved good results. The above video prediction algorithms require a significant amount of memory space to preserve as much information as possible in the image; this greatly limits the application scope of the algorithm. Due to the instability and nonlinearity of cloud movement, based on depth studies, the difficulty of predicting cloud movement using Xi’s method is relatively high. In 2017, Hong et al. [33] combined an autoencoder with ConvLSTM to extract temporal cloud images by incorporating skip connections between the encoder and decoder based on spatio-temporal characteristics, but the accuracy of their prediction results could be improved. In 2021, Cai [34] combined 3D convolution with U-Net to form a cloud image prediction module, and used GAN to train the prediction network to improve the prediction model’s ability to extract spatio-temporal sequence features.

In summary, existing methods are unable to address the issue of irregular cloud image motion in cloud images. Therefore, it is necessary to consider extracting spatial features from both global and local dependencies. In addition, due to the complex shape and spatio-temporal variations of cloud images, it is necessary to pay more attention to the long temporal dependencies between cloud images when extracting cloud image features. In this paper, we introduce the SAM module, which can effectively extract spatial features from both global and local perspectives. Additionally, with the introduction of additional memory units, we can effectively focus on the long temporal and spatial features between cloud images.

3. Methods

In this section, we will provide a detailed introduction to our proposed SAM-Net. The specific network structure diagram is shown in Figure 1. This network is mainly divided into two parts: the light blue dashed box and the red dashed box are the training part, and the orange part is the prediction part of the model. In the training part, eight frames of cloud images are read in each batch. The first four frames are used to learn the features of the cloud images, and the last four frames are the ground truth, used as a comparison for the predicted cloud images. The size of the input cloud images is $128\times 128\times 1$. We seamlessly embed the self-attention memory module (SAM) in PredRNN-v2. In the prediction part, we set the size to $128\times 128\times 1$. The cloud images are input into the trained model to obtain predicted cloud images.

3.1. Self-Attention Module

The first row of Figure 2 shows the process of the standard self-attention module. $H_t$ in the input original feature image, which is mappedto different feature spaces, where $Q_h$, $K_h$, and $V_h$ are, respectively, the query, key, and value. The specific formula for the self-attention module is as follows:

$$\begin{array}{cc}\hfill & Q_h=W_q{H}_t\in {\mathbb{R}}^{\widehat{C}\times N}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & K_h=W_k{H}_t\in {\mathbb{R}}^{\widehat{C}\times N}\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & V_h=W_v{H}_t\in {\mathbb{R}}^{C\times N}\hfill \end{array}$$

(3)

In the above formula, $W_q$,$W_k$,$W_v$ is the weight set of 1 × 1 convolution, and C and $\widehat{C}$ are the number of channels, where $N=H\times W$.

$$\begin{array}{cc}\hfill & e=Q_h^T{K}_h\in {\mathbb{R}}^{N\times N}\hfill \end{array}$$

(4)

Calculate the similarity score of each pair of points through matrix multiplication:

$$\begin{array}{cc}\hfill & {\alpha }_{i,j}={\displaystyle \frac{exp{e}_{i,j}}{{\sum }_{k=1}^{N}exp{e}_{i,k}}},i,j\in \{1,2,...,N\}\hfill \end{array}$$

(5)

where $e_{i,j}$ represents the similarity between the i-th and j-th points, $e_{i,j}=\left(H_{t,i}^T{W}_q^T\right)\left(W_k{H}_{t,j}\right)$, $H_t^i$ and $H_t^j$ represent feature vectors with a shape size of C * 1, and finally, the similarity score is normalized along the column.

$$\begin{array}{cc}\hfill & {\mathbf{Z}}_i=\sum _{j=1}^N{\alpha }_{i,j}\left(W_v{H}_{t;j}\right),\hfill \end{array}$$

(6)

Calculate the position and features of the i-th position by weighting all positions, where $W_v,H_{t,j}\in {\mathbb{R}}^{C\times 1}$ are the j-th columns of the value $V_h$.

3.2. Self-Attention Memory Module

As shown in the second row of Figure 2, introducing an additional spatio-temporal memory cell M into the attention mechanism can capture long-range dependencies between input cloud frames, making it easier to understand the motion trends of the cloud. The self-attention memory module receives two inputs: one is the spatio-temporal memory $M_{t-1}$ from the previous step, and the other is the feature $H_{t-1}$ from the current time step. The entire module is divided into three parts: feature aggregation, memory update, and output.

$$\begin{array}{cc}\hfill & {\mathbf{e}}_m={\mathbf{Q}}_h^T{\mathbf{K}}_{\mathbf{m}}\in {\mathbb{R}}^{N\times N}.\hfill \end{array}$$

(7)

Similar to Formula (4), calculate the similarity score between input and memory by performing matrix multiplication between query $Q_h$ and key $K_m=W_{mk}{H}_t$.

$$\begin{array}{cc}\hfill & {\alpha }_{m;i,j}={\displaystyle \frac{exp{e}_{m;i,j}}{{\sum }_{k=1}^{N}exp{e}_{m;i,k}}},i,j\in \{1,2,...,N\}.\hfill \end{array}$$

(8)

Similar to Formula 5, use the $SoftMax$ function along the row direction to calculate the weight values of aggregated features.

$$\begin{array}{cc}\hfill & {\mathbf{Z}}_{m;i}=\sum _{j=1}^N{\alpha }_{m;i,j}{\mathbf{V}}_{m;j}=\sum _{j=1}^N{\alpha }_{m;i,j}{\mathbf{W}}_{mv}{M}_{t-1;j}.\hfill \end{array}$$

(9)

Calculate the pixel value of the i-th position of feature $Z_m$ by summing the weights of all N positions in the value $V_m$, where $M_{t-1;j}$ is the j-th column of memory, aggregating feature $\mathbf{Z}={\mathbf{W}}_z[{\mathbf{Z}}_h;{\mathbf{Z}}_m].$

$$\begin{array}{cc}\hfill & i_t^{\prime }=\sigma (W_{m;zi}\ast \mathbf{Z}+W_{m;hi}\ast H_t+b_{m;i}).\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & g_t^{\prime }=tanh(W_{m;zg}\ast \mathbf{Z}+W_{m;hg}\ast H_t+b_{m;g}).\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & M_{t+1}^l=(1-i_t^{\prime })M_t^l+i_t^{\prime }\circ g_t^{\prime }.\hfill \end{array}$$

(12)

Use a gating mechanism to update memory M to obtain long-range dependencies in the spatio-temporal domain. Aggregate feature $\mathbf{Z}$ and input $H_t$ to generate input gate and fusion feature $g_t$ values, using 1 − $i_t$ instead of a forget gate to reduce parameters.

$$\begin{array}{cc}\hfill & o_t^{\prime }=\sigma (W_{m;zo}\ast \mathbf{Z}+W_{m;ho}\ast H_t+b_{m;o}).\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & H_{t+1}^l=o_t^{\prime }\circ M_{t+1}^l.\hfill \end{array}$$

(14)

The output feature $H_{t+1}^l$ is the dot product of the output gate $o_t^{\prime }$ and the updated $M_{t+1}^l$.

3.3. SAM-Net Cell

The SAM-Net cell is composed of the self-attention memory module and PredRNN-v2 module. The specific cellular structure is shown in Figure 3, and the formula for the model is as follows:

$$\begin{array}{cc}\hfill & g_t=tanh(W_{xg}\ast X_t+W_{hg}\ast H_{t-1}^l)\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & i_t=\sigma (W_{xi}\ast X_t+W_{hi}\ast H_{t-1}^l)\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & f_t=\sigma (W_{xf}\ast X_t+W_{hf}\ast H_{t-1}^l)\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & C_t^l=f_t\odot C_{t-1}^l+i_t\odot g_t\hfill \end{array}$$

(18)

In the above formula, $g_t$, $i_t$, and $f_t$ respectively represent the input modulation gate, input gate, and forget gate, while $C_t^l$ is the memory states, $H_t^l$ is hidden states, and W and b represent weight and bias, respectively.

$$\begin{array}{cc}\hfill & g_t^{″}=tanh(W_{xg}^{\prime }\ast X_t+W_{mg}\ast M_t^{l-1})\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & i_t^{″}=\sigma (W_{xi}^{\prime }\ast X_t+W_{mi}\ast M_t^{l-1})\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & f_t^{″}=\sigma (W_{xf}^{\prime }\ast X_t+W_{mf}\ast M_t^{l-1})\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill & M_t^l=f_t^{\prime }\odot M_t^{l-1}+i_t^{\prime }\odot g_t^{\prime }\hfill \end{array}$$

(22)

$M_t^l$ is the spatio-temporal memory flow, while $g_t^{\prime }$, $i_t^{\prime }$ and $f_t^{\prime }$ respectively represent the time and space memory flow that passes through it.

$$\begin{array}{cc}\hfill & o_t=\sigma (W_{xo}\ast X_t+W_{ho}\ast H_{t-1}^l+W_{co}\ast C_t^l+W_{mo}\ast M_t^l)\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & H_t^l=\sigma (W_{xo}\ast X_t+W_{ho}\ast H_{t-1}^l+W_{co}\ast C_t^l+W_{mo}\ast M_t^l)\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & H_{t+1}^l,M_{t+1}^l=SAM(H_t^l,M_t^l)\hfill \end{array}$$

(25)

In the above formula, $\sigma$ is the sigmoid activation function, and * and ⊙ represent the product of the convolution operator and Hadamard. $o_t$ is the output gate, $X_t$ represents the input at the current moment, and SAM represents this self-attention memory module. The specific self-attention memory model is shown in Figure 2. The light green part in the picture represents the self-attention memory module that we have integrated into it.

4. Experiments

4.1. Datasets

Given that the MovingMNIST and KTH action datasets are standard benchmarks for video spatio-temporal sequence prediction, we assessed the performance of our SAM-Net on these datasets, in addition to the cloud image dataset, to validate its capability in spatio-temporal sequence forecasting.

The MovingMNIST dataset contains 1000 video sequences, each consisting of 20 frames. In each video sequence, two numbers move independently within the frame, and these numbers are placed in random locations. Numbers are often interlaced, overlapping, and initialized at random speed, and bounce from the edge of the image at a certain angle. The image size of the dataset is $64\times 64\times 1$, and each sequence contains 20 frames, of which 10 frames are used for input and 10 frames are used for prediction. The experimental dataset has a total of 280,000 frames, of which the training dataset contains 20,000 frames. The verification set contains 40,000 frames of images, and the test set contains 40,000 frames of images.

The KTH Action Dataset was obtained from 25 experimenters in four different scenarios: indoor and outdoor scene changes, wearing different clothes, and repeating six types of actions (walking, jogging, running, boxing, waving, and clapping). All video clips were shot using a static camera in a background with a constant frame rate of 25 fps, with an average length of 4 s. All videos were divided into a training set (1–16) and a testing set (17–25) based on the experimenter. The model predicts the next 10 frames of images by learning 10 consecutive frames, including a training set of 108,717 sequences and a testing set of 4086 sequences.

The dataset of typhoon cloud images was taken from the geostationary satellite data of Himawari-8. Himawari-8 is the world’s first new-generation geostationary meteorological satellite with a total of 3 visible, 3 near-infrared, and 10 infrared channels. The area range covers a north latitude of 3°${51}^{\prime }$N to 53°$33\prime$N, and east longitude of 80°E to 135°${05}^{\prime }$E. Our field of vision covered the southeast coast of our country, including part of the western Pacific Ocean and a small part of land. The typhoon cloud image datasets used in this experiment include Typhoon Nida from 20:00 on 29 July 2016 to 20:00 on 2 August 2016, Typhoon Hato from 2:00 on 20 August 2017 to 16:00 on 23 August 2017, Typhoon Mangkhut from 23:00 on 12 September 2018 to 15:00 on 17 September 2018, and Typhoon Higos from 17:00 on 16 August 2020 to 12:00 on 19 August 2020. The spatial resolution of these typhoons is 2 km × 2 km. The dataset used 1200 series data of the water vapor channel, containing 840 training sequences, 180 validation sequences, and 180 test sequences. Each sequence contained eight frames of images, of which the first four frames were used as training data, and the last four frames were used as verification data. In order to save computing resources, each image was converted into a $128\times 128\times 1$-size npz file, and the sampling interval between frames was 1 h.

The Northwest Pacific cloud images dataset is also from Himawari-8. In this experiment, we selected cloud image data from 0:00 on 1 April 2024 to 24:00 on 30 April 2024, and from 0:00 to 24:00 on 31 May 2024, and sampled the dataset once per hour, resulting in a total of 1464 samples. Due to limitations in computing resources, the original images were adjusted to 128 × 128 pixels with a spatial resolution of 20 km × 20 km. In total, 1024 cloud images from this dataset were selected as training samples, 220 cloud images were used as validation samples, and 220 cloud images were used as test samples. The first four frames were used as inputs for the training model, and the last four frames were used as ground truth.

4.2. Implementation

Figure 1 illustrates the flowchart for the typhoon cloud image prediction model. The process is divided into two main components: the training process and the testing process. During the training phase, each batch of input data comprises eight cloud images. The model generates both the actual cloud image data and the four-step predicted cloud image data. The dimensions of the training and test data are (840, 1, 128, 128) and (180, 1, 128, 128), respectively. These dimensions represent the total number of data samples, the number of data channels, and the height and width of each data sample. To ensure a fair comparison with the PredRNN-v2 model, the experimental setup on the MovingMNIST dataset mirrored that of PredRNN-v2. We utilized the ADAM optimizer, set the batch size to 8, and established a learning rate of 0.0001. Each hidden state within the model features a channel dimension of 128. The convolution kernel size within each cell unit is configured at $5\times 5$. The mean squared error (MSE) serves as the loss function, and the training is halted after reaching 80,000 iterations.

In the typhoon cloud image dataset, after obtaining the typhoon dataset, we first convert the .tif-format dataset into .jpg-format data, then remove incomplete data, select data with band 8, normalize the image to the range of [0–1], consider the limitation of computing resources, adjust the image size to 128 × 128, and finally convert it into .npz format as the training and testing dataset. We also used ADAM as the optimizer, the batch size was set to 8, the learning rate was set to 0.001. The number of channels per hidden state was set to 128, the convolution kernel size per intracellular unit was set to $5\times 5$, and MSE loss was used as the loss function. The training process is stopped after 10,000 iterations, and the preprocessing method is used. The cloud image data obtained from the Northwest Pacific are video data. We extracted each frame of cloud image data from the video, converted the cloud images’ size to 128 × 128, and finally converted it into npz-format data. Other parameters during the training process were the same as those of the typhoon cloud image data.

4.3. Evaluating Indicator

For the above experiment, we used evaluation metrics such as mean square error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index measurement (SSIM) [35], image similarity (LPIPS) [36], and Pearson spatial correlation coefficient (PCCS), which have been widely used in image evaluation. The difference between these indicators is that MSE is used to calculate the absolute pixel error value, PSNR is an index used to measure the similarity degree of two images, SSIM is used to measure the similarity of structural information in the spatial neighborhood, and LPIPS is an evaluation index based on deep features that are more in line with human perception. PCCS measures the degree of correlation between two variables. The larger the value of SSIM, PCCS, and PSNR, or the smaller the value of MSE and LPIPS, the better the prediction effect of the model.

$$\begin{array}{cc}\hfill & MSE={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left({\widehat{y}}_i-y_i\right)}^2\hfill \end{array}$$

(26)

where ${\widehat{y}}_i$, $y_i$ represent the predicted and true values at the sampling location, and N represents the number of pixels in the test sample.

$$\begin{array}{cc}\hfill & PSNR=10\times lg\left({\displaystyle \frac{{255}^2}{\mathrm{MSE}}}\right)\hfill \end{array}$$

(27)

$MSE$ stands for mean square error.

$$\begin{array}{cc}\hfill & SSIM(x,y)={\displaystyle \frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)({\sigma }_x^2+{\sigma }_y^2+C_2)}}\hfill \end{array}$$

(28)

Among them, x and y are the two images to be compared, ${\mu }_x$ and ${\mu }_y$ represent the means of these two images, ${\mu }_x^2$ and ${\mu }_y^2$ represent the variances of these two images, ${\sigma }_{xy}$ represents the covariance of these two images, and $C_1$ and $C_2$ are constants used to avoid situations where the denominator is 0. The values of constants vary depending on the color range. Generally, $C_1$ = $(K_1$× L)², $C_2$ = $(K_2$× L)², where L is the range of pixel values. In this study, pixel values were between 0 and 1. $K_1$ and $K_2$ are constants less than 1. In this experiment, we set $K_1$ = 0.01 and $K_2$ = 0.03.

$$\begin{array}{cc}\hfill & d(x,x_0)=\sum _l{\displaystyle \frac{1}{H_l{W}_l}}\sum _{h,w}\left|\right|w_l\odot ({\widehat{y}}_{hw}^l-{\widehat{y}}_{0hw}^l){\left|\right|}_2^2\hfill \end{array}$$

(29)

In the calculation formula of $LPIPS$, d is the distance between $x_0$ and x. Extract feature stacks from the L layer and perform unit normalization in the channel dimension. Use vector $W_l$ to scale the number of activated channels and ultimately calculate the $L_2$ distance. Finally, average in space and sum in channels.

$$\begin{array}{c}PCCS={\displaystyle \frac{\sum (x_i-\overline{x})(y_i-\overline{y})}{\sqrt{\sum {(x_i-\overline{x})}^2\sum {(y_i-\overline{y})}^2}}}\hfill \end{array}$$

(30)

Among them, $x_i$ and $y_i$ are the $i\text{-}th$ pair of observations in the sample data, and $\overline{x}$ and $\overline{y}$ are the means of x and y, respectively.

5. Result and Analysis

5.1. MovingMNIST Dataset

In the MovingMNIST dataset, employing the same hyperparameters and completing 80,000 training rounds, our SAM-Net outperforms PredRNN-v2. As shown in

Table 1. Results on MovingMNIST averaged over 10 future timesteps.

Datasets Models	MovingMNIST
	MSE (↓)	$\mathbf{\Delta }$	SSIM (↑)	$\mathbf{\Delta }$	PSNR (↑)	$\mathbf{\Delta }$	LPIPS (↓)	$\mathbf{\Delta }$
ConvLSTM [4]	103.3	–	0.707	–	–	–	0.156	–
CDNA [37]	97.4	–	0.846	–	–	–	–	–
VPN Baseline [38]	64.1	–	0.87	–	–	–	–	–
MIM [39]	52	–	0.874	–	–	–	0.079	–
Conv-TT-LSTM [40]	64.3	–	0.846	–	–	–	0.133	–
PredRNN [6]	56.8	–	0.867	–	–	–	0.107	–
PredRNN-v2 [23]	48.4	–	0.891	–	–	–	0.071	–
SAM-Net	48.3	−0.1	0.893	+0.02	20.03	–	0.068	−0.03

, our proposed model achieves a lower MSE by 0.01, a higher SSIM by 0.02, and a decrease in LPIPS by 0.003 compared to PredRNN-v2. The images predicted by SAM-Net are clearly distinguishable in both number and trajectory. In the dataset experiment on MovingMNIST in Figure 4, we can see that there is almost no difference in the prediction performance between our proposed model and PredRNN’s model in the simple number movement of “8” and “5”. However, in Figure 5, considering the slightly complex spatio-temporal prediction where the numbers “8” and “0” overlap and separate, we embed a self-attention memory module in the cell unit of PredRNN, and we can see a significant difference between the two models. After the numbers “8” and “0” leave, it is difficult to recognize the number “8” in the image predicted by PredRNN, while our images predicted by SAM-Net can be well distinguished in terms of both number and trajectory.

5.2. KTH Dataset

On the KTH action dataset, our proposed SAM-Net demonstrates superior performance compared to PredRNN-v2. As shown in

Table 2. Results on KTH averaged over 10 future timesteps.

Datasets Models	KTH Action Dataset
	MSE (↓)	$\mathbf{\Delta }$	SSIM (↑)	$\mathbf{\Delta }$	PSNR (↑)	$\mathbf{\Delta }$	LPIPS (↓)	$\mathbf{\Delta }$
ConvLSTM [4]	–	–	0.712	–	23.58	–	0.231	–
Mcnet + Residual [41]	–	–	0.806	–	26.29	–	–	–
TarjGru [5]	–	–	0.79	–	26.97	–	–	–
DFN [36]	–	–	0.794	–	27.26	–	–	–
Conv-TT-LSTM [40]	–	–	0.815	–	27.62	–	–	–
PredRNN [6]	–	–	0.839	–	27.55	–	0.204	–
PredRNN-v2	20.5	–	0.845	–	31.72	–	0.139	–
SAM-Net	19.37	−1.13	0.861	0.016	32.18	0.046	0.109	−0.03

, our model shows improved metric results, with a 1.13 lower MSE, 0.016 higher SSIM, 0.046 higher PSNR, and 0.03 lower LPIPS. As depicted in Figure 6, which illustrates the experiment on the KTH dataset, we observe the experimenter’s arms transitioning from a closed position on the head to an open and extended state. The PredRNN-v2 model’s predictions show the experimenter’s arms remaining closed on the head. In contrast, our proposed SAM-Net accurately captures this transition, with the predicted arms closely resembling the ground truth, indicating a clear progression from a closed to an open and flat state.

5.3. Typhoon Cloud Images

5.3.1. Prediction of 4 Timesteps

As shown n the red box in Figure 7, the white cloud layer generated by our SAM-Net model is more prominent compared to PredRNN-v2. Secondly, in terms of the overall shape of the typhoon, SAM-Net also has more accurate predictions. As seen in

Table 3. Results on typhoon cloud images averaged over 4 and 10 future timesteps.

Datasets Models	Typhoon Image Cloud (4 Timestemps)
	MSE (↓)	$\mathbf{\Delta }$	SSIM (↑)	$\mathbf{\Delta }$	PSNR (↑)	$\mathbf{\Delta }$	LPIPS (↓)	$\mathbf{\Delta }$	PCCS (↑)	$\mathbf{\Delta }$
ConvLSTM	104.19	–	0.448	–	18.32	–	0.242	–	–	–
ConvGRU	89.34	–	0.532	–	20.02	–	0.203	–	–	–
PredRNN	80.86	–	0.621	–	23.15	–	0.162	–	–	–
PredRNN-v2	64.2	–	0.715	–	24.63	–	0.143	–	0.871	–
SAM-Net	60.13	−4.07	0.719	+0.004	24.89	+0.26	0.138	−0.005	0.914	+0.043
Datasets Models	Typhoon Image Cloud (10 Timestemps)
	MSE (↓)	$\mathbf{\Delta }$	SSIM (↑)	$\Delta$	PSNR (↑)	$\mathbf{\Delta }$	LPIPS (↓)	$\mathbf{\Delta }$	PCCS (↑)	$\mathbf{\Delta }$
PredRNN-v2	265.89	–	0.317	–	18.09	–	0.238	–	0.562	–
SAM-Net	85.31	−180.58	0.668	+0.351	23.65	+5.56	0.174	−0.064	0.881	+0.319

, our proposed model demonstrates improvements over the PredRNN-v2 model, with a decrease of 2.39 in MSE, an improvement of 0.025 in SSIM, an improvement of 0.63 in PSNR, and a decrease of 0.005 in LPIPS. Given the relatively short duration of four timesteps, both PredRNN-v2 and SAM-Net capture similar amounts of information in terms of local dependencies. Consequently, the prediction graphs and results of the two models are nearly identical.The visualization of the results of the four indicators is shown in Figure 8.

5.3.2. Prediction of 10 Timestep

As seen in Figure 9 and Table 3, our SAM-Net predicts cloud images more accurately compared to PredRNN-v2, especially in the white clouds marked with red boxes. Our model predicts more clearly, accurately, and clearly. It is evident that our proposed SAM-Net offers more pronounced advantages and more accurate prediction of changes in cloud images. The integration of the self-attention memory module allows for the acquisition of global spatial context within a single layer, which is more effective than traditional convolution operations for capturing global information. The self-attention memory (SAM) leverages a feature aggregation mechanism that calculates pairwise similarity scores, fusing current and memory features to achieve a global receptive field effect. This capability enables SAM to seize long-range dependencies in both spatial and temporal dimensions, thus enhancing the model’s performance in long-term spatio-temporal prediction tasks. This is particularly beneficial in scenarios involving large-scale, irregular, and dispersed distributions of cloud images, as well as in the presence of substantial noisy data, where the model’s performance is further enhanced.The visualization of the results of the four indicators is shown in Figure 10.

5.4. Northwest Pacific Cloud Image Datasets

From Figure 11, it can be seen that our SAM-Net is not only applicable to typhoon cloud images with distinct features, but also to ordinary cloud images with scattered cloud features. The areas marked with red boxes indicate that the two clouds gradually aggregate together, while the air between the two clouds gradually dissipates and becomes smaller.

5.5. Analysis of Ablation Experiment Results

In order to further investigate the superiority of the performance of the self-attention memory module, we conducted ablation experiments on a four-step cloud graph dataset to analyze the memory cells of the internal modules of self-attention memory $M_t$, and thus the necessity and effectiveness of self-attention in this work. In the ablation experiment, comparative experiments were conducted by controlling variables and gradually adding them. The results are shown in

Table 4. Ablation experiments using different methods on cloud image datasets.

Datasets Models	Typhoon Image Cloud (4 Timesteps)
	MSE (↓)	$\mathbf{\Delta }$	SSIM (↑)	$\mathbf{\Delta }$	PSNR (↑)	$\mathbf{\Delta }$	LPIPS (↓)	$\mathbf{\Delta }$
PredRNN-v2	66.28	–	0.762	–	24.61	–	0.137	–
w Mem w/o Attention	64.98	−1.3	0.779	+0.017	25.02	+0.41	0.134	−0.003
w/o Mem w Attention	65.77	−0.51	0.771	+0.009	24.93	+0.31	0.136	−0.001
w Mem w Attention	63.89	−2.39	0.787	+0.025	25.24	+0.63	0.132	−0.005

. The introduction of memory cells and self-attention mechanisms has improved all four evaluation indicators. From the table, it can be seen that compared with the self-attention memory, the introduction of memory cells shows a stronger improvement on various indicators than the self-attention module, because it can extract long-range spatio-temporal dependency relationships, which is very important for SAM-Net.

6. Discussion

This article proposes an innovative typhoon prediction model based on PredRNN-v2 and a self-attention memory mechanism. In the cloud prediction experiment of this article, we used the four-year typhoon cloud map data provided by Himawari-8, including Nida, Hato, Mangkhut, and Higos, as well as ordinary cloud map data in the northwest Pacific from April and May 2024, to conduct experiments on our model. Satellite data have undergone preprocessing steps, including format conversion, spatio-temporal matching, missing value processing, image normalization, and image resizing. After training the model, we evaluated it on the 2020 Higos typhoon dataset. Typhoon Higos formed on Luzon Island, Philippines, at 12:00 on 16 August 2020, then strengthened and moved northwest. It made landfall along the coast of Guangdong, China, at 6:00 on 19 August and dissipated at 8:00 on 19 August. The cloud image data used in this study are the cloud images after entering China’s territorial waters on 17 August. Quantitative evaluation shows that our proposed SAM-Net model has better accuracy than other models in cloud prediction at 4 and 10 time steps. Especially at 10 time steps, we have made significant improvements compared to PredRNN-v2. Compared with 10 time steps, 4 time steps have better performance representation, highlighting the potential of our model in short-term prediction. Due to the introduction of the self-attention memory mechanism, we achieved good performance in short-term typhoon cloud images, indicating that this mechanism can capture the spatio-temporal sequence features between typhoon cloud image frames and the global and local spatial dependency features between each frame of the image well. However, as the time steps increase, these features are gradually lost, resulting in larger prediction errors compared to the four-timestep typhoon cloud map prediction. Therefore, in future research work, we will consider introducing new loss functions or adding super-resolution algorithms to image prediction to improve the accuracy of long-term typhoon cloud image prediction and the clarity of the generated cloud images. In addition, in future research, the weights of cloud image data during the training phase will be adjusted according to the different generation stages of typhoon cloud images. For example, in the stages of typhoon formation, heat storm, recession, and dissipation, features from different stages will be extracted to improve the predictive ability and interpretability of the model. These results will be announced in the near future. To verify the generalization ability of our proposed model, we also conducted experiments on a regular cloud image dataset. The experimental results show that our model is not only applicable to typhoons, but also to ordinary cloud images, and has good generalization performance.

In addition, a major challenge in predicting typhoon cloud imagery is resource consumption, especially when dealing with larger and more complex typhoon cloud imagery data. How to reduce resource consumption and train models more quickly and effectively is an important issue. Therefore, in future work, we will consider how to reduce the consumption of computing resources without affecting performance, such as reducing the number of neurons, weight quantization, and other methods, to generate a lightweight framework. In summary, how to make our model lighter and more efficient will be one of the focuses of our future research directions.

7. Conclusions

In this article, we introduce the SAM-Net model for spatio-temporal prediction of images, specifically targeting the challenges posed by cloud images. To cope with the complex changes in cloud position and shape in cloud images, the prediction module of the model needs to be able to extract both global and local spatial features from the cloud images. In addition, for irregular cloud motion, more attention should be paid to the spatio-temporal sequence features between input cloud image frames in the temporal sequence prediction module, considering the extraction of temporal features with long temporal dependencies, so that the spatio-temporal sequence prediction network can learn cloud motion trends more accurately. To address these issues, we have integrated a self-attention memory module into the PredRNN-v2 framework, enabling the capture of remote spatial and temporal features. This enhancement allows the SAM-Net to learn not only temporal patterns but also spatial correlations through the self-attention mechanism.

Our experiment results indicate that the performance of PredRNN-v2 is not satisfactory in predicting typhoon cloud images. Thus, we proposed an improved prediction model based on the predRNN-v2 and evaluated it on both the MovingMNIST and KTH action datasets, as well as a typhoon cloud image dataset. The results demonstrated an improvement over the PredRNN-v2 model.