Time Series and Non-Time Series Models of Earthquake Prediction Based on AETA Data: 16-Week Real Case Study

Abstract

The Key Laboratory of Integrated Microsystems (IMS) of Peking University Shenzhen Graduate School has deployed a self-developed acoustic and electromagnetics to artificial intelligence (AETA) system on a large scale and at a high density in China to comprehensively monitor and collect the precursor anomaly signals that occur before earthquakes for seismic prediction. This paper constructs several classic time series and non-time series prediction models for comparison and analysis in order to find the most suitable earthquake-prediction model among these models. The long short-term memory (LSTM) neural network, which gains the best results in earthquake prediction based on AETA data extracted from the precursor anomaly signals, is selected for real-earthquake prediction for 16 consecutive weeks.

1. Introduction

Earthquakes can cause huge damage to the natural environment and human society. If the three aspects of earthquakes can be accurately predicted, i.e., time, epicenter, magnitude, the damage can be largely reduced.

Acoustic and electromagnetics to artificial intelligence (AETA) is a multi-component earthquake-monitoring and prediction system developed by the Key Laboratory of Integrated Microsystems of Peking University Shenzhen Graduate School, which is used for earthquake prediction by collecting underground earthquake precursor anomaly signals, including electromagnetic (EM) signals and geoacoustic (GA) signals [1]. The importance of EM signals and GA signals to earthquake prediction has been revealed by numerous studies during the last forty years [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]. The AETA team has completed a large-scale deployment in China and has collected over 50 TB of data which laid a good foundation for the establishment of earthquake-prediction models.

Previously, many scholars attempted to use these algorithmic models for earthquake prediction in the field of seismology. Some of these researchers applied non-time series models to predict earthquakes. In non-time series models, the temporal data were converted to non-temporal data, i.e., the two-dimensional temporal feature matrix was reduced and compressed into a one-dimensional feature vector, and then the extracted temporal features were added to construct a one-dimensional sample set. Zhang, Y. et al. proposed an earthquake disaster image information anomaly detection model based on scale-invariant feature transform (SIFT) feature and support vector machine (SVM) classification in 2019 [17]. Jozinovic et al. proposed a method based on deep convolutional neural networks (CNN) to predict the degree of ground shaking when an earthquake occurred in 2020 [18]. Xiong, P. et al. used light gradient boosting machine (LightGBM) in a five-fold cross-validation test on the benchmarking datasets, which shows a strong capability in discriminating electromagnetic pre-earthquake perturbations in 2020 [19]. Wang, L. et al. used an efficient seismic slope stability analysis approach by introducing LightGBM to analyze the seismic stability of a hypothetical embankment in 2021 [20]. Saad constructed a random forest (RF) model to detect the location of an upcoming earthquake in 2022 [21]. The non-time series models developed in this paper include neural network (NN), SVM, gradient boosting decision tree (GBDT), RF and LightBGM models represented by integrated trees.

Some experts use time series models for earthquake forecasting. Compared with non-time series models, time series models can preserve the richness of the data, and can learn the asymptotic changes of various features in the time scale, which can detect the earthquake precursor anomaly signals. Kanarachos et al. introduced a signal processing algorithm, which combined wavelets, neural networks and the Hilbert transform to predict earthquake activity in 2017 [22]. Zhou, Y. et al. combined convolutional and recurrent neural networks to pick phases from archived continuous waveforms in 2019 [23]. In 2019, Titos et al. also used the gated recurrent unit (GRU) model to exploit temporal and frequency information from continuous seismic data to detect and classify continuous sequences of volcanic seismic events at the Deception Island Volcano, Antarctica [24]. In 2020, Jena et al. developed a recurrent neural network (RNN) model to create an earthquake probability map for the eastern region of India, including the coastal state of Odisha [25]. Xu, Y. et al. proposes a framework based on a long short-term memory (LSTM) neural network architecture for real-time regional seismic damage assessment in 2021 [26]. In 2021, Yan, X. et al. utilized two LSTM models to simulate and forecast hydrological variations based on hydrological time series of data from a monitoring site to identify possible precursors to the Lijiang earthquake [27]. In 2021, Huang, Y. et al. introduced a moving-steps strategy and established three recurrent neural network models: simple-RNN, LSTM, and GRU models to the prediction of the slope dynamic response [28]. The main time series models established in this paper include the LSTM prediction model, the GRU, and the CNN+GRU stacking prediction model. Xue, J. et al. applied deep learning to extract SESs and develop a novel deep learning network based on geoelectric field characteristics by combining the LSTM in 2022 [29]. This paper analyzes and compares various non-time series models and time series models in order to find a more suitable and simpler model for earthquake prediction based on AETA data.

In Section 2, this paper introduces the AETA system developed by the IMS lab and the process of constructing the dataset. In Section 3, the construction of the non-time series and time series models are introduced respectively. Section 4 shows the results of each prediction model. In Section 5, the prediction results of non-time series models and time series models are compared and analyzed, and LSTM in the time series model gets the best results in earthquake prediction based on AETA data. Finally, the research work of this paper is summarized in the conclusion section.

2. AETA System

2.1. AETA Devices and Data Acquisition

AETA, the multi-component earthquake monitoring and prediction system, was developed by the Key Laboratory of Integrated Microsystems of Peking University Shenzhen Graduate School for earthquake prediction. This system consists of two sensors, a terminal, a cloud platform, and a data analysis system. The two sensors are an electromagnetic sensor and a geoacoustic sensor, both of which are used to collect electromagnetic and geoacoustic signals, respectively. Then they transmit data to the data-processing terminal through cables, undergo sampling, compression, and filtering processes, and finally upload the data to the cloud server through the network for subsequent data analysis, the capture of earthquake precursor anomaly signals and research of seismic-related activities [30].

The electromagnetic sensor mainly monitors the electromagnetic signal band containing a very low frequency (VLF) and an ultra-low frequency (ULF), with a frequency range of 0.1 Hz to 10 kHz, an amplitude range of 0.1 to 1000 nT, a sensitivity of >20 mV/nT@0.1 Hz to 10 kHz, a resolution of 18 bits, and a sampling rate of 500 Hz at low frequency and 30 kHz at full frequency [31]. The frequency range of the geoacoustic signals monitored by the geoacoustic sensor is 0.1 Hz to 50 kHz, with a resolution of 18 bits, the sensitivity of 3LSB/pa@ 0.1 Hz to 50 kHz, and the sampling rate of 500 Hz at low frequencies and 150 kHz at full frequencies [32].

The AETA sensor system is low-cost and convenient for large-scale, high-density deployment. Up to now, nearly 300 sets of AETA devices have been deployed in some of the most seismically active regions in China, such as Sichuan, Yunnan, and the surrounding provinces. The AETA system has accumulated over 50 TB of data over 5 years of observation. The rich observation data has laid a good foundation for the establishment and comparative analysis of earthquake-prediction models.

2.2. Data Set Construction

AETA raw data contains electromagnetic signals and geoacoustic signals. Most of the anomalies of AETA data occur within n days before an earthquake, where n is 27 for electromagnetic signals and n is 10 for geoacoustic signals. The reason for the value of n can be confirmed in Section 3. This paper selects the feature data of the first n days as the input sample and the last 7 days as the output sample. The current time is $T_0$, the first n days are $T_{-\mathrm{n}}$, and the last 7 days are $T_{+7}$, earthquake-prediction tasks are defined as follows:

$${\tilde{y}}_1=F_1(x_{T_{-\mathrm{n}}},x_{T_{-\mathrm{n}+1}}, \dots , x_{T_{-1}}),{\tilde{y}}_1\in [0,1]$$

(1)

$${\tilde{y}}_2=F_2(x_{T_{-\mathrm{n}}},x_{T_{-\mathrm{n}+1}}, \dots , x_{T_{-1}})$$

(2)

$${\tilde{y}}_2=[{\tilde{y}}_{mag},{\tilde{y}}_{lat},{\tilde{y}}_{lon}], {\tilde{y}}_{mag}\in [3.5,8], {\tilde{y}}_{lat}\in [22,34], {\tilde{y}}_{lon}\in [98,107],$$

(3)

where ${\tilde{y}}_1$ is the result of earthquake prediction, and $F_1$ is the earthquake prediction model; ${\tilde{y}}_2$ is the prediction result of magnitude and epicenter location, and $F_2$ is the corresponding prediction model, $x$ represents the input feature data, and ${\tilde{y}}_{mag}$, ${\tilde{y}}_{lat}$, ${\tilde{y}}_{lon}$ represent the predicted magnitude, and epicenter latitude and longitude, respectively.

In terms of the extraction range of the dataset, a typical earthquake-prone region in China has been selected for earthquake prediction in this paper. To be specific, this region is located in the Sichuan–Yunnan region ($22.00° \mathrm{N}~34.00° \mathrm{N}, 98.00° \mathrm{E}~107.00° \mathrm{E})$, which has suffered a total of 206 earthquakes with magnitude larger than 3.5 from January 2017 to January 2021, according to the China Earthquake Network (http://www.ceic.ac.cn/history, accessed on 10 March 2021). Figure 1 shows the distribution of stations and earthquakes with magnitude above Ms3.5 in the Sichuan–Yunnan region over 5 years. Moreover, the epicenter distribution shows a typical regional clustering effect.

In order to improve the accuracy of prediction, the Sichuan–Yunnan region is divided into three areas according to the clustering feature of earthquake distribution. The three areas are marked out with different color boxes in Figure 1. Area1 is marked out with green box, area2 is marked out with brown box and area3 is marked out with pink box. The range of the stations are expressed in Equations (4)–(7) in order to reduce the effect of random deviation generated by the division of regions and allows the station data in the overlapping regions to be comprehensively trained on both regional models.

$$\mathrm{area}1=(30° - \Delta \mathrm{N}~34°+\Delta \mathrm{N}, 102° - \Delta \mathrm{E}~106°+\Delta \mathrm{E})$$

(4)

$$\mathrm{area}2=(26° - \Delta \mathrm{N}~30°+\Delta \mathrm{N}, 102° - \Delta \mathrm{E}~106°+\Delta \mathrm{E})$$

(5)

$$\mathrm{area}3=(24° - \Delta \mathrm{N}~28°+\Delta \mathrm{N}, 98° - \Delta \mathrm{E}~102°+\Delta \mathrm{E})$$

(6)

$$\Delta =1°$$

(7)

Earthquake-prediction models are built in each of the three regions with the sample matrices obtained by sliding windows [33]. The step of the sliding window is set to 1 day, all features are arranged in time sequence by day, and the one-dimensional sequence sample set can be generated along the time dimension according to this sliding window. In order to enhance the robustness of the data, this paper checks the data loss rate for each sliding window matrix, and if the loss rate > 0.3, this sample is to be discarded. The loss rate is chosen as 0.3, which is an experimental result. This paper tried to use 0.1~0.5 as the threshold, the step size of which was set to 0.05. The amount of data was sufficient and the validation set obtained was good enough when 0.3 was chosen as the threshold. Figure 2 shows the regional sample composition process.

This paper builds the AETA feature library with a total of 95 kinds of featured data whose validity has been verified [34]. The detailed information is shown in Appendix A.

In addition, due to the infrequent occurrence of earthquakes, the imbalance of positive and negative samples must be considered when constructing the prediction model. Otherwise, it will greatly affect the training of the model, resulting in the model being more likely to predict no earthquakes. For the problem of the lack of few positive samples, the synthetic minority oversampling technique (SMOTE) algorithm must be applied when constructing the sample set for the non-time series prediction model.

To generate new samples by algorithm synthesis, SMOTE iterates through the sparse number of unbalanced samples, calculates the distance of each sample x from all the remaining samples, and gets the k samples closest to it. Then, according to the imbalance ratio of positive and negative samples, some samples are randomly selected from the obtained k samples to be added to the original sample. Finally, a new sample is generated based on each selected sample $\widehat{x}$ and the original sample x [35]. The equation is as follows:

$$x_{\mathrm{new}}=x+\mathrm{rand}(0,1) \times (\widehat{x} - x).$$

(8)

In contrast, the SMOTE algorithm cannot be used directly for the time series prediction model because its samples are two-dimensional matrices. Therefore, this paper designs a two-dimensional matrix SMOTE algorithm to generate new two-dimensional samples. The aim of this is to compress two-dimensional samples into one-dimensional samples in time sequence, then process them by using the SMOTE algorithm, and finally reconstitute the two-dimensional samples in time sequence. The use of the SMOTE algorithm allows for a more productive and balanced model training with a ratio of positive and negative samples between 1:1 and 1:1.5. Figure 3 shows the two-dimensional matrix SMOTE algorithm.

For each station with different installation time and different number of total samples, 85% of the total samples of each station are taken as the training set and 15% as the validation set to build the seismic prediction model and the magnitude and epicenter prediction model respectively.

3. Model Construction

The AETA raw signal: electromagnetic signals and geoacoustic signals, both of which are time series signals. However, it has not been verified whether the featured data after feature extraction retains temporal information [34]. Therefore, this paper constructs both time series and non-time series models for earthquake prediction.

3.1. Non-Time Series Prediction Model

AETA feature data contains rich information of earthquake precursor anomaly signals and have high correlations with earthquakes. Therefore, robust non-time series models can commonly be used to identify the hidden outliers in the featured data for earthquake prediction.

3.1.1. LightGBM

For processing non-time series data, GBDT is often harnessed by researchers because of the distributed training feasibility and low memory consumption. So far, there have been various extended models, including the categorical boosting (CatBoosting) and the extreme gradient boosting (XGBoost) [36], but their efficiency and scalability are not ideal when the amount of data is large. LightGBM is more suitable for seismic prediction that are based on large amounts of data in terms of decision making and scalability by using gradient-based one-side sampling (GOSS), exclusive feature bundling (EFB) [37].

3.1.2. NN

NN is a complex network consisting of multiple layers of neural cells fully connected in parallel. Neural networks can be divided into three main layers, with the first layer as the input layer, the last layer as the output layer, and all the middle ones as hidden layers. To be more specific, the training data is input from the input layer, propagated forward along the network, and the loss is calculated in the output layer. As far as the structure is concerned, the neural network layers are interconnected in parallel, and a particular neuron is interconnected with all neurons in the preceding layer and all neurons in the following layer [38]. Each local neuron is a perceptron consisting of a linear expression $\mathrm{z}={{\displaystyle \sum }}^{ }{\mathrm{w}}_{\mathrm{i}}{\mathrm{x}}_{\mathrm{i}}+\mathrm{b}$ with an activation function $\mathsf{\sigma }\left(\mathrm{z}\right)$. The back propagation (BP) algorithm is regarded as the most basic algorithm for the training of a neural network, which updates the neuron weight matrix w and the bias vector b by the gradient descent (GD) algorithm.

The commonly used activation functions are sigmoid, tanh, relu, and elu. The general loss function includes cross-entropy, mean absolute deviation, and mean square deviation. Among them, cross-entropy is often used in classification tasks, whereas mean absolute deviation and mean square deviation are commonly used in regression tasks.

3.1.3. Other Models

In addition to the two models mentioned above, other well-performing algorithms in machine learning tasks include the SVM and RF. SVM is mainly used for binary classification, seeking to maximize the classification interval, with linear and nonlinear [39]; RF is a kind of bagging integrated tree, taking the principle of minority rule; it is not easy to overfit, and has strong generalization ability [40].

3.2. Time-Series Prediction Models

Time-series prediction models contain the LSTM prediction model, the GRU prediction model, and the CNN+GRU stacking prediction model. These models are fully trained on the sample set to predict the three elements of earthquakes. This paper selects the most suitable model among them and the optimal parameter set for seismic prediction by grid search and a five-fold cross-validation method.

3.2.1. LSTM

LSTM is a time series network that takes the previous output and the current input as the input for the next time, thus allowing the model to take into account historical or future information of the data when performing prediction tasks. Compared with the traditional time-series model RNN, LSTM incorporates the concept of “gates”, which is used to solve the problem of retaining long sequences of historical information and long-range gradient transfer. To be specific, LSTM introduces cell states, input gates, forgetting gates, and output gates, which are used for storing information about historical data, inputting current and historical information, controlling the information that needs to be forgotten, and output predicted results, respectively. The schematic diagram of LSTM network architecture is shown in Figure 4.

The input and output of the LSTM are controlled by gates at each moment. The parameter matrix is gradient-updated by the loss of the final loss function, which affects the weights of the gates. Through the feedback regulation of the three gates, important information can be selectively retained, redundant information forgotten, and then beneficial information passed on. The data are processed as shown in the following equations:

$$f_t=\sigma \left(W_f·\left[h_{t-1},x_t\right]+b_f\right)$$

(9)

$$i_t=\sigma \left(W_i·\left[h_{t-1},x_t\right]+b_i\right)$$

(10)

$${\tilde{C}}_t=tanh\left(W_c·\left[h_{t-1},x_t\right]+b_C\right)$$

(11)

$$h_t=\sigma \left(W_o·\left[h_{t-1},x_t\right]+b_o\right)\ast tanh\left(C_t\right)$$

(12)

$$C_t=f_t\ast C_{t-1}+i_t\ast {\tilde{C}}_t,$$

(13)

where $x_t$ is the input, W_S is the weight parameter, b is bias parameter, $\sigma \left(z\right)$ is an activation function, $C_t$ refers to the previously stored information, $h_{t-1}$ is the output at the previous moment, and $h_t$ is the output.

Figure 5 shows the training structure of the LSTM model in this paper, which mainly contains three LSTM layers with 32 neurons and one fully connected layer with the suitable number of neurons. The LSTM model extracts the anomaly information from input data for n consecutive days and uses it for time sequence analysis.

3.2.2. GRU

GRU has only two gates: the reset gate and the update gate [41]. Because the number of gates is reduced by one, its structure is correspondingly more simplified. The role of the update gate of GRU incorporates the forgetting gate and the input gate in LSTM, and the principle is similar to that of LSTM. Thus, the effects of the two models are not much different on several tasks. The GRU network architecture schematic is shown in Figure 6.

The data are processed as shown in the following equations:

$$z_t=\sigma \left(W_z·\left[h_{t-1},x_t\right]\right)$$

(14)

$$r_t=\sigma \left(W_r·\left[h_{t-1},x_t\right]\right)$$

(15)

$${\tilde{h}}_t=tanh\left(W·\left[r_t\ast h_{t-1},x_t\right]\right)$$

(16)

$$h_t=\left(1-z_t\right)\ast h_{t-1}+z_t\ast {\tilde{h}}_t,$$

(17)

where $x_t$ is the input, W_S is the weight parameter, $\sigma \left(z\right)$ is an activation function, $h_{t-1}$ is the output at the previous moment, and $h_t$ is the output.

3.2.3. CNN+GRU

CNN uses convolutional kernels to effectively grasp local information and perceive the whole from the local by convolutional operation, which has an excellent performance in multidimensional image processing.

This paper proposes a CNN+GRU stacking method for earthquake prediction. First, the one-dimensional convolution kernel performs convolution operation along the time dimension to extract the anomaly information from the two-dimensional feature matrix for n consecutive days. One of the dimensions of the two-dimensional feature matrix is composed of n, and the other is the number of feature data.

After that, the data extracted by multiple convolutional kernels are stitched horizontally to form a new two-dimensional anomaly data matrix. Finally, it is input to the GRU temporal network for classification or regression tasks. The overall architecture of the model is shown in Figure 7.

3.3. Model Parameters

As for the non-time series model, the two-dimensional feature matrix compressed into a one-dimensional dense vector in the time dimension by the principal components analysis (PCA) algorithm, which can reduce the dimension to show the main components of feature data [42]. Then the time-series features extracted by the Tsfresh algorithm are added to non-time series model as the input [43]. The temporality of the compressed data is to be excluded from consideration, so that additional temporal features are added to the feature matrix as compensation.

In addition, the cross-validation method is used to increase the robustness of the model [44], which can make the results more convincing by dividing the total sample into k mutually exclusive subsets and being trained accordingly. Figure 8 shows the schematic diagram of the five-fold cross-validation.

For the time series model, the two-dimensional time series matrix sample needs to be constructed by the time-dimensional sliding window. The optimal model is preserved by observing the change in the loss function, through cross-validation. In terms of the cross-validation of time series data, the problem of data leakage needs to be considered, which needs to be addressed by sliding windows. The quality of the data directly affects the model prediction effect, and the amount of information contained is different in different time scales of the sliding window. If the time scale is too short, the amount of information obtained becomes insufficient. On the other hand, if the time scale is too long, it means that too much useless information must be obtained. Therefore, choosing a suitable sliding window is vital to the final results of the model.

In terms of earthquake prediction, a problem is that there are usually more no-earthquake samples than samples with earthquakes. As a result, the prediction model tends to identify “almost” samples as no-earthquake samples to get a higher area under ROC curve (AUC) index. Therefore, it is necessary to introduce the receiver operating characteristic (ROC) curve when predicting earthquakes in order to make the prediction results more accurate. The ROC curve has the true positive rate (TPR) as the vertical axis and the false positive rate (FPR) as the horizontal axis. AUC is the area under the ROC curve. When the AUC is closer to 1, it indicates that the effect of prediction model is better [45,46,47]. The TPR, FPR, and AUC calculation method is defined as follows:

$$\mathrm{TPR}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(18)

$$\mathrm{FPR}=\frac{\mathrm{FP}}{\mathrm{TN}+\mathrm{FP}}$$

(19)

$$\mathrm{AUC}=\frac{1}{2}{\displaystyle \sum }_{i=1}^{m-1}\left(x_{i+1}{ - x}_i\right)·\left(y_i{+y}_{i+1}\right),$$

(20)

where TP is true positive, FP is false positive, TN is true negative, FN is false negative, and $x_{i+1}$, $x_i$, $y_i$, $y_{i+1}$ represent the FPR and TPR under different thresholds of the ROC curve.

Taking the processing of electromagnetic signals as an example, in this paper, eight control experiments were done for the sliding window time scale parameter with an interval of 5 days each time. This paper verifies the effect of the model and counts the number of stations with an AUC index greater than 0.65. It is obvious that the effect of the model increases at first but decreases as the time scale of the sliding window becomes longer. It achieves the best effect in the time range of [24 days, 30 days], and finally the sliding window size is set to 27 days for electromagnetic signals according to the results in Figure 9a. Similarly, the sliding window size is determined to be 10 days for the geoacoustic signals as shown in Figure 9b.

3.4. Softmax-AUC Index Weighting Method

There are different prediction effects for different stations despite using the same model. Thus, this paper proposes the method of multi-station softmax-AUC with index weighting, which can integrate the information of each station for seismic prediction. The AUC index on the validation set can measure the prediction effects of each station. Therefore, this paper uses the softmax-AUC index weighting method to select suitable stations with AUC above 0.65. The prediction result of each station is 0 or 1. Specifically, 0 means no earthquake has occurred, and 1 means an earthquake has occurred, and the weight is normalized by ${\mathrm{e}}^{{\mathrm{AUC}}_{\mathrm{i}}}$. The threshold is set to 0.5, if the risk value (risk_value) is bigger than the threshold (threshold_value), the region is identified to have an earthquake, and vice versa. The equations are defined as follows:

$${\mathrm{station}\_\mathrm{risk}\_\mathrm{value}}_{\mathrm{i}}=\frac{{\mathrm{e}}^{{\mathrm{AUC}}_{\mathrm{i}}}}{{{\displaystyle \sum }}_{\mathrm{j}}{\mathrm{e}}^{{\mathrm{AUC}}_{\mathrm{j}}}}\ast {\mathrm{pred}}_{\mathrm{i}}$$

(21)

$$\mathrm{risk}\_\mathrm{value}={{\displaystyle \sum }}_{\mathrm{i}}{\mathrm{station}\_\mathrm{risk}\_\mathrm{value}}_{\mathrm{i}}$$

(22)

$$\mathrm{area}\_\mathrm{pred}=\{\begin{array}{c}1 \mathrm{if} \mathrm{risk}\_\mathrm{value} > \mathrm{threshold}\_\mathrm{value}\\ 0 \mathrm{if} \mathrm{risk}\_\mathrm{value} \le \mathrm{threshold}\_\mathrm{value}\end{array},$$

(23)

where ${\mathrm{station}\_\mathrm{risk}\_\mathrm{value}}_{\mathrm{i}}$ is the risk index of the station, ${\mathrm{pred}}_{\mathrm{i}}$ is prediction result of station, $\mathrm{threshold}\_\mathrm{value}$ is the threshold which is set to 0.5, area_pred is the final regional prediction results.

3.5. Model Evaluation Indicators

This paper focus not only on the accuracy of earthquake prediction, but also on the missed prediction and false prediction. Therefore, this paper utilizes the indicators of PA, PP, RP, PN, and RN to evaluate the goodness of the model. The relevant indicators are explained separately below. If these indicators get closer to 1, it indicates that the prediction effect of the model is better. The threshold is set to 0.5, the region is identified to have an earthquake if the risk value exceeds threshold. Based on this judgment condition, the number of predicted earthquakes and predicted no-earthquakes can be counted. According to the prediction results, the indicators can be calculated to illustrate the effects of the prediction model.

Precision All (PA) refers to the total accuracy, the number of correct prediction results divided by the total number of predictions.

$$\mathrm{PA}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FN}+\mathrm{FP}+\mathrm{TN}}$$

(24)

Precision positive (PP) is the accuracy of predicting earthquakes, the number of correct prediction results divided by the number of predicted earthquakes.

$$\mathrm{PP}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(25)

Recall positive (RP) is the recall rate, the number of correct prediction results divided by the total number of actual earthquakes.

$$\mathrm{RP}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(26)

Precision negative (PN) is the accuracy of predicting no-earthquakes, the number of correct prediction results divided by the total number of predicted no-earthquakes.

$$\mathrm{PN}=\frac{\mathrm{TN}}{\mathrm{FN}+\mathrm{TN}}$$

(27)

Recall negative (RN) is the recall rate of no-earthquakes, the number of correctly prediction results divided by the total number of actual no-earthquakes.

$$\mathrm{RN}=\frac{\mathrm{TN}}{\mathrm{FP}+\mathrm{TN}}$$

(28)

where TP is true positive, FP is false positive, TN is true negative, and FN is false negative.

The magnitude prediction, epicenter latitude and longitude prediction belong to the ternary regression task. This paper uses two vital indicators—magnitude of absolute mean deviation (mag_mae) and distance average deviation (distance_average)—to evaluate the effect of the model which focuses on the absolute deviation of magnitude prediction and the distance deviation of the epicenter prediction. The two related indicators are explained below:

Magnitude of absolute mean deviation (mag_mae) represents the average of the absolute deviation between the predicted magnitude and the actual magnitude. It can reflect the accuracy of the magnitude prediction. The calculation method is defined as follows:

$$\mathrm{mag\_mae} = \frac{1}{m}{{\displaystyle \sum }}_{i=1}^m|y_i - {\widehat{y}}_i|,$$

(29)

where $y_i$ represent the actual magnitude and ${\widehat{y}}_i$ represents the predicted magnitude.

Distance average deviation (distance_average) represents the mean value of the difference between the actual and predicted positions of the epicenter. It can reflect the accuracy of the epicenter prediction. The calculation method is defined as follows:

$$\mathrm{distance\_average}=\frac{1}{m}{{\displaystyle \sum }}_{i=1}^m\mathrm{geodesic}\left(\right(y_{i\_lat}, y_{i\_lon}), ({\widehat{y}}_{i\_lat}, {\widehat{y}}_{i\_lon}\left)\right),$$

(30)

where $y_{i\_lat}, y_{i\_lon}$ represents the actual epicenter latitude and longitude, respectively and ${\widehat{y}}_{i\_lat}, {\widehat{y}}_{i\_lon}$ represents the predicted epicenter latitude and longitude, respectively.

4. Results

This paper compares the effects of the models based on the four indicators: AUC, RP, $\mathrm{distance}\_\mathrm{average}$, and $\mathrm{mag}\_\mathrm{mae}$. To be specific, the AUC metric represents the overall performance of the model for earthquake prediction, including correct prediction, false prediction, and missed prediction. The RP measures the recall of earthquake prediction and its expression is the same as TPR. The distance_average and mag_mae evaluate the deviation of epicenter and magnitude prediction, respectively. The higher the number of stations in the same index, the better the model effect is indicated.

4.1. Prediction Results of Non-Time Series Models

The results of the non-time series models were verified in the Sichuan–Yunnan region, which are shown in

Table 1. The results of non-time series models.

Model	$\mathrm{AUC} \ge 0.65$	$\mathrm{RP} \ge 0.70$	$\mathrm{Distance}\_\mathrm{Average} \le 100 \mathrm{km}$	$\mathrm{Mag}\_\mathrm{Mae} \le 0.25$
LightGBM	68	43	52	36
NN	57	49	47	39
SVM	51	39	43	41
GBDT	36	42	39	43
RF	45	37	41	32

. The data in the table represent the number of corresponding stations.

The results of the five non-time series prediction models show that the LightGBM model and the NN model have better prediction results than the other models. The two indicators of the LightGBM model, AUC and distance_average, are better than those of the NN model, but the RP and mag_mae are worse than those of the NN model.

4.2. Prediction Results of the Time Series Models

The results of the time series models are verified in the Sichuan–Yunnan region, which are shown in

Table 2. The results of time series models.

Model	$\mathbf{AUC} \ge 0.65$	$\mathbf{RP} \ge 0.70$	$\mathbf{Distance}\_\mathbf{Average} \le 100 \mathbf{km}$	$\mathbf{Mag}\_\mathbf{Mae} \le 0.25$
LSTM	84	55	64	47
GRU	82	58	61	48
CNN+GRU	79	49	56	40

. The data in the table represent the number of corresponding stations.

Overall, the three time series prediction models all has great prediction results. The two indicators of the LSTM model, AUC, and distance_average, are better than those of other models. In the next section, the paper compares the prediction results of the above eight models in detail.

5. Discussion

5.1. Comparison of Non-Time Series Models and Time Series Models

To determine a more suitable earthquake prediction model for the AETA data, this paper compares the effects of three time series models and five non-time series models on the validation set, as shown in Figure 10. The first five are non-time series prediction models, namely LightGBM, NN, SVM, GBDT, RF, and the last three are time series prediction models, namely LSTM, GRU, and CNN+GRU.

The input of the time series prediction model is two-dimensional time series feature data, which completely contains the information of AETA raw signals to the maximum extent. In contrast, the input data of the non-time series prediction model is compressed, and no longer retains the time series information of the raw data. Therefore, it is difficult for the model to learn the changes of the time dimension.

According to the bar chart, it is obvious that the overall effect of the three time series prediction models is significantly better than that of the five non-time series prediction models in four indicators. Moreover, the LSTM model has the best performance among these prediction models. The LSTM prediction effects of each station are shown in Appendix B. Prediction effects of the LSTM model for magnitude and epicenter of each station are shown in Appendix C.

5.2. Real-Earthquake Prediction

LSTM, which is the model with the optimal effect among these several prediction models, was chosen to predict the earthquake with a magnitude larger than 3.5 in the Sichuan–Yunnan region ($22.00° \mathrm{N}~34.00° \mathrm{N}, 98.00° \mathrm{E}~107.00° \mathrm{E})$ from April 2021 to July 2021 for 16 consecutive weeks. Each prediction was made every Sunday and includes Y/N prediction for next 7 days in the target region, if it was a Y prediction, then the location and magnitude would be given as well.

The whole prediction process is shown in Figure 11. When making a prediction for the next week’s earthquake, the features are first input into the model of all stations in each region to get the risk_value for each region. If the risk_value is less than the threshold_value for the region, it means there is no expected earthquake and the prediction is finished. On the contrary, if it is judged to be an earthquake, then the prediction of magnitude and epicenter latitude and longitude will start.

LSTM is used for real-earthquake prediction for 16 consecutive weeks based on the multi-station decision mechanism method. Most of the large prediction deviation occurred in the first month, as it was difficult to set a reasonable regional risk threshold at the beginning. In order to reduce prediction deviation, this paper made several historical projections of the model, adjusted the regional risk threshold and optimized the model, which finally improved the model prediction results. AETA is also continuously collecting new data, and the model is incrementally learned in batches every two weeks. Then the new collected data is used for sample set construction. Every month, the model is learned in a full batch, and the training and testing sets are regenerated with all the data and the whole model is retrained. Finally, we save the best model with the best effect on the validation set.

This paper focuses on the earthquakes with the largest magnitude within a week.

Table 3. The results of real-earthquake prediction.

	Actual Magnitude	Predicted Magnitude	Actual Epicenter	Predicted Epicenter
1th week (5 April 2021–11 April 2021)	N	N	N	N
2th week (12 April 2021–18 April 2021)	N	Ms4.0	N	$(28.38° \mathrm{N}, 104.76° \mathrm{E})$
3th week (19 April 2021–25 April 2021)	N	N	N	N
4th week (26 April 2021–2 May 2021)	N	N	N	N
5th week (3 May 2021–9 May 2021)	Ms3.6	N	$(32.4° \mathrm{N}, 104.02° \mathrm{E})$	N
6th week (10 May 2021–16 May 2021)	Ms4.7	N	$(24.43° \mathrm{N}, 99.24° \mathrm{E})$	N
7th week (17 May 2021–23 May 2021)	Ms6.4	Ms3.9	$(25.67° \mathrm{N}, 99.87° \mathrm{E})$	$(28.41° \mathrm{N}$$, 104.65° \mathrm{E}$)
8th week (24 May 2021–30 May 2021)	Ms4.1	Ms4.5	$(25.74° \mathrm{N}, 99.95° \mathrm{E})$	$(25.59° \mathrm{N}$$, 99.95° \mathrm{E}$)
9th week (31 May 2021–6 June 2021)	N	Ms4.1	N	$(25.64° \mathrm{N}, 99.98° \mathrm{E})$
10th week (7 June 2021–13 June 2021)	Ms5.1	N	$(24.34° \mathrm{N}, 101.91° \mathrm{E})$	N
11th week (14 June 2021–20 June 2021)	Ms4.2	Ms4.2	$(24.33° \mathrm{N}, 101.91° \mathrm{E})$	$(24.53° \mathrm{N}$$, 99.41° \mathrm{E}$)
12th week (21 June 2021–27 June 2021)	Ms3.8	Ms4.0	$(32.2° \mathrm{N}, 104.94° \mathrm{E})$	$(24.31° \mathrm{N}$$, 101.87° \mathrm{E}$)
13th week (28 June 2021–4 July 2021)	Ms4.6	Ms3.9	$(24.31° \mathrm{N}, 101.89° \mathrm{E})$	$(32.08° \mathrm{N}$$, 104.57° \mathrm{E}$)
14th week (5 July 2021–11 July 2021)	Ms4.7	N	$(24.43° \mathrm{N}, 99.24° \mathrm{E})$	N
15th week (12 July 2021–18 July 2021)	Ms4.8	Ms3.9	$(32.97° \mathrm{N}, 103.84° \mathrm{E})$	$(28.12° \mathrm{N}$$, 104.64° \mathrm{E}$)
16th week (19 July 2021–25 July 2021)	Ms4.1	Ms4.0	$(29.28° \mathrm{N}, 105.44° \mathrm{E})$	$(28.14° \mathrm{N}$$, 104.69° \mathrm{E}$)

shows the results of the real-earthquake prediction for 16 consecutive weeks; the letter “N” means no earthquakes.

The number of correct predictions is 10, the number of false predictions is 2, and the number of missed predictions is 4. The real-earthquake prediction achieves great results with an accuracy of 0.625, which is the number of correct prediction results divided by the total number of weeks. Some prediction results have a small deviation, such as the predictions in the eighth, eleventh, twelfth, thirteenth and sixteenth weeks. In addition, the model has won the top three in the second AETA earthquake prediction competition, which can also prove the effectiveness of our model. The website of the competition is https://competition.aeta.io/, accessed on 1 September 2021.

However, there are still some actual earthquakes that are not predicted. The predicted results may be inaccurate sometimes when predicting earthquakes near Yunnan. After research and discussion, the reasons can be attributed to the fact that the stations in Sichuan province are installed with high density and concentrated distribution, so that the stations can capture the anomaly signals in time, whereas the stations in Yunnan province are mostly scattered, resulting in the lack of abundant monitoring data and anomaly signals. Therefore, the number of stations installed near Yunnan or the weight of Yunnan in the model needs to be increased in the future.

6. Conclusions

AETA, as a multi-component earthquake detection and prediction system, has been developed independently by the Key Laboratory of Integrated Microsystems (IMS) of Peking University Shenzhen Graduate School. After five years of data acquisition, a large amount of data has been accumulated. This paper constructs several non-time series and time-series models for earthquake prediction based on the feature data extracted from the AETA raw signals. After comparison and analysis, it is confirmed that the LSTM model can achieve the best results for earthquake prediction. Then, the LSTM model is selected for real-earthquake prediction for 16 consecutive weeks. In addition, this paper proposes the method of multi-station softmax-AUC with index weighting to solve the situation that the prediction effect of the same model varies in different stations. This model has placed in the top three of the second AETA earthquake prediction competition.