Stock Price Prediction Using a Frequency Decomposition Based GRU Transformer Neural Network

Abstract

Stock price prediction is crucial but also challenging in any trading system in stock markets. Currently, family of recurrent neural networks (RNNs) have been widely used for stock prediction with many successes. However, difficulties still remain to make RNNs more successful in a cluttered stock market. Specifically, RNNs lack power to retrieve discerning features from a clutter of signals in stock information flow. Making it worse, by RNN a single long time cell from the market is often fused into a single feature, losing all the information about time which is essential for temporal stock prediction. To tackle these two issues, we develop in this paper a novel hybrid neural network for price prediction, which is named frequency decomposition induced gate recurrent unit (GRU) transformer, abbreviated to FDGRU-transformer or FDG-trans). Inspired by the success of frequency decomposition, in FDG-transformer we apply empirical model decomposition to decompose the complete ensemble of cluttered data into a trend component plus several informative and independent mode components. Equipped with the decomposition, FDG-transformer has the capacity to extract the discriminative insights from the cluttered signals. To retain the temporal information in the observed cluttered data, FDG-transformer utilizes hybrid neural network of GRU, long short term memory (LSTM) and multi-head attention (MHA) transformers. The integrated transformer network is capable of encoding the impact of different weights from each past time step to the current one, resulting in the establishment of a time series model from a deeper fine-grained level. We appy the developed FDG-transformer model to analyze Limit Order Book data and compare the results with that obtained from other state-of-the-art methods. The comparison shows that our model delivers effective price forecasting. Moreover, an ablation study is conducted to validate the importance and necessity of each component in the proposed model.

1. Introduction

The stock market has been widely recognized as an investment approach with great potential in making profits [1]. Ways have been studied to predict the future prices in order to maximize the returns of the investments and minimize the corresponding risks [2]. However, due to the unpredictable fluctuation of stock market data and complicated trading circumstances in practice [3], it is difficult to interpret the price movement by domain knowledge. Thus, though traditional statistical methods with good interpretation such as auto regressive models [4], hidden Markov models [5] and tree models [6] have been studied, it is still under extensive investigation to train a strong prediction model [7].

In search for powerful prediction models, deep learning techniques have become more and more popular due to the increasing number of available datasets and growing computational power [8,9]. Compared to traditional models, deep learning models are capable of approximating almost all algebra functions [10]. Therefore, they do not have the disadvantage of suffering from model misspecification [11]. Moreover, prior knowledge has been incorporated to the neural networks to improve model performance [12]. Existing methods focus on designing a novel model structure or adaptive loss functions. For example, convolutional neural networks have been proposed to capture the spatially invariant features of the relevant datasets [13]. Resnet has been proposed to guide the process of training the neural networks [14]. Also, loss function has been adjusted according to domain rules to obtain a better network [15].

Meanwhile, recurrent neural networks (RNN) have been established to handle sequential datasets [16]. In RNN family, information is passed recursively along the time axis with a weighted decay, which means, more recent time will have an impact with larger weight. But the importance of the information does not always depend on the happening time in real problems [17]. Hence, in order to store crucial long temporal information, many adaptive RNNs have been proposed for use. For instance, long short-term memory (LSTM) [18], which stores both short and long time information jointly, is considered to further incorporate their interactions with new information. In addition, gated recurrent unit (GRU) has been proposed to supplement LSTM to accelerate the training and mitigate the problem of overfitting [19].

Due to its success in analyzing sequential datasets [20], LSTM and GRU have been extended to perform stock prediction [21,22,23]. But there are two main challenges in this approach. The first challenge is that a subtle change in high frequency financial data may cause a significant increase of difficulty for the neural network to make reliable stock price predictions. As displayed in Figure 1a, the stock price change from 10.235 to 10.215 with a 0.19% fluctuation would be too subtle to be captured. The second challenge lies in that current RNN networks mingle all the past information (from time 0 to time $t-1$) into a single long time cell, thus unable to consider the impact of past time on current time from the fine-grained level. At the same time, due to the extremely low signal-to-noise ratio of the stock data in comparison to the other sequential datasets such as speech recognition and text analysis, it is necessary to take certain targeted adjustments to boost this signal-to-noise ratio [24].

To tackle the first problem, the complete ensemble empirical mode decomposition (CEEMD) [25] will be introduced to decompose the original high-frequency sequential data into several component. As shown in Figure 1, we decompose the signal into one trend component of high frequency data (price trend) and several mode components by frequency revealing explicit fluctuations. The mode components range between $-0.006$ and 0.006, where the relative change can be easily captured. To solve the second problem, we consider impact of different weights from each time step to current time, and establish the time series model from a more fine-grained level. Transformers are such functions that have been proved to possess strong modelling abilities to capture long-time dependencies in sequential data [26], thus have been studied to perform the stock prediction as well [27]. In summary, to combine the strong potentials of RNN and transformer in capturing related signals, we propose in this paper a hybrid GRU-based network with a multi-head transformer layer and the complete ensemble empirical mode decomposition (CEEMD) [25]. To the best of our knowledge, this is the first time that a hybrid deep learning model integrating these three great time series techniques is used to forecast the stock price. In our experiments to be detailed later in the paper, it shows our model outperforms the 3 other strong hybrid models in [28,29,30] regarding high-frequency stock price prediction.

There are three major features in our proposed frequency decomposition based GRU transformer (FDG-Trans) neural network:

• We decompose the feature of sequential prices into 4 series using CEEMD. After applying CEEMD, there are 1 trend component and 3 mode components, which are far more inofrmative and greatly alleviates the model training problems caused by subtle change problem. Hence, it is well designed for the highly noisy stock data.
• In the FDG-Trans hybrid network structure, transformer is used first to extract fine-grained information between different time steps, followed by LSTM and GRU to do further analysis. This architecture does a well trade-off between capturing more information and reducing overfitting.
• When applied to high frequency data such as limit order book (LOB), FDG-Trans distinguishes the timely variant features such as prices and volumes and timely invariant features such as open price and yesterday close price. In FDG-Trans, the invariant features will not be put into the RNN networks, which further reduces the overfitting and makes the training process more efficient.

The rest of this paper is organized as following: In Section 2 we review the related work about applying deep neural networks to stock data. In Section 3, our proposed method, as well as the evaluation criterion, is developed in detail. In Section 4, both high frequency trading datasets and their forecasting results using different models are presented. The conclusions and discussions are given in Section 5.

2. Related Work

In the last 5 years there is rich literature on applying the RNN family to do forecasting about the stock price. LSTM is the most well-known and widely-used one. It has a short-term and long-term cell in each step to capture both the instant information and important information. Due to the complexity of the stock data, some hybrid models including LSTM layers have also been proposed. GRU is an adaptive version of LSTM. It involves less number of parameters so it is computationally faster and more applicable even when the size of stock dataset is overly large. Attention is a mechanism which addresses how the current information relates to the previous information, thus has great potential capturing the sequential interactions. The remaining of this section will review the use of LSTM based, GRU based or attention based models for stock market forecasting, respectively.

2.1. LSTM Based Models

Zhuge et al. [31], Srijiranon et al. [32], and Nguyen and Yoon [33] use LSTM to predict the opening price or close price of the stock. In [31], the input consists of normal stock index data and emotional data obtained by a Bayesian classifier. Then the normal stock index data and emotional data will be transformed by a separate fully connected neural network, respectively. In [32], the news and historical data was pre-processed by FinBERT [34] and Principle Component Analysis(PCA) [35], respectively. In [33], some prior knowledge has been used through transfer learning [36]. Experiment shows that the prediction performances have been greatly improved. Selvin et al. [37] integrates LSTM combined with Convolutional Neural Network(CNN) to do the stock price prediction. The data is first inputed into a CNN layer to capture some locational features, and then followed by a LSTM layer. In the experiment, the model is trained on the Infosys dataset and can be applied to TCS and Cipla with good performance. Kim and Won [38] combines LSTM with several GARCH type models to predict the stock price index volatility. The input consists of GARCH parameters, EGARCH parameters and explain variables. After concatenation, the data will be passed in 1 LSTM layer and 2 dense layers to get the final prediction. In the experiment, the proposed model is superior to GARCH models and LSTM under the dataset of KOSPI 200. Sunny et al. [39] applies Bi-Directional LSTM (BI-LSTM) to forecast the stock price trend. The model consists of an activation RELU layer, 2 hidden BI-LSTM layers, 1 drop out layer and 1 dense layer. After some adjustment of hyper-parameters, lower RMSE can be generated compared to LSTM models, which helps the stock traders to get some financial profits with a sustainable strategy. Zhang et al. [28] uses LSTM based model to forecast the stock price trend with limit order book data, which is a snapshot of instant bid and ask information every 3 s. The model (named DeepLOB) consists of 6 CNN layers, 3 parallel inception layers and 1 LSTM layer. Experimental shows that DeepLOB outperforms 10 time-series related models when applied to London Stock Exchange.

2.2. GRU Based Models

Hossain et al. [40] uses GRU based models to do the stock prediction. The model consists of 1 LSTM layer, 1 dropout layer, 1 GRU layer, 1 another dropout layer, and finally followed by a dense layer. In the experiment, S& P 500 historical time series data is used to train and test the model. It is shown that this model out performances former state-of-art methods including RNN, CNN and Ensemble methods. Shahi et al. [23] uses LSTM and GRU to do the stock market forecasting. The model consists of 3 data transformation process, which pre-process the stock and news data, and then concatenate into a time series sequence generator. After the transformation, the output will be put into an LSTM or GRU layer to get the final prediction. In the NEPSE dataset in the experiment, it is showed that news data helps with the prediction and GRU outperforms LSTM. Gao et al. [21] intends to combine GRU with LASSO or PCA to predict the price trend. The model will first use LASSO or PCA to do the dimension reduction and then apply LSTM or GRU layer to get the final prediction. In the experiment, the data of the Shanghai Composite Index with a total of 3481 days are used to do the training and test. It is shown that GRU using LASSO dimension reduction is better. Wu at al [41] uses GRU with a Tree Regularization technique(GRU-Tree) to forecast the price trend. In this method, a GRU based neural network is converted to a decision tree. In the experiment, it is shown that GRU-Tree is with a higher AUC both than the decision tree and GRU network under a dataset of SSE Composite Index in China. Liu et al. [42] proposes a regularized GRU-LSTM neural network to do short-term closing price prediction. The model consists of 1 $L_2$ regularized GRU layer, 1 another GRU layer, 1 LSTM layer and finally 1 dense layer. In the experiment, 2 stock prices with a time range of 10 years in China are used. It is shown that the hybrid model performs better that the existing GRU and LSTM network.

2.3. Attention Based Models

Li et al. [43] incorporates multi input LSTM to attention based neural networks. In this model, there is a multi input LSTM and a attention layer, followed by several ReLU layers to get the final prediction. The multi input consists of a main stream and 3 auxiliary feature groups in order to avoid dimension expansion problems. It is showed that in the dataset of CSI-300 index with a time range of 4 years, there is a significant improvement compared to several LSTM or CNN based methods. Muhammad et al. [44] develops a transformer-based model with 2 input layers, 3 transformers layers, 1 pooling layer, followed by 2 dropout layers and 2 dense layers, which is used to do the forecasting of Dhaka Stock Exchange(DSE), including daily, weekly and monthly stock prices, with a promising results. Liu et al. [45] proposes CapTE (Capsule network based on Transformer Encoder) model to do the stock movements prediction. The transformer encoder is utilized to filter the deep notable features of text in social media. Combined with the stock data, it will be passed into 2 CNN and 2 dense layers to get the final prediction. It shows significant results when applying to a dataset including 47 stocks with a time range of 10 years. Zhang and Zohren [29] combines the aforementioned DeepLOB with an encoder-decoder structure to do the prediction. The encoder part is a DeepLOB model, through which the stock data is transformed into a vector, followed by a decoder to get the final predictions. Experimental shows that it outperforms DeepLOB and other time-series related models when applied to FI-2010 dataset, especially when the prediction time horizon is long. Sridhar and Sanagavarapu [46] integrates transformer with CNN layers to do the stock movement prediction. The structure of the model consists of 2 multi head layers, 1 transformer and 2 CNN layers. In the experiment, it is showed that this model is superior to several CNN or RNN based models under the dataset of the S& P 500 index.

Compared to other papers above, in this paper, FDG-Trans intends to combine GRU and transformer, which can take advantage of the fast-speed GRU, and capture the semantic signals along different timestamps. In addition, the decomposition of the frequency technique has been added to our model. After the decomposition, the fluctuation of the price will be largely reduced, which leads to more stable training data for the training process of the neural network, thus improves the prediction performance.

3. Methods

3.1. Problem Statement

Let y denote the value of target, $\widehat{y}$ the predicted value and ${\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},\cdots ,{\mathbf{x}}_{\mathbf{T}}$ the values of predictors at different time steps. Then we want to minimize $D(y,\widehat{y})$ where

$$\widehat{y}=f(\mathbf{\theta },{\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},\cdots ,{\mathbf{x}}_{\mathbf{T}})$$

(1)

${\mathbf{x}}_{\mathbf{t}}=\{x_{t1},x_{t2},\cdots ,x_{tJ}\}$ is a vector of J predictors at time t, and $\mathbf{\theta }$ is the hyper parameter in function $f(·)$. $D(·,·)$ is a customized distance discrepancy to measure how close the estimation is to the true value.

3.2. Problem Analysis and Overview

In Equation (1), when ${\mathbf{x}}_{\mathbf{T}}$ represents the stock data, we hope to capture the inter correlation between ${\mathbf{x}}_{{\mathbf{t}}_{\mathbf{k}}}$ and ${\mathbf{x}}_{{\mathbf{t}}_{\mathbf{l}}}$. Hence, it is highly recommended to explicitly model the relationship between ${\mathbf{x}}_{{\mathbf{t}}_{\mathbf{k}}}$ and ${\mathbf{x}}_{{\mathbf{t}}_{\mathbf{l}}}$. Another problem of the stock data aforementioned is the subtle change problem. As such, ideally it is preferred that the same features at different time steps are comparatively more stable. To address this, we hope to do some data transformation to the original scattered features.

In detail, let $x_{tj}$ denote the $j_{th}$ feature at time step t, and assume that the $p_{th}$ column is where the mid price located. We propose frequency decomposition based GRU Transformer (FDG-Trans) framework, consisting of a CEEMD module and a hybrid GRU Transformer. First, we use the CEEMD to do the decomposition for ${\mathbf{x}}_{\mathbf{p}}=\{x_{1p},x_{2p},\cdots ,x_{Tp}\}$ and get ${\mathbf{x}}_{\mathbf{p}\mathbf{1}},{\mathbf{x}}_{\mathbf{p}\mathbf{2}},{\mathbf{x}}_{\mathbf{p}\mathbf{3}},{\mathbf{x}}_{\mathbf{p}\mathbf{4}}$. Then, we put the new data matrix into 1 LSTM and 1 GRU, followed by 1 multi-head transformer, finally concatenated with timely invariant features and 1 linear layer. The sketch of our method can be shown in Figure 2.

3.3. Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMD)

3.3.1. Empirical Mode Decomposition (EMD)

EMD algorithm [47] attends to decompose a complex series data to several basic ones, which is under the assumption that the series data is influenced by several simple factors. The phase of simple factors is composed of Intrinsic Mode Functions (IMFs), which satisfy 2 conditions: (i) in the whole series data, the number of extrema and the number of zero-crossings must be equal or differ by one; and (ii) the mean value of upper and lower envelopes in each signal is zero. The detailed process can be described as following:

(1)

Mark all the local minimums and maximums in the series data $S\left(t\right)$.

(2)

Apply cubic spline method to determine the lower and upper envelops along the time line.

(3)

Obtain the local mean $m\left(t\right)$ according to the lower and upper envelops.

(4)

$tem{p}_1\left(t\right)=S\left(t\right)-m\left(t\right)$, if the variance of $tem{p}_1\left(t\right)$ is less than a pre-defined threshold, then $IM{F}_1\left(t\right)=tem{p}_1\left(t\right)$. Otherwise, replace $S\left(t\right)$ by $tem{p}_1\left(t\right)$ and repeat the above steps.

(5)

$S\left(t\right)=S\left(t\right)-IM{F}_1\left(t\right)$, repeat the above process to get $IM{F}_2\left(t\right),IM{F}_3\left(t\right),\cdots ,IM{F}_K\left(t\right)$. The value of K is determined by the termination condition.

(6)

In sum, we get $S\left(t\right)={\sum }_{i=1}^{K}IM{F}_i\left(t\right)+res\left(t\right)$ where $res\left(t\right)$ is a term to indicate the overall trend of series.

3.3.2. Ensemble Empirical Mode Decomposition (EEMD)

Although widely used, EMD has its own weakness. In practice the signals are sometimes mixed with variational fluctuations, or white noise, which means $S\left(t\right)=P\left(t\right)+\sigma \left(t\right)$. $\sigma \left(t\right)$ is what we do not know but has significant impact on the decomposition. As such, EEMD [41] has been put up.

In EEMD, $S\left(t\right)=P_i\left(t\right)+{\sigma }_i\left(t\right),i=1,2,\cdots ,n$, where ${\sigma }_i\left(t\right)$ is the randomly generated Gaussian noise. For each $P_i\left(t\right)$ and $k_{th}$ mode, we apply EMD to get the $IM{F}_k^i\left(t\right)$. Then $IM{F}_k\left(t\right)={\displaystyle \frac{1}{n}}{\sum }_{i=1}^{n}IM{F}_k^i\left(t\right)$. From the process we can see, EEMD tries to decompose several similar series of $S\left(t\right)$ and get the average one.

3.3.3. Complete Ensemble Empirical Mode Decomposition (CEEMD)

EEMD is a further step to do the data analysis, but it is found that the number of experiments n needed to get a satisfactory decomposition is large [25]. To address this problem, Ref. [25] proposed CEEMD. In EEMD, white Gaussian noise has been added to $S\left(t\right)$ compared to EMD. In CEEMD, white Gaussian noises are added for every mode. In other words, each IMF can be regarded as the original definition plus a Gaussian noise. Overall, we get the following steps:

(1)

$S\left(t\right)=P_0^i\left(t\right)+{\sigma }_0^i\left(t\right),i=1,2,\cdots ,n$, where ${\sigma }_0^i\left(t\right)$ is a white noise series. Decompose $P_0^i\left(t\right)$ by EMD to get $IM{F}_0^i\left(t\right)$. $IM{F}_0\left(t\right)={\displaystyle \frac{1}{n}}{\sum }_{i=1}^{n}IM{F}_0^i\left(t\right)$

(2)

$r_1\left(t\right)=S\left(t\right)-IM{F}_0\left(t\right)$.

(3)

Let $r_1\left(t\right)=P_1^i\left(t\right)+{\sigma }_1^i\left(t\right),i=1,2,\cdots ,n$. Decompose $P_1^i\left(t\right)$ by EMD to get $IM{F}_1^i\left(t\right)$, then $IM{F}_1\left(t\right)={\displaystyle \frac{1}{n}}{\sum }_{i=1}^{n}IM{F}_1^i\left(t\right)$

(4)

Repeat the above process K steps. For each $k=2,\cdots ,K$, $r_k\left(t\right)=r_{k-1}\left(t\right)-IM{F}_k\left(t\right)$, $r_k\left(t\right)=P_k\left(t\right)+{\sigma }_k\left(t\right)$, where ${\sigma }_k\left(t\right),k=0,1,2,\cdots ,K$ is a bunch of noise series, with the variance decreasing in general.

(5)

$S\left(t\right)={\sum }_{i=0}^{K}IM{F}_i\left(t\right)+res\left(t\right)$, where K is determined by a pre-defined stopping criteria.

With the proposed CEEMD, it is easier to capture the signals of the series of prices. The original price series change subtly, which makes it difficult for the neural network to learn how to extract informative information to do the forecasting. After CEEMD, the trend component provides the trend information and 3 mode components indicate the explicit fluctuation of stock price.

Note that after the EEMD is applied several IMFs are generated by the decomposition. Selection of suitable IMFs is always a challenge, Ref. [48] has discussed the selection criteria by calculating the cross-correlation for all the modes considered. However, for RNNs, as the fixed number of input features is required, for simplicity, the price series is decomposed to a certain number of IMFs and all will be chosen.

3.4. Hybrid GRU-Transformer

3.4.1. RNN

Recurrent neural network(RNN) is a class of neural network where its hidden states are connected to the hidden states of the previous outputs, so its output is affected by the previous states [49]. By retaining a state that can represent information from an arbitrarily long context window, RNN exhibits temporal dynamic behavior. As shown in Figure 3, a typical RNN consists of three parts: input units X, hidden units h and output units Y. Mathematically, the RNN model can be formulated as:

$$h_t=f(U{X}_t+W{h}_{t-1})$$

(2)

$$Y_t=f\left(V{h}_t\right)$$

(3)

• $h_t$: the hidden state at time t
• $Y_t$: the output state at time t
• $X_t$: the input state at time t
• U: the weight matrix from input layer to hidden layer
• W: the value of the previous hidden layer is used as the weight matrix for the input of the next layer
• V: the weight matrix from hidden layer to output layer
• f: the activation function, usually nonlinear

3.4.2. LSTM and GRU

Long Short-Term Memory (LSTM) network [18] is a specially designed version of RNN to handle sequential dataset. Previous RNN models only use the past information based on the time line, while gates have been proposed in LSTM to realize the function of capturing the long time important information. In the LSTM unit, there are 1 input gate, 1 forget gate and 1 output gate.

Gate Recurrent Unit (GRU) network [19] is similar to LSTM but with simpler structure. In the GRU unit, there are 1 update gate and 1 reset gate. The structures of LSTM and GRU unit can be visualized in Figure 3.

In the LSTM unit in Figure 3, $h_{t-1}$ and $c_{t-1}$ can be regarded as the short and long term memories, respectively. The forget gate ($f_t$) determines the importance of information stored in the long term memory after obtaining new data at time step t. The input gate ($i_t$) indicates how the long term memory should be updated. The output gate calculates the new short term memory, taking the new long term memory into consideration. Mathematically, it can be formulated as:

$$i_t=\sigma (W_{ii}{x}_t+b_{ii}+W_{hi}{h}_{t-1}+b_{hi})$$

(4)

$$f_t=\sigma (W_{if}{x}_t+b_{if}+W_{hf}{h}_{t-1}+b_{hf})$$

(5)

$$g_t=tanh(W_{ig}{x}_t+b_{ig}+W_{hg}{h}_{t-1}+b_{hg})$$

(6)

$$c_t=f_t⨀c_{t-1}+i_t⨀g_t$$

(7)

$$h_t=o_t⨀tanh\left(c_t\right)$$

(8)

• $i_t$: the input gate at time t
• $f_t$: the forget gate at time t
• $g_t$: the candidate memory state at time t
• $c_t$: the memory state at time t
• $h_t$: the hidden state at time t
• $o_t$: the output gate at time t
• $\sigma$: the activation function
• $W_{x,y}$: the wight matrix from state x to state y, where i, g, f, h, n mean the input state, the candidate memory state, the forget state, the hidden state, the new memory state, respectively
• $b_{x,y}$: the bias term from state x to state y

In the GRU unit, compared to LSTM, $h_{t-1}$ contains integrate information of both long and short terms. The reset gate is similar to the function of the input gate in LSTM, while the update gate realizes the functions of the forget and output gates simultaneously. In practice, it is widely accepted that GRU performs more or less the same as LSTM with less computational burden [50]. The mathematical update process is:

$$z_t=\sigma (W_{iz}{x}_t+b_{iz}+W_{hz}{h}_{t-1}+b_{hr})$$

(9)

$$r_t=\sigma (W_{ir}{x}_t+b_{ir}+W_{hr}{h}_{t-1}+b_{hr})$$

(10)

$$n_t=tanh(W_{in}{x}_t+b_{in}+r_t(W_{hn}{h}_{t-1}+b_{hn}))$$

(11)

$$h_t=(1-z_t)n_t+z_t\ast h_{t-1}$$

(12)

• $z_t$: the update gate at time t
• $r_t$: the reset gate at time t
• $n_t$: the new memory state at time t
• $h_t$: the hidden state at time t
• $\sigma$: the activation function
• $W_{x,y}$: the wight matrix from state x to state y, where i, z, h, n mean the input state, the update state, the hidden state, the new memory state, respectively.
• $b_{x,y}$: the bias term from state x to state y

3.4.3. Transformer

Transformer is designed to have a more fine-grained process of sequential data. It adopts the self-attention mechanism [51], which is used to capture the interaction information between any pair of time step i and j. Compared to LSTM and GRU, it is stronger to make use of long term information at the expense of more parameters.

As sketched in Figure 4, the input at each time step will first be transfered by a Embedding layer to a vector, which is a representation of information a high-dimensional space. Then the vector is combined with the positional information to be the input of the multi-head attention unit. For each head of attention unit, there are 3 matrix parameters for the transformer to learn: the key weights $W_K$, the query to be keyed weights $W_Q$, and the value weights $W_V$. The embedding X is multiplied with the above 3 matrix to get the key matrix K, query matrix Q and value matrix V. Denote $d_k$ the dimension of K, then

$$Attention(K,Q,V)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(13)

One tuple of $(W_K,W_Q,W_V)$ weight matrix is named an attention head, and in a multi-head attention layer there exist several heads. The outcomes of each attention head are added and normalized to be passed into the next layer. The feed-forward layer is the weight matrix which is trained during training and is applied to every respective time step position.

Transformer is originally designed for and widely used in the natural language process (NLP) field [52]. In the stock price prediction and FDG-Trans, we make use of the multi-head self-attention mechanism. After the process by LSTM and GRU, multi-head transformer helps with extract useful information about interactions between different time steps.

3.5. Training Objective

The training objective is to minimize the loss function which measures the discrepancy between the predicted and true values on the training or validation dataset. Some widely used loss functions are:

• Mean Absolute Error (MAE):

  $$L_{mae}={\displaystyle \frac{1}{N}}\sum _{i=1}^N|y_i-\widehat{y_i}|$$
• Mean Squared Error (MSE):

  $$L_{mse}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-\widehat{y_i})}^2$$
• Root Mean Squared Error (RMSE):

  $$L_{rmse}=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-\widehat{y_i})}^2}$$
• Huber Loss:

  $$L_{huber}=\left\{\begin{array}{cc}{(y_i-\widehat{y_i})}^2\hfill & for|y_i-\widehat{y_i}|\le \delta \hfill \\ 2\delta |y_i-\widehat{y_i}|-{\delta }^2& otherwise\hfill \end{array}\right.$$

  where y is the real value, $\widehat{y}$ is the predicted value, n is the number of samples, $\delta$ is a pre-defined threshold.

Among the losses, MAE is the simplest one. It calculates the mean of absolute differences between the predicted and actual values. MDPE can be regarded as a normalized version of MAE, as large scale values will not have a key impact. Meanwhile, MAE and MDPE can be biased. MSE is unbiased and computationally efficient, but it is sensitive to outliers. RMSE is a less extreme loss function for large values compared to MSE but the gradient is less smooth near the extreme point. Huber loss function [53] is more smooth and robust, but with a greater training burden.

4. Experiment

4.1. Dataset

4.1.1. Limit Order Book (LOB)

Limit order book (LOB) is a key dataset for doing research about high frequency trading as a mechanism to match buy and sell orders in the market. The key information of an order consists of price, volume and direction (buy or sell). In most stock markets there are two kinds of orders: limit order and market order. Limit order is the order with fixed price, volume and direction for a specific stock. It is also the object to be stored by LOB. The limit orders will be added to the order queue according to the price level and direction, offering market liquidity. Market order is the order with fixed volume and direction, which will be executed immediately on the best prices in the opposite direction. Hence, market orders can be executed immediately but may with disadvantageous prices. Limit orders will only be executed at the limit price but the execution time cannot be guaranteed. Meanwhile, limit orders extend the information stored by LOB while market orders or cancellations of limit orders shorten the LOB.

LOB contains cumulative information of current limit orders. For high frequency trading analysis, usually at least 5 queues of each direction are recorded, which means, the best 5 five selling prices (Ask), the best 5 five buying prices (Bid) and their corresponding volumes are used. Figure 5 is a snapshot example of LOB.

4.1.2. CSI-300

We choose 100 stocks from CSI-300 as our dataset. Our dataset includes 21 trading days in April, 2021. There are 3 h and 57 min active trading hours in each trading day. We update the LOB every 3 s. Hence, there are 4740 time steps of LOB records each day.

In addition, we employ the data normalization to transform the raw data into an appropriate range for deep learning algorithms. The data normalization maps the data into interval $[0,1]$ according to the following formula:

$$\widehat{P}=\frac{P-P_{min}}{P_{max}-P_{min}},$$

(14)

where the P is the price before normalization, $P_{max}$ and $P_{min}$ are the maximum and minimum price before normalization. $\widehat{P}$ is the price after the normalization.

4.2. Evaluation Criterion

For stock s, let $D_n^s=\{y_i,i=1\cdots n\}$ denote the true rates of return and $\widehat{D_n^s}=\{\widehat{y_i},i=1\cdots n\}$ the predicted values obtained by a method, then

$$r_s^2={\displaystyle \frac{{\sum }_{i=1}^n{(\widehat{y_i}-y_i)}^2}{{\sum }_{i=1}^n{y}_i^2}}$$

(15)

is the r square and

$$r^2={\displaystyle \frac{{\sum }_{i=1}^N{r}_i^2}{N}}$$

(16)

is the r square for the method. In general, a method with an r square higher than 0.01 will be regarded as a valid method.

4.3. Implementation Details

4.3.1. FDG-Trans

In this experiment, one specific FDG-Trans is trained for each stock. The dataset consists of 100 stocks in total for one month period from China Securities Index (CSI 300) market. For each stock, the first 19 trading days are used as training data and the rest 3 days test data. For the input of FDG-Trans, recall that we have 4740 time steps for each stock every trading day. LOB information in every continuous 130 time steps will be used as a single input, and the rate of return after 5 min is the corresponding output. The number of 130 is chosen by the cross validation. For test data, all the possible 130 continuous time step information will be used. In other words, $X_1=({\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},\cdots ,{\mathbf{x}}_{\mathbf{130}})$, $X_2=({\mathbf{x}}_{\mathbf{2}},{\mathbf{x}}_{\mathbf{2}},\cdots ,{\mathbf{x}}_{\mathbf{131}})$. Thus, we have around 4610 pieces of input data every trading day. For training data, every 1 in 10 possible 130 continuous time step information will be used, which means, $X_1=({\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},\cdots ,{\mathbf{x}}_{\mathbf{130}})$, $X_2=({\mathbf{x}}_{\mathbf{11}},{\mathbf{x}}_{\mathbf{12}},\cdots ,{\mathbf{x}}_{\mathbf{140}})$. Thus, we have around 461 pieces of data every trading data. The reason to skip the other 9 is to reduce the auto correlation of our inputs, which would possibly lead to the problem of over fitting.

In the training of FDG-Trans, Adam is used as the optimizer. The initial learning parameter is set to be 0.001, with an exponential linear decay by 0.9 after every epoch. The training dataset is trained for 75 epochs. The detailed hyper-parameters are summarized in

Table 1. Experiment Hyper-parameter Details.

Item	Hyper-Parameter
Optimization	Adam
Initial Learning Rate	0.001
Exponential Linear Decay	0.9
Epoch Number	75

. The demonstrate the success of training process, the training losses on four sub-datasets are shown in Figure 6.

4.3.2. Baseline Method

In previous price stock papers, the hybrid models always outperform traditional models such as linear model, SVM, LSTM, GRU or CNN. Hence, in this paper all the baseline methods will not include traditional models. Instead, 3 strong hybrid models are chosen as baseline models for comparison.

In [28] the model (DeepLOB) consists of 6 CNN layers, 3 parallel inception layers and 1 LSTM layer. In [29], the model (MHF) combines the aforementioned DeepLOB with an encoder-decoder structure to do the prediction. The encoder part is a DeepLOB model, through which the stock data is transformed into a vector, followed by a decoder to get the final predictions. In [54], the model contains 5 sequential CNN layers, batch normalized after each of the layers, followed by 1 LSTM layer.

In addition, wavelet transform and Fourier transform will be tried on the frequency decomposition part for the comparison with CEEMD.

4.4. Experiment Results

4.4.1. Comparison with the State-of-the-Art Method

In this part, we investigate the advantages of the proposed FDG-Trans method. We have compared the proposed FDG-Trans method over 3 other hybrid methods. The results are reported in

Table 2. Performances of FDG-Trans and 3 other hybrid models.

Methods	${\mathit{r}}^2$ (%) ↑	MSE (%) ↓	MAE (%) ↓
DeepLOB [28]	$2.36\pm 0.214$	$0.0302\pm 0.00219$	$0.0945\pm 0.00450$
DeepAtt [54]	$1.75\pm 0.157$	$0.0317\pm 0.00218$	$0.0932\pm 0.00451$
MHF [29]	$1.29\pm 0.104$	$0.0337\pm 0.00219$	0.0896 ± 0.00460
FDG-Trans	2.88 ± 0.168	0.0291 ± 0.00217	$0.0917\pm 0.00446$

. The results of 4 methods are all trained and calculated based on exactly the same training and validation datasets. Note that the average of $r^2$ means the mean of $r^2$s of different stocks. Hence, values in the table with symbol ± does not indicate the confidence interval but the range of $r^2$ of different stocks. It is shown that FDG-Trans method has higher $r^2$ and less errors. For example, our method achieves 2.88 $r^2$, surpassing DeepLOB [28], DeepAtt [54] and MHF [29] with 0.52, 1.13 and 1.59. This result validate the stronger regression capacity of our proposed method.

4.4.2. Comparison with Baseline Frequency Decomposition

The experiments with different frequency decomposition techniques have been done for comparison purpose. CEEMD, together with wavelet transform [55], Fourier transform [56], and no decomposition have been studied respectively for each experiment. The result is showed in

Table 3. Performances of FDG-Trans with different frequency decomposition methods. The results are trained based on MSE loss function.

Methods	${\mathit{r}}^2$ (%)$\uparrow$	MSE (%)$\downarrow$	MAE (%) ↓
No decomposition	$2.79\pm 0.617$	$0.0295\pm 0.00222$	0.0932 ± 0.00447
Wavelet [55]	$2.80\pm 0.159$	$0.0291\pm 0.00218$	$0.0946\pm 0.00450$
Fourier [56]	$2.27\pm 0.176$	$0.0307\pm 0.00220$	$0.0954\pm 0.00443$
CEEMD	2.88 ± 0.168	0.0291 ± 0.00217	0.0917 ± 0.00446

. It is indicated that CEEMD is the optimal choice. Wavelet is also a strong frequency decomposition technique, which can remove high frequency noise from the original time series. However, in this case, it is tricky to decide the threshold for the noise, as it might lose some information compared with CEEMD, while the latter keeps all the information of the price series. As the reason for Fourier transform, it is because that the transform has the assumption that the original series can be divided into several periodical series, but this is not reasonable for stock price series, otherwise everyone can make profits according to the fixed price movement.

4.4.3. Ablation Study

We have also conducted the ablation experiments to verify that each part is necessary. The CEEMD, LSTM, GRU, Transformer and concatenation part have been removed respectively under each trail. The result is showed in

Table 4. Performances of FDG-Trans with ablation experiments.

Methods	${\mathit{r}}^2$ (%) ↑	MSE (%) ↓	MAE (%) ↓
No LSTM	$2.66\pm 0.594$	$0.0298\pm 0.00213$	$0.101\pm 0.00434$
No GRU	$2.65\pm 0.524$	$0.0299\pm 0.00216$	$0.0992\pm 0.00434$
No Transformer	$2.78\pm 0.281$	$0.0294\pm 0.00228$	$0.0960\pm 0.00467$
No concatenation	$2.74\pm 0.589$	$0.0308\pm 0.00218$	$0.0964\pm 0.00448$
No CEEMD	$2.79\pm 0.617$	$0.0295\pm 0.00222$	$0.0932\pm 0.00447$
FDG-Trans	2.88 ± 0.168	0.0291 ± 0.00217	0.0917 ±0.00446

. The $r^2$ decreases more or less under each removal, which indicates that each part has an impact on the performance of the system, consistent with our expectations.

Meanwhile, we compared the proposed FDG-Trans method with other settings from the perspective of different loss functions.

Table 5. Performances of FDG-Trans with different losses. The results are trained based on MSE loss function.

Methods	${\mathit{r}}^2$ (%) ↑	MSE (%) ↓	MAE (%) ↓
MAE	$0.181\pm 0.468$	$0.0295\pm 0.00218$	0.0884 ± 0.00471
RMSE	$2.71\pm 0.501$	$0.0292\pm 0.00217$	$0.0953\pm 0.00451$
Huber	$2.28\pm 0.479$	$0.0292\pm 0.00219$	$0.0950\pm 0.00447$
MSE	2.88 ± 0.168	0.0291 ± 0.00217	$0.0917\pm 0.00446$

shows the performances of the FDG-Trans method with different losses. The results of MAE, RMSE and Huber loss functions show that in this case MSE is the most suitable loss function. MSE can lead to an unbiased estimate and is sensitive to large errors, which is the kind of samples with the most potential profits. Note that MAE resulted in a low $r^2$ but comparatively best MAE loss, it is because that MAE is biased and tends to get the median value instead of the mean. When given a specific input, the rate of return distribution is with long tail in one side, ${\int }_{\left(Y\right|X)}{(y-y_{median})}^{2}dy$ can be much larger than ${\int }_{\left(Y\right|X)}{(y-y_{mean})}^{2}dy$, where Y is the random variable of the rate of return given the predictor matrix X.

4.4.4. Visualization of Mode Decomposition

To validate the effectiveness of the complete ensemble empirical mode decomposition (CEEMD), we visualize several decomposition results and provide analysis for signal decomposition. The results are shown in Figure 7. Series 1 contains 400 time steps and have moderate fluctuations along the time axis. After the decomposition, the price trend is captured by the residual with a smooth curve and the IMFs are a little dense. Series 2 contains 1000 time steps and the price goes up and down frequently. The trend is not ideally smooth, the IMFs are dense. Even in this case, where more IMFs are needed, the series after decomposition is more accessible to a sequential model because of more structured pattern. Series 3 contains 400 time steps with a decrease trend. It is nicely captured by the residual. Series 4 is comparatively stable, but with sudden changes in a short period, which could be recognized as an oscillatory signal by a sequential model. After the decomposition, the problems are alleviated. Series 5 contains 130 time steps, which is exactly the same as our input data. It is originally bounced frequently. After the decomposition, the series are stable and smooth.

These cases show that the CEEMD can decompose stock price series into a main trend component and several mode component revealing explicit fluctuation, e.g., oscillatory and stability.

5. Discussion

It is showed in the experiment that our model outperform 3 other strong hybrid neural networks in the model. We design FDG-Trans based on previous widely-used models, the data characteristic and how to better combine the model with the stock data. In the future, we plan to incorporate prior knowledge into the neural networks. As the hybrid neural network usually contains millions of parameters, it is helpful if we can make use of prior knowledge by artificially tailored structure to relief the training burden. Further more, the experimental results are based on Chinese stock data, we will utilize FDG-Trans to other datasets for further verification. In addition, because the latency in real market trade is super important, we aim to store the latent layer information obtained by RNN networks at time step t for further use at time step $t+1$ to save the computational time at the expense of little decrease of accuracy.

6. Conclusions

In this paper, we have proposed an effective hybrid deep learning neural network, FDG-Trans, consisting of CEEMD, transformer, LSTM, GRU and high frequency data adaptive structure. The CEEMD makes the input extreme low signal-to-noise ratio stock data more stable and stationary. The transformer captures fine-grained level interactions and the LSTM and GRU analyze further sequential information. The design of data concatenation after GRU makes the network more intelligent and particularly suitable for large and diverse datasets including the stock data. We have conducted experiments on real financial market and compared FDG-Trans with several baseline methods. The experiments demonstrate that FDG-Trans is effective and captures the signals efficiently to forecast the price movements.