Sequence Prediction and Classification of Echo State Networks

Abstract

The echo state network is a unique form of recurrent neural network. Due to its feedback mechanism, it exhibits superior nonlinear behavior compared to traditional neural networks and is highly regarded for its simplicity and efficiency in computation. In recent years, as network development has progressed, the security threats faced by networks have increased. To detect and counter these threats, the analysis of network traffic has become a crucial research focus. The echo state network has demonstrated exceptional performance in sequence prediction. In this article, we delve into the impact of echo state networks on time series. We have enhanced the model by increasing the number of layers and adopting a different data input approach. We apply it to predict chaotic systems that appear ostensibly regular but are inherently irregular. Additionally, we utilize it for the classification of sound sequence data. Upon evaluating the model using root mean squared error and micro-F1, we have observed that our model exhibits commendable accuracy and stability.

1. Introduction

The echo state network, which is also referred to as reservoir computing, is a specific type of recurrent neural network that exhibits unique dynamic properties. This type of network was initially introduced in 2001 by Jaeger [1]. These properties allow it to store and process partial information, making it capable of instant memory [2]. A basic echo state network consists of three parts: the input layer, middle reservoir, and output layer [3]. Echo state networks (ESN) work similarly to the way the brain receives and processes external information. The external information is the input, which is transmitted and updated in the ESN’s inherent brain-like structure, known as the reservoir. The output layer then expresses the information in some way. In this way, the reservoir acts like the nervous system of the brain, and the output calculation is analogous to receiving multiple outputs from the neurons of the brain.

ESNs overcome the limitations of traditional recurrent neural networks (RNNs) [4] by using a fixed random structure for the reservoir and training only the output layer. ESNs have gained increasing attention in recent years, leading to numerous improvements proposed by researchers. For example, Yao et al. employed fuzzy weight to optimize the performance of ESNs [5], while Liu et al. utilized the binary grey wolf algorithm to optimize ESN parameters [6]. Aceituno et al. customized ESNs based on datasets to achieve optimal performance [7]. Zhang et al. optimized ESNs based on behavioral spaces [8], Steiner et al. used cluster-based optimization methods [9]. Additionally, Xue et al. employed particle swarm optimization to optimize ESN [10]. Similarly, several researchers have proposed various improvements to the ESN structure. For example, Gallicchio et al. introduced the deep echo state network architecture [11], Wen et al. proposed an ESN architecture based on memristors [12], Alizamir et al. proposed a novel deep ESN architecture [13], and Wang et al. constructed an ESN model using random resistor arrays [14]. These studies have enhanced the performance of ESNs to different extents and have facilitated their development.

Regarding its application, ESNs have been applied in various fields such as time series prediction, pattern recognition, image processing, and so on. Jiang et al. proposed a deep chained echo state network (DCSN) composed of ESN submodules to enhance energy consumption prediction [15], and their experiments demonstrated strong predictive capabilities for multistep and multiscale requirements. Liu et al. introduced an input noise enhanced echo state network (ESN) model to address the challenge of predicting time series with noise [6]. Their experimental findings reveal that their proposed method outperforms existing methods in terms of prediction accuracy. Ghazijahani et al. used ESNs to predict irregular but periodic turbulent flow and achieved accurate results [16]. Their model could simulate the periodicity and instability of flow and make predictions for a longer time span. Electrical load time series prediction was performed using ESNs by Tahir et al. [17]; their proposed algorithm has advantages in forecasting long-term power load. A new deep ESN model was proposed by Gao et al. to learn the dynamic characteristics of effective wave height [18], and was tested on 12 datasets showing statistical superiority over other methods. Lyu et al. utilized ESNs to classify multivariate time series with different scales of variation, and achieved competitive results compared with other state-of-the-art methods [19]. Gardner et al. proposed a modified ESN model that is capable of classifying images, and demonstrated its effectiveness on different datasets [20]. Similarly, Donkor et al. applied ESNs for image segmentation and achieved promising results in segmenting medical images [21]. These studies have expanded the application domains of ESNs to the fields of recognition and image processing, showcasing the versatility of this approach in various data analysis tasks.

On the other hand, there are few studies on brain-like models, such as that of Kim et al., who presented a novel electrokinetic log decoder for the arm motion brain–machine interface and evaluated its performance using an ESN with 24 Gaussian inputs [22]. Their findings indicate that the ESN model with Gaussian inputs for classification can be effectively applied to the electrokinetic log decoder model of the motor brain-machine interface. Damicelli et al. adopted a hybrid method combining the real brain connector and the biological echo state network to solve specific memory tasks, so that they could explore the potential computational impact of the real brain connection mode on task solving [23]. Their study revealed that as long as the minimum randomness and connection diversity are allowed, the performance of the bioheuristic network is comparable to that of the classical echo state network.

Based on the above research and investigation on ESNs, it can be observed that the current frontier of ESNs mainly focuses on parameter optimization, with relatively less attention paid to structural improvements. Some notable advancements include the deep echo state network (DESN) proposed by Gallicchio and the memristor echo state network (MESN) proposed by Wen. For the application field of ESNs, it is primarily used for sequence prediction, with some applications in classification and image processing, and very few in brain-like models. Furthermore, most ESN applications use a single-layer structure. However, the internal structure of ESNs, which is similar to neurons and synapses, as well as its adaptive learning characteristics, make it suitable as a brain-inspired model. Therefore, this article attempts to connect echo state networks with artificial intelligence, laying the foundation for intelligent prediction and classification of security data in network security. We presents a novel structure called the merge input deep echo state Network (MIDESN), which is inspired by the brain-like structure based on previous literature research. The aim of this structure is to overcome the issue of vanishing input data in the deep ESN as the number of layers increases. To achieve this, the input data matrix is integrated into the hidden layer of each layer during training. Moreover, the output weights of the ESN are optimized using the elastic network. Finally, the proposed model is tested using both chaotic sequence data and bird call data.

The content of the article is organized as follows: Section 2 presents the relevant knowledge of echo state networks. The proposed merge input deep echo state network (MIDESN) structure is described in Section 3. The experiment and analysis are given in Section 4. Lastly, the conclusion is drawn in Section 5.

2. Related Work

2.1. Echo State Network

The echo state network (ESN) is a recurrent neural network model that consists of a loosely connected hidden layer and a linearly trained output layer. This design greatly simplifies the training process and avoids issues such as slow convergence and local optima. In contrast to other methods, ESNs are fast to train, have no branches, and are straightforward to implement.

In a basic echo state network, as shown in Figure 1, there are three layers, with the middle reservoir $W_M$ being the most important. It contains numerous neurons that generate the echo state properties of the network and only the output weight $W_O$ needs to be trained. The state update process of the middle layer and the training process of the output weights are given as follows.

$$\begin{array}{c}\hfill m(t+1)=(1-\alpha )m\left(t\right)+\alpha tanh(W_{I}i\left(t\right)+W_{M}m\left(t\right)+\theta )\end{array}$$

(1)

$$\begin{array}{c}\hfill W_O^{\prime }=A{B}^T{(B{B}^T+\beta C)}^{-1}\end{array}$$

(2)

where $m(·)$ represents the state of middle layer neurons at a certain moment. $\alpha$ is the leakage rate parameter; it is a scalar value that determines the speed at which the activity of the reservoir decays over time. $W_I,W_M,W_O$ are input weight matrix, middle layer weight matrix, and output weight matrix, respectively, and $\theta$ is the bias term, and is usually set to be 1. In Equation (2), the ridge regression algorithm is used to train the output weight matrix $W_O$. A represents the label data; the state matrix of the reservoir is represented by B. C is a d-unit matrix of the same size as B. $B^T$ represents the inverse matrix of B and the ridge regression parameter is denoted by $\beta$.

2.2. Deep Echo State Network

Gallicchio et al. [11] introduced the concept of the deep echo state network (DESN), which is an extension of the traditional echo state network (ESN). The DESN architecture involves incorporating multiple reservoirs and increasing the number of neurons in the model, allowing for a deeper and more complex representation of the input data. By increasing the number of reservoirs, the network can perform more sophisticated computations and capture long-term dependencies in the input data.

Figure 2 is the traditional deep echo state network; there are multiple interconnected reservoirs that sequentially process the input data. Each reservoir transforms the input data into a higher-dimensional space, which is then fed into the next reservoir in the sequence. The final reservoir produces an output by performing a linear regression on the transformed data. The overall architecture allows for complex and deep transformations of the input data, leading to improved performance on various tasks. The reservoir state of the traditional deep echo state network is updated as follows in Equation (3).

$$\begin{array}{cc}\hfill m^{\left(l\right)}& (t+1)=(1-{\delta }^{\left(l\right)})m^{\left(l\right)}\left(t\right)+{\delta }^{\left(l\right)}tanh\hfill \\ & (W_I^{\left(l\right)}\times i^{\left(l\right)}\left(t\right)+W_M^{\left(l\right)}\times m^{\left(l\right)}\left(t\right)+ϵ)\hfill \end{array}$$

(3)

$$\begin{array}{c}\hfill i^{\left(l\right)}\left(t\right)=I^{\left(l1\right)}\left(t\right)or{i}^{\left(l\right)}\left(t\right)=m^{(l-1)}\left(t\right)\end{array}$$

(4)

where each layer l has its own state represented by $m^{\left(l\right)}$, and the leakage rate is controlled by $\delta$. The input data $i^{\left(l\right)}\left(t\right)$ are fed into the first reservoir of the DESN, and are then processed through each subsequent reservoir before reaching the final reservoir. $W_I^{\left(l\right)}$ and $W_M^{\left(l\right)}$ are the input and reservoir weight, $tanh$ is the activation function, and a bias term $ϵ$ is added to this calculation. Among it, input and output $i^{\left(l\right)}\left(t\right)$ are shown in Equation (4). Finally, the state of the last reservoir is used to perform linear regression and generate the output.

3. Methods

3.1. Building a Multireservoir Echo State Network

This traditional DESN works better than the basic single-layer ESN [24,25,26]. However, this approach ignores the fact that the input signal may become weaker as the number of layers increases. To address this issue, we propose a novel deep echo state network (DESN) model that merges the input matrix with the current reservoir matrix, as shown in Figure 3. The state update matrix equation for this new model is presented in Equation (5).

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left\{\begin{array}{c}{m}_1^{\left(l\right)}\\ m_2^{\left(l\right)}\\ m_3^{\left(l\right)}\\ :\\ m_M^{\left(l\right)}\end{array}\right\}=& (1-{\alpha }^l)\left\{\begin{array}{c}m{\ast }_1^{\left(l\right)}\\ m{\ast }_2^{\left(l\right)}\\ m{\ast }_3^{\left(l\right)}\\ :\\ m{\ast }_M^{\left(l\right)}\end{array}\right\}+{\alpha }^{l}tanh(\left\{\begin{array}{c}{a}_{1,1}^{\left(l\right)},a_{1,2}^{\left(l\right)},\dots ,a_{1,M}^{\left(l\right)}\\ a_{2,1}^{\left(l\right)},a_{2,2}^{\left(l\right)},\dots ,a_{2,M}^{\left(l\right)}\\ \ddots \\ a_{M,1}^{\left(l\right)},a_{M,2}^{\left(l\right)},\dots ,a_{M,M}^{\left(l\right)},\end{array}\right\}\left\{\begin{array}{c}{m}_1^{(l-1)}\\ m_2^{(l-1)}\\ m_3^{(l-1)}\\ :\\ m_M^{(l-1)}\end{array}\right\}\hfill \\ & +\left\{\begin{array}{c}{b}_{1,1}^{\left(l\right)},b_{1,2}^{\left(l\right)},\dots ,b_{1,M}^{\left(l\right)}\\ b_{2,1}^{\left(l\right)},b_{2,2}^{\left(l\right)},\dots ,b_{2,M}^{\left(l\right)}\\ \ddots \\ b_{M,1}^{\left(l\right)},b_{M,2}^{\left(l\right)},\dots ,b_{M,M}^{\left(l\right)},\end{array}\right\}\left\{\begin{array}{c}m{\ast }_1^{\left(l\right)}\\ m{\ast }_2^{\left(l\right)}\\ m{\ast }_3^{\left(l\right)}\\ :\\ m{\ast }_M^{\left(l\right)}\end{array}\right\}+1)\hfill \end{array}\end{array}$$

(5)

where m is the state of neurons in the reservoir and $m\ast$ represents the state at the last moment. $m^{(l-1)}$ is the state of previous layer. Moreover, a and b stand for the weights corresponding to the input matrix and the reservoir matrix, respectively. $\alpha$ and 1 still represent leak rate and bias. Since the input data are merged into the reservoir of each layer, the input weight and reservoir weight beginning the second layer in the model will be updated accordingly.

To prevent the loss of initial data as the number of layers increases, MIDSN merges the input matrix with the current reservoir matrix. In the traditional DESN model, the input weight of from the second layer is of size $M\times M$ instead of $M\times I$. The input is the previous reservoir state $m^{(l-1)}$, and the reservoir weight has always been $m^{(l-1)}$. However, in our MIDSN, the input and reservoir weight sizes continuously increase as the input data increase. The input weight and reservoir weight sizes in MIDSN are as follows:

$$\begin{array}{c}\hfill a_{size}^{\left(l\right)}=[M+I(l-1)]\times [1+M+I(l-1)]\end{array}$$

(6)

$$\begin{array}{c}\hfill b_{size}^{\left(l\right)}=[M+I(l-1)]\times [M+I(l-1)]\end{array}$$

(7)

where $a_{size}^{\left(l\right)}$ and $b_{size}^{\left(l\right)}$ are the size of the input and reservoir matrix; I and M stand for the dimensions of initial data and the number of neurons of the reservoir separately. Moreover, 1 in Equation (6) is the bias in the input weight matrix.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Y\left(t\right)=W_O^{\prime }\left(enet\right)\times M+\theta ,\\ \hfill M=\left[m_1,m_2,\cdots ,m_l\right]\end{array}\end{array}$$

(8)

It is crucial to recognize that when training an MIDESN, the reservoir must be initialized, which involves a period of idleness for the network. The purpose of initialization is to assign initial values to the weights and biases in the network. Unlike traditional DESNs, the output layer of the proposed model retrieves information from all the reservoirs rather than just the last one (Equation (8)). The output weights are then trained to make the final prediction.

3.2. Optimization of the Output Weight

As we know, the most important of DESN is training $W_O$. Most of the previous researchers used ridge regression to train it [27]. Ridge regression adds an L2 regularization term to the loss function of linear regression. It can avoid overfitting by constraining the size of coefficients and can reduce the variance of the model [28]. Therefore, when there is collinearity in the dataset or there are a large number of features, using ridge regression can alleviate these problems, while lasso regression adds an L1 regularization term to the loss function of linear regression. It can compress some coefficients to 0, thereby achieving the feature selection function. Therefore, when there are many features, using Lasso regression can help select the features that have the greatest impact on the target variable while ignoring the features that have a smaller impact on the target variable. Lasso regression is also suitable for datasets with sparsity or highly correlated features. Is there a regression method that can combine the advantages of the two? The answer is yes, it is an elastic network. Elastic Net is a linear regression model that combines L1 and L2 regularization. Its formula can be expressed as:

$$\begin{array}{c}\hfill min\frac{1}{2}RSS+\alpha \times \left[\xi \parallel W_{out}{\parallel }_1+(1-{\xi }_2){\parallel W_{out}\parallel }_2^2\right]\end{array}$$

(9)

where RSS represents the residual sum of squares, $\parallel W_{out}{\parallel }_1$ and $\parallel W_{out}{\parallel }_2^2$ represent the L1 and L2 regularization terms, $\alpha$ represents the regularization coefficient, and $\xi$ represents the proportion of L1 regularization in the regularization terms.

For a linear regression model, the objective is to minimize the residual sum of squares (RSS):

$$\begin{array}{c}\hfill min\frac{1}{2}RSS=min\frac{1}{2}{\parallel Y-X{W}_{out}\parallel }_2^2\end{array}$$

(10)

where y is the dependent variable vector, X is the independent variable matrix, and $W_{out}$ is the regression coefficient vector.

In lasso regression, the regularization term is the L1 norm, which can be expressed as $\alpha \times \left|\right|W_{out}{\left|\right|}_1$, where $\alpha$ is the regularization coefficient, and $\left|\right|W_{out}{\left|\right|}_1$ is the L1 norm of the regression coefficient vector. Therefore, the objective function of lasso regression can be expressed as:

$$\begin{array}{c}\hfill min\frac{1}{2}RSS+\alpha \times {\left|\right|w\left|\right|}_2^2\end{array}$$

(11)

Elastic Net combines the L1 and L2 regularization terms, and can be expressed as Equation (9), where $\xi$ represents the proportion of L1 regularization in the regularization terms, and $(1-\xi )$ represents the proportion of L2 regularization in the regularization terms. When $\xi =1$, it is equivalent to using only L1 regularization and obtaining Lasso regression; when $\xi =0$, it is equivalent to using only L2 regularization and obtaining ridge regression. Therefore, Elastic Net can balance the advantages of L1 and L2 regularization and improve the robustness and prediction performance of the model.

In the Elastic Net model, the regression coefficient vector w is one of the goals of model optimization. The optimization objective function of the Elastic Net model can be expressed as:

$$\begin{array}{c}\hfill min\frac{1}{2n}{\displaystyle \sum _{i=1}^n}{\left(y_i-\widehat{y_i}\right)}^2+\alpha \left(\xi {\displaystyle \sum _{j=1}^p}\left|w_j\right|+\frac{1-\xi }{2}{\displaystyle \sum _{j=1}^p}{w}_j^2\right)\end{array}$$

(12)

where n represents the number of samples, p represents the number of independent variables, and $y_i$ is the true value of the ith sample, $\widehat{y_i}$ is the predicted value of the ith sample, $\alpha$ is the regularization coefficient, $\xi$ is the hyperparameter that controls the L1 regularization ratio, and $w_J$ represents the regression coefficient corresponding to the jth independent variable.

During the optimization process, we will obtain an optimized regression coefficient vector of $w^{\ast }$, which is the regression coefficient vector that minimizes the objective function. Therefore, $w^{\ast }$ can be expressed as:

$$\begin{array}{cc}& w^{\ast }=\arg \underset{w}{\min }\frac{1}{2n}{\sum }_{i=1}^n{\left(y_i-\widehat{y_i}\right)}^2+\alpha \left(\xi {\sum }_{j=1}^p\left|w_j\right|+\frac{1-\xi }{2}{\sum }_{j=1}^p{w}_j^2\right)\end{array}$$

(13)

4. Experiments

4.1. Evaluation and Data Preparation

In our experiment, we tested the MIDESN model using nonlinear chaotic systems and bird sounds, and compared its performance with traditional ESN and DESN models. We employed the RMSE and Micro-F1 to evaluate the performance of these models.

RMSE stands for root mean squared error, which is a commonly used metric for evaluating the accuracy of predictions or estimates. The RMSE value can range from 0 to infinity, with lower values indicating better accuracy. The equation for calculating RMSE is:

$$RMSE=\sqrt{\frac{1}{n}\sum _{i=1}^n{(y_i-\widehat{y_i})}^2}$$

(14)

where $y_i$ represents the actual value for the i-th observation, $\widehat{y_i}$ represents the predicted value for the i-th observation, and n is the total number of observations.

Micro-F1 is a performance metric used to evaluate multiclass classification models, especially in imbalanced datasets. It is an extension of the F1 score, which is a combination of precision and recall. It is computed using the following equation:

$$Micro-F1=\frac{2\times Micro-Precision\times Micro-Recall}{Micro-Precision+Micro-Recall}$$

(15)

where

$$Micro-Precision=\frac{{\sum }_{i=1}^{N}T{P}_i}{{\sum }_{i=1}^N(T{P}_i+F{P}_i)}$$

(16)

and

$$Micro-Recall=\frac{{\sum }_{i=1}^{N}T{P}_i}{{\sum }_{i=1}^N(T{P}_i+F{N}_i)}$$

(17)

where N is the total number of classes, $T{P}_i$ is the number of true positives for class i, $F{P}_i$ is the number of false positives for class i, and $F{N}_i$ is the number of false negatives for class i.

Chaos refers to a seemingly irregular and complex motion in the real world [29]. It is a perplexing behavior generated by a deterministic nonlinear dynamic system, which is considered to be a deterministic system with inherent randomness. We use different dimensions of the Mackey–Glass chaos system (MG) and Lorenz chaotic system to test the model (Figure 4a,b). There is a wide variety of bird species, and the calls of different species exhibit high similarity. Using a bird call dataset provides an excellent means to test the model’s classification capability. The bird sound dataset we use is the public part of the Birdsdata dataset. This collection consists of 14,311 natural audio recordings, each lasting 2 s, and a text file that lists the standard Chinese names of the bird species featured in the recordings (Figure 4c,d).

4.2. Results and Analysis

First, we predict a one-dimensional Mackey–Glass (MG) chaotic system. For the MG system, we set the initialization length to 100, the training length to 2000, and the prediction length to 1500. To conduct an overall evaluation of our established model, we vary the number of reservoir layers from 1 to 5 and the reservoir size from 100 to 600 in intervals of 50. We perform experiments based on these configurations and obtain the results, which are presented in

Table 1. One-dimensional MG series prediction with our model.

Size	100	150	200	250	300	350	400	450	500	550	600
Layer = 2	204	550	305	400	400	353	574	518	510	256	400
Layer = 3	716	217	398	399	600	1288	511	600	896	357	193
Layer = 4	560	896	516	230	398	696	1307	400	397	496	594
Layer = 5	397	303	400	598	304	694	263	856	34	5	6

.

Table 1 shows the steps that our MIDESN model can accurately predict under different layers and reservoir sizes. The predictions with a bold font indicate forecast values exceeding 1000 in step length. As can be seen from the figure, when the number of layers is 3–4 and the reservoir size is 300–400, better prediction results can be obtained. One of the results is shown in Figure 5.

In order to further evaluate our model, we next keep the total number of neurons in the reservoir constant at 600. This means that for a single-layer ESN, there are 600 neurons in the hidden layer; for a two-layer DESN, there are 300 neurons in each layer; for a three-layer DESN, there are 200 neurons in each layer; and for a four-layer DESN, there are 150 neurons in each layer. We test the impact of the model’s depth and reservoir size on the prediction results over a runtime of 1 second, as time progresses. To assess the performance, we employ the RMSE metric. The results are shown in Figure 6a–d. Figure 6a illustrates the impact of the depth of the MIDESN model on prediction performance. The x-axis represents the time of operation, and the y-axis represents the RMSE value that evaluates the prediction performance at the given time. Despite having four layers and a reservoir size of 400, our model operates very fast, taking only 1.09 s. As depicted in the figure, the RMSE value decreases as time increases, but for some curves, the RMSE value increases after the time exceeds 0.9 s. This is because chaotic systems can only be predicted accurately in the short term and not in the long term, resulting in a higher RMSE value for subsequent predictions. Based on the observations from Figure 6a, we found that the model performs better when the depth is 4, and the prediction step is also longer. We further analyzed and adjusted the parameters for the MIDESN and tested the effect of reservoir size on the model. As shown in Figure 6b–d, it displays the effect of reservoir size on the model at a depth of 2–4. We found the model performs best when the reservoir size is depth is 4 and reservoir size is 200.

In order to ensure the reliability of the experiment, we also use the three-dimensional Lorenz chaotic system to carry out the experiment. The processing of this sequence is to discard the length of 100, the training length of 2500, and the test length of 2000. After continuous debugging, we finally set the leakage rate to 0.01 and the sparsity to 0.682; the size of the reservoir increases from 100 to 600 at intervals of 50, and the number of layers increases from 1 to 5 for experiments. It is worth noting that the input and output dimensions in this case are set to 3. We used the Lorenz system to conduct an overall test on the MIDESN model, as shown in

Table 2. Multidimensional time series measurement with our model. The predictable step sizes are shown in it.

Size	100	150	200	250	300	350	400	450	500	550	600
Layer = 2	730	880	795	990	940	810	950	970	900	890	850
Layer = 3	740	850	1010	950	910	950	500	570	650	1100	750
Layer = 4	440	350	670	750	760	670	620	710	1030	670	700
Layer = 5	540	560	750	740	600	1060	570	700	650	750	520

. It shows the predicted length of the Lorenz chaotic system by the MIDESN at different depths and reservoir. The predictions with a bold font indicate forecast values exceeding 900 in step length. When the numbers of layers are 3, 4, and 5, the prediction effect is better, and the accurate prediction step size has reached more than 1000 when the number of layers is 3. One of the results is shown in Figure 7.

Similarly, in order to further assess our model, we maintain the total number of neurons in the reservoir at a constant 600. We examine the influence of the model’s depth and reservoir size on the prediction outcomes over a duration of 0.6 seconds, as time advances. We utilize the RMSE metric to evaluate performance. The results are depicted in Figure 8a–d.

Figure 8a depicts how the depth of the MIDESN model affects its performance in predicting the Lorenz system. While a chaotic system can be predicted in the short term, long-term prediction remains challenging. As shown in Figure 8a, our model can achieve good prediction results in the short term, regardless of the depth. However, as time exceeds 0.4 s, the advantages and disadvantages of different layer depths become apparent. Notably, when the number of layers is 4 (indicated by the purple line marked with a square), the model can still produce good results even over a longer period of time. Figure 8b–d present our additional analysis of the MIDESN model with different depths. The plot reveals that the smallest RMSE value can be obtained in the short term when the number of layers is 4 and the reservoir size is 2.

After that, we used the MIDESN model to conduct a bird sound recognition test. To extract the features from the sound data, we performed MEL spectral feature extraction (as shown in the Figure 9). The recognition results are presented in

Table 3. Birdsdata data classification with our model; this table shows their accuracy.

Deep	layer = 1	layer = 2	layer = 3	layer = 4
Size = 100	0.616	0.621	0.643	0.662
Size = 200	0.685	0.720	0.758	0.782
Size = 300	0.842	0.853	0.862	0.871
Size = 400	0.874	0.882	0.884	0.898

, the bold font indicates situations where the accuracy exceeds 0.8. The results indicate that the performance of our model generally improves with an increase in the number of layers and reservoir size. Specifically, the model achieves the best performance when the number of layers is 4 and the reservoir size is 400.

We also compared our model with traditional ESN and DESN, and the results are shown in

Table 4. Different model application tests.

Dataset	Models	Layer = 1	Layer = 2	Layer = 3	Layer = 4	layer = 5
MG	ESN	0.1517	/	/	/	/
	DESN	/	0.1472	0.1462	0.1451	0.1462
	MIDESN	/	0.1477	0.1423	0.1398	0.1321
Lorenz	ESN	0.1488	/	/	/	/
	DESN	/	0.1463	0.1486	0.1434	0.1418
	MIDESN	/	0.1419	0.1377	0.1311	0.1252
Birds	ESN	0.681	/	/	/	/
	DESN	/	0.953	0.922	0.901	0.867
	MIDESN	/	0.80467	0.8781	0.932	0.958

, the bold sections represent the performance results of our model on different datasets. RMSE was used to evaluate the performance of the models on the MG and Lorenz datasets, while micro-F1 was used for the Birds dataset. It can be observed that our model consistently achieves good performance and stability across all datasets.

5. Conclusions

With the development of networks and technology, network security faces significant threats. In the process of handling security data, sequence prediction and classification are focal and challenging issues. Echo state networks (ESNs), as classical neural networks, have shown outstanding predictive performance and are known for their efficiency. Consequently, ESNs have been at the forefront of research. In recent years, some scholars have attempted to draw connections between ESNs and the human brain. Inspired by this research, we sought to relate ESNs to intelligent networks and test their performance in prediction and recognition. This study lays the groundwork for automatic identification and classification of security sequences in network security. So in this paper, we first improved the traditional echo state network model. Secondly, we tested the model’s predictive performance using chaotic systems of different dimensions, which can be short-term predicted but not long-term, and we also assessed the model’s recognition performance using bird sounds from different species with high similarity.

We proposed a novel deep echo state network model, called the MIDESN. Our model takes into account the gradual attenuation of inputs with increasing depth, by merging an input matrix into each reservoir. The output weights of the model, which contain all the reservoir information, are trained using elastic regression. We evaluated the model using chaotic systems that are challenging to predict in the long term, as well as bird sound data. The chaotic systems we used included one-dimensional Mackey–Glass and three-dimensional Lorenz data. Our results demonstrate that the MIDESN outperforms traditional ESN and DESN models in terms of effectiveness, while also consuming less energy and processing faster.

The architecture and speed advantages of echo state networks make them suitable for simulating artificial intelligence networks. Through our model, improvements and validations, the prediction and classification performance of echo state networks has been enhanced. In future applications, echo state networks can deliver better results in areas such as network security, artificial intelligence, and brain-inspired models, and offer new insights for the fields of neural computation and nonlinear complex systems.