In this section, the results generated in the search for the optimal LSTM architecture are presented. Then, the predictions obtained from the optimal architecture for the testing set are validated. Finally, the influence of rainfall patterns, which could affect the generalization of the method, is discussed.
4.1. Search for Optimal LSTM Architecture and Hyperparameters
The optimal LSTM architecture was achieved by optimizing the LSTM hyperparameters. LSTM hyperparameters include the numbers of LSTM layers, units in each LSTM layer, fully connected layers (or dense layers), units in each dense layer, and epochs, as well as the batch size and the optimization learning algorithm. The number of epochs defines the number of times the algorithm completes the learning process on the entire dataset (training and testing sets). The dataset is usually divided into a number of batches to complete one epoch. The batch size refers to the number of training samples used in a single batch. By optimizing the results of the evaluation metrics, the number of epochs and the batch size for all models were determined to be 300 and 50, respectively. It is noted that each LSTM model was trained using an adaptive optimization learning algorithm, (i.e., Adam), with a learning rate of 0.001.
Hyperparameters were chosen from a set of selected candidate values, as shown in
Table 2. In the experimented models, the number of LSTM layers was set to be 1, 2, or 3 layers, and the number of units for an LSTM layer were selected from values of 100, 50, 30, and 20. The models have one or two dense layers. In the sequence of dense layers, the number of units for the last dense layer represents the number of outputs, or predictions. In the case of two dense layers, 500 was chosen as the number of units was chosen to be 500 and 363 was selected for the first and second dense layers, respectively. In the case of one dense layer, the number of units was set the same as the output dimension of 363. The output of this model includes 120 min (the next 121time steps) of flow rates for three pumps (3 × 121 = 363 data points).
Models with the hyperparameters mentioned above were evaluated using performance metrics of RMSE and R
2 as presented in
Table 3. The optimal hyperparameters for the proposed model were achieved by evaluating the RMSE and R
2 scores. It can be seen from
Table 3 that increasing the LSTM layers to two or three did not improve the RMSE or R
2 scores. Thus, the number of LSTM layers was optimized to one. For the unit size of the LSTM layer, it was found that a unit number smaller than 50 affected the learning ability of the model. For 20 and 30 units, the recorded R
2 scores are 0.949 and 0.932, respectively, which are smaller than R
2 = 0.962 for 50 units. A higher number of units for the LSTM layer (i.e., 100) did not improve the model’s accuracy. Based on the evaluation results, 50 was determined the optimal value for the number of units of the LSTM layer. With an R
2 score of 0.962, the model with only one dense layer showed better performance than two dense layers which had an R
2 score of 0.952.
A summary of the optimized hyperparameters is reported in
Table 4. The optimal architecture comprises one LSTM layer with 50 units and a fully connected layer (or dense layer) at the top of the LSTM layer. The number of units for the dense layer is 363, the same as the output size. The epoch and the batch size selected for this model were 300 and 50, respectively. The proposed LSTM model has the best performance in the evaluation metrics: RMSE = 0.015 GPM and R
2 = 0.962.
4.2. Training and Testing with the Optimal LSTM Architecture
The model was trained with the optimal architecture introduced in
Table 4, and the trained model was then validated on the testing set.
Figure 10 shows the learning curves of the training and the testing sets for the optimal LSTM architecture. The MAE score, which was used as a loss function for both training and testing sets, decreases to the point of stability as the number of epochs increases. Additionally, there is a small gap between the training and testing loss curves. Meeting these two criteria indicates a good fit for the model.
The predictions of flow rates for pumps for the testing sets with the trained model were validated using the results of seepage analysis.
Figure 11a–c compares the predicted flow rates from the proposed LSTM model with the values from the seepage analysis for pumps 1, 2, and 3, respectively. The input for both the LSTM model and the seepage analysis is the rainfall data, and the output is the pump’s flow rate. In the seepage analysis, the pump flow rate was obtained based on the predefined pumping policy for each pump, while the proposed LSTM model could learn the hidden pumping policies for each pump within the training dataset.
Figure 11a–c supports the idea that the trained model can predict the required pumps’ flow rate for the testing set to keep the water table at the target level in response to the rainfall events. The proposed LSTM model can closely map the input data (i.e., set of rainfall data) to the expected output data (i.e., corresponding pumps’ flow rate) without requiring conducting a seepage analysis with prescribed pumping policies. An evaluation of the predictions, which is presented in
Table 5, also confirmed this finding.
From the proposed model, predictions regarding the flow rate of pumps were evaluated using the performance metrics of R2, RMSE, and MAE. Regardless of the output’s unit, values of R2 close to 1 represent high prediction accuracy. The R2 score of the flow rate predictions for pumps 1, 2, and 3 are 0.958, 0.962, and 0.954, respectively. The achieved high values of the R2 score indicate how well the proposed LSTM model fits the observed data from the seepage analysis.
Unlike the R2, RMSE and MAE carry the same unit as the predicted values for the pump flow rates in this study. The RMSE is the square root of the average squared differences between the predicted values from the proposed LSTM model and the actual values from the seepage analysis. The RMSE of the flow rate predictions for pumps 1, 2, and 3 are 0.007, 0.021, and 0.014, respectively. The RMSE values close to 0 represent a better prediction. Also, taking the range of a pump’s flow rate into consideration can help interpret the RMSE values. The range of observed flow rates of pumps 1, 2, and 3 from the seepage analysis in the testing set is 0–0.24 GPM, 0–0.65 GPM, and 0–0.37 GPM, respectively. Thus, the RMSE of the predicted flow rates for the three pumps is relatively small compared with the range of changes in the flow rates of the pumps.
The MAE is the average of differences between the predicted values from the proposed LSTM model and the actual values from the seepage analysis. The MAE of the flow rate predictions for pumps 1, 2, and 3 are 0.003, 0.009, and 0.006, respectively. Values of MAE are very small, which means the model can predict the pump’s flow rate with high accuracy. The evaluation of the predictions using various measurements verifies the promising performance of the proposed LSTM model in learning the pumping policy and predicting a pump’s flow rate.
4.3. Discussion 1: Application of the LSTM Model in Controlling the Groundwater
This subsection discusses different approaches for employing the proposed LSTM model in controlling groundwater level as an intervention method for the geotechnical CPSs studied in this paper. For this purpose, the predicted flow rates from the LSTM model were applied to the pumps in the slope to control the groundwater table. A seepage analysis was then conducted to obtain the groundwater levels at the three points of P1, P2, and P3 during the rainfall events.
The pump flow rates for the 12 events in the testing set can be continuously forecasted using the LSTM model trained with 36 events. In the method discussed in previous sections, any error present in the initial prediction will be carried forward to subsequent predictions. While this method is flexible and easy to implement, the accuracy of predictions will be reduced in later predictions. To reduce accumulated errors, pump flow rates for each rainfall event can be predicted independently. In this method, a separate LSTM model with the optimal architecture, introduced in
Section 4.1, was created to forecast the pump flow rate for each event. After forecasting the pump flow rate for one event, the actual data for that event was added to the training set to reduce the accumulated error for the flow rate predictions of the next rainfall event. By moving forward, the size of the training set increased, which improved the model’s performance on the next event’s predictions.
There are two options for applying the predicted pump flow rate in the seepage analysis to control the groundwater. One option is to reset the groundwater table to the initial condition at the beginning of each rainfall event. While this option can help evaluate the pump flow rate for each rainfall event separately, it cannot represent the real-world application. Alternatively, the groundwater level reset could be omitted during the 12 rainfall events to simulate reality.
Based on the above descriptions, three approaches were investigated for employing the proposed LSTM model by combining the available options for predicting the flow rates of pumps and applying them in the seepage analysis to control the groundwater level. In the first approach, pump flow rates for 12 rainfall events in the testing set were continuously predicted using the LSTM model trained with 36 events. Then, the groundwater table was reset to the initial condition at the beginning of each event to apply the pump flow rate in the seepage analysis. In the second approach, pump flow rates were predicted similar to the first approach. Then, the predicted flow rates were continuously implemented in the seepage analysis without resetting the groundwater condition at the beginning of each event. In the third approach, the training set for the LSTM model was recalibrated after forecasting the pump flow rate for each rainfall event to reduce the accumulated error in predictions. Then, the predicted values were continuously applied to the pumps during the rainfall events to control groundwater level.
Figure 12a–c displays the groundwater level from the three approaches at points of P
1, P
2, and P
3, respectively. To evaluate the application of the LSTM model, the groundwater levels from the three approaches were compared with the groundwater levels obtained with the prescribed pumping policies corresponding to the rainfall events at the points mentioned above.
A comparison of the groundwater levels using the prescribed pumping policies and approach 1 for employing the LSTM model shows that approach 1 has a discontinuity in control of the water table between the rainfall events. Despite positive results during the rainfall events at the three points of P1, P2, and P3, this approach cannot represent real conditions since the groundwater level was reset to the initial condition at the beginning of each event. An advantage of this approach is that the predictions for each event can be independently evaluated from the other events.
Another comparison of the groundwater levels using the prescribed pumping policies and approach 2 for employing the LSTM model provides insight into the accumulated error in the predictions. While this approach is easy to implement, the accuracy of results will be reduced in time. To improve the accuracy of results, approach 3 was suggested for employing the LSTM model. The results for approach 3 demonstrate this approach can boost the performance of the LSTM model in controlling the groundwater level at each of the three points (P1, P2, and P3).
To better evaluate the proposed three approaches for employing the LSTM model, the differences between the groundwater level from each approach and the prescribed pumping policies are shown in
Figure 13. The groundwater level using the prescribed pumping policies performs as a benchmark for evaluation.
Figure 13a–c displays the differences at points P
1, P
2, and P
3, respectively.
The maximum difference from the benchmark for approach 1 at points of P1, P2, and P3 are 13.3%, 16.0%, and 17.0%, respectively. This approach shows a difference of less than 10% in the groundwater level for most of the rainfall events. The average difference of the results for approach 1 from the benchmark is approximately 4.6% at all of the three points.
Approach 2 shows greater differences compared with the other two approaches. The differences from the benchmark for approach 2 at points P1, P2, and P3 increase up to 34.4%, 37.5%, and 40.7%, respectively. The average difference of the results for this approach from the benchmark is approximately 12.7% at all of the three points. The results for approach 3 demonstrate noticeably lower values of differences from the benchmark. The average difference of the results for this approach from the benchmark is approximately 3.5% at all of the three points.
The above discussions conclude the LSTM model has an indisputable impact on controlling the groundwater table in the slope. The evaluation of the groundwater level from different approaches of applying the predictions indicates that the model’s performance may decrease during long-sequential rainfall events. Thus, it is worthwhile to evaluate the predicted pump flow rates in controlling the groundwater table, in addition to the model evaluation metrics such as R2, RMSE, and MAE.
4.4. Discussion 2: Influence of Rainfall Patterns
This subsection is intended to evaluate the generalizability of the proposed LSTM model. A generalized model means the model is capable of performing well on an unseen dataset. Since a small set of data with limited typical rainfall patterns was used in this study to train the LSTM model, it is beneficial to prove the model can be applied to real-world rainfall events with more complex patterns. Thus, the influence of the rainfall patterns on the accuracy of the model predictions was investigated. For this purpose, two rainfall datasets with different numbers of patterns were considered, as shown in
Figure 14a,b. Both datasets include 24 rainfall events with the same variation in rainfall duration and depth. However, dataset 2 comprises more rainfall patterns types than dataset 1. The rainfall events in dataset 1 consist of only two types of patterns (i.e., types “a” and “d”), while the rainfall events in dataset 2 consist of four types of patterns (i.e., types “a”, “b”, “c”, and “d”).
Figure 15a,b show the pumps’ flow rates for pumps 1, 2, and 3 corresponding to the rainfall data generated in
Figure 14a,b, respectively.
An LSTM model was trained with the first 12 rainfall events and the corresponding pump flow rates for both datasets. Then, the model was validated with the next 12 rainfall events and their corresponding pump flow rates. The rainfall events in the testing set have the same type of pattern (“d”) in datasets 1 and 2.
Table 6 presents the evaluation results of the model trained with datasets 1 and 2. The evaluation metrics of RMSE and MAE for dataset 2 are lower than the values for dataset 1 in all three pumps. The R
2 score can better demonstrate the difference between the accuracy of the pump flow rate predictions for dataset 1 and dataset 2. The R
2 values for pumps 1, 2, and 3 in dataset 1 were 0.891, 0.922, and 0.910, respectively. By increasing the number of rainfall patterns for training the LSTM model from 1 in dataset 1 to 3 in dataset 2, the R
2 values for pumps 1, 2, and 3 were improved to 0.913, 0.935, and 0.925.
Additionally, comparing the comparison of the evaluation metrics for dataset 2 in
Table 6 with the corresponding values in
Table 5, it was found that the model trained with 36 rainfall events including the same types of patterns as dataset 2 has a better performance in predicting the pump’s flow rate. This means that the model trained with a dataset consisting of a larger number of rainfall events and pattern types improves the model’s performance in the prediction of the pump flow rates for a given new type of pattern.