1. Introduction
Cloud infrastructure is the aggregation of hardware and software resources that make up cloud computing, which forms the global data centers of different service providers. In light of Gartner’s latest report, global end-user spending on public cloud services is expected to grow by 20.4% to 678.8 billion by 2024, up from 563.6 billion in 2023 [
1]. Moreover, most emerging techniques, e.g., remote sensing, Internet of Things, fog and edge computing, and cyber–physical systems, emphatically rely on cloud computing because storage and computing capabilities for these systems are insufficient [
2,
3,
4,
5,
6]. While users’ demand for these software systems is growing daily, concerns about the availability, reliability, and Quality of Services (QoS) of cloud infrastructure are also rapidly increasing. However, the dramatic variations in workloads tend to trigger the over- or under-provisioning of resources, which drives the need for cloud service providers (CSPs) to agilely formulate strategies to allocate the resources, preventing the violation of service-level agreements (SLAs). Therefore, to further improve energy consumption costs and resource usage efficiency, workload predictions are highly needed for dynamic resource provisioning [
7,
8,
9].
In the cloud computing paradigm, resource-provisioning methods mainly include the following two classes: reactive and proactive methods [
10]. Reactive scaling methods allocate the resources after demands arrive, but this is not advantageous due to non-instantaneous initialization and dynamic migration for hosted virtual machines. This will easily lead to problems, including the over-provisioning and under-provisioning of computing resources and will ultimately cause resource waste, increased power consumption, and SLA violations [
11]. While proactive scaling techniques allow for the scaling of computing resources before the arrival of practical demands, such methods need to know the upcoming workload information in advance. To pursue system availability and effective resource usage, resource demands in the near future can be predicted by the analysis of historical usage and request patterns and the current state of the cloud, which can use predicted resource demands to assist in assorting the requisite physical resources in a proactive manner [
12]. This effectively reduces the number of active servers by allowing only the required number of physical machines to be active, which can promote resource utilization, reliability, and service availability. In addition, the intensity of the workload will also affect the aging problem of the software itself [
13,
14] and play a crucial role in the formulation of software rejuvenation strategies [
15]. Nevertheless, the heterogeneous nature of workloads in cloud infrastructure (i.e., end-users submit different types of application requests, which will require heterogeneous resource capacities), high burstiness (i.e., local irregular fluctuations), and long-term dynamic variations (i.e., long-running characteristics of cloud systems), will make accurate workload prediction an intractable task.
A large number of methods have been developed to apply to the workload prediction tasks in the cloud, e.g., the autoregressive integrated moving average model (ARIMA) [
11,
16], the multi-layer perceptron (MLP) [
17,
18], the convolutional neural network (CNN) [
19,
20], the recurrent neural network (RNN) and its variants [
21,
22,
23,
24,
25], and others. However, workload prediction tasks are intractable since the workload time series dynamically changes in the face of various user demands. Traditional statistical analysis methods, e.g., linear regression, exponential smoothing, the autoregressive moving average model (ARMA) [
26], and ARIMA [
11,
16], are easy to understand and have been widely adopted for cloud workload prediction scenarios. However, these methods often assume a linear relationship among the changes in workload time series data. Therefore, nonlinear variations and long-term temporal correlation may not be fully modeled under the linear function constraints [
16].
Moreover, traditional machine learning methods, such as support vector regression (SVR) [
27] and MLP [
17,
18], have also been widely adopted in cloud workload prediction fields. SVR and MLP can capture the mapping relationships of input and output by using nonlinear transformation. However, these models may sometimes encounter challenges during the training process, as they can become trapped in local optimal solutions and experience gradient problems. This will inescapably cause prediction performance degradation for these traditional machine learning methods. Currently, some deep learning methods, because of their powerful ability in representation learning, have also been employed in workload modeling tasks. These models, such as RNN [
21] and its variants, including long short-term memory (LSTM) [
25] and gate recurrent units (GRUs) [
19], CNN [
20], bidirectional LSTM (BiLSTM) [
22] and their hybrid models [
9,
20,
28,
29,
30], have been frequently used in this fields. In these models, CNNs are used to capture spatial or local relationships in workload time series data [
20]. RNN and its variants are applied to model temporal dependencies in workload data [
19,
25]. However, CNN-based methods generally possess limited ability to model long-term and temporal dependencies due to the receptive field limited by the size of the convolution kernel [
31,
32]. Moreover, RNN and its variants (e.g., LSTM) are confronted with the degradation problem of computational efficiency, since they do not have the ability to execute parallel computing. In addition, they are confronted with the problems of the gradient vanishing and exploding when modeling long-range temporal relationships [
33,
34]. In a real-world cloud environment, the majority of workloads show short-term irregular fluctuation phenomena and long-term dynamic variations in cloud workload traces [
35]. The sudden spikes in the incoming workload represent unprecedented variations in heterogeneous request patterns of different users over time. These problems will cause the loss of useful information during the feature-extraction stage. Ultimately, they may struggle to extract these features’ information effectively [
33]. Therefore, the designed model should possess comprehensive extraction abilities for both local irregular fluctuations and long-term dynamic variation characteristics to provide high prediction accuracy.
For this purpose, this paper presents a new framework, called DuCFF, to handle the issues mentioned above in workload-prediction tasks. DuCFF integrates the temporal convolution network (TCN) channel and transformer channel in parallel to achieve the feature extraction capacities of local irregular fluctuations and long-term dynamic variations. TCN can weaken the influence of the size of the convolution kernel and effectively capture intrinsic local information [
31], which is broadly adopted in natural language processing (NLP) [
31], time series prediction, and classification [
36,
37], and many others [
38]. TCN compensates for the weaknesses of CNN architecture by using hierarchically structured temporal convolutional filters with bigger receptive fields and fewer parameters. The transformer network [
34] is an innovative architecture that has parallel computing and long-term dependency modeling capabilities. It only designs and adopts the self-attention mechanism to replace recurrent or convolutional layers in sequence-to-sequence prediction tasks. Currently, the transformer architecture has been broadly employed in numerous fields, including NLP [
34], time series prediction [
39], computer vision, pattern recognition [
40,
41]. In the workload-prediction tasks studied in this article, its ability to process any part of the workload time series data without distance limits enables it to better capture the coupled long-range temporal relationships.
In particular, DuCFF first adopts the commonly used variational mode decomposition (VMD) [
9] method to decouple the different implicit characteristics of the workload time series, which are further adopted as auxiliary information and combined with the original workload as the model input. Then, DuCFF employs stacked multi-scale TCN blocks to individually extract local irregular fluctuation features from the original input. In addition, adopting a parallel architecture, the proposed model adopts stacked transformer blocks to extract long-term dynamic variation characteristics. By adopting this divide-and-conquer strategy, the robustness of the model can be enhanced by effectively reducing the loss of vital information while maintaining the integrity of the information extraction. Afterward, DuCFF fuses the high-level features extracted from the above two channels, and these fused features are further input to two fully connected layers to accomplish workload prediction for the next time slot. The experimental analysis has been performed to analyze the efficiency of the proposed DuCFF, achieving improvements of 65.2%, 70%, 64.37%, and 15% on average in prediction performance among four evaluation metrics, including Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE) and R-squared (
), compared with CNN-LSTM. This study makes the following contributions:
A workload-prediction framework named DuCFF is presented to fully extract the characteristics of local irregular fluctuation and long-term dynamic variations implied in the original workload time series data of the cloud.
The TCN channel is designed to capture local irregular fluctuation information from raw input. In parallel, the transformer channel is adopted to model long-term dynamic variations features, which can promote workload prediction accuracy.
The prediction performance of the proposed DuCFF is confirmed by carrying out extensive experimental evaluation among three real-world benchmark datasets. The numerical results verify that DuCFF is superior to its nine competitors.
The rest of the paper is organized as follows.
Section 2 introduces related research on workload prediction.
Section 3 describes the details of the DuCFF framework, and
Section 4 depicts the numerical results of the models through extensive experiments. Finally,
Section 5 summarizes the conclusion and future work plan.
2. Related Work
Workload prediction can predict upcoming resource utilization and workload demands to help the cloud data center run flawlessly. To date, many methods have been developed to predict the future evolution of workloads in cloud infrastructure. By sorting out the existing workload prediction methods, this article intends to divide them into the following three dimensions: statistical analysis-based, machine learning-based, and hybrid models in workload prediction.
- A.
Statistical analysis models
Workload prediction using statistical analysis models has a long history. Traditional statistical analysis models usually have extremely high computational efficiency and relatively simple model structures [
16]. The modeling idea of this type of method is to use the known historical data and current data in the workload time series as input samples to estimate various parameters that need to be configured in the model. After the parameters are determined, the constructed model can fit the future evolution trend of the workload data to a certain extent, thereby realizing the process of trend extrapolation. For instance, Calheiros et al. [
16] applied ARIMA to estimate workload based on the historical real traces of requests to web servers and further evaluated the influence of the achieved accuracy in terms of efficiency in resource utilization and QoS. Zharikov et al. [
42] constructed six alternative prediction models, Simple Exponential Smoothing (SES), Holt’s Linear Trend, Holt’s Damped Trend, ARIMA, Linear Regression with Trend (LR), and TBATS, to determine the best combination that generates the most accurate prediction in a cloud data center. Farahnakian et al. [
43] employed the LR model to predict the short-term future CPU usage based on the usage history in each host, which can determine the over-loaded and under-loaded hosts by prediction in the live migration process. Debusschere and Bacha [
44] adopted seasonal ARIMA (SARIMA) to estimate the future 168 h server workload, which allows the ARIMA model to directly address the seasonal patterns.
Nevertheless, due to the complex nature of workload time series, statistical models cannot accurately give reasonable prior explanations. Therefore, the setting of premise assumptions restricts the application of statistical models in workload prediction to a certain extent, making it impossible to achieve better predictions.
- B.
Machine learning models
Compared with traditional statistical models, machine learning models have more powerful nonlinear modeling capabilities [
19,
25]. They can directly approximate any mapping function through the training and learning of historical data and can effectively model nonlinear and non-stationary workload time series. For example, Moghaddam et al. [
17] adopted multiple prediction models, such as LR, MLP, SVR, decision tree regression, and boosted decision tree regression, to predict over-utilized hosts in cloud data centers. The virtual machines were then migrated based on the prediction results, virtual machine migration time, and other constraints to achieve energy-aware VM consolidation. Kumar et al. [
18] developed a neural-network-based workload-prediction framework and used an improved and adaptive differential evolution algorithm to improve the learning efficiency of the proposed model; the experimental results confirmed the superiority of this method on four real-world data traces in comparison with the average, back-propagation, and self-adaptive differential evolution (SaDE) algorithms. Duggan et al. [
21] considered the fact that cloud resource consumption is constantly changing and constructed a workload prediction model based on the RNN model. Compared with traditional prediction methods, i.e., back-propagation neural network, random walk, and moving average, RNN obtained better prediction accuracy. Recently, Ruan et al. [
25] developed a storage workload prediction method called CrystalLP based on the LSTM model and implemented the sensitivity analysis of the hyperparameters in LSTM. Compared with baseline methods, i.e., ARIMA, SVR, and RNN, CrystalLP achieved improvements in prediction performance. Xu et al. [
19] used the GRU model (named esDNN) to perform workload prediction and used a rolling window method to convert the workload data into a supervised learning time series. Compared with BiLSTM, esDNN achieved better prediction results. Chen et al. [
45] constructed a resource usage prediction method (named RPTCN), which is based on temporal convolutional networks (TCNs) in cloud systems, and further combined the fully connected layer and attention mechanism with TCNs to improve the prediction accuracy. The experimental results confirmed that the prediction performance of RPTCN is significantly better than LSTM. In addition, Selvan Chenni Chetty et al. [
46] proposed a hierarchical tree-based deep convolutional neural network (T-CNN) model with sheep flock optimization (SFO) to enhance CDC’s power efficiency and workload prediction. Also, the proposed model was evaluated using Saskatchewan HTTP traces and NASA benchmark datasets, showing promising predictive accuracy.
The above works verify that the prediction performance of machine learning models is superior to traditional statistical analysis models. However, considering that workload data often have multiple overlapping characteristics, when faced with different scenarios, these models cannot model different characteristics of workload time series simultaneously, resulting in prediction accuracy still having room for improvement to a certain extent. Our work differs from these works, and we use multiple techniques to capture different features of the workload time series to achieve complete feature extraction and prediction.
- C.
Hybrid models
The hybrid model can alleviate the problems of insufficient prediction ability and unstable prediction results for a single model by integrating a variety of different technologies, thereby effectively achieving workload prediction [
11]. Currently, a variety of hybrid models have been developed by researchers. For instance, Xie et al. [
47] constructed a hybrid model by combining the ARIMA and Triple Exponential Smoothing (TES) to model both the linear and nonlinear correlations in the docker container workload time series in a cloud platform and confirmed that this hybrid model can achieve the performance promotion. Devi and Valli [
48] considered the linear and nonlinear components hidden in the resource utilization patterns, and they adopted the hybrid model ARIMA-ANN to perform the prediction in the cloud environment. Experimental results indicated that the proposed ARIMA-ANN is better than the single models ARIMA and ANN. For the high-dimensional and highly variable cloud workload data, Chen et al. [
49] integrated the top-sparse auto-encoder (TSA) and gated recurrent unit (GRU) with the TSA block (named L-PAW) to address the high-dimensional and highly variable features, respectively, and extensive experiments demonstrated the effectiveness and adaptability of the L-PAW. In recent years, hybrid models based on CNN, LSTM, and their variants have also received widespread attention. For example, Patel and Kushwaha [
20] adopted three parallel dilated 1D CNN layers to extract the noise and used the LSTM layer to learn the temporal dependencies from the host CPU usage, and performance comparisons verified the superiority of the proposed model compared to several competitors. In addition, Patel and Kushwaha [
28] further integrated parallel and stacked 1-Dimensional CNN (1DCNN) and LSTM networks (named RCP-CL) to model random fluctuations and novel continuous and periodic patterns from contiguous and non-contiguous CPU load values augmented with daily and weekly time patterns, and their experimental results confirmed that the proposed RCP-CL performed better than LSTM and 1D-CNN. Bi et al. [
22] integrated both bi-directional and grid long short-term memory networks to perform the prediction of workload and resource time series, and they adopted the data preprocessing operations, including Savitzky–Golay (SG) filter and Min–Max scaler, to remove the impact of noise and extreme points before model training. Experimental analysis verified the model’s advantage compared to its competitors. Chen et al. [
6] adopted multiple technologies, including the Savitzky–Golay filter (SG filter), CNN, and BiLSTM with an attention mechanism, to construct the hybrid model SG-CBA, which is used to predict workload in edge data centers, and achieved the better performance relative to baseline models, e.g., RNN, GRU, and BiLSTM. Zhang et al. [
50] proposed a hybrid model combining the TES method and LSTM network to perform a Docker container workload proactive prediction, which can not only capture both short-term and long-term dependencies in container resource time series but also smooth the container resource utilization data. For complex and non-linear workload and resources time series, Bi et al. [
9] combined variational mode decomposition (VMD), an SG filter, a multi-head attention mechanism, and bidirectional and grid versions of LSTM networks to achieve the accurate prediction, and extensive experiments confirmed that the proposed model outperformed several advanced prediction models in terms of prediction accuracy. We further illustrate the differences between existing studies and this study based on four dimensions, i.e., proposed models, predicted parameters, datasets source, and metrics used, in
Table 1.
Unlike these works, which combine CNN and LSTM and their variants, and the class of hybrid models, which combine ARIMA and machine learning methods to perform cloud workload predictions, in this work, we use TCNs and Transformer networks to construct a new workload-prediction framework, DuCFF, which can model the local irregular fluctuation and long-term dynamic variations features in parallel in the workload time series. This framework is committed to further improving the workload prediction accuracy while making up for the weaknesses of traditional CNN and LSTM.
4. Experiments and Results
In this section, we perform extensive experiments using real-world workload trace datasets to evaluate the prediction performance of the proposed DuCFF. Specifically, this section consists of the introduction of trace datasets and evaluation metrics, baselines and hyperparameters settings, prediction performance analysis and comparison, parameter sensitivity analysis, and performance overhead analysis.
4.1. Experimental Data Description and Evaluation Metrics
We collect and collate three workload datasets from two real-workload traces (i.e., ClarkNet–HTTP and Google Cloud trace) that include the different application categories.
ClarkNet-HTTP trace: This trace contains all two weeks’ HTTP requests to the ClarkNet WWW server. The details of the workload data consist of the host, timestamp, request, HTTP reply code, and bytes in the reply [
53]. Originally, the requested workload time series was not intuitively structured in the desired time series with regular and uniform timestamps. For experimental purposes, this paper aggregated the size of requests every five minutes to build up a univariate time series that contains 4009 samples, abbreviated as
CNH.
Google Cloud trace: These cluster trace data include a total of 1600 log files and recorded different virtual machine operating data. Each file is recorded every 5 min by the virtual machine monitor, resulting in 288 data points per day. Each group of data consists of two columns: CPU and memory utilization. The data containing only machine events are already accessible on GitHub (
https://github.com/HiPro-IT/CPU-and-Memory-resource-usage-from-Google-Cluster-Data, accessed on 11 January 2024). This paper selects the CPU utilization data of two groups of VMs for ten consecutive days, with 2880 samples, respectively, abbreviated as
GC1 and
GC2.
In conclusion, this paper collects three workload datasets to evaluate the performance and generalization ability of the model, as depicted in
Figure 6. These three workloads are selected since they have special characteristics and dynamics in the workload patterns. For instance, workload, e.g., ClarkNet–HTTP trace, shows dynamic spikes and high fluctuations, and workloads, e.g., Google trace, show seasonality and high fluctuation. In the implemented experiments, 70% of the data are adopted as the training set, while the remaining 30% are used as the testing set.
Evaluation metrics: we adopt four performance evaluation metrics, including MAE, RMSE, MAPE, and
, to measure whether the model is promising, and the computations are expressed as shown below:
where
indicates the prediction value and
is the corresponding ground truth. For the MAE, MAPE, and RMSE metrics, the lower the value, the better the model performance. In contrast, for the metric
, the larger the value, the better the model performance, with an upper limit value of 1.
4.2. Baselines and Parameters Settings
Some existing models possessing state-of-the-art prediction capacities are constructed and compared with the proposed DuCFF. The details of these models are described as below:
SVR [
27]: This is a traditional supervised machine learning model that aims to find a regression plane, making the shortest distance across all data in a collection. We implement the algorithm by calling the Scikit-learn library and configure key parameters, such as the radial basis function (RBF) kernel, ‘
epsilon’, and ‘
C’, based on the prediction accuracy.
MLP [
17]: This model can capture the non-linear relationships in workload data with better accuracy, amd it is also named artificial neural network (ANN). We implement this model using a traditional three-layer network structure.
LSTM [
25] and
GRU [
19]: These models can capture the long-term temporal dependency, and they are variants of classical RNN. We implement these two models using a single LSTM/GRU layer structure.
1DCNN [
28]: This model can capture the local relationships hidden in workload time series by adopting the one-dimensional convolution operation. We implement this model using a single convolution layer.
BiLSTM [
22]: This is the combination of forward LSTM and backward LSTM [
6], which can learn the bidirectional long-range context dependencies in workload time series. We implement this model in the form of a single BiLSTM layer.
BiGRU–Attention [
19]: This model is the combination of BiGRU and attention mechanism, which can further capture long-term dependencies in the workload sequence and weaken the interference of unimportant elements on the prediction. We implement this model using a single BiGRU layer and attention layer.
1DCNN–LSTM [
28]: A workload prediction model aims to model the random fluctuations and novel continuous and periodic patterns from contiguous and non-contiguous CPU load values augmented with daily and weekly time patterns, based on CNN and LSTM. We implement this model using a single 1DCNN layer and LSTM layer.
1DCNN–BiLSTM–Attention [
6]: A workload prediction model that integrates the 1DCNN, BiLSTM, and attention mechanism to achieve feature extraction and prediction. We implement the model in the form of a one-step-ahead prediction.
To proceed, the selection of parameters in the constructed model for workload prediction plays a crucial role. The parameters involved in the proposed DuCFF that have a significant impact on the prediction performance include time lag, epoch, number of heads, kernel size, and number of filters. We adopt the grid search strategy to determine the model parameters one by one. Specifically, for all workload datasets, we define the search range [6, 12, 18, 24, 30] for time lag, [50, 80, 100, 150, 200] for epoch, [2, 3, 4, 5, 6] for the Head, [1, 3, 5] for kernel size, [16, 32, 64, 128] for the number of filters, and for hidden dimensions. By performing the parameter search, we select the parameters that achieved satisfactory prediction performance to equip the proposed DuCFF framework and detailed parameter settings are given in
Table 2. Meanwhile, for all baseline models, the search strategy of parameters is consistent with the proposed model. After the parameters of all models (including baseline models) are determined, considering the randomness of the initial weight assignment of the deep neural network, each experiment was performed 10 times repeatedly, and this paper records and reports the average results.
4.3. Prediction Performance Analysis and Comparison
This section investigates the prediction performance of DuCFF and baseline models by performing extensive experiment evaluations. The following observations can be obtained.
We first report the numerical prediction results for each model in three workload datasets, as depicted in
Table 3. It is easy to observe that the proposed DuCFF is able to reach the best predictive performance, and in all cases (12/12), the best performance gains are achieved relative to the baseline model. This is mainly due to DuCFF’s powerful feature extraction capability. After VMD preprocessing, these complex features contained in the workload time series can easily be captured by a well-designed network structure so as to achieve effective prediction. It is also not difficult to find, for all the baseline models, that performing workload prediction is intricate work, and they cannot obtain low prediction errors and better fitting capabilities. Specifically, these baseline models cannot obtain consistent prediction performance across different workload datasets. For example, in the comparisons of baseline models, SVR achieves the best prediction performance in three cases (3/12), including the MAPE value (18.38) of
CNH and the MAE value (1.10) and MAPE value (3.09) of
GC2. MLP only obtains the optimal prediction performance in one case (1/12), i.e., a MAE value (1.11) of
GC2. GRU obtains the superior prediction performance in three cases (3/12), i.e., RMSE, MAPE, and
values (1.43/2.48/0.9523) of
GC1. 1DCNN obtains the best performance in two cases, i.e., RMSE and
values (1.92/0.9540) of
GC2. 1DCNN–LSTM can achieve the optimal prediction in three cases, i.e., MAE, RMSE and
values (1214999/1736089/0.7293) of
CNH.
However, the prediction performance of these baseline models on remaining cases is unsatisfactory. Moreover, we can also observe that although 1DCNN–BiLSTM–Attention has the capacity to capture local and long-term dynamic variations features, it still cannot achieve effective prediction due to its serial architecture. This is mainly because the integrity of information may not be ensured, causing the loss of helpful information during the feature extraction process. Furthermore, LSTM, BiLSTM, and BiGRU–Attention perform worse than the GRU model in most cases, which illustrates that the complexity of the model structure does not necessarily lead to significant performance gains. Relative to these baseline models, the constructed DuCFF adopts a dual channel with different scales to capture representative features that are beneficial to better predictions, hidden in workload data, thereby achieving remarkable prediction results.
Secondly, we further calculate the average improvement proportion of DuCFF relative to baseline models, to more intuitively verify the prediction ability of DuCFF and its generalization ability on different workload time series, as shown in
Table 4. We can observe that DuCFF obtains a significant performance boost in terms of four metrics. In this regard, relative to CNN-LSTM, the four evaluation metrics (i.e., MAE, RMSE, MAPE, and
) for DuCFF can be improved by 65.20%, 70.00%, 64.37%, and 15.00%, respectively; compared with GRU model, DuCFF obtains performance promotion of 64.63%, 69.65%, 64.14%, and 15.09%, respectively. It is interesting that for the hybrid model 1DCNN–BiLSTM–Attention, its average prediction performance is still inferior to the proposed DuCFF and is even worse than that of GRU and CNN-LSTM. As emphasized earlier, the serial structure of 1DCNN–BiLSTM–Attention extracts the features in a layer-by-layer manner, which will easily lead to the loss of helpful information. In summary, the elaborate network structure of DuCFF can effectively avoid information loss, which contributes to extracting the local and long-term variation features from the workload data.
Finally, the fitting effects of all models (i.e., DuCFF and baseline models) on the
GC2 dataset are selected as an example for visualization due to page limitations, as depicted in
Figure 7. We can observe that the DuCFF model can always accurately model the future variations trend when faced with highly fluctuating workload data, showing excellent fitting ability with low fitting error, and this conclusion can be confirmed by the fitting effect of some mutation points in
Figure 7. However, for the baseline models, local irregular fluctuations make all baseline models go astray and be unable to achieve effective fitting with high fitting errors, which is also the root cause of large model prediction errors. The baseline models cannot weaken and filter the influence of these irregular variations on model learning, and excessive attention to these irregular data points will greatly reduce the prediction ability of the model. The above fine-grained comparisons from numerical results, average improvement proportion, and fitting effect confirm the advantages of the proposed model, and from the experimental results, adequate feature extraction is helpful in achieving effective workload prediction.
4.4. Parameter Sensitivity Analysis
This section discusses how parameter combinations influence the prediction performance of the proposed model to explain the rationality of model parameter selections. For this purpose, the sensitivity analysis on the
CNH dataset as an example is carried out, and we change each parameter in a reasonable range. The parameters involved in this section consist of the number of IMFs, the time lag, the epoch, the number of filters, and the number of Heads since they have a significant impact on model performance. The numerical results are depicted in
Table 5 and
Figure 8, where small a MAPE and large
indicate that the model has better prediction performance. Note that we only exhibit the MAPE and
results because the remaining two metrics have similar changes and page limits. From
Table 5 and
Figure 8, we can conclude with the following findings:
For parameter n: In a certain range, the model performance can be improved to some extent by increasing the number of decomposed subcomponents (IMFs). This is mainly due to more decomposition components that can extract different variation characteristics contained in the raw workload data at a finer granularity. However, more decomposed components as the input also increase the complexity of the model. Therefore, this paper chooses to use n = 9 as a candidate parameter because it can also bring considerable performance improvement relative to the baseline model.
For the parameter time lag: Different input lengths (time lag) have different effects on model performance. In the prediction task of this paper, increasing the length of time lag deteriorates the model performance, while model performance achieves optimal results after using a specific time lag. For example, when the time lag is equal to 30, the model performance is worse than that when the time lag is 24. In general, the historical workload data often contain irregular changes, which make the model learn ineffectively. The noisy information hidden in workload data cannot be fully captured, so the model performance is affected.
For the parameter epoch: Increasing the number of epochs does not mean improving model performance. This is because the model has a performance bottleneck. After a certain number of epochs, the model performance will be saturated, and epochs that are too large will also increase the model training overhead. Therefore, the epoch is set to 200 in this paper.
For the parameter filter: Due to the parallel multi-scale architecture, the proposed model can achieve satisfactory prediction ability with a small filter number. However, increasing the number of TCN filters cannot obtain the ideal performance improvement, and this will further lead to the increase in model parameters, easily causing the overfitting phenomenon. Therefore, the number of filters is set to 32 in this paper.
For the parameter heads: The different number of heads in the Transformer block has a slight impact on model performance. In this paper, heads = 5 is selected as the model parameter as it can reduce the model prediction error.
4.5. Performance Overhead Analysis
This section investigates the performance overhead of each model. We first report the runtime of the training and testing stages for each model on three workload datasets and further analyze the trainable parameters of models. The detailed results are reported in
Figure 9 and
Table 6, and we have the following observations.
On the identical training set, many models involved in this paper spend much time training offline, except for the SVR model (with less time). Specifically, the CNN-LSTM, BiLSTM, and GRU models take an average of 95.3 s, 103.3 s, and 64.5 s, respectively, to train the models. However, the proposed DuCFF takes an average of 154.5 s to train on the three workload datasets, which is higher than the baseline models. It should be noted that the runtime of model training is not only related to the model network architecture but also closely related to the setting of hyperparameters, such as the lengths of time lag and epoch. This is mainly due to the fact that the greater the time lag, the more historical information the model needs to learn, which will cause the higher computational complexity of the model; furthermore, the larger the epoch, the longer the model training time. Moreover, in the evaluation phase, the online testing time for each model is minimal and acceptable, all below 0.2 s. Specifically, the average runtime of the test stage for DuCFF on the three workload datasets is 0.18 s, which presents a small gap compared with baseline models.
Table 6 summarizes the trainable parameter size for all models on three workload datasets, which is exported by the summary() method in the Keras environment. As observed in the numerical results, we can note that DuCFF’s trainable parameters are less than 1DCNN–BiLSTM–Attention, BiLSTM, and BiGRU–Attention in all datasets, but its performance is significantly better than these baseline models. In fact, this benefit comes from the rationality of the design of the DuCFF architecture, using parallel and multi-scale manners to reduce model complexity while achieving efficient information extraction and better prediction.
4.6. Discussion
In summary, from the above horizontal and vertical comparison results obtained in this work, the superiority of the proposed DuCFF can be explained as follows. On the one hand, the prediction performance for some machine learning models (e.g., SVR, MLP) is insufficient for the workload data with high fluctuations and long-term dynamic variations, and they cannot effectively extract useful features to perform prediction. For example, the prediction performance of DuCFF can achieve 65.65%, 70.95%, 64.50%, and 16.11% improvements in terms of four metrics compared with SVR. On the other hand, some advanced models adopting cascaded network architectures cannot achieve consistent prediction performance due to the loss of useful information in the feature-extraction process, such as 1DCNN–LSTM and 1DCNN–BiLSTM–Attention. Finally, in the comparison of models, the DuCFF model is generally the optimal one. It verifies the rationality of the divide-and-conquer strategy adopted by the proposed architecture, which can more effectively capture the variation characteristics of the workload data for better prediction. It presents that the preprocessing process of the workload data using the popular VMD technique before further feature extraction can effectively improve the prediction ability in the case of workload prediction tasks. In addition, the analyses of parameter sensitivity and performance overhead also further confirm the rationality of the proposed DuCFF model in parameter selection and architecture design. The observed numerical results mean that the proposed DuCFF can satisfy the high-precision requirements for workload prediction problems and provide decision support for task scheduling strategies. However, since we only used three sets of data to evaluate the model performance, the diversity of the datasets is lacking, which is also a limitation in this study. Further meaningful research can be conducted in the future is by collecting more workload datasets to verify the generalization ability of the model and confirm the robustness of the model.
5. Conclusions and Future Work
This work constructs a novel workload prediction method, named DuCFF, by integrating data preprocessing technology, multi-scale TCNs, and the transformer. DuCFF employs stacked multi-scale TCN blocks to individually extract local irregular fluctuation features from the original input. Additionally, adopting a parallel architecture, the proposed model adopts stacked transformer blocks to extract long-term dynamic variation characteristics. By adopting this divide-and-conquer strategy, the loss of vital information could be lessened effectively to enhance the performance of the model on the premise of intact information. Extensive experiments are carried out on three datasets collected from ClarkNet–HTTP and Google Cloud traces to evaluate the validity of the proposed model. Specifically, DuCFF can achieve improvements of 65.2%, 70%, 64.37%, and 15% on average in prediction performance among four evaluation metrics, namely MAE, RMSE, MAPE, and , compared with CNN-LSTM. Furthermore, by performing the analysis of parameter sensitivity and time overhead, DuCFF shows consistently superior performance, which provides a new attempt at workload prediction research.
Nevertheless, the runtime cost of the proposed model is relatively high compared to the baseline models, as mentioned in the previous section. In addition, the generalization ability of the model needs to be further verified by using more diverse workload datasets. Future work may go on to collect workload data from different traces to verify model generalization and explore techniques to simplify the model complexity to reduce the runtime cost. We also plan to use a more practical dynamic decomposition strategy to improve model performance. Furthermore, we intend to apply some feature selection algorithms to mine the importance of different features for workload prediction.