An Efficient Multivariate Autoscaling Framework Using Bi-LSTM for Cloud Computing

Abstract

With the rapid development of 5G technology, the need for a flexible and scalable real-time system for data processing has become increasingly important. By predicting future resource workloads, cloud service providers can automatically provision and deprovision user resources for the system beforehand, to meet service level agreements. However, workload demands fluctuate continuously over time, which makes their prediction difficult. Hence, several studies have proposed a technique called time series forecasting to accurately predict the resource workload. However, most of these studies focused solely on univariate time series forecasting; in other words, they only analyzed the measurement of a single feature. This study proposes an efficient multivariate autoscaling framework using bidirectional long short-term memory (Bi-LSTM) for cloud computing. The system framework was designed based on the monitor–analyze–plan–execute loop. The results obtained from our experiments on different actual workload datasets indicated that the proposed multivariate Bi-LSTM exhibited a root-mean-squared error (RMSE) prediction error 1.84-times smaller than that of the univariate one. Furthermore, it reduced the RMSE prediction error by 6.7% and 5.4% when compared with the multivariate LSTM and convolutional neural network-long short-term memory (CNN-LSTM) models, respectively. Finally, in terms of resource provisioning, the multivariate Bi-LSTM autoscaler was 47.2% and 14.7% more efficient than the multivariate LSTM and CNN-LSTM autoscalers, respectively.

1. Introduction

Cloud computing is currently playing a significant role in improving the efficiency of enterprises, industries, and governments [1,2]. Virtualization technology, which allows cloud providers to execute programs on virtual machines (VMs), is an essential component of cloud computing [3]. By renting VMs, stakeholders can deploy their applications on the VMs using pay-per-use prices instead of purchasing expensive physical servers. Elasticity is a key property of cloud computing as it allows cloud providers to create and remove resources as needed, increasing performance, while lowering costs [4]. Autoscaling is an automatic process that dynamically adds and removes resources such as VMs and containers. Autoscaling can be categorized into two types: reactive and proactive. Reactive autoscaling refers to processes that scale resources based on predefined rules or thresholds, whereas proactive autoscaling analyzes and predicts future workloads based on historical data. It then generates scaling decisions based on the predicted workloads. Reactive autoscaling has the advantage of its ease of implementation, but has a significant disadvantage in that it cannot handle the change in workload, leading to a waste of resources, which affects service level agreements and costs [5]. With the proactive approach, service providers can add or remove resources in advance. However, workload demands fluctuate continuously over time, which makes their prediction difficult. To address this problem, many studies have proposed a technique called time series forecasting [3,5,6,7,8,9,10,11]. However, most of these studies focused on univariate time series analysis, that is they only analyzed the measurement of a single feature (or variate or variable), such as CPU utilization or memory usage, and produced predictions. However, there are multiple features to be considered in the cloud system, some of which might have hidden correlations that depend on each other. Multivariate time series analysis can be used to study these hidden correlations. Thus, it helps the prediction model better understand the system and improve the prediction accuracy.

The large dimensionality and spatial–temporal dependence features of multivariate time series data, as well as the existence of noisy data make traditional statistical methodologies difficult to model effectively [12]. Owing to the advancement of artificial intelligence (AI), especially in deep learning, the technology has been applied to various industries, including computer vision, natural language processing, and medical image analysis [13]. Although traditional statistical methodologies can model time series data, deep-learning-based time series forecasting is becoming prevalent [3,9]. In addition, it is promising for designing autoscalers [14]. This paper proposes a multivariate deep-learning-based autoscaling framework using the proactive approach for cloud computing. There are various types of deep learning approaches used in time series forecasting, including recurrent neural networks (RNNs), artificial neural networks, convolutional neural networks (CNNs), and long short-term memory (LSTM). Bidirectional LSTM (Bi-LSTM) [15] is a special type of LSTM with two LSTM layers. These layers run the input data in opposite directions, which helps the model preserve information from the past and the future. Thus, Bi-LSTM can better accumulate knowledge and improve prediction accuracy. The contributions of this study are as follows:

• First, we propose an efficient multivariate proactive autoscaling framework architecture using Bi-LSTM designed for cloud computing;
• Second, through the implementation of a multivariate Bi-LSTM prediction model for time series forecasting, we demonstrate that it outperforms both the univariate Bi-LSTM model and other multivariate time series prediction algorithms;
• Thirdly, we investigated the under-provisioning and over-provisioning of resources by the proposed multivariate Bi-LSTM autoscaler and other well-known multivariate autoscalers.

By utilizing different datasets, our studies revealed that the proposed multivariate Bi-LSTM model outperforms not only the univariate Bi-LSTM model, but also other well-known multivariate deep learning models such as LSTM and CNN-LSTM in prediction accuracy. Furthermore, this work also shows the advantage of the multivariate Bi-LSTM autoscaler compared with other multivariate deep-learning-based autoscalers in terms of resource provisioning efficiency.

The rest of the paper is organized as follows. In Section 2, we review research on autoscaling using the time series forecasting technique and its data. Section 3 presents the system framework architecture followed by the experiments and evaluation in Section 4. Finally, Section 5 concludes the paper.

2. Related Work

In this section, we review recent works related to autoscaling and resource estimation using time series forecasting in cloud computing. Related studies using multiple performance metrics, such as CPU utilization, memory usage, disk I/O, and network throughput, are still limited [11].

Table 1. Autoscaling- and resource estimation-related studies using time series forecasting in cloud computing.

Related Studies	Performance Metric(s)	Type	Method
[8]	HTTP requests	Univariate	ARIMA
[7]	CPU utilization	Univariate	ARIMA
[6]	HTTP requests	Univariate	LSTM and MLP
[9]	HTTP requests	Univariate	LSTM
[10]	HTTP requests	Univariate	AR, HTM, LSTM
[3]	HTTP requests	Univariate	Bi-LSTM
[5]	CPU utilization, memory usage	Multivariate	Bi-LSTM
[11]	CPU utilization, memory usage,	Multivariate	CNN-LSTM
	network usage and disk usage

lists some of these recent related studies.

Time series forecasting is a popular technique for analyzing the behavior of temporal data and predicting future values. It has been widely used in business [16], as well as traffic flow forecasting [17] and anomaly detection [18]. In general, time series data have a natural temporal ordering and can be categorized into two types: univariate and multivariate time series data.

Univariate time series data are a collection of measurements taken for a single variable over time. Univariate refers to a single variable, variate, or feature.

Calheiros et al. [8] proposed a cloud workload prediction module for software as a service (SaaS) using the autoregressive integrated moving average (ARIMA) method to adjust the number of VMs dynamically. They evaluated the proposed prediction model with a realistic HTTP request workload dataset from the Wikimedia Foundation [19]. For seasonal data, the ARIMA model had a 91% accuracy rate. However, the authors did not compare this with other approaches.

Li and Xia [7] presented a hybrid cloud autoscaling architecture known as the cloud resource prediction and provisioning scheme (RPPS). Their proposed prediction model used ARIMA to analyze the historical data of CPU utilization and predict its future usage. The proposed autoscaling architecture, evaluated using varying workloads, showed that their model could outperform the Kubernetes horizontal pod autoscaler (HPA) [20].

Prachitmutita et al. [6] presented a novel autoscaling framework using LSTM and multilayer perceptron (MLP) models to analyze and predict the number of HTTP requests coming to the system. They evaluated the models on a realistic HTTP request workload dataset of the FIFA World Cup 98 [21], and the results showed that the LSTM model could outperform MLP models in terms of accuracy.

LSTM was also used in [9] as a prediction service to predict the number of HTTP requests in the near future. The experiments were conducted on the FIFA World Cup 98 dataset. The authors showed that the LSTM model had a slight prediction error on the root-mean-squared error (RMSE) metric compared with the ARIMA model, indicating that ARIMA had a slightly better accuracy. However, the prediction speed of LSTM was 100-times faster than that of ARIMA.

Toka et al. [10] proposed to use a proactive forecasting method for proactive scaling decisions. The proactive forecasting method consisted of three different prediction models: autoregression (AR), hierarchical temporal memory (HTM), and LSTM. These models were used to learn and predict the incoming HTTP request workload to the system. The authors also proposed HPA+, a backtesting plugin that alternates between the AI-based forecasting technique and HPA automatically. It switches to the HPA if the AI-based forecasting method yields a poor prediction accuracy. The results showed that HPA+ could dramatically reduce the number of rejected requests at the cost of slightly higher resource usage.

Dang-Quang and Yoo [3] proposed a deep-learning-based autoscaling framework for Kubernetes. The proposed autoscaler used Bi-LSTM to analyze and predict the number of HTTP requests and then generate scaling actions. The results of the experiments on the FIFA World Cup 98 dataset showed that Bi-LSTM could outperform LSTM and ARIMA in terms of prediction accuracy, while maintaining the same prediction speed as that of LSTM.

Multivariate time series data are a collection of measurements taken for multiple variables across time. Studying multivariate time series data helps researchers examine and compute the relationship between features. To manage the relationship between variables, they can utilize cross-tabulation, partial correlation, and multiple regressions, as well as add other variables to determine the relationships between the independent and dependent variables or to show the conditions under which the association occurs. The advantage of multivariate analysis over single-variable analysis is that it provides a more realistic picture, thus providing a more powerful test of significance [22].

Yan et al. [5] proposed a hybrid autoscaling system for Kubernetes [20]. The proposed prediction model used Bi-LSTM to analyze and predict the CPU utilization and memory usage of both VMs and pods. To produce elastic scaling actions, the proposed system integrated online reinforcement learning into the proposed Bi-LSTM prediction model, which is a reactive model. The results of the experiments showed that the proposed Bi-LSTM model had better accuracy than those of the ARIMA, RNN, and LSTM models. However, no comparison with the univariate model was performed to analyze the improvement, and the configuration of the proposed Bi-LSTM model was not available. Furthermore, this work did not compare its proposed model with the most recently proposed CNN-LSTM model, and no technique for oscillation mitigation was addressed.

Ouhame and Hadi [11] proposed a multivariate prediction model for cloud computing using a CNN-LSTM network. The proposed model receives multiple features, such as CPU utilization, memory usage, network usage, and disk usage, as inputs to the prediction model. The proposed model was evaluated on the GWA-T-12 dataset from the Bitbrains data center [23] and showed that it could outperform the ARIMA-LSTM model in terms of accuracy. However, this study did not use any feature selection method to select the suitable subset features for the inputs of the model.

Most of the elated studies focused only on univariate time series analysis, which means that they only analyzed the measurement of a single feature (or variate or variable), such as CPU utilization, or memory usage, or network throughput (HTTP requests), and produced predictions. However, there are multiple features to be considered in the cloud system, some of which might have hidden correlations that depend on each other. Multivariate time series analysis can be used to study these hidden correlations. Thus, it helps the prediction model to better understand the system and improve the prediction accuracy. Furthermore, as mentioned earlier, the huge dimensionality and spatial–temporal dependence features of multivariate time series data, as well as the existence of noisy data make traditional statistical methodologies difficult to model effectively [12]. In addition, statistical methods have been relatively slow in matching dynamic workload demands [3,9], which is not suitable for designing real-time autoscaling systems. For these reasons and given the great success of AI, especially deep learning in various fields, this paper proposes an efficient multivariate autoscaling framework using Bi-LSTM for cloud computing environments.

3. Proposed Autoscaling Framework

The proposed system framework is presented in Figure 1. It has two main components: managed resource and autonomic manager. The managed resource component monitors and controls the resources in the system. The autonomic manager component is responsible for estimating needed resources and providing scaling actions.

3.1. Managed Resource

3.1.1. Effector Component

The effector component receives scaling action commands (from the autonomic manager) and executes them. Furthermore, after each scaling action, it validates the currently managed resources to determine whether they meet the desired ones.

3.1.2. Collector Component

In time series forecasting, a multivariate prediction model usually consists of multiple feature values at each time step. The collector component collects the resource utilization data of multiple features such as the CPU, memory, disk I/O, and network received throughput. All the collected data are sent to the monitor server of the autonomic manager.

3.2. Autonomic Manager

3.2.1. Monitor Phase

• Monitoring server: In the first phase of the MAPE loop, the monitoring server continuously gathers the different types of data from the collector component of the monitor server and stores them in the Prometheus time series database;
• Prometheus time series database: Prometheus is a powerful open-source time series database developed by SoundCloud that allows users to leverage the collected data and use them to make decisions. The collected data can be easily exposed and accessed through application programming interface (API) services.

3.2.2. Analyze Phase

• Feature selection service:
  Feature selection enables a machine learning algorithm to train faster, reduces the complexity, and makes the interpretation easier. Furthermore, it can improve the prediction accuracy if the right subset of features is chosen. Finally, it can prevent the overfitting of the model.
  The feature selection component accesses the collected data through the exposed API. Each feature value depends not only on its previous values, but also on other feature values. This correlation between features is important when modeling a multivariate prediction model. In this study, we used the well-known Pearson correlation feature-selection method to measure the linear relationship between two features, where the change in one feature is related to a corresponding change in other features. This is measured by the ratio between the covariance of two different features and the product of their standard deviations. Equation (1) presents the Pearson correlation, $r_{pear}$, and the coefficient of two resource metrics: $X_i$ and $X_j$.

  $$\begin{array}{cc}\hfill r_{pear}=& corr(X_i,X_j)=\frac{{\sum }_{t=1}^T(x_i^t-{\overline{\mathbf{x}}}_{\mathbf{i}})(x_j^t-{\overline{\mathbf{x}}}_{\mathbf{j}})}{\sqrt{{\sum }_{t=1}^T{(x_i^t-{\overline{\mathbf{x}}}_{\mathbf{i}})}^2}\sqrt{{\sum }_{t=1}^T{(x_j^t-{\overline{\mathbf{x}}}_{\mathbf{j}})}^2}}\hfill \end{array}$$

  (1)
  We denote ${\overline{\mathbf{x}}}_{\mathbf{i}}$ and ${\overline{\mathbf{x}}}_{\mathbf{j}}$ as the average values of the features $X_i$ and $X_j$, respectively. The Pearson correlation outputs a value between −1 and 1, where 1 indicates the maximum positive correlation and −1 denotes the negative correlation between the two features. The Pearson correlation will output a value of 0 if the two features are independent of each other. Finally, the Pearson correlation is symmetric; therefore, $corr(X_i,X_j)=corr(X_j,X_i)$. In this study, we chose the network received throughput as the desired prediction metric. Then, we calculated the Pearson correlation value of every resource metric pair (CPU utilization, memory usage, etc.) with the network received throughput. Finally, we selected the resource metric with the highest correlation. The chosen resource metric and network received throughput were selected as the input features of the multivariate Bi-LSTM forecasting service;
• Multivariate Bi-LSTM forecasting service:
  Time series forecasting takes the historical time series data as the input and outputs the future temporal value $p_{t+1}$ at time step $t+1$ or $f_{t+p}$ at time step $t+p$. The model is fed data that include the variable $f_i$ itself and other multivariate time series variables. Using the network received throughput as an example, the model not only includes the network received throughput, but also includes other features such as CPU usage, memory usage, and disk I/O. Network received throughput refers to the amount of data that can be received from the source to the destination within a given time frame. The number of packets that successfully arrive at their destinations is measured by the throughput. Throughput capacity is usually expressed in bits per second, although it can be alternatively expressed as data per second. By predicting the network received throughput, we can estimate the number of incoming packets to our system and better provision or deprovision resources (VMs) to process those packets. For this reason, in this study, we chose the network received throughput as the desired prediction metric for preparing the scaling action described in the planning phase in Section 3.2.3.

  $$MD=\left(\begin{array}{ccc}m{d}_{1,t-w}& \cdots & m{d}_{1,t}\\ m{d}_{2,t-w}& \cdots & m{d}_{2,t}\end{array}\right)\stackrel{\mathrm{trained}\mathrm{model}}{\to }\left(m{d}_{1,t+1}\right)$$

  (2)
  As expressed in Equation (2), the multivariate forecasting model in this paper attempts to predict the value at the next time step $t+1$, given a history of multivariate time series dataset $MD=m{d}_{i,j}|i=1,2;j=t-w,\cdots ,t-1,t$ in the past, where i and w denote the numbers of features and historical time step data, respectively. The proposed model comprises two features: the network received throughput and the feature with the highest correlation chosen from the feature selection service above.
  The multivariate Bi-LSTM forecasting service proposed in this paper collects historical metric data with size w from the Prometheus time series database. Then, the trained model uses the current collected data to predict the data at the next time step $t+1$ of the network received throughput, $m{d}_{1,t+1}$.
  Although the dependencies of long-term states can be established theoretically, an RNN can only learn short-term dependencies because of the gradient vanishing problem [24]. This problem is called long-term dependence. LSTM is a special type of RNN. It was proposed by Hochreiter and Schmihuber in 1996 [25] to solve the RNN problem. Furthermore, it is used in various fields, including weather forecasting, image processing, video analysis, and time series forecasting. The long-term temporal dependence properties of time series data can be learned successfully using LSTM [25]. A typical LSTM unit consists of five components, as shown in the right corner of Figure 2. The inputs of the multivariate Bi-LSTM model include two features: $X_1$ and $X_2$, which are the network received throughput and the chosen feature from the feature selection service. Each red dashed box area presents a matrix of time series data of both input features at a specific time step. For example, the red dashed box area of $X_{t-w}$ is a matrix [$x_{1,t-w}$, $x_{2,t-w}$] that contains two separate historical time series data of features $X_1$ and $X_2$, respectively, at time step $t-w$. We use $i_t,f_t,c_t,o_t,$ and $h_t$ to respectively denote the input gate, forward gate, memory cell, output gate, and hidden state of a basic LSTM unit. The gates and memory cell of the LSTM unit allow it to memorize or forget information.

  $$i_t=\sigma (M_i{x}_t+N_i{h}_{t-1}+b_i)$$

  (3)

  $$f_t=\sigma (M_f{x}_t+N_f{h}_{t-1}+b_f)$$

  (4)

  $$o_t=\sigma (M_o{x}_t+N_o{h}_{t-1}+b_o)$$

  (5)

  $$\widetilde{c_t}=tanh(M_c{x}_t+N_c{h}_{t-1}+b_c)$$

  (6)

  $$c_t=f_t\odot c_{t-1}+i_t\odot \widetilde{c_t}$$

  (7)

  $$h_t=o_t\odot tanh\left(c_t\right)$$

  (8)
  We denote $h_t$ and $c_t$ as the hidden state and memory state of the LSTM, respectively. The LSTM computes $h_t$ and $c_t$ for an input time series value at time step t. We denote $\sigma$ and ⊙ as the activation function and Hadamard product (elementwise product), respectively. The Hadamard product is a binary operation that takes two matrices of the same dimensions and outputs a new matrix of the same dimensions as the operands. However, because LSTM can only learn the past context of time series data, it has a restriction. Thus, in this study, we propose to use Bi-LSTM. Bi-LSTM is a special type of LSTM [15] that has two LSTM layers for simultaneously processing the input time series data in two directions. The first LSTM processes sequence input data in a forward direction from t = 1 to t = T. In contrast, the second LSTM processes the input data in the backward direction from t = T to t = 1. This Bi-LSTM architecture executes the input in two directions: from the past to the future and from the future to the past. Applying LSTM twice results in further learning of the long-term dependencies. Therefore, it improves the accuracy of the model. The equations for Bi-LSTM are as follows:

  $$\overrightarrow{i_t}=\sigma (\overrightarrow{M_i}\overrightarrow{x_t}+\overrightarrow{N_i}\overrightarrow{h_{t-1}}+\overrightarrow{b_i})$$

  (9)

  $$\overrightarrow{f_t}=\sigma (\overrightarrow{M_f}\overrightarrow{x_t}+\overrightarrow{N_f}\overrightarrow{h_{t-1}}+\overrightarrow{b_f})$$

  (10)

  $$\overrightarrow{o_t}=\sigma (\overrightarrow{M_o}\overrightarrow{x_t}+\overrightarrow{N_o}\overrightarrow{h_{t-1}}+\overrightarrow{b_o})$$

  (11)

  $$\overrightarrow{\widetilde{c_t}}=tanh(\overrightarrow{M_c}\overrightarrow{x_t}+\overrightarrow{N_c}\overrightarrow{h_{t-1}}+\overrightarrow{b_c})$$

  (12)

  $$\overrightarrow{c_t}=\overrightarrow{f_t}\odot \overrightarrow{c_{t-1}}+\overrightarrow{i_t}\odot \overrightarrow{\widetilde{c_t}}$$

  (13)

  $$\overrightarrow{h_t}=\overrightarrow{o_t}\odot \overrightarrow{tanh\left(c_t\right)}$$

  (14)

  $$\overleftarrow{i_t}=\sigma (\overleftarrow{M_i}\overleftarrow{x_t}+\overleftarrow{N_i}\overleftarrow{h_{t-1}}+\overleftarrow{b_i})$$

  (15)

  $$\overleftarrow{f_t}=\sigma (\overleftarrow{M_f}\overleftarrow{x_t}+\overleftarrow{N_f}\overleftarrow{h_{t-1}}+\overleftarrow{b_f})$$

  (16)

  $$\overleftarrow{o_t}=\sigma (\overleftarrow{M_o}\overleftarrow{x_t}+\overleftarrow{N_o}\overleftarrow{h_{t-1}}+\overleftarrow{b_o})$$

  (17)

  $$\overleftarrow{\widetilde{c_t}}=tanh(\overleftarrow{M_c}\overleftarrow{x_t}+\overleftarrow{N_c}\overleftarrow{h_{t-1}}+\overleftarrow{b_c})$$

  (18)

  $$\overleftarrow{c_t}=\overleftarrow{f_t}\odot \overleftarrow{c_{t-1}}+\overleftarrow{i_t}\odot \overleftarrow{\widetilde{c_t}}$$

  (19)

  $$\overleftarrow{h_t}=\overleftarrow{o_t}\odot \overleftarrow{tanh\left(c_t\right)}$$

  (20)

  $$h_t=\overrightarrow{h_t}\odot \overleftarrow{h_t}$$

  (21)
  As shown by the above equations, the arrow shows the processing direction. We denote $h_t$ as the final hidden output of Bi-LSTM, which is calculated by concatenating the hidden forward output $\overrightarrow{h_t}$ and hidden backward output $\overleftarrow{h_t}$. The dense layer is a neural network layer that is deeply connected, which means that each neuron in the dense layer receives input from all neurons of its previous layer and then it outputs the predictions.

3.2.3. Planning Phase

The resource estimation service calculates the number of resources (VMs or containers) based on the predicted network received throughput of the previous step, $networ{k}_{t+1}$, as shown in Algorithm 1. The autoscaler has to deal with oscillation problems that frequently perform opposite scaling actions in a short time. This problem will result in a wastage of resources and budget. We applied the control period time (CPT) between each scaling action to deal with this problem. In addition, we set the CPT value to 1 min, which is fine grained for the VM setting. $VM{s}_{t+1}$ and $networ{k}_{VM}$ are the number of estimated VMs in the next time step and the maximum network workload that a VM can handle in a minute, respectively. $VM{s}_{min}$ is the minimum number of VMs, and the system cannot scale down the number of VMs to a number lower than this value. For each CPT (Line 3), we calculated $VM{s}_{t+1}$ (Line 4). If $VM{s}_{t+1}$ is higher than the current number of VMs $VM{s}_{current}$ (Line 5), for the next time step, we scheduled the scale-up command to provision the number of VMs required to fulfill the resource requirement (Line 6). Otherwise, if $VM{s}_{t+1}$ is lower than $VM{s}_{current}$ (Line 7), we obtained the maximum value between $VM{s}_{t+1}$ and $VM{s}_{min}$ (Line 8), and then scheduled the scale-down command to deprovision unneeded VMs (Line 9). Otherwise, the autoscaler sleeps and waits until the next interval to keep recalculating the number of needed VMs and performing scaling actions (Line 11).

Algorithm 1 Adaption strategy service algorithm.

Input: $networ{k}_{t+1}$ // predicted workload in the next interval             $VM{s}_{min}$ // minimum number of VMs that the system has to maintain                                                   Output: Scheduled scaling actions   1: Initialization   2: while system is running do   3:      for each CPT do   4:            $VM{s}_{t+1}$ = $\frac{networ{k}_{t+1}}{networ{k}_{VM}}$   5:            if $VM{s}_{t+1}$ > $VM{s}_{current}$ then   6:                   $SCHEDULE\_SCALE\_UP\_COMMAND$ ($VM{s}_{t+1})$   7:            else if $VM{s}_{t+1}$ < $VM{s}_{current}$ then   8:                   $VM{s}_{t+1}$$=max(VM{s}_{t+1},VM{s}_{min})$   9:                   $SCHEDULE\_SCALE\_DOWN\_COMMAND$ ($VM{s}_{t+1})$  10:           else  11:                  Sleep  12:           end if  13:     end for  14: end while

3.2.4. Execution Phase

The execution phase has two cases. If it receives a scheduled scale-up command, the execution phase first sends the scale-up command to the effector component to spawn a number of VMs to satisfy the demand. Subsequently, it schedules the time required to assign the spawned VMs to a cluster at the next time step. Otherwise, when it receives a scheduled scale-down command, it waits until the next time step and begins to discard the VMs according to the demand.

4. Experiment and Evaluation

To evaluate the proposed system framework described in Section 4, all experiments were performed using real workload datasets obtained from the Bitbrains and Materna data centers.

4.1. Experiment Detail

First, we calculated the Pearson correlation value of every resource metric pair of other resource metrics (CPU utilization, memory usage, etc.) with the network received throughput to select the resource metric with the highest correlation value.

Second, we used the chosen resource and network received throughput metrics as inputs to the proposed multivariate Bi-LSTM model and compared them with the univariate one. The univariate Bi-LSTM model only has a network received throughput metric as its input. Furthermore, the multivariate Bi-LSTM model was compared with other well-known multivariate deep learning models (LSTM and CNN-LSTM) [11] to evaluate its performance. All prediction models performed prediction and output the future network received throughput data of the system. We used the Python programming language and trained the models on the tensor processing unit from Google Colab [26]. Furthermore, the code for implementing these models followed the code in the Deep Learning for Time Series Forecasting book [27]. The configuration of the proposed multivariate Bi-LSTM model is presented in

Table 2. Multivariate Bi-LSTM model configuration.

Number of layers	2 layers (forward and backward)
Number of features	2
Input size	3 neural cells
Number of hidden units	50
per neural cell
Loss function	MSE
Batch size	64
Epochs	10
Activation function	ReLU

.

Finally, we simulated the environment to evaluate the resource provisioning efficiency of the proposed multivariate Bi-LSTM autoscaler compared with those of other multivariate deep-learning-based autoscalers, as mentioned above. In this experiment, we used the first 200 observations in the testing set of the GWA-T-13 dataset to estimate the resource provision efficiency of each multivariate model. In addition, we assumed that each VM could handle 3300 KB/s of network received throughput, $networ{k}_{VM}=3300$, and that the workload was balanced between VMs.

4.2. Dataset

We evaluated the proposed multivariate Bi-LSTM model on two real-world workload datasets: GWA-T-12 [23] from the Bitbrains data center and GWA-T-13 [28] from the Materna data center.

4.2.1. GWA-T-12 Bitbrains

The GWA-T-12 [23] logs contain the performance metrics of 1750 VMs from the Bitbrains data center. These logs are organized according to the following traces: fastStorage and Rnd. This study used the first trace—fastStorage—which contains the performance metrics of 1250 VMs that are connected to the storage devices of a fast storage area network.

4.2.2. GWA-T-13 Materna

The GWA-T-13 [28] dataset contains three traces of the performance metrics of 520, 527, and 547 VMs from the Materna data center. Materna is a premium full-service supplier that has successfully implemented innovation-through-collaboration projects for its clients for over 35 years. We chose the first trace that contained the performance metrics of 520 VMs.

4.3. Metric for Validation

Three standard statistical metrics, mean absolute error (MAE), MSE, and RMSE, were used to evaluate the prediction accuracy of the models. The equations of these metrics are expressed in Equations (22)–(24), respectively.

$$MAE=\frac{1}{n}\sum _{i=1}^n\left|(v_i-l_i)\right|$$

(22)

$$MSE=\frac{1}{n}\sum _{i=1}^n{(v_i-l_i)}^2$$

(23)

$$RMSE=\sqrt{MSE}=\sqrt{\frac{1}{n}\sum _{i=1}^n{(v_i-l_i)}^2}$$

(24)

We denote $v_i$ and $l_i$ as predicted and true values, respectively. The MAE is the difference between the predicted and true values. The MSE calculates the average squared difference between the predicted and true values. The RMSE is the square root of the average of the squared differences between the prediction and true observation. These metrics are the three most commonly used metrics for measuring the accuracy of continuous data, where smaller values indicate higher prediction accuracy and vice versa.

To evaluate the resource provisioning efficiency, we applied the system-oriented elasticity metrics presented by the Research Group of the Standard Performance Evaluation Corporation (SPEC) [29]. Various studies [3,9,30] have also used these metrics to evaluate the resource provisioning efficiency of their autoscalers. The equations of the metrics are shown in Equations (25)–(29), where $d_n$ and $p_n$ are the resource demand and provisioned resources, respectively.

The under-provisioning resource metric, ${\mathsf{\Theta }}_U$, calculates the number of required VMs that must be provisioned to meet the service-level objectives normalized by the measured time. The over-provisioning resource metric, ${\mathsf{\Theta }}_O$, calculates the number of excess VMs that are provisioned by the autoscaler. The values of ${\mathsf{\Theta }}_U$ and ${\mathsf{\Theta }}_O$ range between 0 and infinity, where 0 is the best value. $T_U$ and $T_O$ are the durations the system is under-provisioned and over-provisioned, respectively. $T_U$ and $T_O$ have values ranging from 0 to 100, with 0 being the optimal value when no under-provisioning or over-provisioning occurs throughout the measurement period. Finally, the elastic speed-up metric, ${ϵ}_{na}$, is the efficiency of the autoscaling compared with that of the no-autoscaling case, where $na$ indicates no autoscaler is used and a indicates otherwise. The default ${ϵ}_{na}$ value of $na$ is 1.0. If the proposed autoscaler (a case) has an ${ϵ}_{na}$ value greater than 1.0, it means that the autoscaler has an autoscaling gain over the no-autoscaler case; otherwise, it has none.

$${\mathsf{\Theta }}_U[\%]=\frac{100}{T}\sum _{t=1}^T\frac{max(d_n-p_n,0)}{d_n}\mathsf{\Delta }t$$

(25)

$${\mathsf{\Theta }}_O[\%]=\frac{100}{T}\sum _{t=1}^T\frac{max(p_n-d_n,0)}{d_n}\mathsf{\Delta }t$$

(26)

$$T_U[\%]=\frac{100}{T}\sum _{t=1}^{T}max(sgn(d_n-p_n),0)\mathsf{\Delta }t$$

(27)

$$T_O[\%]=\frac{100}{T}\sum _{t=1}^{T}max(sgn(p_n-d_n),0)\mathsf{\Delta }t$$

(28)

$${ϵ}_{na}={\left(\frac{{\mathsf{\Theta }}_{U,na}}{{\mathsf{\Theta }}_{U,a}}\frac{{\mathsf{\Theta }}_{O,na}}{{\mathsf{\Theta }}_{O,a}}\frac{T_{U,na}}{T_{U,a}}\frac{T_{O,na}}{T_{O,a}}\right)}^{\frac{1}{4}}$$

(29)

4.4. Experiment Results

4.4.1. Feature Selection Results

• GWA-T-12 dataset:
  Table 3. Feature selection of the GWA-T-12 dataset.

  Pair of	Pearson Correlation
  Metrics	Value
  corr(network, CPU)	0.714
  corr(network, memory)	0.746
  corr(network, disk read)	0.637
  corr(network, disk write)	0.63

  The bold value indicates the best value.

  shows the Pearson correlation values of the network received throughput metric with other metrics for the GWA-T-12 dataset. As can be seen in the table, memory usage had the highest correlation value of 0.746. Thus, the network received throughput and memory usage were used as inputs of the multivariate models to evaluate the prediction accuracy;
• GWA-T-13 dataset:
  Table 4. Feature selection of the GWA-T-13 dataset.

  Pair of	Pearson Correlation
  Metrics	Value
  corr(network, CPU)	0.388
  corr(network, memory)	0.422
  corr(network, disk read)	−0.019
  corr(network, disk write)	0.393

  The bold value indicates the best value.

  presents the Pearson correlation values of the network received throughput metric with other metrics for the GWA-T-13 dataset. The memory usage metric in this dataset also had the highest correlation value. Thus, the memory usage and network received throughput were used as inputs to the multivariate models to evaluate their prediction accuracy.

We preprocessed the dataset to a value in the range [0, 1] based on Equation (30). Subsequently, we divided each dataset into training (80%) and evaluation (20%) datasets for both datasets.

As mentioned earlier, the feature selection can attempt to prevent overfitting of the prediction model. Figure 3 presents the model loss of the proposed multivariate Bi-LSTM model during the training process. The training dataset (which contains 80% of the entire dataset) was split into two parts: training (80%) and validation (20%). As shown in Figure 3, the loss during the training and validation processes decreased over time. Thus, the model did not have an overfitting problem.

$$z=\frac{x-min\left(x\right)}{max\left(x\right)-min\left(x\right)}$$

(30)

4.4.2. Prediction Performance Results

• GWA-T-12 dataset:
  The MAE, MSE, and RMSE metric values for the multivariate Bi-LSTM model were lower than those for the univariate Bi-LSTM model, as shown in

  Table 5. Comparison of univariate Bi-LSTM and multivariate LSTM using the GWA-T-12 dataset.

  Prediction Model	MAE	MSE	RMSE
  Univariate Bi-LSTM	0.000191	0.00000007	0.000257
  Multivariate Bi-LSTM	0.000105	0.0000002	0.000139

  The bold value indicates the best value.

  . The proposed multivariate Bi-LSTM had an RMSE prediction error 1.84-times smaller than that of the univariate one. These results indicate that Bi-LSTM had a better prediction accuracy. Figure 4 shows the absolute error values for the first 100 observations of the univariate and multivariate Bi-LSTM models. We can see in the figure that the multivariate Bi-LSTM had smaller absolute error values in every observation than those of the univariate Bi-LSTM.
  Table 6. Comparison of different multivariate models using the GWA-T-12 dataset.

  Prediction Model	MAE	MSE	RMSE
  Multivariate Bi-LSTM	0.000105	0.0000002	0.000139
  Multivariate LSTM	0.000451	0.00000022	0.000468
  Multivariate CNN-LSTM	0.000219	0.00000005	0.000234

  The bold value indicates the best value.

  also indicates that the multivariate Bi-LSTM model had smaller prediction error values for the MAE, MSE, and RMSE metrics than those of the multivariate LSTM and CNN-LSTM models. Moreover, the multivariate Bi-LSTM had better accuracy than those of the other models. We plotted the absolute error values of the first 100 observations using different multivariate prediction models, as shown in Figure 5. In the figure, we can clearly see that the multivariate Bi-LSTM had smaller absolute error values in all 100 observations. This proves that the time series data in the forward direction do not provide effective information other than to integrate the backward direction of the time series, causing the LSTM and CNN-LSTM models to become less effective;
• GWA-T-13 dataset:
  Table 7. Comparison of univariate Bi-LSTM and multivariate LSTM using the GWA-T-13 dataset.

  Prediction Model	MAE	MSE	RMSE
  Univariate Bi-LSTM	0.048584	0.0059453	0.077106
  Multivariate Bi-LSTM	0.048427	0.005538	0.074419

  The bold value indicates the best value.

  lists the results obtained by the univariate and multivariate Bi-LSTM models on the GWA-T-13 dataset. The multivariate Bi-LSTM model outperformed the univariate one, reducing the RMSE prediction error by 3.4% compared with that of the univariate model. Figure 6 and Figure 7 show the results obtained on the testing dataset. The multivariate Bi-LSTM model had better prediction results. We can clearly see this by observing some high peak points where the multivariate model had closer predicted values to the actual values than those of the univariate model. Figure 8 visualizes the absolute error for the first 100 observations of both models and shows that the multivariate Bi-LSTM had smaller prediction absolute error values than those of the univariate one.
  When compared with other well-known multivariate models listed in

  Table 8. Comparison of different multivariate models using the GWA-T-13 dataset.

  Prediction Model	MAE	MSE	RMSE
  Multivariate Bi-LSTM	0.048427	0.005538	0.074419
  Multivariate LSTM	0.051780	0.0063723	0.079827
  Multivariate CNN-LSTM	0.049105	0.006196	0.078715

  The bold value indicates the best value.

  , the Bi-LSTM model also had better prediction accuracy than those of the LSTM and CNN-LSTM models. It reduced the RMSE prediction error by 6.7% and 5.4% when compared with the LSTM and CNN-LSTM models, respectively. Figure 9 and Figure 10 show the predicted values versus the actual values on the testing dataset. Figure 9 tends to over-predict over time when compared with Figure 7 and Figure 10. Figure 11 shows the absolute error for the first 100 observations of the three multivariate models, where Bi-LSTM had smaller prediction absolute error values than those of the LSTM and CNN-LSTM models.

4.4.3. Evaluation of the Autoscaler

Figure 12, Figure 13 and Figure 14 show the number of provisioned VMs versus the number of required VMs of the different multivariate models calculated using Algorithm 1. A comparison of the resource provisioning of the autoscaler using different multivariate models is also shown in

Table 9. Comparison of autoscalers using different multivariate prediction models.

Type	No-Autoscaling	Multivariate	Multivariate	Multivariate
		LSTM	CNN-LSTM	Bi-LSTM
${\mathsf{\Theta }}_O[\%]$	0.235	0.762	0.699	0.419
$T_O$ [%]	8.0	28.0	26.5	25.5
${\mathsf{\Theta }}_U[\%]$	7.983	0.702	0.455	0.458
$T_U$ [%]	84.5	41.5	26.98	27.0
${ϵ}_{na}$	1.0	1.195	1.533	1.759

The bold value indicates the best value.

.

The Bi-LSTM autoscaler performed better by having smaller over-provisioned (${\mathsf{\Theta }}_O$, $T_0$) and under-provisioned (${\mathsf{\Theta }}_U$, $T_U$) resource metric values than those of the LSTM autoscaler. Furthermore, it had better elastic speed-up ${ϵ}_{na}$ values: 1.759 > 1.195 (increase of 47.2%), which indicate that it had better autoscaling gain than that of the LSTM autoscaler.

Although the CNN-LSTM autoscaler exhibited a similar under-provisioning value to that of Bi-LSTM, its over-provisioning value was higher. Consequently, the Bi-LSTM autoscaler exhibited better autoscaling gain than that of the CNN-LSTM autoscaler. This was demonstrated by the 14.7% increase in the ${ϵ}_{na}$ values when compared with those of the CNN-LSTM autoscaler.

5. Conclusions

Cloud computing plays a significant role in improving the efficiency of enterprises, industries, etc. Autoscaling is an essential mechanism in cloud computing that automatically adds or removes resources. However, over time, the workload demands fluctuate continuously, which makes their prediction difficult. To predict the resource workload, many studies have proposed a technique called time series forecasting. However, most of these studies have focused only on univariate time series forecasting. This study proposed an efficient multivariate autoscaling framework using Bi-LSTM for cloud computing environments. The proposed autoscaling system framework can be flexibly implemented for VM platforms (such as OpenStack [31]) and container platforms (such as Docker Swarm [32] and Kubernetes [33]). The experiments were conducted on actual trace workload datasets from the Bitbrains and Materna data centers. The evaluation results showed that our proposed multivariate Bi-LSTM model can outperform not only the univariate Bi-LSTM model, but also other well-known multivariate deep learning models such as LSTM and CNN-LSTM in terms of prediction accuracy. Furthermore, this work also showed the advantage in the resource-provision efficiency of the proposed multivariate Bi-LSTM autoscaler, which had better autoscaling gains than those of the other multivariate deep-learning-based autoscalers.

In future work, we plan to extend our work to hybrid autoscaling (which fuses reactive and proactive autoscaling) and extend the inputs of the proposed prediction model to evaluate its performance.