**6. Experimental Results**

### *6.1. Comparison of Prediction Performance with Various STLF Models*

In this paper, we compare popular machine learning algorithms such as decision tree (DT), gradient boosting machine (GBM), bagging algorithm, and so on, to explain why we chose XGBoost and RF models in the first stage. Besides, we compare the performance with the prediction model (Persistence) which is actually using in the data collection environment. Persistence model uses the previous day (or the corresponding day in the previous week) as a prediction. Persistence implies that future values of the time series are calculated on the assumption that conditions remain unchanged between the current time and future time. As the second stage of our proposed model uses the predicted values of these two models, we divide the dataset into training set 1 (training the first-stage model), training set 2 (training the second-stage model), and test set (forecasting electric energy consumption and economic analysis), at a ratio of 50:25:25. Specifically, data collected from January 2015 to December 2016 was used as training set 1, data collected from January 2017 to December 2017 was used as training set 2, and data collected from January 2018 to December 2018 was used as test set. The performance of each machine learning algorithm was measured using the training set 2. Figure 3 shows monthly energy consumption and divided dataset. In addition, we compare our proposed model with various STLF models composed of different machine learning algorithms in the first-stage, and several forecasting models from our previous studies in the second-stage. To do this, we divided the dataset into training and test sets, at a ratio of 75:25.

**Figure 3.** Box plots by the monthly electric load for separation of training and test sets.

Additionally, we selected a coefficient of variation of the root mean square error (CVRMSE) and mean absolute percentage error (MAPE) because they are easier to understand than other performance indicators such as the root mean square error (RMSE) or mean squared error (MSE) [64]. They were then used to evaluate the prediction performance of the proposed model. The CVRMSE and MAPE equations are shown in (7) and (8), respectively. Here, *n* is the number of time observed, *Y* is an average of the actual values. *Yi* and *Y* ˆ *i* are the actual and predicted values, respectively. Figure 4 exhibits the comparison of CVRMSE and MAPE results for each machine learning algorithm.

$$\text{CVRMSE} = \frac{100}{\overline{\chi}} \sqrt{\frac{\sum\_{i=1}^{n} \left(\chi\_i - \hat{Y}\_i\right)^2}{n}} \tag{7}$$

$$MAPE = \frac{100}{n} \sum\_{i=1}^{n} \left| \frac{\mathbf{Y}\_i - \mathbf{\hat{Y}}\_i}{\mathbf{Y}\_i} \right| \tag{8}$$

As shown in Figure 4, XGBoost and RF models show better prediction performance in training set 2 compared with other machine learning algorithms. The performance of machine learning techniques is better than the persistence model which is a statistical model. In addition, the performance of the XGBoost and RF models was better than the other machine learning algorithms. XGBoost performed well because it allows users to choose an appropriate loss function depending on the characteristics of the data. RF performed well because it can handle high-dimensional data well. Table 8 summarizes the Pearson correlation coefficients between the forecasted values of machine learning algorithms and actual electric energy consumptions. We found that the forecasted values of XGBoost and RF present higher correlation coefficients than those of other machine learning algorithms. Therefore, we used the forecasted values of XGBoost and RF as new input variables for the second stage.



Tables 9–11 summarize the comparison of our proposed model with other 2-stage models and several forecasting models of our previous studies in terms of CVRMSE and MAPE. As summarized in Tables 9–11, our proposed model showed an almost better prediction performance than other forecasting models.

**Forecasting Model Jan. Feb. Mar. Apr. May Jun. Jul. Aug. Sep. Oct. Nov. Dec. Avg.** Moon et al. (2018) [33] 28.58 37.49 39.13 39.55 45.00 42.30 36.39 39.71 50.19 50.85 45.27 40.77 41.27 Son et al. (2018) [34] 23.11 34.42 33.57 23.31 22.13 33.06 29.28 26.43 29.68 28.49 23.71 27.47 27.84 Moon et al. (2019) [36] 22.35 30.79 37.78 30.89 32.88 33.15 26.51 25.77 30.23 33.11 30.27 32.19 30.48 Park et al. (2019) [37] 15.45 20.92 26.78 18.67 22.96 40.55 28.34 23.48 18.38 32.17 23.89 20.13 24.32 2-stage RF 22.39 37.06 37.08 27.38 40.23 48.22 43.18 41.88 47.28 42.41 25.80 24.24 36.42 2-stage XGBoost 22.36 36.93 33.73 25.06 37.20 35.47 31.31 28.66 34.96 32.67 21.00 22.94 30.15 Proposed Model **14.62 13.98 20.35 18.58 18.95 16.50 19.42 18.45 21.09 18.33 19.81 18.24 18.22**

**Table 9.** MAPE comparison for each month (The best in bold).

**Table 10.** CVRMSE comparison for each month (The best in bold).




Finally, to ensure the significant contribution in terms of forecasting accuracy improvement for the proposed model, the Wilcoxon test and the Friedman test are conducted [30]. Wilcoxon test was used to test the null hypothesis by setting the null hypothesis to determine whether there was a significant difference between the two models. If the *p*-value is less than the significance level, the null hypothesis is rejected and the two models are judged to have significant differences. Friedman test is a multiple comparisons test that aims to detect significant differences between the results of two or more algorithms model. The results of the Wilcoxon test with the significance level set to 0.05 are shown in Table 12. Since the *p*-value in all cases is below the significance level, it was proven that proposed model is superior to the other models.



### *6.2. Economic Analysis Based CCHP Operation Scheduling*

In this section, we describe how CCHP operation scheduling is made based on economic analysis. To maximize the annual economic benefits, it is also essential to determine the electric rate and amount of contract demand at the same time. We perform an experiment to find the optimal electric rate and contract demand to maximize on economic benefits.

A monthly economic analysis using the test set confirms that the economic benefits are similar to the monthly energy consumption, as shown in Figure 5. We can see that high economic benefits can be obtained in summer and winter when energy consumption is high.

**Figure 5.** Monthly economic benefits.

Because the industrial building where the electric energy consumption data was collected is equipped with advanced meters, the electric rate of industrial service (A) II and industrial service (B) can be chosen. In addition, since the building's supply voltage is between 3300 V and 66,000 V, we choose the high voltage A as the electric rate of the building. Industrial service (A) II has two options, and industrial service (B) has three options. As a result, five di fferent electric rates are compared in the experiment. Figure 6 shows the annual economic benefit of each electric rate based on contract demand.

**Figure 6.** Annual economic benefit of each electric rate based on contract demand.

Figure 6 shows that "industrial service (A) II / high voltage A / Option II" electric rate with 160 kW contract demand can make the highest annual economic benefit and Figures 7–9 show the scheduling result of the CCHP operation according to this electric rate. In the figure, the yellow boxes represent electric energy supplied by the public power system and the green boxes represent electric energy generated by the CCHP system.

(**a**) CCHP operation scheduling based on predicted electric loads.

(**b**) Results of economic analysis based on predicted electric loads. **Figure 7.** Example of CCHP operation scheduling.

 **Figure 8.** Example of CCHP operation scheduling in winter.

 **Figure 9.** Example of CCHP operation scheduling in summer.

According to the schedule, an economic benefit of USD 195 can be made when using CCHP with a public power system for three days. Moreover, economic benefits of more than USD 14,000 annually can be achieved by using CCHP with the public power system.
