The proposed hybrid model has two components that are connected sequentially. The first one is called the basic model and includes ARIMA forecasting to capture the linear relationship of the input data. The output of the ARIMA is fed into the second component. A neural network model capable of capturing both linear and nonlinear patterns is used in the second component. As most data sets are composed of linear and nonlinear elements, two separate models that work in tandem will be more effective in capturing different portions of the data. Before moving into details of the hybrid model, brief descriptions of ARIMA, and ANN models are given below.
3.2. Artificial Neural Networks (ANNs)
ANNs are computational systems based on the operational structure of a biological brain. ANN architecture consists of units (i.e., neurons) that are organized in layers. In addition to the input and output layers, there are hidden layers between the input and the output. ANN is a nonparametric approach that can extract nonlinear dynamics from the data. Because of the limitations of statistical tools, ANN has long been regarded as an alternative route to deal with nonlinear problems in various fields. There are various architectures used in ANN, some of which mimic the function of the brain, but others are mathematical representations constructed according to the problem requirements. Among these varieties, MLP (multi-layer perceptron) has long been used for nonlinear regression problems.
A three-layer feed-forward neural network is commonly used in time series modeling (
Figure 2). The input layer includes
p nodes, each representing an input to the model (
through
). The middle (i.e., hidden) layer includes
q units, and there is a single output
. Each unit in the hidden layer processes the information fed from the input units. Links between the input and hidden layers are model parameters. Each link between the input and hidden layer has a weight
. Similarly, the
represents the weight of the link between the hidden and output layers.
3.4. Hybrid Model
The proposed hybrid arrangement combines ARIMA and ANN models in series. As given in Equation (5), the original time series is considered the sum of two components: one is linear and the other is nonlinear. The linear component is represented by a multi-period lagged ARIMA model (M
t). This component captures the linear relationship in the data. The second component (N
t) is built on the leftover error of the first model. Whatever cannot be explained by ARIMA is fed into the ANN.
Based on the assumption that, the data
can be split into two components, representing linear (
) and nonlinear (
) portions, a hybrid forecasting approach is developed to exploit the unique characteristics of each component. Processes involved in hybrid modeling can be divided into six sub-tasks performed in sequential order: (1) The routine starts with the
basic model; (2) the error output of the basic model is processed in the
classification; (3) based on the classification,
the input lists are prepared for the second model; (4)
the secondary model runs on each list separately; (5) the output of the second model is consolidated in
hybridization; (6) in the final stage, projections are generated with the hybrid model.
Figure 3 shows tasks and relationships graphically.
The modeling process starts with the division of the original time series into training and test series. Next, the basic model is developed using only the training data, which includes the first portion of the data set (i.e., the chronologically older). In terms of size, the training data are typically several times larger than the test portion. In selecting the split point, we followed the guidelines used in previous studies on the same data sets. It should be noted that the number of observations in the training data is further reduced because of the lag effect in the data. The formula for the basic model is given in Equation (6).
where
and
represent the actual and forecast values at point t. The offset between the two is the error
, and it is used as a basis to construct a new time series for the second component.
In the second stage, the error output of the first component (Equation (7)) is classified into three distinct groups. The classification is done according to an a priori selected threshold value. Over-shooters include non-negative errors that are greater than the threshold. Under-shooters are negative errors that are smaller than the negative value of the threshold limit. Anything that falls between these two limits is categorized as neutral. This type of categorization was first proposed by Khashei et al. [
25], where the threshold limits are defined as the desired level of error (DLE). The value of this parameter could be set by the user based on the analysis of the error coming from the basic model. In particular, the distribution of error coming from the basic model can provide insight regarding when or how the basic model over- or under-estimates. If there is no insight into the performance of the basic model, it could be set to a fixed value. For example, the DLE is set to 5% in experimental runs by Khashei et al. [
25]. The error output of the basic model is categorized into three groups: A, B, and C, with the following definitions:
, Category A
, Category B
, Category C
Based on the categorization of the second step, three separate data lists are constructed from the original error output in Equation (7). To explain the process, let us assume that the output of the basic model is stored in a list, E, with n members,
. Three separate lists, A, B, and C, are generated from list E. While lists A and B represent underestimated and overestimated values of E, list C holds neutral error values. The content of E is directly copied to A, B, and C to initialize the lists. Then, with the guidelines given in Equation (8), adjustments are made to each list. In list A, which represents underestimated values, all category A values are kept as they are, but everything else is set to zero. Similarly, list B, representing overestimated errors (i.e., category B), retains its original values but overwrites everything else with zero. For list C, all elements with over- and underestimated values are set to zero.
The second model performs a complementary function. It uses leftover error values from the basic model to extract additional explanatory information. As explained in the previous step, the error output is categorized into three separate lists. The lists A and B represent under- and overestimated values, and they are the inputs for the second model. List C contains error values that remain below the threshold (i.e., DLE) and are excluded from the run. The second model includes an artificial neural network (ANN) to identify nonlinear patterns and any high-order linear relationships that the basic model missed. In the original approach proposed by Zhang [
26], the primary and secondary models are assigned specific tasks. While the primary model deals with the linear component of the data, the leftover error is assumed to be the nonlinear component. This type of classification of the domain may not be applicable in every situation, and the residual error of the primary model cannot be presupposed as a nonlinear component. It is likely that the leftover error values are a combination of linear and nonlinear error values.
The ANN model is run separately on A and B lists to generate two sets of forecasts, representing the under- and overestimation portions of the model. As stated in Equations (9) and (10), the ANN uses
n period-lagged values from
t to
t −
n to generate the forecast for
t + 1.
where
and
represent functions used by ANN. The terms
and
are random errors.
Hybridization involves integrating basic and secondary models in a schema to produce a single (i.e., hybrid) forecast. In the proposed schema, the hybrid forecast for
t + 1 is the sum of outputs generated by
and
.
Forecasts given in (11) and (12) can be considered a modified version of the basic model forecast. The in (11) is the corrected version when the basic model underestimates the true value. Underestimation produces positive errors in the basic model. When positive values are fed into the secondary model, the secondary model, in turn, is likely to produce positive-valued forecasts. This could be considered compensation for the underestimation of the basic model. A reverse mechanism works in (12). When the basic model overestimates, errors are likely to be negative. To compensate, the second model will produce negative-valued forecasts ().
In addition to these two forecasts, the basic model’s direct output can be considered a third forecast. The second model is bypassed, and the output of the basic model is the final forecast:
. The
can be considered category C, and it represents the mid-value of three forecasts. While category C determines the midpoint, forecasts for A and B are upper and lower estimates. Among three candidate forecast points (i.e.,
), one must be selected as the final forecast. The selection is based on the error metrics discussed in
Section 3.3. The combination with the lowest score is chosen as the final hybrid model.
The hybrid model selected in the previous step is used for projection. Model selection is based on the performance of the training data, which are exclusively used for model development. Once the model is finalized, its forecasting performance is evaluated on the test data. This data set is a continuation of the training data and is used for model validation only.
In terms of platform and software, we used IBM’s SPSS statistical software to build ARIMA models. The MATLAB Toolbox is used for ANN models. Development and test runs were conducted on a Windows-based laptop with an Intel Core i5-7200U CPU @ 2.50GHz. For both ARIMA and ANN models, the run times were trivial.