3.4.5. Model Selection

Each model type, which is defined by one of the parameter combinations shown in Tables 5 and 6, will be instantiated 10 times. The reason for this is to investigate the stability of each model type and to reduce the impact of randomness, which is prevalent in each of the statistical model frameworks. In RF, the randomness is introduced by random selection of data points for each tree and the random selection of input variables to determine each split. In ANN, the randomness is partly governed by the random selection of the initial values of the network weights.

The aggregate statistical metrics based on the 10 model instances of each model type are presented in Table 7.

**Table 7.** The adjusted-*R*<sup>2</sup> and error metric variants that are used to evaluate the performance of the aggregated model instances.


To determine the stability of the models, which is influenced by the underlying randomness, the idea behind the model selection criteria was to keep the difference between *R*¯ <sup>2</sup> *max* and *R*¯ <sup>2</sup> *min* as low as possible. Hence, the algorithm for selection the best model type of each variable batch can be expressed as follows:


The adjusted-*R*<sup>2</sup> was chosen to determine the stability of the models because it indicates the goodness of fit. The other error metrics, based on the model error, do not relate to goodness of fit.

Using the above algorithm, the number of models will be reduced to 8, which is equal to the number of variable batches. From this model subset, the model with the fewest number of input variables will be selected, given that more than one model have the same *R*¯ <sup>2</sup> *<sup>μ</sup>*-value. This ensures that the model selection adheres to the concept of model parsimony, i.e., that the simplest model, which is the model with least number of input variables, is selected when more than one model have the same performance.
