Optimal Feature Set Size in Random Forest Regression

Abstract

One of the most important hyper-parameters in the Random Forest (RF) algorithm is the feature set size used to search for the best partitioning rule at each node of trees. Most existing research on feature set size has been done primarily with a focus on classification problems. We studied the effect of feature set size in the context of regression. Through experimental studies using many datasets, we first investigated whether the RF regression predictions are affected by the feature set size. Then, we found a rule associated with the optimal size based on the characteristics of each data. Lastly, we developed a search algorithm for estimating the best feature set size in RF regression. We showed that the proposed search algorithm can provide improvements over other choices, such as using the default size specified in the randomForest R package and using the common grid search method.

1. Introduction

Random forest [1] is one of the most successful ensemble machine learning methods because it has many advantages. It is suitable for large data with fast model fitting and evaluating, is applicable to both regression and classification, is robust to outliers, can deal with simple linear and complicated nonlinear associations, and produces competitive prediction accuracy for high-dimensional data [2,3]. These unique advantages lead RF to gain increasing interests in a variety of research fields. In particular, it has been actively used and shown great accomplishment in many biomedical applications. For example, RF has been successfully utilized in cancer prognosis and diagnosis [4,5], to predict infectious diseases with high accuracy [6,7], and to recognize disease associated genes in microarray data [8,9].

The RF prediction performance can be further improved by tuning its hyper-parameters, a factor to determine the learning process, which are usually set by users. In the RF algorithm, there are three main hyper-parameters that highly affects the performance [10]: (1) ntrees, which controls how many trees are constructed in the ensemble; (2) nodesize, which decides the size of each tree by controlling minimum number of observations in terminal nodes; and (3) mtry, which controls the feature set size to search for the best split rules at each node of trees. Many studies have been conducted to explore the influence of the three hyper-parameters on the RF prediction performance. Oshiro et al. [11] and Probst and Boulesteix [12] investigated how to obtain the optimal number of trees. Han and Kim [13] examined the effect of tree size and developed a new ensemble method for improving the RF model by growing deeper trees. Bernard et al. [14], Goldstein et al. [15], and Han and Kim [16] studied the impact of the feature set size on the prediction performance in the context of classification.

Feature set size, i.e., mtry, can be considered as the salient hyper-parameter in the sense that it controls the accuracy of individual trees and the diversity between pairs of trees in the ensemble [1]. Setting a large value makes accurate individual trees, but the trees would be similar to each other; setting a small value enhances the diversity between trees, but each tree would have poor prediction accuracy. The two indicators have a trade-off relationship, which must be balanced to achieve the best RF prediction accuracy. Hence, using the appropriate feature set size is the most important task in the RF model fitting to obtain good prediction accuracy. Unfortunately, no theoretical proof has been developed to select the optimal feature set size that provides the highest accuracy, and, in most cases, the default value specified in a software package is used. For example, the default mtry value is $p/3$ for the randomForest R package and p for the RandomForestRegressor in Python’s sklearn.ensemble package, where p denotes the number of features.

Various hyper-parameter optimization algorithms have been applied to obtain the appropriate feature set size. (1) Manual tuning is a traditional optimization method which is performed by users manually. It depends on the trial and error process, so it would be effective only for experienced users. (2) Random search is a method of randomly selecting feature set sizes, evaluating the RF model for each size, and then estimating the optimal size based on RF predictions. It is very fast since only a small number of candidates is considered, but the true optimal size can be missed. (3) Grid search, which has a similar mechanism with the random search, except that every feature set size is explored. It can estimate the true optimal size with high accuracy, but it is computationally intensive. (4) Recently, a Bayesian model-based hyper-parameter optimization algorithm called BOA [17] has been increasingly implemented in machine learning and deep learning communities [18,19]. This method selects the hyper-parameters of the next step using the probability model constructed by the previous step. We propose a novel algorithm that can more efficiently replace the grid search method without using a probability model.

We investigated the effect of feature set size on the RF prediction performance in the context of regression because most existing studies have been developed focusing on classification problems. There are not enough references in regression problems, even though classification and regression are quite different analysis task. Using many datasets, a total of 56 real and artificial datasets, we first investigated whether the RF prediction performance is affected by the feature set size, and then we found a rule associated with the optimal size for each dataset. Finally, we developed a search algorithm that combines a typical grid search and two unique concepts for estimating the best size in RF regression. We compared the proposed method with the typical grid search algorithm. In addition, we compared the size estimated by the proposed method with the optimal size and the default size specified in the randomForest R package.

The rest of the paper is organized as follows. Section 2 begins with a brief introduction to the RF algorithm in regression. Through experimental studies, we first explored the influence of feature set size on the RF prediction performance, and then we examined relationships between an optimal feature set size and characteristics of given datasets. In Section 3, we propose a search algorithm for estimating the best size in regression. In Section 4, we study the impact of our proposed search algorithm on the RF prediction by comparing it with a default size and the optimal size. In addition, we compare the proposed algorithm with a typical grid search method. The paper ends with conclusion in Section 5.

2. Feature Set Size (mtry)

2.1. Random Forest Algorithm in Regression

RF, a popular ensemble machine learning method, is a modified version of bagging [20] since they share conceptually same algorithms to each other, but the main difference is that RF further reduces the variance of a prediction model by combining more de-correlated trees than bagging. The RF algorithm starts with generating bootstrapped samples from a training dataset. Multiple regression trees are fitted on the bootstrapped samples. When the trees are constructed, RF uses a random feature subset instead of all features to find the best split rule at each split of trees. The results of each tree are combined to produce a final prediction. The detailed RF algorithm in regression is described in Algorithm 1.

Algorithm 1 The RF algorithm in regression.

Training Phase: Given:   - D: training set with n observations, p features, and the response variable.   - B: number of regressors in the ensemble. Procedure:   For b = 1 to B   1. Generate bootstrapped sample $D_b^{*}$ from training set D.   2. Grow a regression tree using the bootstrapped sample $D_b^{*}$.    For a given node t,    (i) Randomly sample mtry features from the full features.    (ii) Find the best split rule using the random feature subset.    (iii) Split the node t into two child nodes using the best split rule.    Repeat (i)–(iii) until stopping rules are met.   3. Obtain a trained regressor $R_b$. Test Phase:   For a test instance x, the prediction estimated by the B regressors is given as: $$R\left(x\right)=\frac{1}{B}\sum _{b=1}^B{R}_b\left(x\right)$$

2.2. Influence on Prediction Performance

In the literature, three mtry values, 1, $lo{g}_2\left(p\right)+1$, and $p/3$, have been primarily used in the RF model fitting. The first two were introduced by Breiman [1], and the last was recommended for regression problems by Hastie et al. [2] and Liaw and Wiener [21]. We consider $p/3$ as the default mtry value throughout this paper.

We conducted experimental studies to investigate the influence of the feature set size on the RF prediction accuracy in regression. The experiments were based on 56 real or artificial datasets that were used in other studies or came from the UCI data repository [22].

Table 1. The description of 56 datasets for regression problems.

Name	Observation	Feature	Source
aba	4177	8	UCI (Abalone)
air	1503	5	UCI (Airfoil Self-Noise)
ais	202	12	[23]
alc	2462	18	[24]
ame	3044	21	[25]
app	19,735	25	UCI (Appliances energy prediction)
att	838	9	[26]
aut	398	7	UCI (Auto MPG)
beh	135	17	UCI (Behavior of the urban traffic in Sao Paulo)
blo	208	134	R Library caret (Blood-Brain Barrier)
bos	506	13	R Library mlbench (Boston Housing data)
bre	198	32	UCI (Breast Cancer Wisconsin)
bsd	731	11	UCI (Bike Sharing with count daily)
bsh	17,379	12	UCI (Bike Sharing with count hourly)
bud	1729	10	[27]
can	3775	9	[28]
car	804	17	R Library caret (cars)
cdo	378	9	[29]
com	1994	100	UCI (Communities and Crime)
con	1030	8	UCI (Concrete Compressive Strength)
cou	3114	13	[30]
cox	462	255	R Library caret (Cox-2 Activity)
cps	534	10	[31]
dee	696	13	[31]
dia	366	15	[30]
eec	1296	8	UCI (Energy efficiency with Cooling)
eeh	1296	8	UCI (Energy efficiency with Heating)
ele	10,000	12	UCI (Electrical Grid Stability Simulated)
fac	500	18	UCI (Facebook metrics)
fam	1318	22	[32]
fat	252	17	[33]
fir	517	8	UCI (Forest Fires)
fis	6806	24	[34]
hat	100	13	[35]
hit	322	18	R Library ISLR (Hitters)
ins	5822	85	UCI (Insurance Company Benchmark)
ist	538	7	UCI (ISTANBUL STOCK EXCHANGE)
lab	2953	18	[36]
lah	200	16	[37]
mdp	1495	8	R Library COUNT (medpar)
med	4406	21	[38]
ozo	366	12	R Library mlbench (Ozone)
pks	5875	16	UCI (Parkins Telemonitoring with total-UPDRS)
rat	144	10	[38]
rwm	3874	14	R Library COUNT (rwm1984)

and

Table 2. The description of 56 datasets for regression problems.

Name	Observation	Feature	Source
sce	113	10	[39]
sms	141	10	[39]
sol	1066	10	UCI (Solar Flare with C-class)
spm	395	30	UCI (Student Performance with Math)
spp	649	30	UCI (Student Performance with Portuguese)
tre	100	8	[40]
tri	186	60	[41]
wag	3380	17	[42]
wqr	1599	11	UCI (Wine Quality with red)
wqw	4898	11	UCI (Wine Quality with white)
yac	308	6	UCI (Yacht Hydrodynamics)

contain information about the datasets.

The design of the experiments is as follows: we randomly split a dataset into 60% training set for fitting and 40% test set for evaluating. For the RF model fitting, we used the full range of values from 1 to p as the value of mtry. We set the number of trees as 100, which is a common choice in many RF applications [15,16,43]. For evaluation, root mean squared error (RMSE) was calculated to measure the RF prediction performance. All experiments were repeated 100 times to obtain stable results.

The results are organized with three figures. First, Figure 1 compares the RMSE depending on different mtry values using four representative datasets.

The large fluctuation of RMSE means that the performance of the RF is severely affected by mtry values. Moreover, the results clearly show that the optimal mtry that achieves the smallest RMSE differs from the common choices in all four datasets. Second, in Figure 2, we explore a relative distance, which is defined as (optimal mtry—default mtry)/p, to measure the difference between the optimal size and the default size. There are only four datasets where the default size is matched to the optimal size, and, in most cases, the default size fails to offer the best RF prediction accuracy. In addition, there are more datasets where the optimal size is larger than the default size than there are datasets in the opposite case.

Last, Figure 3 compares the optimal size and the default size by using a relative RMSE, defined as $log\left(\frac{\mathrm{RMSE}\mathrm{with}\mathrm{default}mtry}{\mathrm{RMSE}\mathrm{with}\mathrm{optimal}mtry}\right)$. The relative RMSE greater than 0 means that the optimal size achieves more accurate prediction than the default size. The RF model with the optimal size is obviously more accurate than that with the default size because almost all boxes are located above 0. The performance difference is huge when the optimal size is larger than the default size, but the difference is quite small when the optimal size is smaller than the default size. In summary, through the experimental results, we observed that: (1) the mtry value has an important role in the RF prediction performance; (2) the default mtry value cannot guarantee the best RF performance in regression problems; (3) the optimal mtry value differed dataset to dataset; and (4) there is no clear pattern for the optimal size, thus it is difficult to guess the best mtry value in advance.

2.3. Rule for Optimal Feature Set Size

In this section, we further try to find a rule associated with the optimal mtry using characteristics of datasets. In the context of classification, Bernard et al. [14] and Goldstein et al. [15] observed that the optimal size tends to be related to the number of relevant features, where the relevance means that a feature is significantly associated with the target. If there are many relevant features in a given dataset, a smaller mtry value is preferred to utilize the relevant features equally for splitting nodes in the tree construction. As a result, the trees will be more diverse. In this situation, larger mtry value would cause similar trees in the ensemble because the most dominant feature is repeatedly selected for the split rule at each node. On the other hand, if there are only a few relevant features in a given dataset, a larger mtry value may be advantageous to increase the accuracy of individual trees. However, these observations are limited on classification problems and may not be valid in regression problems. Hence, we sought a relationship between the optimal mtry and the characteristics of given datasets, focusing on the relevance of features, in the context of regression.

How do we measure the relevance between the response variable Y and a feature $X_i$, where $i=1,\cdots ,p$, in regression? The typical Pearson correlation coefficient between Y and $X_i$ can be ineffective because it deals with linear associations only. Hence, we consider a modified Pearson correlation coefficient between Y and $\widehat{Y_i}$, which is a predicted response for $X_i$, obtained by a regression tree between Y and $X_i$. In detail, a decision tree between Y and $X_i$ is fitted with pruning, and then the predicted response $\widehat{Y_i}$ is obtained by applying $X_i$ on the fitted tree. We can capture both linear and nonlinear associations by considering the decision tree [2]. We denote the modified Pearson correlation coefficients as $R_i$, $i=1,\cdots ,p$, and the larger $R_i$ means stronger evidence that $X_i$ is a relevant feature to the response Y.

To investigate whether the optimal mtry is related to the relevance of features, we consider a classification tree, where the target variable $Y_{dt}$ is a binary outcome that is 1 if the optimal size is larger than the default size, and is 0 if the optimal size is smaller than or equal to the default size. We also create a variety of factors $X_{dt}$ to be used in the decision tree: (1) $nrel$, number of relevant features; (2) $nirr$, number of irrelevant features; (3) $scor$, standard deviation of $R_i$; (4) $mcor$, mean of $R_i$; (5) $ccor$, coefficient of variation of $R_i$; and (6) $pn$, ratio of features and observations. The definitions of $Y_{dt}$ and $X_{dt}$ are summarized in

Table 3. Description of $Y_{dt}$ and $X_{dt}$ for classification tree analysis. The definitions of $X_{dt}$ are as follows: (1) $nrel$, the number of relevant features ($Q_3\left(R\right)$: the third quartile of R); (2) $nirr$, the number of irrelevant features ($Q_1\left(R\right)$: the first quartile of R); (3) $scor$, the standard deviation of R; (4) $mcor$, the mean of R; (5) $ccor$, the coefficient of variation of R; and (6) $pn$, the ratio of features and observations. Note that R is the collection of $R_i$, $i=1,...,p$.

Attribute	Name	Definition
Target ($Y_{dt}$)	$Y_{dt}$	$Y_{dt}=1$ if Optimal $mtry$ > Default $mtry$
		$Y_{dt}=0$ if Optimal $mtry$ ≤ Default $mtry$
Factor ($X_{dt}$)	$nrel$	$Length(R>Q_3\left(R\right))/p$
	$nirr$	$Length(R<Q_1\left(R\right))/p$
	$scor$	$Sd\left(R\right)$
	$mcor$	$Mean\left(R\right)$
	$ccor$	$Sd\left(R\right)/Mean\left(R\right)$
	$pn$	$p/n$

.

Figure 4 depicts the classification tree with pruning between $Y_{dt}$ and $X_{dt}$ based on a total of 56 datasets, where $scor$, standard deviation of $R_i$, is the most significant factor to discriminate the 56 datasets. The result shows that, for $scor<0.135$, there are many datasets with the optimal size less than or equal to the default size; for $scor\ge 0.135$, most datasets have an optimal size larger than the default size. These results can be interpreted as follows. If features in a given dataset have similar relevance to each other, the optimal size tends to be small, thus setting smaller mtry value can be helpful to achieve good prediction performance. Conversely, if features are widely distributed from irrelevant to relevant, the optimal size tends to be large, thus setting larger mtry value may be preferred.

We found that the optimal feature set size may be related to the relevance of features in a given dataset, but it is not enough to select a specific mtry in RF model fitting. Hence, in the next section, we develop a search algorithm for estimating the best mtry in RF regression.

3. Search Algorithm for Optimal Feature Set Size in Random Forest Regression

We develop a search algorithm for estimating the best mtry in RF regression using the findings in Section 2. This algorithm is also motivated by Han and Kim [16].

Our proposed algorithm combines a typical grid search method and two unique concepts: (1) SearchDirection, which controls the direction of search, “forward” and “backward”; and (2) SearchSize, which decides how many features are searched at a time, set $ceiling(p/10)$ by default. These two concepts make our proposed search algorithm more efficient than the typical grid search method by reducing the number of searches. The proposed algorithm searches using the out-of-bag mean squared error (OOB-MSE), which is computed on the out-of-bag samples. The OOB-MSE estimate is known as a good estimate for the true MSE [1]. In the SearchDirection = “forward” setting, the proposed algorithm starts the searches from $mtry=floor(p/3)$ to p in increasing order. Note that, if SearchDirection is “backward”, the searches are performed from $mtry=floor(p/3)$ to 1 in decreasing order.

We now describe our proposed search algorithm using a simple example. Suppose that SearchDirection is “forward" and a dataset has 15 features ($p=15$); then, SearchSize is set as $ceiling(15/10)=2$ by default. The proposed method first computes the OOB-MSE of the RF model with $mtry=(15/3)=5$, which is denoted as ${\widehat{e}}_5$. Then, it compares ${\widehat{e}}_5$ with the next two OOB-MSEs, ${\widehat{e}}_6$ and ${\widehat{e}}_7$, which are obtained by the RF model with $mtry=6$ and $mtry=7$, respectively, given that SearchSize is 2. If ${\widehat{e}}_5$ is smaller than the minimum of ${\widehat{e}}_6$ and ${\widehat{e}}_7$, the algorithm estimates the best mtry as 5 and terminates the search. If ${\widehat{e}}_5$ is larger than one of ${\widehat{e}}_6$ and ${\widehat{e}}_7$, the minimum of ${\widehat{e}}_6$ and ${\widehat{e}}_7$ is continuously compared with the next two OOB-MSE, ${\widehat{e}}_8$ and ${\widehat{e}}_9$. The algorithm is reiterated until stopping criteria are met. The detailed search algorithm is in Algorithm 2.

Algorithm 2 A search algorithm for estimating the best mtry in RF regression.

Require:$SearchDirection\leftarrow ‘‘forward"$ or $‘‘backward"$, and $F\leftarrow floor(p/3)$ Require: $SearchSize$←$ceiling(p/10)$ Require: ${\widehat{e}}_f$: out-of-bag Mean Squared Error with $mtry=f$ Procedure:   1: if $SearchDirection=‘‘backward"$ then   2:  Compute ${\widehat{e}}_F$   3:  for x from $(F-1)$ to $(F-SearchSize)$ with decrement 1   4:   if there is ${\widehat{e}}_x$ already then skip   5:   else compute ${\widehat{e}}_x$   6:   end if   7:  end for   8:  j←$argmi{n}_j$(${\widehat{e}}_j$|j∈ [$F-SearchSize,\cdots ,F-1$])   9:  if ${\widehat{e}}_j<{\widehat{e}}_F$thenF←j and go to 3   10: endif   11: if $SearchDirection=‘‘forward"$ then   12:  Compute ${\widehat{e}}_F$   13:  for x from $(F+1)$ to $(F+SearchSize)$ with increment 1   14:   if there is ${\widehat{e}}_x$ already then skip   15:   else compute ${\widehat{e}}_x$   16:   end if   17:  end for   18:  j←$argmi{n}_j$(${\widehat{e}}_j$|j∈ [$F+1,\cdots ,F+SearchSize$])   19:  if ${\widehat{e}}_j<{\widehat{e}}_F$thenF←j and go to 13   20: endif   21: Return F

4. Experimental Study

This section studies the impact of our proposed search algorithm by comparing the RF prediction accuracy using an estimated mtry by our proposed algorithm, with that using the optimal mtry, the default mtry, and an estimated mtry by a typical grid search method. We determine the SearchDirection by the results of the classification tree in Figure 4. Specifically, if a dataset falls into the node number 2, the SeachDirection is assigned as “backward”; if a dataset falls into the node number 3, the SearchDirection is assigned as “forward".

4.1. Estimated Size vs. Optimal Size

The results of comparison between the estimated mtry by our proposed algorithm and the optimal mtry are presented with two figures. First, Figure 5 compares the relative distance between the estimated mtry and the optimal mtry, where the relative distance between the optimal mtry and the default mtry is added above as a reference. The two subfigures clearly show that the estimated mtry is closer to the optimal mtry than the default, and, in 23 out of 56 datasets, the estimated mtry is exactly matched to the optimal mtry. It demonstrates that the proposed search algorithm performs well to estimate the optimal mtry. Second, Figure 6 compares the relative RMSE between the estimated mtry and the optimal mtry. Since almost all boxes are located near 0, it seems that the estimated mtry and the optimal mtry produce similar RF prediction performances.

4.2. Estimated Size vs. Default Size

Figure 7 compares the relative RMSE between the estimated mtry and the default. The results show that using the estimated mtry offers improvement over the default size in terms of the RF prediction because many boxes are located above 0. Interestingly, the distribution of the boxes is quite similar with that in Figure 3. It confirms that the optimal mtry and the estimated mtry have similar performance. We further conducted the paired t-tests based on the 100 RMSE pairs of the estimated mtry and the default mtry for each dataset. We observed that the RF prediction using the estimated mtry is statistically better than that using the default mtry on 38 out of 56 datasets, and, for the other 18 datasets, their differences are not statistically significant. There is no dataset where the default mtry produces better prediction than the estimated mtry.

4.3. Comparison between the Proposed Search and Typical Grid Search

Figure 8 compares the estimated mtry by the proposed search algorithm and the one by the typical grid search method in terms of prediction accuracy and computational cost. The typical grid search method can be considered as a special version of the proposed search algorithm, where the searches are always performed from $mtry=1$ to p with the Searchsize of p. The results demonstrate that, while the RF prediction accuracies using the two estimated sizes are not different because almost all boxes are located near 0, the proposed search algorithm is much faster than the typical grid search method.

5. Conclusions

We focused on the influence of mtry on the RF prediction in regression problems because most existing studies about mtry have been primarily conducted focusing on classification problems. Using many datasets, a total of 56 real or artificial datasets, we carried out several experiments and observed that mtry has a significant impact on the RF regression prediction. In addition, the default mtry of $p/3$ cannot guarantee the highest accuracy in regression.

We also found that it is difficult to find a specific rule for estimating the best mtry since the optimal size seems domain dependent. Fortunately, we found a relationship between the optimal size and characteristics of datasets. If the features of the dataset have similar relevance to each other, the optimal size tends to be relatively small, thus smaller mtry values are preferred. In the opposite case, larger mtry values are preferred.

We developed a search algorithm for estimating the optimal mtry in RF regression, which is motivated from a method for RF classification [16]. The proposed search algorithm combines a typical grid search method and two unique concepts of “SearchDirection” and “SearchSize”. The two additional concepts allow our proposed search algorithm to estimate the best mtry more efficiently than the typical grid search method. In the experimental studies, we demonstrated that the estimated size by the proposed algorithm is close to the optimal size and produces better prediction accuracy than the default size. We also confirmed that the proposed algorithm provides similar prediction accuracy to the typical grid search method, but it is more efficient in terms of computational cost.