2.4.1. Bagging

Bagging (bootstrap aggregation) is used when our goal is to reduce the variance of a decision tree. In this process, the basic idea is to generate several subsets of data from the training sample, which is selected randomly by replacement. Each subset of data is used to train the corresponding decision tree model. As a result, we end up with a set of different models. Finally, the average of all predictions obtained from different trees is used, which is more powerful and accurate than a single decision tree (Figure 1).

## 2.4.2. Boosting

Boosting is another ensemble technique that aims to improve the accuracy of predictions generated by one or many models. This technique starts by fitting an initial model (e.g., a tree or linear regression) to the data. Then, a second model is constructed that focuses on accurately predicting cases where the first model does not perform well by using a weighted data sample. The combination of these two models is better than either individual model separately. The boosting process is then repeated several times. Each

successive model attempts to correct the weaknesses and errors of the combined boosted set of all of the previous models (Figure 2). Combining the entire set at the end converts the weak learners into a better performing model.

**Figure 1.** Bagging [16].

#### **Figure 2.** Boosting [16].
