*Article* **Tree-Based Classifier Ensembles for PE Malware Analysis: A Performance Revisit**

**Maya Hilda Lestari Louk <sup>1</sup> and Bayu Adhi Tama 2,\***


**\*** Correspondence: bayu@umbc.edu

**Abstract:** Given their escalating number and variety, combating malware is becoming increasingly strenuous. Machine learning techniques are often used in the literature to automatically discover the models and patterns behind such challenges and create solutions that can maintain the rapid pace at which malware evolves. This article compares various tree-based ensemble learning methods that have been proposed in the analysis of PE malware. A tree-based ensemble is an unconventional learning paradigm that constructs and combines a collection of base learners (e.g., decision trees), as opposed to the conventional learning paradigm, which aims to construct individual learners from training data. Several tree-based ensemble techniques, such as random forest, XGBoost, CatBoost, GBM, and LightGBM, are taken into consideration and are appraised using different performance measures, such as accuracy, MCC, precision, recall, AUC, and F1. In addition, the experiment includes many public datasets, such as BODMAS, Kaggle, and CIC-MalMem-2022, to demonstrate the generalizability of the classifiers in a variety of contexts. Based on the test findings, all tree-based ensembles performed well, and performance differences between algorithms are not statistically significant, particularly when their respective hyperparameters are appropriately configured. The proposed tree-based ensemble techniques also outperformed other, similar PE malware detectors that have been published in recent years.

**Keywords:** portable executable malware; tree-based ensemble; performance comparison; statistical significance test

## **1. Introduction**

Malware (e.g., malicious software) is commonly recognized as one of the most potent cyber threats and hazards to modern computer systems [1,2]. It is an overarching word that refers to any code that potentially has a destructive, harmful effect [3]. On the basis of their behavior and execution processes, malicious softwares are categorized as worms, viruses, Trojan horses, rootkits, backdoors, spyware, logic bombs, adware, and ransomware. Computer systems are hacked for a variety of reasons, including the destruction of computer resources, financial gain, the theft of private and confidential information and the use of computing resources, as well as the inaccessibility of system services, to name a few [4].

Malware is recognized using signature-based or behavior-based methods. The signaturebased malware detection techniques are quick and effective, but obfuscated malware can quickly circumvent them. In contrast, behavior-based methods are more resistant to obfuscation. Nonetheless, behavior-based methods are relatively time-intensive. Therefore, in addition to the signature-based and behavior-based malware detection techniques, numerous fusion techniques exist that contain the benefits of both [5,6]. The goal of these fusion strategies is to address the shortcomings of signature and behavior-based approaches.

While we work to defend ourselves from malware, cybercriminals continue to create increasingly complex techniques to obtain and steal data and resources. Conventional

**Citation:** Louk, M.H.L.; Tama, B.A. Tree-Based Classifier Ensembles for PE Malware Analysis: A Performance Revisit. *Algorithms* **2022**, *15*, 332. https://doi.org/10.3390/a15090332

Academic Editors: Francesco Bergadano and Giorgio Giacinto

Received: 29 August 2022 Accepted: 14 September 2022 Published: 17 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

methods (i.e., rule-based, graph-based, and entropy-based) for analyzing and detecting malware focus on matching known malicious signatures to alleged malicious programs. Such static solutions require a known harmful signature, rendering them unsatisfactory against new (e.g., zero-day) attacks, and depend on end users to maintain system updates. Attackers are aware that these methods may also be vulnerable to obfuscation, such as code obfuscation to avoid detection against known signatures [7]. Hence, it is necessary to update and build malware detection mechanisms that are capable of withstanding significant attacks [8].

Machine learning offers the potential to construct malware detectors that are capable of combating newer versions of malware, and different supervised and unsupervisedalgorithm-based machine learning methods have been reported in the literature [9–11]. More specifically, ensemble learning approaches have been utilized and achieved excellent results in malware detection [12–17]. In most cases, ensemble learning algorithms yield superior results as compared to individual classification algorithms, i.e., support vector machine, decision tree, naive Bayes, and neural networks. However, although classifier ensembles demonstrate a significant performance, the majority of these ensembles are deployed in a restricted manner without adequate hyperparameter tuning. Moreover, the performance of classifier ensembles is validated using a single dataset; consequently, no generalizable results are produced.

The tree-based ensemble technique is an ensemble learning paradigm in which a collection of base learners (e.g., decision trees or CART) are constructed and combined from the training data [18]. For instance, random forest [19] is comprised of a large number of individual decision trees that operate as an ensemble. It uses feature randomness to generate an uncorrelated forest of decision trees. In a similar fashion, the gradient boosting decision tree algorithms combine a collection of individual decision trees to form an ensemble. However, unlike random forest, the decision trees in gradient boosting are constructed serially (e.g., additively). Gradient boosting decision tree algorithms have recently been proposed and have demonstrated remarkable results in many domains, such as protein–protein interaction prediction [20], neutronic calculation [21], human activity recognition [22], etc. However, their performance in classifying and detecting malware remains questionable. This motivated us to employ ensembles of tree-based algorithms to classify PE malware. This paper makes the following contributions to the current literature.


The remainder of the paper is structured as follows. An overview of PE malware detection based on classifier ensembles is provided in Section 2. Next, we present the background of tree-based classifier ensembles and datasets in Section 3. Section 4 discusses the experimental results, and in the end, Section 5 concludes the paper.

## **2. Related Work**

Ucci et al. [7], Maniriho et al. [10] provide the machine learning taxonomy for malware analysis, while [11] present an overview of malware analysis in CPS and IoT. Malware analysis can be accomplished via either static or dynamic analysis, or a mix of the two, depending on how the information extraction procedure is carried out. Approaches based on static analysis evaluate the content of samples without necessitating their execution, whereas dynamic analysis examines the behavior of samples by executing them. This study analyzes a static analysis of PE files, since it can yield a plethora of useful information, e.g., the compiler and symbols used.

Meanwhile, machine learning techniques were largely employed in malware detection [29,30]. Malware samples were examined and the extracted features are used to train the classification algorithm. An overview of the machine learning techniques used for the classification of malware is provided in the following. We particularly explore malware detectors that employ at least one ensemble learning technique. Vadrevu et al. [31], Mills et al. [17], Uppal et al. [32], Kwon et al. [33] utilized random forest for malware detection based on PE file characteristics and networks. Furthermore, Mao et al. [34], Wüchner et al. [35], Ahmadi et al. [36] developed a random forest classifier to detect malware using various features, such as system calls, file system, and Windows registry. Amer and Zelinka [13] proposed an ensemble learning strategy to address the shortcomings of the existing commercial signature-based techniques. The proposed technique was able to focus on the most salient features of malware PE files by lowering the dimensionality of the data. Dener et al. [37] and Azmee et al. [38] compared the use of various machine learning algorithms to detect PE malware and showed that XGBoost and logistic regression were the best-performing methods.

Liu et al. [39] employed data visualization and adversarial training on ML-based detectors to effectively detect the various types of malware and their variants in order to address the current issues in malware detection, such as the consideration of attacks from adversarial examples and the massive growth in malware variants. In [40], a deep feature extraction technique for malware analysis was addressed in light of the current progress in deep learning. Deep features were obtained from a CNN and were fed to an SVM classifier for malware classification. Moreover, a CNN ensemble for malware classification was proposed in [15,16]. The proposed architecture was constructed in a stacked fashion, with a machine learning algorithm providing the final classification. A meta-classifier was selected after various machine learning algorithms were analyzed and evaluated. Most recently, Hao et al. [41] proposed a CNN-based feature extraction and a channel-attention module to reduce the information loss in the process of feature image generation of malware samples. Specific deep learning architectures, such as a deep belief network and transformer-based classifier, were also considered when classifying Android [42,43] and PE malware [44], respectively. Table 1 presents a summary of the existing malware detectors described in the literature.

**Table 1.** Summarization of the existing PE malware detectors.



**Table 1.** *Cont.*

## **3. Materials and Methods**

This study evaluates the performance of ensembles of tree-based classifiers in detecting PE malware. Figure 1 depicts the stages involved in our comparative analysis. Several tree-based ensemble approaches are trained on three distinct PE malware training datasets in order to generate classification models. The performance of classification models is then determined by validating them on a testing dataset. Finally, a two-step statistical significance test is then utilized to evaluate the performance benchmarks. In the following section, we provide a brief summary of the malware datasets and tree-based classifier ensembles utilized in this study.

**Figure 1.** Performance comparison methodology of tree-based ensembles for PE malware detection.

## *3.1. Datasets*

One of the most problematic aspects of using machine learning to solve malware detection problems is producing a realistic feature set from a large variety of unidentified portable executable samples. In essence, the dataset used to train machine learning models determines their level of sophistication. Hence, developing a solid, labeled dataset that represents all analyzed samples is more helpful for malware detection. In light of this, we utilize more recent public datasets that depict the characteristics and attack behaviors of contemporary malware:

(a) BODMAS [27]

The dataset contains 57,293 malicious and 77,142 benign samples (134,435 in total). The malware samples were arbitrarily picked each month from the internal malware database of a security company. The data were collected between 29 August 2019 and 30 September 2020. The benign samples were gathered between 1 January 2007 and 30 September 2020. In order to reflect benign PE binary distribution in realworld traffic, the database of the security company is also processed for benign samples. In addition, SHA-256 hash, the actual PE binary, and a pre-extracted feature vector were given for each malicious sample, whereas only SHA-256 hash and the pre-extracted feature vector were provided for each benign sample. BODMAS is comprised of 2381 input feature vectors and 1 class label feature, of which 0 is labeled as benign and 1 is labeled as malicious.

(b) Kaggle (https://tinyurl.com/22z7u898, access on 25 August 2022)

The dataset was developed using a Python library called *pe f ile* (https://tinyurl. com/w75zewvr, accessed on 25 August 2022), which is a multi-platform module used to parse and work with PE files. Kaggle dataset contains 14,599 malicious and 5012 benign samples (19,611 in total). The dataset is comprised of 78 input features, denoting PE header files and one class label attribute.

(c) CIC-MalMem-2022 [28]

Unlike the two above-mentioned datasets, CIC-MalMem-2022 is an obfuscated malware dataset that is intended to evaluate memory-based obfuscated malware detection algorithms. The dataset was designed to mimic a realistic scenario as accurately as possible using reowned malware. Obfuscated malware comprises malicious software that conceals itself to escape detection and eradication. The dataset consists of an equal ratio of malicious and benign memory dumps (58,596 samples in total). In addition, CIC-MalMem-2022 is made up of 56 features that serve as inputs for machine learning algorithms.

## *3.2. Tree-Based Ensemble Learning*

The tree-based ensemble is a non-ordinary learning paradigm that constructs and combines a set of base learners (e.g., decision trees or CART) as opposed to the commonplace learning paradigm that attempts to construct individual learners from training data. Normally, an ensemble is formed in two processes, i.e., by first producing the base learners and then integrating them. For a decent ensemble, it is commonly considered that the base learners must be as accurate and diversified as possible [18]. This study considers four tree-based ensemble learning algorithms. It is worth mentioning that tuning the hyperparamters for each algorithm is carried out using *random search* approach [45].

(a) Random forest [19]

As its name implies, a random forest is a tree-based ensemble in which each tree is dependent on a set of random variables. The original formulation of random forest algorithm provided by Breiman [19] is as follows. A random forest employs trees *hj*(X , Ω) as its base learners. For training data D = {(**x**1, *y*1), . . . ,(**x***α*, *yα*)}, where **x***<sup>i</sup>* = (**x***i*,1, . . . , **x***i*,*p*) *T* represents the *p* predictors and *y<sup>i</sup>* represents the response, and a specific manifestation *ω<sup>j</sup>* of Ω*<sup>j</sup>* , the fitted tree is given as ˆ*hj*(**x**, *ω<sup>j</sup>* , D). More precisely, the steps involved in the random forest algorithm are described in Algorithm 1.

We use a fast random forest implementation called *ranger* [46], available in *R*, which is suitable for high-dimensional data such as ours. The list of random forest's hyperparameters for each malware dataset is provided in Table 2. We set the search space for each hyperparameter tuning is as follows. Number of trees = {50, 100, 250, 500, 750, 1000}, split rule = {'gini','extratrees'}, minimum node size = {1, 2, . . . , 10}, *mtry* = number of features × {0.05, 0.15, 0.25, 0.333, 0.4}, sample fraction = {0.5, 0.63, 0.8}, and *replace* = {TRUE, FALSE}.


**Training**: **Require**: Original training set D = {(**x**1, *y*1),(**x**2, *y*2), . . . ,(**x***α*, *yα*)}, with **x***<sup>i</sup>* = (**x***i*,1, . . . , **x***i*,*p*) *T* 1. **for** *j* = 1 to *J* 2. Perform a bootstrap sample D*<sup>j</sup>* of size *α* from D. 3. Using binary recursive partitioning, fit a tree on D*<sup>j</sup>* . 4. **end for Testing**: **Require**: An instance to be classified **x**. 1. ˆ *f*(**x**) = arg max*<sup>y</sup>* ∑ *J j*=1 *I*( ˆ*hj*(**x**) = *y*) where ˆ*hj*(**x**) denotes the response variable at **x** using the *j*-th tree.

**Table 2.** The final learning parameters of random forest used for each dataset after performing a *random search*.


(b) Gradient Boosting Decision Trees

In this paper, we also considered various tree-based boosting ensemble approaches for malware detection, such as XGBoost [23], CatBoost [24], GBM [25], and LightGBM [26]. As a rule, GBDT ensembles are a linear additive model, where a tree-based classifier (e.g., CART) was utilized as their base model. Let D = {(**x***<sup>i</sup>* , *yi*)|*i* ∈ {1, . . . , *α*}, **<sup>x</sup>***<sup>i</sup>* <sup>∈</sup> <sup>R</sup>*<sup>η</sup>* , *y<sup>i</sup>* ∈ R} denote the malware dataset comprising *η* features and *α* samples. Considering a collection of *j* trees, the prediction output *y*(**x**ˆ) *j* for an input **x** is obtained by calculating the predictions from each tree *y*(**x**ˆ) *j* , as shown in the following formula.

$$y(\hat{\mathbf{x}})^j = \sum\_{i=1}^j f\_i(\mathbf{x}) \tag{1}$$

where *f<sup>i</sup>* represents the output of the *i*-th regression tree of the *j*-tree ensemble. GBDTs minimize a regularized objective function *Obj<sup>t</sup>* in order to create the (*j* + 1)-th tree, as follows.

$$\min\{Obj(f)^{t}\} = \min\{\Omega(f)^{t} + \Theta(f)^{t}\} \tag{2}$$

where Ω(*f*) *t* represents loss function and Θ(*f*) *t* is a regularization function to control over-fitting. The loss function Ω(*f*) *<sup>t</sup>* measures the difference between the prediction *y*ˆ*<sup>i</sup>* and the target *y<sup>i</sup>* . On the other hand, the regularization function is defined as Θ(*f*) *<sup>t</sup>* = *γT* + <sup>1</sup> 2 *λ* k *w* k 2 , where *T* and *w* indicate the number of leaves and leaf weights in the tree, respectively.

## (i) XGBoost [23]

XGBoost is a scalable end-to-end tree-boosting strategy that generates a large number of sequentially trained trees. Each succeeding tree corrects the errors made by the preceding one, resulting in an efficient classification model. Through sparsity-aware metrics and multi-threading approaches, XGBoost not only addresses the algorithm's overfitting problem, but also boosts the speed of most real-world computational tasks. This study utilizes two different XGBoost implementations, such as native implementation in *R* [47] and *H*2*O* [48]. We set the search space of native XGBoost's hyperparameters are as follows. Maximum depth = {2, 3, . . . , 24}, eta = {0, 0.1, 0.2, . . . , 1.0}, subsample = {0.5, 0.6, 0.7, 0.8}, and column sample by tree = {0.5, 0.6, 0.7, 0.8, 0.9}. Moreover, we set the search space of XGBoost's hyperparameters implemented in *H*2*O* are as follows. Maximum depth = {1, 3, 5, . . . , 29}, sample rate = {0.2, 0.3, . . . , 1}, column sample rate = {0.2, 0.21, 0.22, . . . , 1}, column sample rate per tree = {0.2, 0.21, 0.22, . . . ,1}, and minimum rows = {0, 1, . . . , log2× number of rows-1}. The final learning parameters for both XGBoost implementations are presented in Table 3.

**Table 3.** The final learning parameters of XGBoost used for each dataset after performing a *random search*.


## (ii) CatBoost [24]

CatBoost is built with symmetric decision trees. It is acknowledged as a classification algorithm that is capable of producing an excellent performance and ten times the prediction speed of methods that do not employ symmetric decision trees. CatBoost, unlike other GBDT algorithms, is able to accommodate gradient bias and prediction shift to increase the accuracy of predictions and generalization ability of large datasets. In addition, CatBoost is comprised of two essential algorithms: ordered boosting, which estimates leaf values during tree structure selection to avoid overfitting, and a unique technique for handling categorical data throughout the training process. An implementation of CatBoost in *R* is employed in this paper, whereas the search space of each hyperparameter is considered as follows. Depth = {1, 2, . . . , 10}, learning rate = {0.03, 0.001, 0.01, 0.1, 0.2, 0.3}, l2 leaf regularization = {3, 1, 5, 10, 100}, border count = {32, 5, 10, 20, 50, 100, 200}, and boosting type = {"Ordered", "Plain"}. The final learning parameters of CatBoost for each malware dataset are given in Table 4.


**Table 4.** The final learning parameters of CatBoost used for each dataset after performing a *random search*.

## (iii) Gradient boosting machine [25]

GBM is the first implementation of GBDT to utilize a forward learning technique. Trees are generated in a sequential manner, with future trees being dependent on the results of the preceding trees. Formally, GBM is achieved by iteratively constructing a collection of functions *f* 0 , *f* 1 , . . . , *f t* , given a loss function Ω(*y<sup>i</sup>* , *f t* ). We can optimize our estimates of *y<sup>i</sup>* by discovering another function *f <sup>t</sup>*+<sup>1</sup> = *f <sup>t</sup>* + *h t*+1 (**x**), such that *h t*+1 reduces the estimated value of the loss function. In this study, we adopt GBM implementation in H2O, whereas the hyperparameters' search space is specified as follows. Maximum depth = {1, 3, 5, . . . , 29}, sample rate = {0.2, 0.3, . . . , 1}, column sample rate per tree = {0.2, 0.21, 0.22, . . . , 1}, column sample rate change per level = {0.9, 0.91, . . . , 1.1}, number of bins = 2{4,5,...,10} , and minimum rows = {0, 1, . . . , log2× number of rows − 1}. Table 5 shows a list of all the final GBM hyperparameters that were used on each malware dataset.

**Table 5.** The final learning parameters of GBM used for each dataset after performing *random search*.


## (iv) LightGBM [26]

LightGBM is an inexpensive gradient boosting tree implementations that employs histogram and leaf-wise techniques to increase both processing power and prediction precision. The histogram method is used to combine features that are incompatible with each another. Before generating a *n*width histogram, the core idea is to discretize continuous features into *n* integers. Based on the discretized values of the histogram, the training data are scanned to locate the decision tree. The histogram method considerably reduces the runtime complexity. In addition, in LightGBM, the leaf with the greatest splitting gain was found and then divided using a leaf-by-leaf strategy. Leaf-wise optimization may result in overfitting and a deeper decision tree. To ensure great efficiency and prevent overfitting, LightGBM includes a maximum depth constraint to leaf-wise. In this study, we employed a LightGBM implementation in *R* with the following hyperparameter search space; Maximum bin = {100, 255}, maximum depth = {1, 2, . . . , 15}, number of leaves = 2({1,2,...,15} , minimum data in leaf = {100, 200, . . . , 1000}, learning rate = {0.01, 0.3, 0.01}, lambda l1 = {0, 10, 20, . . . , 100}, lambda l2 = {0, 10, 20, . . . , 100}, feature fraction = {0.5, 0.9}, bagging fraction = {0.5, 0.9}, path smooth = {1 <sup>×</sup> <sup>10</sup>−<sup>8</sup> , <sup>1</sup> <sup>×</sup> <sup>10</sup>−<sup>3</sup> }, and minimum gain to split = {0, 1, 2, . . . , 15}. Table 6 contains the list of all final LightGBM hyperparameters used for each malware dataset.


**Table 6.** The final learning parameters of LightGBM used for each dataset after performing *random search*.

## **4. Result and Discussion**

This section analyzes and discusses the results of the tree-based classifier ensembles applied to malware classification. The results of exploratory analysis are presented first, followed by a performance comparison between the tree-based ensemble models.

## *4.1. Exploratory Analysis*

We first provide a correlation analysis between multiple variables in each malware dataset. Figure 2 shows the correlation coefficient score matrix measured by Pearson correlation. Correlation analysis is useful to understand the relationship between variables in a dataset, since the Good input features of a dataset should have a high correlation with target features, but should be uncorrelated with each other. Figure 2 confirms that both BODMAS and Kaggle datasets have fewer uncorrelated features than CIC-MalMem-2022. Hence, to mitigate the curse of dimesionality, it is strongly recommended to employ feature selection before employing a machine learning method on CIC-MalMem-2022. Highly correlated features have a negligible effect on the output prediction but raise the computational cost.

**Figure 2.** Feature correlation analysis of each malware dataset: (**a**) BODMAS, (**b**) Kaggle, and (**c**) CIC-MalMem-2022.

In addition, we ran a t-SNE algorithm [49] with a learning rate = 5000 and perplexity = 100. The t-SNE is an approach that converts a set of high-dimensional points to two dimensions in such a way that, ideally, close neighbors remain close and far points remain far. Figure 3 provides a spatial representation of the dataset in two dimensions. The t-SNE provides a pliable border between the local and global data structures. It also estimates the size of each datapoint's local neighborhood based on the local density of the data by requiring each conditional probability distribution to have the same perplexity (e.g., Gaussian kernel). Furthermore, Figure 3 demonstrates that both BODMAS and Kaggle datasets are highly imbalanced as compared with CIC-MalMem-2022.

**Figure 3.** Two-dimensional visualization of instance pairs using t-SNE technique of each malware dataset: (**a**) BODMAS, (**b**) Kaggle, and (**c**) CIC-MalMem-2022.

## *4.2. Comparison Analysis*

In the experiment, we employed a *k* cross-validation technique (*k* = 10), where the final performance outcome for each tree-ensemble model is the mean of the ten folds. The performance of each model was measured based on six performance metrics, such as accuracy, MCC, precision, recall, AUC, and F1. These metrics are chosen to provide more accurate estimates of the behavior of the classifier ensembles under the experiment. Especially, Chicco et al. [50] have shown that MCC is more informative than accuracy and F1, which yield reliable estimates when used to balanced datasets, but misleading outcomes when applied to imbalanced data sets. For a binary classification problem, the outcome of a tree-based classifier ensemble is typically derived from a contigency matrix, T = *TP FN FP TN* , where *TP* is true positive, *FN* is false negative, *FP* is false positive, and *TN* is true negative. Let *ξ*<sup>+</sup> = *TP* + *FN* and *ξ*<sup>−</sup> = *TN* + *FP* be the number of samples labeled as malware and non-malware, respectively. Hence, the performance

$$\stackrel{\smile}{\text{of samples labeled as malware and non-malware, respectively. Hence, the performance metrics used in this study can be calculated as follows.}$$

$$\text{Accuracy} = \frac{TP + TN}{\mathfrak{f}\_{+} + \mathfrak{f}\_{-}} \tag{3}$$

$$\text{MCC} = \frac{TP \times TN - FP \times FN}{((TP + FP) \times \xi\_- \times (TN + FN) \times \xi\_+)^{1/2}} \tag{4}$$

$$\text{Precision} = \frac{TP}{TP + FP} \tag{5}$$

$$\text{Recall} = \frac{TP}{\mathfrak{f}\_+} \tag{6}$$

$$\text{AUC} = \int\_0^1 \text{Recall} \times \frac{FP}{\tilde{\xi}\_-} d\frac{FP}{\tilde{\xi}\_-} = \int\_0^1 \text{Recall} \times \left(\frac{FP}{\tilde{\xi}\_-}\right)^{-1} (x) dx \tag{7}$$

$$\text{F1} = \frac{\text{2TP}}{\text{2TP} + \text{FP} + \text{FN}} \tag{8}$$

Figure 4 presents the performance score of each algorithm on each dataset. Overall, considering MCC as a performance indicator, LightGBM is the worst-performing algorithm, while XGBoost (native) is the best-performing on BODMAS and Kaggle datasets, followed by GBM (H2O). Interestingly, random forest has also performed well on the remaining dataset. Using accuracy as a performance metric, it is also apparent that there are modest performance disparities amongst algorithms (e.g., all algorithms achieve 100% accuracy). Consequently, our results support the findings stated by [50]. In Table 7, we provide the performance average of each algorithm over various datasets and demonstrate that XGBoost (native) is superior to any competitors on the board in terms of accuracy, MCC, and precision metrics. On the other hand, when recall, AUC, and F1 metrics are utilized, GBM (H2O) shows a superior performance.

**Figure 4.** Performance comparison of various tree-based ensemble models on different datasets, i.e., (**a**) BODMAS, (**b**) Kaggle, and (**c**) CIC-MalMem-2022.


**Table 7.** Performance average of each ensemble technique over various malware datasets.

This section includes a two-step statistical significance test using Quade omnibus test and Quade post-hoc test [51] to better comprehend the performance difference between tree-based ensemble models. Using a significant threshold *α* = 0.05 and MCC as a performance indicator, the Quade omnibus test demonstrates that at least one classifier performs differently than others (*p*-value = 0.01725). Since we found significance in the previous test, we then applied the Quade post-hoc test to determine the pairwise performance difference between classifiers. Here, we considered XGBoost (native) as a control classifier for comparison with the remaining algorithms. Table 8 depicts the *p*-value of the pairwise

comparison. It is clear that the performance differences between XGBoost and the other algorithms are not statistically significant (*p*-value > 0.05).

**Table 8.** The *p*-value of Quade post-hoc, in which XGBoost (native) is used as a control algorithm.


To demonstrate the efficacy of tree-based ensemble models for malware detection, we compared our performance findings to those of previous studies for each dataset. Table 9 denotes the performance comparisons in terms of several performance measures, such as accuracy, precision, recall, and F1. Please note that the comparison is conducted as objectively as possible, given that the prior experiment may have been conducted under different settings, such as validation techniques and the number of training and testing samples. Nevertheless, this study shows that the top-performing tree-based ensemble examined for each dataset outperforms prior research, with a comparable result. More precisely, GBM (H2O), XGBoost (native), and random forest are the best performers on the Kaggle, BODMAS, and CIC-MalMem-2022 datasets, respectively, which also outperform other state-of-the-art malware detection techniques available in the recent literature.

**Table 9.** Performance comparisons over existing studies. The best performance value on each dataset is shown in bold.


## **5. Conclusions**

This article examined tree-based ensemble learning algorithms that analyze PE malware. Several tree-based ensemble techniques, including random forest, XGBoost, CatBoost, GBM, and LightGBM, were assessed based on a number of performance criteria, such as accuracy, MCC, precision, recall, AUC, and F1. In addition, we incorporated cutting-edge malware datasets to comprehend the most recent attack trends. This work contributed to the prior research in several ways, including by providing a statistical comparison of fine-tuned tree-based ensemble models utilizing several malware datasets. Furthermore, this article can be expanded in a number of ways, including by looking at the explainability of tree-based ensemble models and signature-based malware classification. Furthermore, a deep neural network model for tabular data, such as TabNet [52], has been underexplored in this application domain, providing a new direction for future research. Finally, it is anticipated that tree-based PE malware detection will be deployed in various real-world settings, such as in host, network, and cloud-based malware detection components.

**Author Contributions:** Conceptualization, M.H.L.L. and B.A.T.; methodology, B.A.T.; validation, M.H.L.L.; investigation, M.H.L.L.; writing—original draft preparation, M.H.L.L.; writing—review and editing, M.H.L.L. and B.A.T.; visualization, B.A.T.; supervision, B.A.T. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

## **List of Acronyms**


TPR True Positive Rate.

## **References**

