Detecting Web-Based Attacks with SHAP and Tree Ensemble Machine Learning Methods

Abstract

Attacks using Uniform Resource Locators (URLs) and their JavaScript (JS) code content to perpetrate malicious activities on the Internet are rampant and continuously evolving. Methods such as blocklisting, client honeypots, domain reputation inspection, and heuristic and signature-based systems are used to detect these malicious activities. Recently, machine learning approaches have been proposed; however, challenges still exist. First, blocklist systems are easily evaded by new URLs and JS code content, obfuscation, fast-flux, cloaking, and URL shortening. Second, heuristic and signature-based systems do not generalize well to zero-day attacks. Third, the Domain Name System allows cybercriminals to easily migrate their malicious servers to hide their Internet protocol addresses behind domain names. Finally, crafting fully representative features is challenging, even for domain experts. This study proposes a feature selection and classification approach for malicious JS code content using Shapley additive explanations and tree ensemble methods. The JS code features are obtained from the Abstract Syntax Tree form of the JS code, sample JS attack codes, and association rule mining. The malicious and benign JS code datasets obtained from Hynek Petrak and the Majestic Million Service were used for performance evaluation. We compared the performance of the proposed method to those of other feature selection methods in the task of malicious JS code content detection. With a recall of 0.9989, our experimental results show that the proposed approach is a better prediction model.

1. Introduction

Websites are very popular; hence, cybercriminals find these platforms to be perfect tools for launching their attacks. Web-based attacks remain a significant challenge, as evasion techniques are continuously evolving. Attackers compromise Uniform Resource Locators (URLs) and their JavaScript (JS) content to perform malicious activities on the Internet. Such activities include phishing, URL redirection, spamming, social engineering, botnets, and drive-by-download exploits [1,2,3]. The attacks are delivered through emails, malware advertisements, texts, pop-ups, malicious scripts, and search results. The Domain Name System (DNS) [1,2,3,4,5] provides cybercriminals the flexibility to easily migrate their malicious servers, as they hide the IP addresses behind domain names [2,3]. Securing websites is vital for maintaining confidentiality, integrity, and availability, and an equally adaptive strategy is required to detect such attacks effectively.

Methods such as maintaining a blocklist [6], client honeypots, domain reputation inspection [4], and domain and web metrics analysis [5,7,8,9] can detect malicious URLs and their JS code content [4,10,11]. However, new URLs are registered every day, and a blocklist can be evaded through URL obfuscation, fast flux, cloaking, and URL shortening. Recently, machine learning approaches that employ feature extraction and representation learning for malicious URLs and their JS code content detection have been proposed [2,3,12,13,14]. Machine learning algorithms learn a prediction function based on features such as lexical, host-based, URL lifetime, and content-based features that include HyperText Markup Language and JS code. A classifier is then used to predict whether given content is malicious or benign using a test dataset. Models such as these can be generalized to new data, unlike the blocklist approach. However, there are challenges in crafting fully representative features for model training.

In our previous work [15,16,17], we developed a system to detect malicious JS codes using fixed-length vectors and the Abstract Syntax Tree form of the JS code (AST-JS). This study extends our work by detecting malicious JS code content through feature selection. Using atomic features and allowing the model to determine the relationships between them is preferable instead of using composite features [2,3]. This study proposes using Shapley additive explanations (SHAP) values and tree ensemble methods. While other methods learn a set of manually extracted or engineered features, our approach adopts features that are automatically selected based on their contributions to the model’s output.

Malicious URLs and their JS code content can stealthily perpetrate web-based attacks [15,16,17,18]. Other studies [12,19,20] have shown that malicious URLs and their JS code content exhibit features that are distinct from those of legitimate URLs. Therefore, we focused on JS code content-based features to detect malicious websites. Our premise is that malicious JS code content originating from attackers exhibits different AST-based features than legitimate content. A SHAP-based model would learn such features from the distribution of continuously evolving malicious JS code content.

The main contributions of this study are the following:

• This study proposes using SHAP and tree ensemble methods for detecting web-based attacks.
• We detail the process of obtaining features using AST-JS node sets and patterns, sample JS attack codes, and association rule mining.
• We compared the performances of different classifiers in malicious JS code detection using SHAP selected features and achieved good detection performance for the tree ensemble methods.
• We compared the performance of SHAP selected features to the performance of those selected by other feature selection methods: Boruta, ELI5, RandomForest, and SelectKBest.
• The proposed web-based attack detection method outperformed the other feature selection methods in all three evaluation metrics.

The remainder of this paper is organized as follows. Section 2 highlights the related work. We present our approach in Section 3. Section 4 presents the performance evaluations of our own and other models; and we present our discussion and conclusions in Section 5 and Section 6, respectively.

2. Related Work

Researchers have put forward many prevention techniques to keep up with the increase in attacks and new attack methods. Blocklists are repositories of known malicious URLs and have long been employed to detect malicious URLs and their JS codes. Heuristic and signature-based systems search for signs of standard attack patterns. The weakness of these approaches is that new or variant URLs and their JS codes and zero-day attacks can easily evade detection. Further, cybercriminals widely use obfuscation to perpetrate attacks and evade detection.

Some studies [12,13,14] have proposed embedding the bag-of-words model in a URL or features of a URL and its JS code content. Ma et al. [12,13,14] proposed an approach to analyzing a URL to predict the maliciousness of websites using an online confidence-weighted algorithm for lexical feature learning. Individual feature vectors were manually engineered. Ma et al. [12] acknowledged that other potentially useful sources of information for features could improve classification accuracy. However, they did not examine a web page’s actual content, citing reasons such as user safety, the model’s operation costs, the classifier’s applicability to the URL context, and poor reliability when obtaining a malicious web page version. They also used bag-of-words features and therefore did not preserve token order. Conversely, our proposed approach uses JS code content features. Since AST-JS is resistant against perturbations in JS code content, it makes our model applicable for various JS code contexts and varying content.

Other systems [2,3,5,21,22] have been proposed that use constructed features or feature engineering. Bilge et al.’s [2,3] EXPOSURE conducted a passive DNS analysis to identify malicious domains. It is a dynamic reputation system based on passive recursive DNS monitoring. They extracted 15 behavioral features to train the J48 decision tree classifier. Their study assumed that large volumes of DNS data requests should exhibit sufficient behavioral differences to distinguish between benign and malicious domains. In another study, a system to detect malicious domains using DNS records and domain name features was proposed by Al Messabi et al. [5] using J48, a C4.5 decision tree. This approach studied prior DNS activities for each domain and the relationship between defective and legitimate domains’ physical behavior [21], combining several existing DNS-based and domain name-based features from previous work [2,3,22]. They assumed that eight unique behavioral features could accurately identify malicious websites and proactively detect an attack. However, feature engineering is challenging, even for domain experts. We performed automatic feature selection to address these challenges.

Kuyama et al. [23,24] sought to detect targeted attacks by monitoring the communication between the Command and Control (C&C) server and the computers in the local area network. Their proposed method identified new C&C servers using supervised machine learning; extracted the WHOIS feature points and DNS information; and searched the site. For performance comparison, the malicious domains were detected separately using a neural network and a support vector machine. Since manual feature selection is a tedious task, even for domain experts, we propose an autonomous feature-selection method using SHAP values and tree ensemble methods.

Other studies have attempted to detect specific URL attack types, such asphishing [25,26,27,28,29,30], malicious advertisements, and click fraud [1,31,32,33,34,35,36]. These approaches are suited for specific attack types and do not generalize well to other attacks. Masri et al. [1] proposed a system for the automatic classification and detection of malicious advertisements. They used VirusTotal [37], URLVoid [38], and TrendMicro [39]. Their study relied heavily on other tools that employ signatures and blocklist services. Therefore, it was prone to problems inherent to signature and blocklist-based systems [4].

We propose an automatic feature selection method of malicious JS code content using SHAP values. A more thorough analysis of JS code content-based features may help detect threats [10,40]. These features provide more rich information for the feature learning process than URL-based ones, as much information is extracted from a web page. Safety concerns may arise if the JS code content is executed. However, we used AST-JS for the code structure representation. Further, AST-JS enables the capturing of more details regarding a web page. The assumption is that more information would lead to a better prediction model. We assumed that correlations between various web-based attack features exist, and a machine learning model can identify these relationships with minimal effort.

3. Proposed Method

When machine learning is employed to detect malicious URLs and their JS code content, certain considerations have to be addressed, such as privacy and safety concerns, feature learning methods, and the development of a fully representative feature set.

Given the common vulnerabilities, characteristics, and features employed by malicious JS codes, it is evident that malicious websites are different from benign ones, as they each have different objectives. For example, a malicious JS code contains functions or a combination of functions such as $document.write$, $location.replace$, and $document.getElementById$. Among other things, these functions are used to perform cross-site scripting attacks by injecting malicious JS code into the document object model, direct concatenation of user input with the query string, and building a database query via string concatenation used to perform structured query language injections. Therefore, developing a malicious JS code detection system that can generalize well to detect different attack types is imperative. We propose automatic feature selection for AST-JS features, leveraging the global and local features that contribute to the model performance. Adequate details of a JS code structure are also presented without executing actual code.

This section details the proposed method for web-based attack detection using SHAP and tree ensemble methods. The method includes the following three main stages: preprocessing, feature selection, and classification, as shown in Figure 1. The following sections explain the JS code preprocessing, feature selection, the SHAP values used for feature selection, and the machine learning classifiers used for classification.

3.1. Preprocessing

This section describes the AST-JS node sets and patterns, sample JS attack codes, and association rule mining.

3.1.1. AST-JS Node Sets and Patterns

Preprocessing is essential to enhancing the feature learning process. To define our first input set, we parsed each JS code in our dataset to obtain the $expression$, $pattern$, $statement$, and $declaration$ AST-JS nodes.

3.1.2. Sample JS Attack Codes

Malicious JS code rarely uses raw values, as attackers endeavor to evade static analyzers. Therefore, it is essential to capture features that employ covert implementations to hide their values. To capture these features, we analyzed the feature importance of our first feature set using SHAP interaction values. This feature analysis exposed each AST-JS node and the combination’s contribution to the malicious JS code detection model. To identify such combinations, we obtained sample JS attack codes and performed association rule mining using the Frequent Pattern growth (FP-growth) mlxtend library [41,42,43]. The sample JS attack codes are explained next.

Listing 1 shows a JS code that lacks a definition of the index of an object in the current scope. When $foo\left(\right)$ is called, $x=0$ is passed, which is a local index of the function. We capture this attack pattern using the AST node $type$ $EmptyStatement\_ExpressionStatement\_CallExpression\_Identifier$.

Listing 2 shows a JS code where the argument of $setTimeout\left(\right)$ is a string concatenation with binary expression. We capture this attack pattern using the AST node $type$ $CallExpression\_Identifier\_BinaryExpression$.

Listing 1. A snippet of a JS code with EmptyStatement_ExpressionStatement_CallExpression _Identifier.

Listing 1. A snippet of a JS code with EmptyStatement_ExpressionStatement_CallExpression _Identifier.

Listing 2. A snippet of a JS code with CallExpression_Identifier_BinaryExpression.

Listing 2. A snippet of a JS code with CallExpression_Identifier_BinaryExpression.

Listing 3 shows nested JS code function calls, where an attacker hides the function arguments in another function call to evade static analyses. We capture this attack pattern using the AST node $-type$ $CallExpression\_Identifier\_Literal\_CallExpression$.

Listing 4 shows variable declarations, where e in $line2$ is assigned to a function c. We capture this attack pattern using the AST node $type$ $AssignmentExpression\_Identifier\_FunctionExpression$.

Listing 5 shows a JS code where the attacker attempts to detect the user environment, such as the operating system or the browser version. We capture this attack pattern using the AST node $type$ $VariableDeclarator\_Identifier\_LogicalExpression$.

Listing 6 shows the JS code sample string concatenation and function obfuscation codes. We capture this attack pattern using the AST node $type$ $BinaryExpression\_BinaryExpression$.

Listing 3. A snippet of a JS code with CallExpression_Identifier_Literal_CallExpression.

Listing 3. A snippet of a JS code with CallExpression_Identifier_Literal_CallExpression.

Listing 4. A snippet of a JS code with AssignmentExpression_Identifier_FunctionExpression.

Listing 4. A snippet of a JS code with AssignmentExpression_Identifier_FunctionExpression.

Listing 5. A snippet of a JS code with VariableDeclarator_Identifier_LogicalExpression.

Listing 5. A snippet of a JS code with VariableDeclarator_Identifier_LogicalExpression.

Listing 6. A snippet of a JS code with BinaryExpression_BinaryExpression.

Listing 6. A snippet of a JS code with BinaryExpression_BinaryExpression.

3.1.3. Association Rule Mining

Algorithm 1 shows AST-JS’s association rule mining procedure using the FP-growth algorithm. The generated association rules for benign and malicious JS codes are transformed into a list (lines 14–17). For AST-JS feature selection, the lists are converted into individual benign and malicious data frames (lines 19–22).

Algorithm 1 Mining frequent AST-JS node sets using the FP-growth algorithm.

Input:D—a database of benign and malicious JS codes defined as AST-JS nodes; $min\_support$—the minimum support count threshold. Output: Benign and malicious DataFrames of AST-JS nodes and node combinations. 1: if$Tree$ contains a single path P then 2:    for each combination $\beta$ of the nodes in the path P do 3:      generate pattern $\beta \cup \alpha$ with $support=min\_support$ of nodes in $\beta$; 4:    end for 5: else 6:    for each $a_i$ in the Tree header do 7:      generate pattern $\beta =a_i\cup \alpha$ with $support=a_i.support$; 8:      construct $\beta$’s conditional pattern base and then $\beta$’s conditional FP_tree ${Tree}_{\beta }$; 9:    end for 10:    if ${Tree}_{\beta }$≠0 then 11:      call FP_growth($Tree$$\beta$,$\beta$); 12:    end if 13: end if 14: for each benign 0 and malicious 1 frequent patterns do 15:    0.antecedents.apply(sorted(list(i)))+0.consequents.apply(sorted(list(i))); 16:    1.antecedents.apply(sorted(list(i)))+1.consequents.apply(sorted(list(i))); 17:    return list0, list1 18: end for 19: for each benign list0 and malicious list1 frequent patterns do 20:    TransactionEncode.fit(list0).transform(list0); 21:    TransactionEncode.fit(list1).transform(list1); 22:    pd.DataFrame(list0, list1, columns=TransactionEncode.columns_) 23: end for

Support is defined as the frequency of an AST-JS node or set in the JS code dataset, and confidence is the probability of an AST-JS node set’s occurrence with its set of nodes. Confidence measures how often an association rule is found to be true. $min\_support$ is the minimum support for an AST-JS node set to be identified as frequent. $confidence$’s $min\_threshold$ is the minimum confidence for generating an association rule.

Finally, we analyzed the feature importance of each AST-JS feature selected using SHAP values to understand how different features influence our malicious JS code detection model’s performance. This process was achieved in two parts: first, we used our original AST-JS features and then the AST-JS combination features extracted through association rule mining. Next, we explain the feature selection process.

3.2. Feature Selection

The feature selection process comprises sample JS attack codes, association rule mining, and SHAP values. The sample JS attack codes are used to obtain AST-JS feature combinations based on expression, pattern, statement, and declaration nodes. AST-JS feature combinations augment the AST-JS node features by capturing JS-based attacks. The JS-based attacks include variable obfuscation, attacks using eval and unescape, attacks using the current context or browser objects, user environment detection, string obfuscation, string concatenation, encoding functions, nested function calls, function obfuscation, and attacks combining string concatenation with function obfuscation.

Association rule mining is used to measure how often AST-JS nodes appear together in benign and malicious JS codes. Features obtained using Association rule mining are used for performance comparison. SHAP values are used to determine AST-JS feature interactions, and a feature set is selected based on the contribution to the detection performance for malicious JS code. Next, we explain the SHAP features’ importance.

3.3. Shapley Additive Explanations’ Feature Importance

SHAP is a game-theoretic approach that explains any machine learning model’s output that connects optimal credit allocation with local explanations using the classical Shapley values and their related extensions [44,45,46]. The values show the extent to which a feature is responsible for a change in the model’s output. SHAP values either increase or decrease the model’s prediction values and balance out the input’s actual prediction. The prediction starts from the base value, the mean derived from the entire prediction. Our premise is that interpreting the malicious JS code detection model’s output will guide AST-JS feature selection, further improving its performance. Identifying the most-impactful features may enable us to derive other features with additional information to improve the detection performance. This knowledge will also provide necessary insights into the distribution of specific benign and malicious JS code features.

A JS code input is represented as Z with a set of AST-JS features z₁, z₂, ⋯, z_n, and its corresponding output $Z^{\prime }$ and predicted features $z{}^{\prime }1$, $z^{\prime }$₂, ⋯, $z^{\prime }$_n, using the tree ensemble model g. Algorithm 2 shows the procedure for calculating tree SHAP values using tree ensemble methods that return AST-JS features (line 6) sorted in descending order of their importance. First, we obtained AST-JS features from the JS code dataset and saved them as AST-JS M features. For each feature $z^{\prime }$_i in the AST-JS M features, we used tree SHAP to obtain the SHAP values. When predicting the observed feature $z^{\prime }$_i, the SHAP importance values for each feature, z₁, z₂, ⋯, z_n, excluding z_i, were calculated. Tree SHAP receives g and a background set with j instances to build the local explanation model and calculate the SHAP values. Then, g takes Z and i as input and predicts $Z^{\prime }$; the value in the i-th feature, a feature in the AST-JS M features, is returned by Algorithm 3. This results in a two-dimensional list of the $SHAPsortedM$ features. Each row in the list represents the SHAP values one of each AST-JS feature in M features.

We divided the AST-JS features into those that enhanced the performance of the malicious JS code detection model and those that did not by determining whether their SHAP values moved the predicted value toward or away from the true value. Algorithm 4 shows how the SHAP values were used to select relevant AST-JS features. For each AST-JS feature (line 1), we checked whether the true feature value given by the input JS code was greater than the predicted value (line 2); a positive SHAP value indicates a contributing feature (line 3) and vice versa (line 4). If the predicted feature value was less than the input value for a JS code (line 5), the contributing features is indicated by a negative SHAP value and vice versa. This algorithm returned two lists, the $SHAPSelected$ and $SHAPnotSelected$ features, containing the contributing and non-contributing AST-JS features, respectively, along with the SHAP values for each of the AST-JS M features.

Algorithm 2 Calculating tree SHAP values for AST-JS M features.

Input:Z—a malicious or benign JS code we want to explain, Z$1..j$—AST-JS instances that tree SHAP uses as background examples, the g—tree ensemble model. Output:$SHAPsortedM$ features - SHAP values for each AST-JS feature in JS code dataset sorted in descending order of their importance. 1: M features ← feature values based on AST-JS. 2: for each$i\in$M features do 3:    explainer ← shap.TreeExplainer(g,Z$1..j$) 4:    $SHAPsortedM$ features $\left[i\right]$← explainer.shapvalues(Z,i) 5: end for 6: return $SHAPsortedM$ features

Algorithm 3$g(Z,i)$—predicting the ith AST-JS feature.

Input:Z—a malicious or benign JS code to explain, i—the AST-JS feature to get prediction for, g—tree ensemble model. Output:$z^{\prime }$i—the value of the ith AST-JS feature in $Z^{\prime }$. 1: $z^{\prime }$i←g.$predict($Z$)$[i] 2: return $z^{\prime }$i

Algorithm 4 Selecting contributing AST-JS features’ SHAP values.

Input:$SHAPsortedM$ features—SHAP values for each AST-JS feature in JS code dataset sorted in descending order of their importance, Z—a malicious or benign JS code we want to explain, $Z^{\prime }$—the prediction for Z. Output: $SHAPSelected$ features and $SHAPnotSelected$ features. 1: for each$i\in$$SHAPsortedM$ features do 2:    if zi > $z^{\prime }$i then 3:      $SHAPSelected$[i] ←$SHAPsortedM$ features[i] > 0 4:      $SHAPnotSelected$[i] ←$SHAPsortedM$ features[i] < 0 5:    else 6:      $SHAPSelected$[i] ←$SHAPsortedM$ features[i] < 0 7:      $SHAPnotSelected$[i] ←$SHAPsortedM$ features[i] > 0 8:    end if 9: end for 10: return $SHAPSelected$ features, $SHAPnotSelected$ features

The benefits of computing SHAP values are global and local interpretability, which shows the manner and degree of each AST-JS feature’s contribution to the prediction, and each observation obtains its own set of SHAP values, enabling the evaluation of the features’ impacts.

3.4. Machine Learning Classifiers

For the classification task for web-based attack detection, seven machine learning classifiers from the Scikit-learn library [47,48,49,50] were used in the experiments, i.e., XGBoost, LightGBM, RandomForest, DecisionTree, LogisticRegression, KNeighbors, and GaussianNB. The best algorithm was selected based on the k-fold cross-validation evaluation results.

4. Experiments

This section presents the experimental setup and performance comparison results for web-based attack detection.

4.1. Experimental Setup

We used a dataset of 39,443 malicious and 40,000 benign JS codes in the experimental setup. The malicious JS codes were obtained from Hynek Petrak’s dataset [51], and the benign codes were obtained from the Majestic Million Service [7]. We used a JS code parser to preprocess the dataset into AST-JS nodes, resulting in 32,430 malicious and 38,891 benign JS codes. This resulted in 25 $expression$ and $pattern$ and 23 $statement$ and $declaration$ AST-JS features.

Next, we visualized and investigated each AST-JS feature’s contribution to the malicious JS code detection model’s performance using SHAP values. This analysis indicated the need to define more concrete features. Using sample JS attack codes, we added ten more AST-JS features based on combinations of the original ones. These features represent JS-based attacks and contribute to the model’s detection performance. We then used association rule mining and a confidence metric to measure how often AST-JS nodes appear together in benign and malicious JS codes. This resulted in 33 features with parameters $min\_support=0.4$ for malicious JS codes, $min\_support=0.53$ for benign codes, and $confidence$’s $min\_threshold=1$ for both benign and malicious codes. This selection reduced our original features and formed the second input set. Finally, we selected a final set of AST-JS features using the SHAP values. This selection resulted in forty-four features, comprising thirty-four AS-JS nodes and ten node combinations.

Performance evaluation was conducted using the three-input feature sets, i.e., the 48 AST-JS features obtained after the initial preprocessing, the 33 features obtained using association rule mining, and the 44 features obtained by feature selection using the SHAP value. The parameters obtained using $GridSearchCV$ for the two best-performing models, XGBClassifier and LGBMClassifier, using the three AST-JS feature sets, are listed in

Table 1. XGBClassifier and LGBMClassifier parameters.

Parameter	AST-JS Nodes	Association Rule	SHAP Value
XGBClassifier
$colsample\_bytree$	0.5	0.7	0.4
$gamma$	0.0	0.1	0.0
$learning\_rate$	0.2	0.2	0.2
$max\_depth$	10	8	13
$min\_child\_weight$	1	1	1
LGBMClassifier
$lambda\_l1$	0	0	0
$lambda\_l2$	0	0	0
$max\_depth$	9	6	8
$n\_estimators$	520	520	520
$num\_leaves$	10	10	10

.

We found that these parameters are optimal for malicious JS code content detection using the three-input feature sets. We performed 10-fold cross-validation to evaluate the performance of our approach and computed precision, recall, and F1-score evaluation metrics. Precision is the classifier’s ability to not label a benign JS code as malicious, and recall is the classifier’s ability to find all malicious JS codes. The F1-score can be interpreted as the weighted harmonic mean of precision and recall. Given that $TP$ is the number of malicious JS codes correctly classified as malicious, $TN$ is the number of benign JS codes correctly classified as benign, $FN$ is the number of malicious JS codes classified as benign, and $FP$ is the number of benign JS codes classified as malicious. $Precision$, $recall$, and $F1-score$ are given by:

$$\begin{array}{cc}\hfill Precision& =\frac{TP}{TP+FP}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill Recall& =\frac{TP}{TP+FN}\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill F1-score& =2\frac{Precision\times Recall}{Precision+Recall}\hfill \end{array}$$

(3)

4.2. Performance Comparisons

The results of these experiments are presented in this section.

Table 2. Performance comparison using AST-JS node features.

Model	Recall	Precision	F1
XGBoost	0.9981 ± 0.0008	0.9698 ± 0.0033	0.9838 ± 0.0018
LightGBM	0.9979 ± 0.0008	0.9691 ± 0.0032	0.9833 ± 0.0019
RandomForest	0.9986 ± 0.0005	0.9702 ± 0.0032	0.9842 ± 0.0018
DecisionTree	0.9983 ± 0.0005	0.9687 ± 0.0029	0.9833 ± 0.0016
LogisticRegression	0.9839 ± 0.0021	0.9414 ± 0.0039	0.9622 ± 0.0025
KNeighbors	0.8448 ± 0.0062	0.9986 ± 0.0006	0.9153 ± 0.0037
GaussianNB	0.9968 ± 0.0013	0.5493 ± 0.0029	0.7083 ± 0.0025

presents the precision, recall, and F1-score values obtained using features defined as AST-JS nodes and 10-fold cross-validation. Each model performed well on the recall metric, especially the tree ensemble methods. The best-performing model, XGBoost, could detect malicious JS code content with an error rate of 0.0019 for recall, 0.0302 for precision, and 0.0162 for F1. The lower precision metric revealed that the model misclassified some benign JS codes. Even though this scenario is not harmful, it may lead to threat-alert fatigue if users and security analysts receive many false alarms. Therefore, there is a need to improve this model further.

We analyzed the AST-JS node features using the SHAP values. Figure 2 presents a SHAP summary plot that shows the relative impacts of AST-JS features on the JS code dataset.

The SHAP values are plotted on the x-axis for each AST-JS feature on a row sorted by the sums of their SHAP value magnitudes. The vertically piled points represent the feature density, and the colors show the feature values. The values give the distribution of each AST-JS feature’s impact on the model’s output. The red and blue colors represent high and low AST-JS feature values. The color allows us to visualize how changes in the value of an AST-JS feature would affect a change in prediction; for example, high SHAP values for the $SequenceExpression$ feature would indicate a high risk of maliciousness for a JS code.

Using such plots, we can deduce that features such as the $ObjectExpression$ would influence the model’s prediction more than the $LabeledStatement$. A feature such as $SequenceExpression$ has a significant, positive effect on AST-JS prediction, and therefore, a high $SequenceExpression$ SHAP value may indicate a higher risk for maliciousness. On the contrary, a feature such as the $FunctionExpression$ has a significant, negative effect on the AST-JS class prediction, and therefore, a low $FunctionExpression$ SHAP value may indicate a higher risk for maliciousness.

Additionally, interesting patterns can be observed, such as high values of the $ExpressionStatement\_CallExpression\_Identifier$ feature clustered in a very dense region represented by the red blob. Additionally, low values of the $BlockStatement$ feature are clustered in a very dense region, as shown by the blue blob. However, features such as the $ReturnStatement$ and $FunctionExpression$ have a much more uniform distribution with high and low SHAP values, respectively, pushing the prediction to 1. The red and blue blobs on the left and right indicate an even distribution of that feature in the JS code dataset. Some features such as $EmptyStatement\_ExpressionStatement\_CallExpression$ and $CallExpression\_Identifier\_Literal$ are not crucial for most JS codes. However, these features significantly impact a subset of JS codes in the dataset. This scenario highlights how a globally important feature is not necessarily the most critical feature for attack detection in JS codes.

Figure 3 shows the SHAP dependence plot for the top AST-JS feature before (a) and after (b) feature selection.

Every dot represents a JS code. Vertical dispersion at an AST-JS feature value results from interaction effects in the model. The color highlights the high or low forces behind the interactions. The y-axis represents the SHAP values. The SHAP summary plot is obtained by projecting the SHAP dependence plot points onto the y-axis and recoloring the value’s feature. The $TryStatement$ and $VariableDeclaration$ features were automatically selected for coloring based on a potential interaction in the model. Plot (b) shows that low SHAP values of the $ObjectExpession$ feature influence the model’s output more significantly for observations where the $VariableDeclaration$ feature has high SHAP values.

Figure 4 shows the SHAP interaction values plot for the top two AST-JS features before (a) and after (b) feature selection.

It shows the main effects and interaction effects for the $ObjectExpression$ feature. These effects capture all vertical dispersions. Plot (a) shows that high SHAP values for the $ObjectExpression$ and $FunctionExpression$ feature significantly influence the model’s output. Plot (b) shows that low SHAP values for the $ObjectExpression$ feature and high SHAP values for the $ExpressionStatement\_CallExpression\_Identifier$ feature significantly influence the model’s output.

Table 3. Performance comparison using association rule mining AST-JS features.

Model	Recall	Precision	F1
XGBoost	0.9763 ± 0.0025	0.9825 ± 0.0024	0.9794 ± 0.0018
LightGBM	0.9762 ± 0.0023	0.9820 ± 0.0025	0.9791 ± 0.0018
RandomForest	0.9757 ± 0.0024	0.9833 ± 0.0025	0.9795 ± 0.0018
DecisionTree	0.9758 ± 0.0024	0.9812 ± 0.0025	0.9785 ± 0.0019
LogisticRegression	0.9600 ± 0.0033	0.9602 ± 0.0040	0.9601 ± 0.0025
KNeighbors	0.8516 ± 0.0044	0.9991 ± 0.0007	0.9195 ± 0.0026
GaussianNB	0.8221 ± 0.0048	0.7845 ± 0.0044	0.8029 ± 0.0038

presents precision, recall, and F1-scores obtained using features defined by association rule mining and 10-fold cross-validation.

By including these features, precision was improved by 0.0127, 0.0129, 0.0131, 0.0125, 0.0188, and 0.2352 for the XGBoost, LightGBM, RandomForest, DecisionTree, LogisticRegression, and GaussianNB, respectively. This improvement indicates a reduction in the number of misclassified benign JS codes. However, there was a reduction in the recall metric, which indicates an increase in misclassified malicious JS codes. Leveraging features from the malicious JS code samples and SHAP selected features based on their contributions to the model’s output was pursued to improve the detection performance.

Table 4. Performance comparison using SHAP selected AST-JS features.

Model	Recall	Precision	F1
XGBoost	0.9989 ± 0.0004	0.9832 ± 0.0024	0.9909 ± 0.0014
LightGBM	0.9986 ± 0.0007	0.9820 ± 0.0027	0.9902 ± 0.0015
RandomForest	0.9985 ± 0.0006	0.9840 ± 0.0024	0.9912 ± 0.0014
DecisionTree	0.9986 ± 0.0005	0.9815 ± 0.0024	0.9900 ± 0.0013
LogisticRegression	0.9895 ± 0.0013	0.9632 ± 0.0037	0.9762 ± 0.0017
KNeighbors	0.8779 ± 0.0056	0.9995 ± 0.0003	0.9347 ± 0.0031
GaussianNB	0.9373 ± 0.0056	0.6889 ± 0.0031	0.7941 ± 0.0030

presents precision, recall, and F1-scores obtained using features selected based on SHAP values and 10-fold cross-validation.

Compared to the association rule mining features, each model’s detection performance was improved in all three evaluation metrics, with notable improvements in recall and F1-score. The best performing model, the XGBoost model, had error rates of 0.0011, 0.0168, and 0.0091 for recall, precision, and F1, respectively. The model outperformed the LightGBM and DecisionTree models by 0.03%, and the RandomForest model by 0.04% in the recall metric. A high recall rate translates to low misclassification and false-negative rates. AST-JS features selected using the SHAP values capture global and local feature importance. Therefore, these features enhance the machine learning model’s feature learning process and lead to a better prediction model. As evidenced by the experiments, high performance was achieved by all the malicious JS code detection models, with tree ensemble methods yielding the best results. Consequently, these models had the lowest false positive and false negative rates.

Figure 5 shows the feature importance assigned to the AST-JS features by other feature selection methods: Boruta, ELI5, RandomForest, and SelectKBest [47,52].

Each method resulted in different feature values for each AST-JS feature, and therefore, a different number of features was selected for each feature selection method. Boruta, a wrapper around RandomForest, found all relevant features carrying information that can be used to predicting malicious JS. AST-JS features proven by a statistical test to be less relevant are rejected iteratively. ELI5 is used for explaining predictions. It is also referred to as permutation importance or Means Decrease Accuracy. The method measures how the score decreases when an AST-JS feature is eliminated. The RandomForest feature importance method measures each AST-JS feature’s importance using the $entropy$ function for the information gain. SelectKBest ranks AST-JS features by the k highest scores. This method measures the dependency between features using the mutual information score function.

Table 5. Performance comparison of SHAP and other feature selection methods.

Model	Recall	Precision	F1
SHAP	0.9989 ± 0.0004	0.9832 ± 0.0024	0.9909 ± 0.0014
Boruta	0.9983 ± 0.0008	0.9698 ± 0.0033	0.9839 ± 0.0019
ELI5	0.9982 ± 0.0008	0.9699 ± 0.0033	0.9839 ± 0.0019
RandomForest	0.9984 ± 0.0007	0.9698 ± 0.0032	0.9839 ± 0.0018
SelectKBest	0.9982 ± 0.0007	0.9687 ± 0.0034	0.9832 ± 0.0019

presents precision, recall, and F1-scores obtained using SHAP selected features compared to the other feature selection methods; Boruta, ELI5, RandomForest, and SelectKBest.

Our proposed detection model performed better than the other feature selection methods in all three evaluation metrics, with notable differences in precision and F1. The other feature selection methods have limitations because different model iterations assign different feature values to the AST-JS features. Additionally, Boruta assigns a $True$ or $False$ value to each AST-JS feature, as shown in Figure 5a. The permutation-based methods are computationally expensive and can have problems with highly-correlated AST-JS features, resulting in loss of important information. SHAP-selected AST-JS features have consistency in the feature values assigned to each feature. Unlike Boruta, SHAP values show the degree and manner of each AST-JS feature’s contribution to the model prediction and are model-agnostic. These features also provide interaction graphs that are instrumental in getting information on AST-JS feature combinations, further boosting the model’s detection performance.

Table 6. Training and detection time.

Model	Training Time (s)	Detection Time (s)
XGBoost	0.8685	1.498 × 10${}^{-6}$
LightGBM	0.5716	2.114 × 10${}^{-6}$
RandomForest	1.6350	1.035 × 10${}^{-5}$
DecisionTree	0.0895	1.179 × 10${}^{-6}$
LogisticRegression	0.6384	1.871 × 10${}^{-6}$
KNeighbors	0.0037	0.0015
GaussianNB	0.0425	1.334 × 10${}^{-6}$

shows the training and detection times for the various classifiers on the JS code dataset using SHAP selected features. XGBoost yielded the highest detection performance and the third-fastest detection time, making it the best classifier for JS-based attack detection. KNeighbors yielded the lowest training time; however, it achieved the lowest recall rate, rendering it ineffective for this detection task. DecisionTree yielded the lowest detection time; however, XGBoost outperformed DecisionTree in all three evaluation metrics.

5. Discussion

Blocklists, and heuristic- and signature-based systems are widely used to detect malicious URLs and JS codes. Current detection methods have various challenges stemming from fast-flux, cloaking, and zero-day attacks. Some services that can be used for copyright and privacy reasons in benign JS codes, such as obfuscation, can also create new or variant JS-based attacks. Therefore, it is biased to classify transformed URLs and JS codes as malicious without further detailed analysis. A system to analyze and accurately detect web-based attacks is needed.

As part of our research findings, feature selection using SHAP values resulted in a better prediction model. Other researchers limited their scope by studying specific malicious URLs and JS-based attacks, such as those containing phishing attacks, social engineering, malvertising attacks, and denial of service attacks, among others. For this, they used manually engineered features unique to the type of attack they intended to identify. Such approaches have challenges in that they cannot be generalized to other attack types. Our premise is that features selected using SHAP values can overcome such challenges, as the analyst can easily identify which features significantly impact the overall model performance. Figure 6 shows SHAP interaction values with combinations of AST-JS features shown using $Featur{e}^{*}-Feature$ patterns.

Using SHAP interaction values, it is easier to tell which AST-JS feature combinations or interactions contribute more toward the model’s detection performance and in which direction. Some of these interactions are directly interpretable as attack types present in the dataset. Therefore, our proposed approach does not focus on specific categories of web-based attacks or threats.

6. Conclusions

Web-based attacks remain a significant challenge, as evasion techniques are continuously evolving. An equally adaptive strategy is required to detect such attacks effectively. This study proposes AST-JS feature selection using SHAP values and tree ensemble methods to detect these attacks. We also investigated how often AST-JS nodes appear together in benign and malicious JS codes using association rule mining. One expectation was that this approach would result in fully representative features that generalize well to other attacks. We used AST-JS nodes to represent the JS code structure and their SHAP values for feature selection. We experimented with features selected using AST-JS nodes, association rule mining, and SHAP values. We compared the performances of different machine learning classifiers in malicious JS code detection and achieved good detection performance with the tree ensemble methods. Additionally, we compared the performance of SHAP-selected features with the performances of those selected by other feature selection methods: Boruta, ELI5, RandomForest, and SelectKBest. The proposed web-based attack detection method outperformed the other feature selection methods in all three evaluation metrics. AST-JS nodes are resistant to perturbation in JS codes, and this ensures that the model is applicable in various JS code contexts, and therefore, to different JS-based attacks. We can extend this study to investigate malicious URL features in future work.