A Novel Static Analysis Approach Using System Calls for Linux IoT Malware Detection

Abstract

The proliferation of Internet of Things (IoT) devices on Linux platforms has heightened concerns regarding vulnerability to malware attacks. This paper introduces a novel approach to investigating the behavior of Linux IoT malware by examining syscalls and library syscall wrappers extracted through static analysis of binaries, as opposed to the conventional method of using dynamic analysis for syscall extraction. We rank and categorize Linux system calls based on their security significance, focusing on understanding malware intent without execution. Feature analysis of the assigned syscall categories and risk ranking is conducted with statistical tests to validate their effectiveness and reliability in differentiating between malware and benign binaries. Our findings demonstrate that potential threats can be reliably identified with an F1 score of 96.86%, solely by analyzing syscalls and library syscall wrappers. This method can augment traditional static analysis, providing an effective preemptive measure to enhance Linux malware analysis. This research highlights the importance of static analysis in strengthening IoT systems against emerging malware threats.

1. Introduction

The Internet of Things (IoT) is reshaping cyber–physical systems with unprecedented impact. By the end of this decade, an estimated 25.44 billion devices will be interconnected, predominantly as IoT devices, comprising 75% of the total device count [1]. Recent reports, including the 2023 Zscaler ThreatLabz Enterprise IoT and OT Threat Report, highlight a staggering 400% surge in attacks targeting IoT devices [2]. The rapid integration of IoT technologies across various industries signals a continued escalation in such security incidents impacting various domains such as transportation, healthcare, and energy management. However, IoT devices lack security features and are inherently complex in terms of hardware and software design, making them vulnerable to cyber attacks [3].

Linux systems are the backbone of numerous IoT devices [4]. This ubiquity has led to a noticeable uptick in malware designed to exploit Linux-based environments, as evidenced by a 50% increase in new Linux malware reported by AV-ATLAS in just one year [5].

The focus of malware analysis is to examine malicious binaries (malware) to discern their behavior, purpose, and impact using a range of techniques and tools. This is typically carried out through static analysis, dynamic analysis or a hybrid of both these approaches. The extent of analysis is only limited by the objective—to quickly detect the malware or understand its functionality and behavior patterns such as network traffic, interactions with the system, evasion techniques and other features.

In this paper, we focus solely on static analysis to extract system calls (syscalls) and library syscall wrappers made by binaries to the Linux kernel for ARM architecture-based Linux files, commonly known as ELF (Executable and Linkable Format) binaries. We analyze 1117 ARM-based Linux IoT malware samples and 1214 benign or non-malicious binaries. The syscall dataset created is evaluated using common machine learning classification models, such as Logistic Regression, Neural Networks, and Random Forest, to classify these binaries as malicious or benign.

While there are numerous research works where syscalls are used to analyze malware through dynamic analysis, to the best of our knowledge, there are no known research works that extract syscalls and library syscall wrappers from static analysis.

Static analysis has the limitation of not capturing all the system calls made by a binary, as it cannot detect indirect system calls. Static analysis is also not effective when the binaries use extensive obfuscation or encryption to pack malware. In this study, we evaluate whether the limited syscalls and library syscall wrappers that can be extracted from the disassembly of the binary are still adequate for malware detection. We limit the scope of static analysis to syscalls extracted from reverse engineering the ELF binary to determine if this approach is a viable option.

The main contributions of this paper are as follows:

• Systematic Linux syscall Categorization and Security Risk Ranking: We categorize and assign risk rankings to all extracted calls based on their potential security risks. The validity of these categories and rankings is confirmed through statistical analysis in this study.
• Static Analysis Dataset: We reverse-engineered 2331 ARM architecture-based ELF binaries to extract syscalls and library syscall wrappers using static analysis and correlated with syscall categories and security risk ranking to create a comprehensive dataset, which includes the syscalls, category and security risk ranking for each syscall, along with added statistical features for each binary.
• Malware Detection: To demonstrate the reliability of using static analysis for malware detection based on system calls extracted from the disassembled binaries, we evaluate the dataset created with standard ML classification models such as Logistic Regression, Random Forest classifier, Support Vector Classification (SVC), and Multi-Layer Perceptron (MLP) Neural Networks.

Although we focus on ARM binaries, our syscall extraction approach is architecture-agnostic due to the use of the popular reverse engineering tool radare2. Radare2 employs the ESIL (Evaluable Strings Intermediate Language) framework, which abstracts the underlying architecture details, allowing for consistent and accurate syscall extraction across different hardware platforms [6].

Binary lifting in static analysis has been successfully employed for malware variant classification by Ramamoorthy et al. [7], where the Intermediate Representation (IR) of opcode sequences was used to create a dataset. In their study, a Random Forest classification model achieved an F1 score of 97%.

The organization of this paper is as follows: Section 2 provides the background for the study, followed by Section 3, which describes relevant research work in the field of IoT malware detection. The methodology is explained in Section 4, including statistical analysis and machine learning models used for evaluation. Section 5 presents the results obtained using the proposed methods and includes an analysis of these results. Section 6 summarizes the conclusions, followed by Section 7, which discusses the limitations and future scope.

2. Preliminaries

In this section, we discuss and provide an overview of malware static analysis and system and library syscall wrappers.

2.1. System and Library Calls

In ELF ARM binaries, both library and system calls are crucial and play an important role in program functionality. They enable binaries to interact with external resources, perform operations and access system-level functionalities.

• Syscalls: In Linux, a system call (syscall) is the primary interface through which user-space programs request services and access functionalities from the operating system kernel. It acts as a bridge between the user space, where applications run, and the kernel space, where core system functions reside. System calls enable programs to perform privileged operations and access restricted system resources, such as hardware interaction, process management, file I/O operations, network communication, and memory management as shown in Figure 1. Some system calls are architecture-specific, and their implementation and availability can vary between different hardware platforms (e.g., x86, ARM, MIPS). These differences are due to the unique characteristics and requirements of each architecture, necessitating specific handling within the kernel to optimize performance and compatibility [8].
• Library syscall wrappers: When using programming languages like C++ or Java, developers often use pre-built functions from libraries such as the GNU C Library glibc, which contains routines for file management, memory allocation, and computational tasks. When a program calls one of these library functions, it may require system-level functionalities that reside in kernel space, like hardware interaction or process management. Consequently, library functions often make system calls and provide wrapper functions to syscalls in order to perform these tasks as shown in Figure 1.
• Virtual syscalls:
  Many Linux distributions also provide optimization of certain syscalls called virtual syscalls. Virtual syscalls, or vDSO (virtual dynamic shared object) calls, are a set of performance-optimized routines provided by the Linux kernel that user-space applications can use to execute certain system calls more efficiently. These virtual syscalls are mapped directly into the process address space, allowing some system call functionalities to be executed without the overhead of a traditional syscall. vDSO calls are generally captured through dynamic analysis, as they involve runtime components and optimizations. This study does not capture virtual syscalls.

2.2. Linux Malware Static Analysis

Static malware analysis examines malicious software without execution, focusing on the code and structure of the malware to identify malicious patterns, behaviors, and potential threats. This method analyzes API calls, function calls, and data structures to understand malware logic and potential behaviors, such as file modifications, network communications, and privilege escalation attempts.

Static analysis, although powerful, has limitations—it cannot detect all system calls made during runtime, which are essential for fully understanding interactions with the operating system. Dynamic analysis complements this by capturing all interactions and changes, providing a comprehensive view of malware capabilities. The approach we propose does not seek to replace dynamic analysis; rather, it offers crucial insights into malware behavior without the need for execution.

Anti-evasive malware techniques, such as obfuscation and encryption, and packed binaries can diminish the effectiveness of static analysis significantly as demonstrated by a recent study [9].

In this paper, we introduce a novel approach that studies call patterns by extracting syscalls and library syscall wrappers from binaries through static analysis, forming the basis of the dataset used in our research.

3. Literature Review

In the past decade, malware has continued to pose a significant threat to global infrastructure. As the majority of the infrastructure systems heavily rely on the Internet of Things (IoT), the importance of Linux malware analysis has become crucial, particularly because Linux is the preferred operating system for major IoT devices and servers. In this section, we present some of the previous literature on the usage of syscalls to classify malware.

Asmitha et al. [10,11] propose a novel non-parametric statistical approach using machine learning techniques for identifying previously unknown malicious Executable Linkable Files (ELF). They used a dataset of 226 Linux malware samples and 442 benign samples from which syscalls (features) were dynamically extracted. Their proposed method prioritizes features through non-parametric statistical techniques such as the Kruskal–Wallis ranking test (KW) and Deviation From Poisson (DFP). Three learning algorithms (J48, Adaboost, and Random Forest) were used to create a prediction model using a minimal feature set extracted from the system calls. This approach yielded an optimal feature vector, resulting in an overall classification accuracy of 97.30% in identifying unknown malicious files.

Additionally, Phu et al. [12] introduce a framework for analyzing MIPS ELF files based on syscall behaviors. They compiled a database of 3773 MIPS ELF malware samples from Detux, IoTPOT, and VirusShare and executed them in a QEMU-based sandbox called F-sandbox, derived from Firmadyne and Detux. For their final feature set, the authors took the 30 most common syscalls and applied the n-gram technique to construct the feature vectors. The authors then employed machine learning classifiers such as Support Vector Machine (SVM), Random Forest, and Naive Bayes to analyze the syscalls. Their experiments demonstrated that Random Forest achieves high accuracy in classifying MIPS malware, with an accuracy of 97.44%.

Taking it a level further, Tahir and Qadir [13] proposed a detection scheme for IoT malware using cross-architectural (MIPS, ARM and x86) analysis based on system calls. They had 69 system calls dynamically extracted from 1048 samples of both IoT malware and benign binaries. They then trained five popular machine learning models: Support Vector Machine, Random Forest, Logistic Regression, Bagging, and Multi-Layer Perceptron to detect the malware. The results showed that Random Forest achieved the highest detection accuracy at 99.04%. Additionally, their findings also suggested that feature sets based on system calls hold significant potential for detecting multi-architectural IoT malware.

Shobana and Poonkuzhali [14] carried out the same using deep learning techniques. In their approach, the authors identified malware by analyzing its behavior through the sequence of system calls it generates during execution. The system calls made by IoT malware are captured using the strace tool on Ubuntu. These recorded malicious system calls undergo preprocessing using n-gram techniques to extract necessary features. The resulting system calls are then categorized into two classes—normal and malicious sequences using a Recurrent Neural Network (RNN). The results reveal that using this technique produced 98% classification accuracy.

Abderrahmane et al. [15] present a solution that will classify whether an Android application is malicious or not. The application is installed and executed simulating its usage. After execution, system calls generated by the Linux kernel are collected, processed, and fed into the Neural Network model that will be used to classify the application. The authors used a convolutional Neural Network for this research and their findings reveal that their solution could detect malicious applications with an accuracy of 93.29%.

Lastly, Ramamoorthy et al. [7] focus on using binary lifting methods to perform static analysis and extract Intermediate Representation (IR) opcode sequences for analysis. The study introduces a series of statistical entropy-based characteristics derived from these opcode sequences. By concentrating solely on function metadata and opcode entropy, their approach is versatile across different architectures. It effectively identifies malware and accurately categorizes its variations, achieving an impressive F1 score of 97%.

To the best of our knowledge, existing research predominantly focuses on syscall extraction through dynamic analysis, using tools like strace to generate extensive syscall logs as shown in

Table 1. Comparative study of research works that use syscalls for malware detection.

Research	Statistical Analysis	Arch	Features Used	Dynamic/Static	Models	Accuracy	Comparison
Asmitha Vinod [10]	non-parametric statistical methods like Kruskal–Wallis ranking test (KW), Deviation From Poisson (DFP)	-	syscalls	Dynamic	J48, Adaboost, Random forest	97.30%	Dynamic analysis *
Asmitha Vinod [11]	-	-	syscalls	Dynamic	Naïve bayes, J48, Adaboost, RF, IBK-5	97%	Dynamic analysis *
Phu et al. [12]	n-gram, chi square	MIPS	syscalls	Dynamic	RF, NB, SVM	97%	Dynamic analysis *, MIPS architecture-specific
Tahir Qadir [13]	-	x86, MIPS, ARM	syscalls	Dynamic	SVM, LR, RF, MLP, Bagging	Bagging, RF (99%)	Dynamic analysis *
Shobana Poonkuzhali [14]	-	-	syscalls	Dynamic	RNN	98.7%	Dynamic analysis *
Abderrahmane et al. [15]	-	ARM	log files, syscalls	Dynamic	CNN	93.3%	Dynamic analysis *, ARM architecture-specific
DeMarinis et al. [16]	-	x86	syscalls extracted from assembly	Static	-	-	This research is a completely different use case and not directly related since it proposes a proactive defensive mechanism, but we reference this work since syscalls are filtered from disassembly. The authors use a binary analysis-based framework that automatically limits syscalls that attackers can (ab)use.
Our research	chi-squared test, Wilcoxon test	Multi architecture	syscalls, library syscall wrappers	Static	Linear Regression, Random Forest, SVC, Neural Network	96.86%	We could not find any recent works that use syscalls for static malware analysis. Static analysis requiring no execution or environment setup. Multi-architecture support. Can be combined with other static features for a higher accuracy. We conduct statistical analysis to validate syscall ranking and category assignment. These scores can be adjusted for various architectures.

‘*’ These works use dynamic analysis which requires a compatible environment to be set up and configured for the malware to execute. Moreover, dynamic analysis tends to generate a high volume of syscalls within a short period, as the kernel performs various internal operations during its execution. This rapid generation of syscalls can lead to significant noise, with numerous irrelevant and redundant syscalls that can obscure meaningful patterns.

. This method requires executing malware within a controlled environment, which poses significant challenges in Linux due to the diversity of architectures and runtime environments. In contrast, our study uses a static analysis approach, extracting syscalls and library syscall wrappers directly from disassembled code. Although this method may not capture indirect syscalls to the kernel, we demonstrate that it remains an effective strategy for identifying malicious behavior.

4. Methodology

In this section, we provide a detailed explanation of the workflow that was utilized throughout our research. This workflow is structured into several distinct steps, each of which is outlined in this section and visualized in Figure 2 below.

4.1. Iot Malware Dataset

This paper concentrates exclusively on employing static analysis to identify system calls (syscalls) and library syscall wrappers utilized by binaries targeting the Linux kernel in ARM architecture-based Linux files, which are typically referred to as ELF (Executable and Linkable Format) binaries. Our study involves the examination of 1117 samples of ARM-based Linux IoT malware and 1214 binaries categorized as benign or non-malicious, with 251 unique syscalls as shown in Figure 3. The raw binaries are extracted from open-source IoT malware ARM architecture binary dataset from [17,18]. We selected random samples of binaries containing fewer than 2000 functions to ensure that the reverse engineering process was batched to a manageable size. Subsequent stages are illustrated as shown in Figure 2.

The proportion of syscalls by ranking and categories are shown in Figure 3 and Figure 4 which are discussed in the Section 4.7.

4.2. Reverse Engineering

This study explores the identification of malicious behavior in ARM-based Linux IoT devices by analyzing syscalls and library syscall wrappers extracted solely from static analysis of ELF binaries. We employ radare2 [19], a popular reverse engineering library, to extract syscalls and library syscall wrappers from global symbols, exports, imports, and call references within the binary.

4.3. Heuristics

Additionally, each syscall is assigned a risk level—High, Medium, or Low, reflecting its potential for malicious use. The Linux syscalls are categorized as outlined in

Table 2. Extended categorization of Linux syscalls.

Syscall Category	Description
FileSystem	Handles file management operations such as reading, writing, and permissions.
Process	Manages process lifecycle such as creation, execution, and termination.
Memory	Controls memory allocation, deallocation, change and access critical for process management.
Network	Encompasses syscalls for network communication such as socket management, send, recv.
System	General system calls for services, system configuration and management.
Metadata	Involves retrieval and manipulation of file or system metadata.
Signal	Inter-process communication, and signal-driven interruptions.
Security	Key management, encryption, and access controls.
NonblockingIO	Non-blocking operations for input/output.
Time	Measuring time, manipulation.

. To validate the effectiveness of our categorization and ranking system, we perform a chi-squared analysis [20] to compare the distribution of syscalls between benign and malware samples. This statistical validation confirms the significant differentiation in syscall usage patterns between the malware and benign subsets.

Our dataset distribution is outlined in

Table 3. Counts of binary samples.

Class	Binary Count	Syscall Counts
Malware	1117	163,288
Benign	1214	45,284

. The raw binaries are reverse-engineered to extract the the following attributes: binary_id, hash, call_api, call_description, call_category, and call_rank.

Unlike Windows environment, Linux binaries have built-in debugging symbols with DWARF which is an integral part of the ELF format. This rich context contributes to gathering relevant symbols and their usage patterns, which is crucial for assessing the potential maliciousness of each call. We have systematically categorized all Linux system calls into the following ten categories based on their functionality:

4.4. Statistical Feature Engineering

To enhance the predictive power of our models, we enriched our dataset with several engineered features derived from the syscall and library call data. These features aim to capture the nuances of syscall behavior that distinguish malware from benign software. The engineered features include the following:

• Call Count: The total number of system calls made by each binary. This feature reflects the general activity level of the binary, which can indicate suspicious behavior.
• Distinct Call Count: The number of unique syscalls made by each binary. A high variety of calls can be indicative of complex or unusual binary behavior.
• Category Frequency: For each of the ten syscall categories, we calculated the frequency of syscalls falling into each category per binary. This helps in understanding which types of operations are predominant in a binary, aiding in profiling typical and atypical behaviors.
• High-Risk Call Proportion: The proportion of syscalls that are ranked as ’High’ risk relative to the total number of syscalls. This feature specifically targets the detection of syscalls more commonly associated with malicious activities.
• Entropy of Calls: We calculated the entropy of syscall distribution within each category of a binary to measure the unpredictability and randomness of syscall usage, which can be higher in malware due to evasion techniques or diverse functionalities.
• Weighted Risk Score: By assigning weights to syscalls based on their assigned risk levels (High, Medium, and Low), we computed an overall risk score for each binary. This score provides a quantitative measure of the potential threat posed by the binary based on the observed syscalls.

Each syscall in a binary is assigned a weight based on its risk rank (High, Medium, and Low). The weighted rank score for each syscall can be calculated as:

$${\mathrm{weighted}\_\mathrm{rank}\_\mathrm{score}}_i={\mathrm{weight}}_i·{\mathrm{count}}_i$$

(1)

where ${\mathrm{weight}}_i$ is the risk weight for syscall i, and ${\mathrm{count}}_i$ is the count of syscall i within the binary.

The total rank score is the sum of all weighted rank scores across all syscalls in a binary:

$$\mathrm{total}\_\mathrm{rank}\_\mathrm{score}=\sum _{i=1}^n{\mathrm{weighted}\_\mathrm{rank}\_\mathrm{score}}_i$$

(2)

where n is the number of different syscalls recorded for the binary.

The diversity score, calculated using Shannon entropy, measures the unpredictability or diversity of syscall ranks in a binary. It is defined as:

$$\mathrm{diversity}\_\mathrm{score}=-\sum _{j=1}^m{p}_j{\log }_2(p_j)$$

(3)

where $p_j$ is the proportion of syscalls of rank j (e.g., High, Medium, and Low), and m is the number of different ranks.

The final score is a composite metric that combines the total rank score and the diversity score, providing an overall score for each binary:

$$\mathrm{final}\_\mathrm{score}=\mathrm{total}\_\mathrm{rank}\_\mathrm{score}+\mathrm{diversity}\_\mathrm{score}$$

(4)

These features were integrated into our analytical framework to provide a robust foundation for subsequent machine learning models to classify the samples effectively. By leveraging a combination of count-based, frequency-based, and information-theoretic metrics, we aim to capture a comprehensive profile of each binary’s behavior, significantly enhancing our malware detection capabilities.

4.5. Data Organization

We index the data with a binary_id for organizing syscalls within each binary, to facilitate aggregation and analysis at the binary level. By grouping syscalls by their respective binary_id, we ensure that behavioral patterns are accurately captured for the entire binary. This structured approach enhances the reliability of our analysis and improves the model’s ability to effectively learn and differentiate between distinct syscall patterns. The resulting syscall dataset created has the features as outlined in

Table 4. Description of the IoT Malware Syscall Analysis dataset.

Dataset Feature	Description
binary_id	Unique identifier for each binary
hash	Hash of the binary
Architecture	ARM—although our metholodolgy supports multiple architectures with binary lifting strategy
isMalware	Whether the binary is malicious or beningn
prc	32-bit/64-bit
Endian	The endianess of the binary (LSB or MSB)
Stripped	Binary attributes indicating whether it is stripped of symbol information
call_api	Name of the syscall
call_desc	Description of the function of the syscall
call_type	syscall or lib (library syscall wrapper)
call_cat	Category of the syscall, Refer Table 2
call_rank	Security risk level of the syscall based on its potential for malicious use (High, Medium, and Low)
bin_all_call_cnt	Count of all the syscalls and lib calls in the binary
bin_dist_call_cnt	Count of the distinct syscalls and libcalls in the binary
bin_dist_cat_cnt	Count of each category of syscalls and libcalls
bin_each_api_cnt	Count of unique syscalls in the binary
bin_calls_per_cat	Number of calls per category in a binary
bin_dist_call_per_cat	Number of calls in each distinct category in a binary
call_rank_n	Number of syscalls in the binary with the current syscall rank
bin_calls_per_rank	Number of syscalls and libcalls in each rank per binary
std_dev_calls_per_cat	Standard deviation of the syscalls per category in a binary
mean_calls_per_cat	Mean of calls per category in a binary
average_rank	Average rank of the syscalls and libcalls in a binary
weighted_rank_score	Weight of the syscall, Refer Equation (1)
total_rank_score	Sum of all weighted rank scores across all syscalls in a binary, Refer Equation (2)
diversity_score	Unpredictability of the syscall ranks in the binary, Refer Equation (3)
final_score	Total rank score added to the diversity score, Refer Equation (4)
cat_concentration_index	Number of syscall categories in the binary (Unused)

for each binary_id.

4.6. Handling of Outliers

Outliers highlight extreme behaviors or unusual patterns that are characteristic of malicious activities. Therefore, we retain outliers in our dataset to capture such behavior patterns.

4.7. Statistical Feature Analysis

To validate the effectiveness of the categorized and ranked syscalls in differentiating malware from benign binaries, we employ a chi-squared test. This statistical method evaluates whether there is a significant difference in the distribution of syscall ranks between malware and benign samples.

The test procedure is as follows:

• Data Collection: We calculate the frequency of each syscall rank for both malware and benign binaries.
• Normalization: The frequencies are normalized to proportions within each group.
• Chi-squared Test: We perform the chi-squared test to determine if the observed differences in the distribution of syscall ranks are statistically significant.

  $${\chi }^2=\sum {\displaystyle \frac{{(O_i-E_i)}^2}{E_i}}$$

  (5)

  where $O_i$ are the observed frequencies of risk rank, and $E_i$ are the expected frequencies. We use the same methodology to assess the importance of syscall categories as well.
  Figure 3 shows the proportion of each syscall category for malware and benign datasets along with the chi-squared test results. Similarly, Figure 4 shows the syscall risk rank proportion along with the chi-squared test results for the malware and benign subsets.

The results show p-value < $2.2\times {10}^{16}$ in both cases, implying that the rankings are not just arbitrary, and assigned rankings correspond to observable differences in behavior between malware and benign binaries. This reinforces the idea that higher risk ranked syscalls and certain categories of syscalls such as network, process and signal category syscalls are more critical in the context of malware activity. Also, the high-risk ranked syscalls are statistically higher in the malware subset than benign binaries.

To validate the statistical significance of the differences observed between the calculated final_score based on the entropy of the syscalls and their risk ranking, we employ the Mann–Whitney (Wilcoxon) U test. This non-parametric test is suitable for skewed data for comparison from two independent groups. Our analysis reveals a negligibly low p-value < $2\times {10}^{-16}$, confirming that the final_score computed based on Equation (4), can reliably differentiate between malware and benign samples as shown in Figure 5.

4.8. Machine Learning Classification Based on Static Extraction of Syscalls

The primary focus of this study is to identify malware based on syscalls and library syscall wrappers extracted from static analysis as opposed to dynamic execution of the malware. Based on the extracted syscalls, we outlined the statistical features added to the dataset in previous sections, which are based on syscall categorization and security risk ranking of the syscalls used by the binary.

In this section, we discuss the feasibility of machine learning (ML) classification models on the created dataset. We have evaluated the approach and dataset with commonly used ML classification models: Logistic Regression, Random Forest, LinearSVC, and Multi-Layer Perceptron (MLP) Neural Networks.

• Logistic Regression: To evaluate a simple yet effective baseline for binary classification.
• Random Forest: Random Forest classifier is robust in handling outliers and anomalies typical of malware, and suitable for complex interactions between features.
• LinearSVC: LinearSVC is often effective with high-dimensional data, and used for interpretability.
• Multi-Layer Perceptron (MLP) Neural Network: MLP is effective in capturing complex patterns and interactions in the data through a layered architecture.

As discussed in Section 5, the strong performance of all the models suggests that our approach and dataset are well suited for malware detection.

To preserve the context of syscalls and library syscall wrappers within each binary, we train our models using grouped data as discussed in Section 4.5. Syscall categories and risk ranking are One-Hot-encoded to handle categorical data effectively. We then compute the Variation Inflation Factor (VIF) to identify and mitigate multicollinearity in our feature set.

The Generalized Linear Model (GLM) using the ’glmnet’ statistical model package [21] is employed with Lasso Regression to train the glm model. Regularization is used to minimize overfitting and improve generalization.

The dataset is split in an 80:20 ratio, ensuring a balanced representation through shuffling of batches grouped by binary_id to preserve the frequency and group of syscalls.

4.9. Hyper Parameters

Most of the standard default values from the ‘scikit-learn’ library [22] were retained and minimal tuning was required for the ML models. In this section, we discuss the vectorization strategy and some of the hyperparameter settings of the individual models.

Vectorization

For text feature ‘call_api’ (syscall name), we use TF-IDF (Term Frequency–Inverse Document Frequency) vectorizer, with a maximum of 500 distinct tokens (unique Linux system calls) to extract, and an n-gram range of (1, 1). For numeric features, the mean value is used for imputation and numeric values are standardized (normalized) by removing the mean and scaling to unit variance. Categorical features, syscall category (call_cat) and syscall risk ranking (call_rank), are One-Hot-encoded. The pipeline for the data preprocessing and vectorization strategy is shown in Figure 6.

• Linear Regression A Linear Regression model is implemented with the preprocessing pipeline as shown in Figure 6 The syscall imbalance between malware and benign binaries is handled with class weights assigned by the Linear Regression model’s balanced parameter.
• Random Forest Classifier:
  Through GridSearch, we identified the optimal number of trees as 200. The default values were retained for the maximum depth of the tree, which was set to None, and Gini impurity is used for the split criteria.
• LinearSVC:
  For LinearSVC, we used the default ‘squared-hinge’ loss function with an L2 regularization and set the tolerance of $1\times {10}^{-4}$ for the stopping criteria.
• MLP Classifier:
  For the Neural Network model, we employed the rectified linear unit function (ReLU) for activation and the ‘adam’ solver for weight optimization, with an initial learning rate of 0.001.

Our dataset is supported by statistical features and validation which is evident from the comparable results across all ML models as discussed in the next section.

5. Results and Analysis

In this section, we compare and analyze the performance of various machine learning classification models on the syscall dataset comprising IoT malware and benign binaries. The primary objective is to evaluate the effectiveness of the ML models in distinguishing between malware and benign ELF binaries.

The performance metrics used to evaluate the models include accuracy, precision, and F1 score, defined as follows:

• Accuracy: The proportion of true results (both true positives and true negatives) among the total number of cases examined.

  $$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

  (6)

  where $TP$, $TN$, $FP$, and $FN$ represent the number of true positives, true negatives, false positives, and false negatives, respectively.
• Precision: The proportion of true positive results in all positive predictions.

  $$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

  (7)
• F1 score: The harmonic mean of precision and recall.

  $$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

  (8)

  $$\mathrm{F}1\mathrm{score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

  (9)

The performance metrics of the ML models are shown in

Table 5. Performance metrics of machine learning models.

Model	Accuracy	Precision	F1 Score
Random Forest	93.34%	94.71%	96.86%
Logistic Regression	92.34%	96.0%	95.06%
SVC	92.48%	96.16%	95.14%
MLP NN	92.07%	93.34%	95.03%

. The Random Forest classifier has the best performance with an F1 score of about 97%. The optimal threshold for the final prediction was found to be 0.72, and the maximum F1 score of 96.86%.

5.1. Discussion

As observed from the results in Table 5, batching syscalls for a single binary in the course of training likely plays a crucial role in all ML model’s ability to detect malware binaries accurately since the syscall frequency and order are maintained. The Random Forest classifier performed the best, with a 96.8% F1 score likely due to the model’s ability to handle outlier data which is characteristic of malware binaries. For example, there could be rarely used syscalls, or a direct syscall instead of library wrappers in malware which are anomalous behaviors.

The confusion matrices for the machine learning classifiers provide detailed insights into the performance of each model in distinguishing between malware and benign binaries. The confusion matrix is critical in understanding the classification performance, as it breaks down the results for the true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). By examining the confusion matrices, we can better understand the types of errors each model makes.

• True positives (TP): Correctly identified malware binaries.
• False positives (FP): Benign binaries incorrectly classified as malware.
• True negatives (TN): Correctly identified benign binaries.
• False negatives (FN): Malware binaries incorrectly classified as benign.

5.2. Error Analysis

The results suggest that most ML classifier models are able to accurately identify malware behavior based on static syscall analysis since the dataset includes statistical scores for the binaries based on syscall ranking and category.

The LinearSVC classifier, Logistic Regression and MLP Neural Network all show a similar number of true positives as shown in Figure 7, Figure 8 and Figure 9, respectively.

Analyzing Confusion Matrix of Random Forest Classifier which is the best performing model as shown in Figure 10, the labels with value 1 indicate malware and value 0 indicates benign syscalls. The Random Forest classifier accurately classified 158,223 malware syscalls with only 5065 misclassifications. The model accurately classified 36,464 benign syscalls with 8820 misclassifications. Although malware binaries are characterized by extreme outliers, the Random Forest model demonstrates a higher degree of accuracy and precision. It should be noted that all the syscalls in a single binary are set to malware or benign by the model based on the scoring and other features. The performance of the model is conducted on an individual syscall level as shown by the confusion matrix, indicating the actual performance of the models is even better.

Logistic Regression and SVC models demonstrated strong performance but had slightly higher false negative rates. This suggests that while they are effective at identifying benign binaries, they may miss certain types of malware that exhibit less common syscall patterns.

The MLP Neural Network, while effective, showed a marginally higher rate of false positives, indicating it might be more prone to overfitting on benign syscall patterns. This could be mitigated by further tuning the network architecture or using regularization techniques.

The calculated AUC (Area Under the Curve) score for the ROC (Receiver Operating Characteristic) curve of the Random Forest classifier is 0.964, which demonstrates robust performance under various threshold as shown in Figure 11. The optimal threshold was found to be 0.72, with a F1 score of 96.8%.

6. Conclusions

This study successfully demonstrates the viability of static analysis for detecting malware in Linux IoT systems by analyzing call patterns (syscalls and library syscall wrappers) extracted from ELF binaries. By ranking and categorizing calls based on their security implications, we have established a methodological framework that distinguishes between benign and malicious binaries effectively. The application of a chi-squared test further validates the significant differences in call patterns between these groups, reinforcing the reliability of our categorization and ranking system.

We demonstrate the efficacy of call pattern analysis from static analysis by using machine learning models such as glmnet with Lasso Regression, Random Forest classifier, SVC and MLP Neural Network. The models were able to perform malware detection with a high level of accuracy, as indicated by a Random Forest classifier F1 score of 96.86%. This is particularly significant given that this study solely uses system calls extracted using static analysis without any other significant contributing features that are important in static analysis such as sections, functions, strings and file header information. Our approach mitigates the risks associated with dynamic analysis and offers an effective alternative for malware detection.

7. Future Works

While this study demonstrates the viability of static analysis for detecting malware in Linux IoT systems with a high accuracy rate, it primarily focuses on syscalls and library syscall wrappers. However, this narrow focus may not capture all of the malware behavior patterns. Future research could consider incorporating additional static features, such as section headers, functions, and strings, for a comprehensive detection strategy.

Additionally, static analysis does not capture dynamically generated syscalls, potentially leading to incomplete data. The limitations of dynamic analysis are already discussed in this study. Augmenting static analysis with dynamic analysis in future studies could provide a more comprehensive understanding of malware behavior and improve detection rates.

Building on the findings of this study, future research will aim to extend the scope of the static analysis framework. The study could also be extended to multiple architectures for evaluation since the methodology proposed is architecture-agnostic, thereby broadening its applicability in the field of IoT malware detection.