**6. Ransomware Detection**

#### *6.1. Ransomware-Detection Methods*

The two main types of ransomware-detection methods are automated and manual. Employing technologies to identify and report ransomware attacks is a prerequisite for automated methods. These tools are typically software programs that have the potential to be able to stop attacks. Techniques for manual detection focus on routinely scanning data and devices for indicators of attacks. Checking to see if a malware attack has not modified data or stopped authorized users from accessing their devices or files includes looking at any changes to file extensions, the accessibility of devices and files by authorized users, and any changes to file extensions. The flow of the presentation in this section is illustrated in Figure 7.

**Figure 7.** Ransomware detection taxonomy.

#### 6.1.1. Manual Ransomware Detection

Manual ransomware detection refers to the process of detecting ransomware through human analysis and intervention rather than automated systems. This approach involves analyzing system logs, network traffic, and other indicators of compromise to identify patterns and behaviors associated with ransomware attacks. While manual detection can be time-consuming and resource-intensive, it can be an effective complement to automated detection methods, as it can help identify new or unknown types of ransomware that may not be detected by automated systems [30].

Despite its effectiveness, manual ransomware detection has some limitations. It can be labor-intensive and requires highly trained personnel to analyze system logs and network traffic. Additionally, manual detection may not scale well in large organizations or networks, where automated detection methods may be more efficient [30].

#### Scanning

Manual ransomware-detection scanning is a technique used to detect ransomware through the manual analysis of files and systems. This approach involves scanning individual files or systems for signs of ransomware activity, such as encrypted files or abnormal network traffic. Manual scanning can be a complementary approach to automated scanning methods, as it can help detect new or unknown types of ransomware that may not be detected by automated systems [30].

While manual ransomware-detection scanning can be effective, it has some limitations. It can be time-consuming and labor-intensive, especially when scanning large networks or systems. Additionally, manual scanning may generate false positives, which can be disruptive to normal system operations [30].

#### 6.1.2. Automated Ransomware Detection

The current methods for detecting ransomware primarily involve monitoring the system at the file system level. Automated approaches to detecting ransomware can be categorized into two main groups: those based on artificial intelligence (AI) and those that are not based on AI. AI-based methods typically employ machine learning (ML), deep learning (DL), and artificial neural network (ANN) techniques to detect ransomware. Some tools utilize variations of these techniques or a hybrid approach that combines two or more techniques to combat the threat of ransomware attacks. Non-AI methods rely on packet inspection and traffic analysis to detect ransomware. One of the major advantages of automated approaches is their ability to detect, block, and recover from ransomware attacks without human intervention. Additionally, these tools are highly accurate and reliable in terms of detecting, preventing, and recovering from ransomware attacks [31].

#### Artificial-Intelligence-Based Approaches

Artificial intelligence (AI) techniques, including machine learning, deep learning, and artificial neural networks, have been utilized for automated ransomware detection. These techniques involve the use of behavioral techniques, as well as static and dynamic analysis, to identify and prevent ransomware attacks. Machine learning algorithms can learn from previous ransomware attacks and detect new variants by analyzing patterns and behaviors. On the other hand, deep learning methods can leverage neural networks to detect ransomware attacks by analyzing large amounts of data. Artificial neural networks can also be used to identify ransomware by processing and analyzing multiple data sources. These AI-based approaches offer a more efficient and reliable way to detect and prevent ransomware attacks, reducing the potential impact on businesses and individuals [31]. AI-based approaches include the following:

#### 1. Machine Learning Approaches

Machine-learning-based detection is a more advanced approach that relies on training a machine learning model to detect ransomware based on its behavior patterns or features. This approach is based on collecting a large dataset of benign and malicious samples, extracting relevant features from them, and then training a machine learning model to classify new samples as peaceful or hostile based on their characteristics [32,33].

Machine-learning-based detection has several benefits, including its ability to detect new or unknown ransomware variants that do not match existing signatures or patterns and to adapt to changing ransomware behavior patterns over time. Moreover, this approach is less prone to false positives than signature-based and heuristic-based detection, as it relies on detecting actual behavior patterns rather than static code signatures or predefined rules. However, machine-learning-based detection is limited by its reliance on a large and representative dataset of training samples and by its susceptibility to adversarial attacks that can manipulate the features or behavior of the ransomware to evade detection [31].

a. Machine Learning Algorithms for Ransomware Detection

A particular kind of artificial intelligence known as machine learning enables computer systems to improve their performance on a given job without being explicitly taught. Malicious ransomware malware encrypts a victim's files and demands payment for the decryption key. Due to their rising prevalence and severity, machine learning techniques are increasingly needed to identify and stop ransomware attacks. Table 4 lists the machine learning algorithms that are employed. Support vector machines, decision trees, random forests, k-nearest neighbors, XGBoost, and logistic regression are just a few machine learning approaches that can detect ransomware. Each method has advantages and disadvantages, and the best approach depends on the situation and the data [1,6].


**Table 4.** Machine learning algorithms.

Decision trees are a simple and intuitive machine learning algorithm that can be used for classification tasks, including ransomware detection. Decision trees work by recursively partitioning the data into subsets based on the values of the features and creating a treelike structure representing the decision-making process. Both categorical and continuous

components can be handled by decision trees, which are simple to interpret but susceptible to overfitting and sensitive to minute changes in the data [13,31,34].

Random forests are an extension of decision trees that improve performance and reduce overfitting. By randomly selecting features and data, random forests create multiple decision trees and combine their predictions. They are better-equipped to handle highdimensional data and are less likely to overfit. However, they can be computationally demanding and difficult to interpret [17].

Support vector machines are reliable machine learning techniques that can be utilized for ransomware detection and classification and regression applications. Support vector machines operate by identifying the hyperplane that divides the data into distinct classes according to the values of the features as thoroughly as possible. Support vector machines can effectively handle high-dimensional data. They can accept both linear and nonlinear borders, but the choice of the kernel function and its parameters may impact them [14].

k-NN is a non-parametric algorithm used for classification and regression tasks. It works by finding the k closest data points in the training set to a given input, and then predicting the label of the input based on the most common label among those k neighbors. It is a simple but effective algorithm that can be used in a wide range of applications [36,37]

XGBoost (short for "Extreme Gradient Boosting") is a powerful machine learning algorithm that is especially popular for gradient boosting tasks. It uses a combination of decision trees and gradient boosting to create a highly accurate model that can handle large datasets and complex feature interactions. XGBoost has become widely used in the industry [38].

Logistic regression is a parametric algorithm used for binary classification tasks (i.e., where the output is one of two possible classes). It works by modeling the probability of the output class as a function of the input features. The algorithm is trained to find the optimal parameters that maximize the likelihood of the training data and can be regularized to prevent overfitting [39].

The choice of a machine learning algorithm for ransomware detection depends on the specific problem and data available. Decision trees, random forests, support vector machines, and neural networks are all effective options, and researchers have successfully used each of these algorithms for ransomware detection in different contexts [5,31].

#### 2. Deep Learning Approaches

Deep learning techniques have been proposed as a solution to address the limitations of traditional supervised ransomware-detection tools to enhance the accuracy and reliability of ransomware detection. These algorithms utilize automatic feature generation and are wellsuited to handle unstructured datasets, requiring minimal or no human intervention due to their self-learning capabilities. Their effectiveness in classifying audio, text, and image data makes them particularly useful in detecting textual and image-based ransomware data. However, training deep learning algorithms demand a considerable amount of data, which may render them unsuitable for general-purpose applications, particularly those involving small datasets or sizes. Other challenges associated with deep learning include the need for high processing power and difficulty with adapting to real-world datasets [6,40].

3. Artificial Neural Network Approaches

Artificial neural network approaches are well-suited for detecting various types and variants of ransomware data, including text and image ransomware variants, due to their wide range of applications. Neural networks are an excellent choice for adapting to new ransomware data and identifying zero-day attacks because of their ability to continuously learn. The versatility of neural networks makes them highly effective in detecting different forms of ransomware data and adapting to new threats. However, these techniques are dependent on hardware and can be vulnerable to data dependencies, as well as the blackbox nature of the technology, which limits the ability of human analysts to monitor data processing and identify deviations in the process [5,6,41].

Non-Artificial-Intelligence-Based Methods

Non-AI techniques such as packet inspection and traffic analysis can be utilized to detect ransomware. Anomaly detection is one effective algorithm used to detect ransomware. These algorithms analyze network traffic and identify patterns that deviate from normal behavior. Unusual patterns of network traffic, such as a sudden increase in file encryption activity or a large number of outbound network connections to suspicious IP addresses, are indications of ransomware activity. By comparing network traffic to a baseline of normal behavior, anomaly-detection algorithms can quickly identify and alert security teams to potential ransomware attacks [2].

Other non-AI techniques include signature-based detection, which involves comparing network traffic to known ransomware signatures, and behavior-based detection, which looks for patterns of behavior consistent with known ransomware attacks [2].

Another approach involves the use of honeypots to monitor network activity and detect the presence of ransomware. This method entails the establishment of a honeypot folder and observing any changes that may indicate the presence of ransomware. The early detection of ransomware is critical in mitigating its impact and preventing further damage [2].

It is important to note that these detection techniques are not foolproof and should be used in conjunction with other security measures such as user education, regular backups, and security patches [2].

Antivirus software is an example of a non-AI-based approach for detecting and preventing malware, including ransomware. It typically uses a combination of signaturebased detection and behavior-based detection to identify and block malicious software. Signature-based detection involves comparing files against a database of known malware signatures, while behavior-based detection looks for patterns of behavior that are indicative of malware activity. While antivirus software has been an effective tool for detecting and preventing malware, it has some limitations. For example, signature-based detection is only effective against known malware signatures, meaning that new or unknown forms of malware can bypass this detection method. Additionally, some types of malware can be designed to evade behavior-based detection methods [42].

In recent years, AI-based approaches, such as machine learning and deep learning, have been introduced to enhance the accuracy and effectiveness of malware detection. However, antivirus software continues to be a critical component of cybersecurity, particularly for organizations with limited resources or expertise in AI-based techniques. By using a combination of signature-based and behavior-based detection, antivirus software can provide an effective defense against known and unknown forms of malware, including ransomware [42].

#### 1. Packet Inspection

Packet inspection refers to examining individual data packets' contents as they move through a network. This technique can be used to detect the presence of malware by identifying packets that contain suspicious data or have characteristics that are inconsistent with normal network traffic. For example, packets containing large amounts of encrypted data or sent from suspicious IP addresses may indicate ransomware activity [43,44].

#### 2. Traffic Analysis

Traffic analysis, on the other hand, involves the examination of patterns of network traffic over a period of time. This technique can be used to detect ransomware by identifying patterns of behavior that are consistent with known ransomware attacks. For example, traffic analysis may reveal a sudden increase in network traffic during off-hours or a large number of outbound network connections to suspicious IP addresses. Packet inspection and traffic analysis are two important techniques used in detecting malicious software, including ransomware. These techniques involve the examination of network traffic to identify potentially harmful data packets and patterns of behavior that may indicate the presence of malware. By examining network traffic and identifying patterns of behavior

indicative of malicious activity, these techniques can help organizations detect ransomware attacks and protect their critical data and systems [45,46].

Packet inspection and traffic analysis are two essential techniques for detecting ransomware and other forms of malware. By examining network traffic and identifying behavior indicative of malicious activity, these techniques can help organizations detect ransomware attacks and protect their critical data and systems. They should be used alongside other security measures, such as regular backups and security patches, as they are not completely infallible. Furthermore, these techniques necessitate specialized tools and expertise, which can pose a challenge for organizations without dedicated cybersecurity resources [43–46].

#### *6.2. Ransomware-Detection Techniques*

Ransomware detection is a critical component of cybersecurity, and various techniques have been developed to detect ransomware attacks. This section will discuss different ransomware-detection techniques proposed in the literature and their strengths, weaknesses, and limitations.

### 6.2.1. Signature-Based Detection

Signature-based detection is a traditional approach that relies on identifying known ransomware signatures or patterns in the code or behavior of the malware. This approach is based on creating a database of known ransomware signatures or marks and scanning the system or network for matching signatures or patterns. If a match is found, the ransomware is flagged as malicious and appropriate actions are taken [32,33].

One benefit of signature-based detection is its simplicity and effectiveness in detecting known ransomware variants. However, this approach is limited by its inability to detect new or unknown ransomware variants that do not match existing signatures or patterns. Moreover, attackers can easily evade signature-based detection by modifying the code or behavior of the ransomware to avoid detection [31].

## 6.2.2. Heuristic-Based Detection

Heuristic-based detection is a more advanced approach that identifies ransomware behavior patterns or anomalies indicative of malicious activity. This approach is based on creating rules or heuristics that describe typical ransomware behavior and then monitoring the system or network for any deviations or anomalies from these rules. If such variations or abnormalities are detected, the ransomware is flagged as suspicious or malicious, and appropriate actions are taken [32,33].

One of the advantages of heuristic-based detection is its ability to detect new or unknown ransomware variants that do not match any existing signatures or patterns. Moreover, this approach is less prone to false positives than signature-based detection, as it relies on detecting actual behavior patterns rather than static code signatures. However, heuristic-based detection is limited by its reliance on predefined rules or heuristics, which may only capture some possible ransomware behavior patterns or anomalies. Moreover, attackers can easily evade heuristic-based detection by modifying the behavior of the ransomware to avoid detection [31].

#### 6.2.3. Network-Based Detection

Network-based detection is an approach that relies on monitoring the network traffic for suspicious or malicious activity that may be indicative of a ransomware attack. This approach is based on analyzing the network traffic for anomalies or patterns characteristic of ransomware, such as large volumes of outbound traffic, unusual network connections, or network traffic encryption [32,33].

One of the advantages of network-based detection is its ability to detect ransomware activity even if the malware has not yet infected the system or if the ransomware is using non-standard encryption methods. Moreover, this approach is less prone to false positives than other detection approaches, as it relies on detecting actual network traffic patterns rather than static code signatures or predefined rules. However, network-based detection is limited by its reliance on network traffic analysis tools that may not be available or may not capture all ransomware activity. Moreover, attackers can easily evade network-based detection by encrypting their network traffic or using stealthy communication channels [31].
