Chaotic-Based Shellcode Encryption: A New Strategy for Bypassing Antivirus Mechanisms

Abstract

This study employed chaotic systems as an innovative approach for shellcode obfuscation to evade current antivirus detection methods. Standard AV solutions primarily rely on static signatures and heuristic analysis to identify malicious code. However, chaotic systems employ dynamic and unpredictable encryption methods, significantly obstructing detection efforts. The utilization of various chaotic maps for shellcode encryption facilitates the generation of multiple unique variations from the same functional code, each exhibiting distinct unpredictability due to the inherent nonlinearity and sensitivity of chaotic systems to initial conditions. The unpredictability of these situations poses a considerable challenge for antivirus software in recognizing consistent patterns, resulting in decreased detection rates. The findings from our experiments demonstrate that chaos-driven encryption methods significantly outperform traditional encryption techniques in terms of evading detection. This paper emphasizes the potential of chaos theory to enhance malware evasion strategies, offering a sophisticated approach to bypassing modern antivirus protections while ensuring the effectiveness of malicious payloads.

1. Introduction

Innovative methods for concealing malicious code are continually evolving in the ongoing conflict between malware authors and antivirus (AV) solutions [1]. Malware, an abbreviation for malicious software, encompasses a wide range of harmful programs designed to disrupt, damage, or gain unauthorized access to computer systems. These include viruses, worms, trojans, and ransomware that can steal private information, damage files, or take over computers. Malware assaults usually result in significant financial losses, reputational damage, and operational disruptions for both individuals and enterprises, which can have catastrophic consequences. Shellcode, commonly used to exploit software vulnerabilities, consists of code segments that execute arbitrary instructions on a target system, constituting a significant element of malware attacks. It is an effective weapon in the cybercriminal’s arsenal; once injected into the target machine’s memory, it enables attackers to seize control or execute malicious actions. Investigating malware and shellcode is crucial for developing more effective defensive strategies, as malware significantly contributes to cybersecurity threats. Malware developers must enhance their techniques to maintain efficacy as AV technology advances [2].

Obfuscation techniques such as metamorphism and polymorphism focus on altering code structure to hide it from detection. These approaches adhere to detectable transformation patterns that AV systems can finally learn to recognize [3], regardless of their capacity to generate numerous versions of the same virus. Chaotic encryption provides a unique benefit due to the inherent nonlinearity and unpredictability of chaotic systems which introduce an additional degree of complexity. Chaotic encryption generates highly randomized, dynamic encryptions that lack established protocols, in contrast to traditional obfuscation. This significantly complicates the ability of AV softwareto recognize consistent indicators in the code, hence enhancing the malware’s capability to circumvent even the most sophisticated detection systems. By examining chaotic encryption, researchers may explore more robust evasion techniques that not only outperform traditional methods but also establish new frontiers in malware defense and cybersecurity.

This study’s main contribution is demonstrating how chaotic encryption techniques can be used to make shellcode more stealthy, so it can avoid detection by AV programs. Using the nonlinearity and high sensitivity of chaotic systems, our suggested method creates an encrypted shellcode that is both dynamic and unexpected. The experimental results reveal that this method is light years ahead of more conventional methods of evasion. This research presents a fresh viewpoint on how current AV technology could be sidestepped by showcasing the possibilities of chaos theory in malware evasion tactics.

This study is organized as follows: Section 2 examines pertinent literature, Section 3 delineates the technique and encryption procedure, Section 4 presents the experimental findings, and Section 5 concludes with prospective research avenues.

2. Related Work

2.1. Antivirus Detection Methods

Commonly used in malware creation, obfuscation is used to hide malicious code [4]. Traditionally, AV systems have mostly depended on signature-based detection, which detects malware by comparing a file against a database of known hostile signatures [5]. This method struggles to identify new or altered malware, including polymorphic or metamorphic versions that change their appearance with each iteration [6], even though it is successful against established threats. Heuristic-based detection tries to identify malware depending on suspicious behavior patterns in order to overcome this constraint. However, this approach may overlook malware that dynamically alters its own behavior [7] and is prone to false positives.

Antivirus (AV) systems have progressively included artificial intelligence (AI) and machine learning (ML) to improve their detection power as malware develops in sophistication. With anomaly detection as a basic use, ML-based systems examine patterns in large-scale datasets to identify previously unidentified threats. These complex algorithms highlight deviations that might point to hostile activities and train models on proper software behavior. Complementing this method, behavioral-based detection tracks file access patterns, network traffic, and system calls to identify malware activities. Leading this technological arms race, deep learning models—including convolutional neural networks (CNNs) and recurrent neural networks (RNNs)—are being used to forecast hostile behavior before it occurs [8,9].

2.2. Polymorphic and Metamorphic Malware

Malware creators have devised metamorphic and polymorphic approaches [10] to avoid signature-based detection. Complicating detection, polymorphic malware encrypts its payload, decrypting it during runtime, and changes its encrypted form with every iteration. By rewriting its own entire code during each infection, metamorphic malware goes a little further and generates functionally comparable but structurally distinct variations [11]. Though they rely on modifications that AV systems can ultimately learn to recognize [12], these techniques can evade both signature-based and heuristic detection methods.

Recent years have seen hackers merging metamorphic and polymorphic methods with packing and virtualization-based obfuscation [13]. Packers encrypt or compress executables, therefore, preventing AV analysis until runtime. Virtualization-based obfuscation adds complexity to the execution environment [14] by converting malware code into bespoke virtual machine (VM) bytecode, therefore, complicating detection. These hybrid methods challenge even contemporary AV systems, necessitating advanced detection methods such as dynamic analysis in sandbox settings to monitor malware’s interactions with the operating system and uncover evasive behavior [15].

2.3. Obfuscation Techniques for Malware Evasion

Malware analysis and detection are complicated by obfuscation methods including instruction replacement, garbage code insertion, control flow flattening, and instruction sequencing [16]. These techniques complicate the reverse-engineering of the code and prevent its discovery. Different tools and frameworks apply these approaches, so traditional AV solutions lose their efficiency.

Nevertheless, AV technologies are evolving to counter obfuscation techniques as they become more advanced. Modern AV systems use methods such as machine learning and dynamic analysis to identify unusual execution patterns or excessive complexity in program behavior [17], thereby detecting obfuscated code. Although obfuscation is still useful to some extent, the constant arms race between malware creators and antiviral programs emphasizes the drive for constant innovation in both malware methods and cybersecurity protections.

2.4. Well-Known Evasion Tools

2.4.1. TheFatRat

Particularly used to produce payloads that elude AV detection [18], the popular utility TheFatRat aids in constructing obfuscated malware. Using a range of obfuscation and encryption methods, it lets users develop backdoors and payloads challenging for AV systems to detect. TheFatRat adds additional levels of obfuscation to evade signature-based detection and automates the injection of malicious payloads into permitted programs.

2.4.2. Shikata Ga Nai

This is a somewhat popular encoding method in the Metasploit framework. Shikata Ga Nai employs polymorphic encoding to produce several variations of the same payload, therefore, altering the shellcode’s appearance without compromising its performance. This method continually modifies the structure of the shellcode, consequently defeating signature-based AV detection [19].

2.4.3. Veil Framework

Veil is another obfuscation tool designed to generate payloads that can evade AV systems. It allows users to encode payloads using different methods, including encryption and polymorphic techniques, to avoid detection by static and heuristic analysis. Veil supports various languages and formats for payload generation and focuses on bypassing real-time protection [20].

2.4.4. ROPInjector

Using return-oriented programming (ROP), ROPInjector is a tool for translating shellcode into its ROP equivalent and injecting it into portable executable (PE) files with the aim of AV evasion [21,22,23]. ROPInjector uses innocuous code gadgets already existing in the target PE, unlike conventional polymorphism techniques requiring writable code sections, therefore, making the injection process more difficult for AV software to detect. By converting shellcode into ROP code, the tool essentially thwarts behavioral detection techniques [24] and signature-based detection systems. New static analysis methods for ROP code have recently been presented, allowing for the examination of challenging payloads without first having to replicate the ROP activation environment. Furthermore, improving the identification of complex ROP-based assaults [25], a guessing technique helps in the discovery of gadget origins in payloads from network traffic or documents.

2.4.5. Movfuscator

Designed to replace traditional x86 instructions with equivalent MOV instructions, Movfuscator is an obfuscation tool [26]. The Turing completeness specific to the MOV instruction makes this transformation feasible, as it lets all the instructions be replaced with sequences of MOV instructions while maintaining the original logic of the program. MOV-based obfuscation is unique, so no particularly effective and accessible deobfuscation techniques have yet been developed [27]. Deobfuscation tools such as de-movfuscator can only partially deobfuscate the code, offering a starting point, but they are not yet able to completely eradicate this kind of obfuscation [28].

2.5. Chaotic Systems in Cryptography

Chaotic systems have been investigated across several domains, including cryptography, data encryption, and secure communications, owing to their intrinsic unpredictability, sensitivity to the beginning circumstances, and nonlinear dynamics [29]. Chaotic systems in cryptography have been used to create encryption algorithms that produce extremely unpredictable key streams, which are challenging to anticipate or replicate without access to the starting parameters [30,31]. Investigations in this domain have demonstrated that chaotic systems can generate secure and efficient encryption methods appropriate for settings with constrained computational capabilities [32]. J. Fridrich suggested the encryption of $N\times N$ images using two-dimensional chaotic maps [33]. The author demonstrated that two-dimensional chaotic maps when modified and discretized, can facilitate the development of symmetric block encryption systems characterized by robust diffusion qualities, exemplified by the Baker map for safe encryption. Another study presented a multimedia communication method utilizing 2D alteration models and integrating several chaotic maps for shuffling and substitution [34]. This exhibits robust resilience to assaults and enhanced performance relative to conventional approaches. The use of chaos theory in shellcode obfuscation remains inadequately examined, presenting the potential for innovative strategies in malware avoidance.

Despite these developments, the use of chaos theory for obfuscating shellcodes is still unexplored territory. In order to improve malware evasion tactics, this research proposes a new route for current malware creation and tests these techniques in real-world AV systems using lightweight chaotic maps.

Table 1. Comparison of evasion techniques: advantages and disadvantages of various methods for shellcode detection evasion.

Method	Advantages	Disadvantages
TheFatRat	Automates payload injection and applies multiple obfuscation techniques, challenging AV detection.	Susceptible to detection by advanced AV systems, as many AV tools recognize common patterns in generated payloads.
Shikata Ga Nai	Polymorphic encoding alters the appearance of shellcode without changing functionality, defeating static analysis.	Limited effectiveness, as AV systems can learn to detect the encoding pattern with multiple iterations.
Veil Framework	Supports multiple languages and formats for payload generation, using encryption and polymorphic techniques.	Less effective against dynamic analysis and sandboxing; may still be flagged by heuristic or behavioral-based AV systems.
ROPInjector	Converts shellcode into return-oriented programming (ROP) gadgets, complicating detection.	Complex to implement and limited by the availability of suitable ROP gadgets within the target environment.
Movfuscator	Replaces instructions with equivalent MOV instructions, creating unique obfuscated code.	Limited deobfuscation techniques exist, but AV systems may still detect certain patterns; high performance overhead.
Chaotic Encryption (Proposed)	Generates randomized, dynamic encryption with chaotic systems, offering high unpredictability and evasion capabilities.	Sensitive to parameter tuning; if the parameters are not optimized, the system may stabilize, reducing evasion effectiveness.

provides a comparative analysis of various evasion techniques, highlighting the advantages and limitations of each method in circumventing AV detection.

3. Methodology

This section delineates the encryption procedure using a chaotic system. Algorithm 1 employs a chaotic system to produce random values (chaotic values) for the encryption of a specified plaintext (shellcode) P. The method processes each byte of the plaintext, extracting chaotic values from the system, which are then normalized to a range between 0 and 255. If the chaotic value is zero, a new chaotic value is created, as XOR with zero does not effectuate any encryption as $p\oplus 0=p$. The resultant ciphertext C is derived by executing the XOR operation between the plaintext byte and the chaotic key.

Algorithm 1: Chaotic Encryption Algorithm

Input: Plain text (shellcode) P Output: Cipher text C

As shown in Algorithm 2, because XOR is a reversible operation, the plaintext P can be recovered from the ciphertext C using the same chaotic key k obtained from the system [35].  

Algorithm 2: Chaotic Decryption Algorithm

Input: Cipher text (encrypted shellcode) C Output: Plain text P

Chaotic systems have the benefit of producing consistent random values k from identical beginning circumstances, hence guaranteeing the congruence of encryption and decryption keys. The values produced by chaotic systems demonstrate significant randomness and unpredictability, making it difficult for AV to identify discernible patterns or signatures inside the encrypted ciphertext.

This ensures that the encryption is reversible and the original data may be recovered when needed, given that the identical parameters for the chaotic system are used for decryption. The Lorenz system [36], Chua’s circuit system [37], and the Rossler system [38] use the Runge–Kutta fourth Order (RK4) method for solving first-order ordinary differential equations (ODEs) [39,40], offering an effective equilibrium between precision and computational efficiency [41]. It is especially advantageous for chaotic systems because of its stability and accuracy in extended simulations.

$$k_1=f(t_n,y_n)$$

(1)

$$k_2=f\left(t_n+{\displaystyle \frac{dt}{2}},y_n+{\displaystyle \frac{k_1·dt}{2}}\right)$$

(2)

$$k_3=f\left(t_n+{\displaystyle \frac{dt}{2}},y_n+{\displaystyle \frac{k_2·dt}{2}}\right)$$

(3)

$$k_4=f\left(t_n+dt,y_n+k_3·dt\right)$$

(4)

$$y_{n+1}=y_n+{\displaystyle \frac{dt}{6}}\left(k_1+2k_2+2k_3+k_4\right)$$

(5)

In this method, $k_1$, $k_2$, $k_3$, and $k_4$ are intermediate slopes used to estimate the solution at the next step $y_{n+1}$. The final result is a weighted average of these slopes, making RK4 highly accurate for solving first-order ODEs. The method allows for higher precision and stability compared to lower-order methods, especially when applied to chaotic systems such as the Lorenz, Chua’s circuit, and Rössler systems.

3.1. Logistic Map Encryption/Decryption

The logistic map is a mathematical function that models chaotic behavior in dynamic systems. It is widely studied in chaos theory because it demonstrates how complex, unpredictable behavior can emerge from simple, nonlinear equations. The logistic map is defined as follows:

$$x_{n+1}=r·x_n·(1-x_n)$$

(6)

• $x_n$: The current state of the system, typically within the range $0<x_n<1$.
• r: The control parameter.
• n: The iteration index.

The most interesting dynamics occur when r is between 3.57 and 4, where the system transitions from periodic behavior to full chaos [42]. The implementation source code is from Listing A1 in Appendix A.1.

3.2. Henon Map Encryption/Decryption

For the Henon Map, the values of x (and y) can vary depending on the parameters a and b. The Henon Map is defined as follows:

$$x_{n+1}=1-a{x}_n^2+y_n$$

(7)

$$y_{n+1}=b{x}_n$$

(8)

• a and b: The control parameters that influence the system’s dynamics.
• $x_n$ and $y_n$: The state variables at iteration n.

The system exhibits chaotic behavior for typical values $a=1.4$ and $b=0.3$ and is widely studied in chaos theory [43]. The implementation pseudocode is from Listing A2 in Appendix A.1.

3.3. Lorenz System Encryption/Decryption

The Lorenz system is a system of three coupled, nonlinear differential equations originally developed to model atmospheric convection. The system is known for its chaotic behavior, which emerges under specific parameter conditions. The Lorenz system is defined as follows:

$${\displaystyle \frac{dx}{dt}}=\sigma (y-x)$$

(9)

$${\displaystyle \frac{dy}{dt}}=x(\rho -z)-y$$

(10)

$${\displaystyle \frac{dz}{dt}}=xy-\beta z$$

(11)

• $\sigma$, $\rho$, and $\beta$: Control parameters that influence the system’s dynamics.
• x, y, and z: Initialize the start value.

For $\sigma =10$, $\rho =28$, and $\beta =8/3$, the system exhibits the well-known Lorenz attractor. The implementation source code is from Listing A3 in Appendix A.1.

3.4. Chua’s Circuit System Encryption/Decryption

Chua’s circuit, composed of standard electronic components, became one of the first recognized circuits to demonstrate chaos. Chua’s circuit is highly sensitive to its initial conditions, which is a hallmark of chaotic systems. This can be described by the following set of coupled nonlinear ordinary differential equations as

$${\displaystyle \frac{dx}{dt}}=\alpha \left(y-x-g\left(x\right)\right)$$

(12)

$${\displaystyle \frac{dy}{dt}}=x-y+z$$

(13)

$${\displaystyle \frac{dz}{dt}}=-\beta y$$

(14)

where $\alpha$ and $\beta$ are system parameters. $g\left(x\right)$ is a piecewise linear function modeling the nonlinearity introduced by Chua’s diode and is defined as

$$g\left(x\right)=\left\{\begin{array}{cc}{m}_{1}x+m_1-m_0,\hfill & \mathrm{if}x\le -1\hfill \\ m_{0}x,\hfill & \mathrm{if}-1\le x\le 1\hfill \\ m_{1}x+m_0-m_1,\hfill & \mathrm{if}x\ge 1\hfill \end{array}\right.$$

• $\alpha$ and $\beta$: Control parameters for the system’s dynamics.
• $m_0$ and $m_1$: Parameters defining the nonlinearity of $g\left(x\right)$.
• x, y, and z: Initialize the start value.

The typical values for chaotic behavior are $\alpha =9.0$ and $\beta =14.286$ [44]. For the above parameter values, Chua’s circuit exhibits chaotic behavior, making it a widely studied system in chaos theory. The implementation source code is from Listing A4 in Appendix A.1.

3.5. Rössler System Encryption/Decryption

The Rössler system is a simple three-dimensional dynamical system that exhibits chaotic behavior. It was introduced as a model of chemical reactions but has since been widely used to study chaos. The Rössler system is defined as follows:

$${\displaystyle \frac{dx}{dt}}=-y-z$$

(15)

$${\displaystyle \frac{dy}{dt}}=x+ay$$

(16)

$${\displaystyle \frac{dz}{dt}}=b+z(x-c)$$

(17)

• a, b, and c: Control parameters that determine the system’s behavior.
• x, y, and z: Initialize the start value.

For $a=0.2$, $b=0.2$, and $c=5.7$, the system exhibits the well-known Rössler attractor, a chaotic attractor that is widely studied in chaos theory [45]. The implementation source code is from Listing A5 in Appendix A.1.

4. Experiment

For this test, the computer language C# was used with .NET Framework 4.7.2 on Windows and .NET 8.0 on Linux. The goal of this study was to evaluate the efficiency of chaotic encryption in evading AV detection on Kleenscan.com, an online sandbox platform that aggregates data from over 30 AV companies. This platform enables users to submit files and analyze the detection probabilities for various AV engines. The platform was chosen because it enables a comprehensive evaluation of the effectiveness of chaotic encryption in escaping detection, and it offers a wide choice of commercial AV solutions. Penetration testers and red teams can benefit greatly from utilizing this method because it ensures a thorough and fair evaluation of the effectiveness of encryption across a variety of AV systems. The shellcode is created with the msfvenom tool; the unencrypted original shellcode, generated via the msfvenom command, is depicted in Figure A1 and Figure A2 in Appendix B.2, and will be shown for 64-bit Windows and Linux staged reverse TCP [46].

4.1. Windows Shellcode Execution

As described in Listing A6 in Appendix A.2, this code demonstrates the dynamic execution of shellcode by using platform invoke (P/Invoke) to call functions from kernel32.dll, such as VirtualProtect, CreateThread, and WaitForSingleObject. These functions are used to allocate memory, change the memory protection settings, and create a thread to execute the shellcode [47,48].

4.2. Linux Shellcode Execution

As described in Listing A7 in Appendix A.2, this code uses P/Invoke (Platform Invocation Services) in .NET 6.0 to execute unmanaged code by interfacing with Linux’s system calls via libc. Specifically, it allocates memory, copies executable code into that memory, changes the memory’s protection to allow execution, and then runs the code [49].

4.3. Parameters in Chaotic Systems

We conducted experiments on encryption using the logistic map with the parameter $r=3.99$ and the initial condition $x_0=0.5$; the Henon map with the parameters $a=1.4$, $b=0.3$, and initial conditions $x_0=0.1$, $y_0=0.0$; the Lorenz system with the parameters $\sigma =10$, $\rho =28$, $\beta =2.667$, and initial conditions $x_0=0.1$, $y_0=0.0$, $z_0=0.0$; Chua’s circuit with the parameters $\alpha =10$, $\beta =14$, $m_0=-0.14$, $m_1=0.28$, and initial conditions $x_0=0.1$, $y_0=0.0$, $z_0=0.0$ and the Rössler system with the parameters $a=0.2$, $b=0.2$, $c=5.7$, and initial conditions $x_0=0.1$, $y_0=0.0$, $z_0=0.0$. We also conducted the worst case for encryption using the logistic map with the parameter $r=20$ and the initial value condition $x=2$.

5. Results

5.1. Execution Time

This experiment involved measuring the encryption duration of a shellcode on both Windows and Linux platforms utilizing various chaotic systems. The shellcodes for Windows and Linux were produced using msfvenom, measuring 510 bytes and 130 bytes, respectively. We performed encryption with five distinct chaotic systems: the logistic map, the Henon map, the Lorenz system, Chua’s circuit, and the Rössler system. The execution durations for each system were assessed utilizing an AMD Ryzen Threadripper 2950X 16-Core Processor (3.5 GHz).

Table 2. Processing times for encrypting/decrypting Windows and Linux shellcodes using different chaotic systems.

Chaotic System	Windows Execution Time (µs)	Linux Execution Time (µs)
Logistic Map	425.2	463.8
Henon Map	754.5	668.9
Lorenz System	1677.5	1368.3
Chua’s Circuit	1268.3	1431.4
Rössler System	1855.3	1313.1

summarizes the execution timings for both the Windows and Linux shellcodes.

5.2. Ensuring Chaotic Behavior in Encryption

The data were analyzed pre- and post-encryption using the logistic map encryption method, as shown in Figure 1; corresponding chaotic keys were produced throughout the procedure. Although the capacity to reverse the encryption with the appropriate keys is retained, the chaotic nature of the logistic map significantly alters the shellcode. This demonstrates the efficacy of chaotic encryption in altering the original data’s structure while guaranteeing accurate decryption. For example, the first byte from the shellcode is 0xFC and the key is 0xFE, while the end first encrypted byte is 0x02. This process continues for each subsequent byte.

The shellcode files exhibit keys and encrypted data, as shown in Figure 2. The value’s derivation from the Henon map, which might have negative outputs, affects the result. The value from the second byte 0xFFFFFFAC is negative in the two complement representations. The XOR operation using 0xFF only examines the least significant 8 bits of the negative integer, yielding 0xFF ⊕ 0xAC = 0x53. This transpires because limited to a byte (8 bits), the negative integer 0xFFFFFFAC (in two’s complement) is interpreted as 0xAC, leading to the XOR operation producing 0x53.

5.3. Worst Encryption Case

The worst-case scenario of the experiment occurs when the chaotic system stabilizes and becomes predictable. As shown in Figure 3, this sample has been identified as malicious and represents the worst sample for logistic map encryption with the parameter $r=20$ and the initial value condition $x=2$. In this case, the value from the third byte 0x80000000 is negative in the two complements. The XOR operation with 0x83 considers only the least significant 8 bits of the negative integer, resulting in 0x83 ⊕ 0x00 = 0x83. This occurs because constrained to a byte (8 bits), the negative integer 0x80000000 (in two’s complement) is interpreted as 0x00, resulting in the XOR operation yielding 0x83. The corresponding byte in the encrypted data remains identical to the original shellcode byte. This results in sections of unencrypted data, which could reduce the overall encryption effectiveness.

5.4. Handling 0x00 Values in Chaotic Encryption

Figure 4 illustrates the significance of omitting 0x00 in the design of Algorithms 1 and 2, using the logistic map as a case study. To determine the system’s resilience, it is important to assess whether the chaotic sequence produced by the logistic map converges to stability under specific parameters. Through the examination of the sequence’s behavior regarding stability in different places, we can ascertain that omitting 0x00 preserves the desired chaotic characteristics. This verification phase guarantees that the system does not settle into a discernible pattern, hence maintaining the encryption’s unpredictability. When XOR encryption meets 0x00 chaotic value at $r=0.35,x=0.5$, in the Logistic Map, the process fails to transform the original data, leading to unchanged bytes and compromising the encryption’s efficacy. Upon configuring the method to exclude 0x00, each XOR operation consistently conceals the data, guaranteeing that the encryption process remains resilient and safe. This idea is widely applicable to encryption methods that employ chaotic maps to generate encryption keys.

5.5. Disassemble

Part of the original shellcode and encrypted data were disassembled using Defuse Security’s Online x86/x64 Assembler and Disassembler (https://defuse.ca/online-x86-assembler.htm accessed on 4 November 2024) [50]. As shown in Figure 5 and Figure 6 (the disassembled codes are in Appendix B.1), following CISC (complex instruction set computing) architecture, this assembly language uses a precise understanding of past instructions to ascertain the limits of later ones in decoding. The variable-length character of CISC instructions means that a single bit mistake may interfere with the decoding of the next instructions, therefore, influencing the whole program’s run. In the disassembly of the encrypted data, certain segments are marked as BAD. These labels indicate that the encrypted bytes do not translate to valid opcodes, resulting in decoding errors when attempting to interpret them as executable instructions. This is an expected outcome in the encryption process, as the XOR operation with chaotic keys produces byte sequences that no longer represent meaningful or valid instructions.

In Figure 7 (with the disassembled code in Appendix B.1), the red-highlighted parts denote code segments that are identical in both the original shellcode and the encrypted variant. This consistency suggests that some segments of the shellcode remain unaltered by the encryption process. This partial encryption produces discernible patterns in the encrypted data, which may be detected by antivirus. The continued existence of identifiable parts of code shows the importance of an enhanced encryption strategy that completely hides the payload, thereby decreasing the chance of antivirus detection.

5.6. Scan Report

5.6.1. Comparison Other Proposed Method

The comprehensive report is included in Appendix B.3. As indicated in

Table 3. Scan results for other evasion tools for Windows reverse shell.

Vendor	Msfvenom Native (exe)	shikata_ga_nai in 5 Iterators	PwnWind in FatRat	Veil Framework
AdAware	Trojan.Metasploit.A	Trojan.Metasploit.A	Gen:Variant.Bulz.632241	Gen:Variant.Kryptik.175
Alyac	Trojan.Metasploit.A	Trojan.Metasploit.A	-	Gen:Variant.Razy.575638
Arcabit	Trojan.Metasploit.A	Trojan.Metasploit.A	Trojan.Bulz.D9A5B1	Trojan.Kryptik.175
Avast	Win32:MsfShell-V	Win32:ShikataGaNai-B	Win32:TheFatRat-A	Win32:Evo-gen
AVG	Win32:MsfShell-V	Win32:ShikataGaNai-B	Win32:TheFatRat-A	Win32:Evo-gen
Avira	TR/Crypt.XPACK.Gen7	TR/Crypt.XPACK.Gen7	HEUR/AGEN.1234945	TR/Crypt.XPACK.Gen7
Bullguard	Win32:MsfShell-V	Win32:ShikataGaNai-B	Win32:TheFatRat-A	Win32:Evo-gen
ClamAV	Win.Malware.Metasploit-10022275-0	Win.Trojan.MSShellcode-6360728-0	-	-
Comodo Linux	-	-	TrojWare.MSIL.Rozena.C	TrojWare.MSIL.TrojanDownloader.Small.H
Crowdstrike Falcon	Threat detected.	-	Threat detected.	-
Emsisoft	Trojan.Metasploit.A	Trojan.Metasploit.A	Gen:Variant.Bulz.632241	Gen:Variant.Kryptik.175
F-Prot	W64/S-c4a4ef26!Eldorado	W64/S-c4a4ef26!Eldorado	-	-
F-Secure	TR/Crypt.XPACK.Gen7	Trojan.TR/Crypt.XPACK.Gen7	Heuristic.HEUR/AGEN.1306275	Trojan.TR/Crypt.XPACK.Gen7
G Data	Trojan.Metasploit.A	Trojan.Metasploit.A	Gen:Variant.Bulz.632241	Gen:Variant.Kryptik.175
IKARUS	Trojan.Win64.Meterpreter	Trojan.Win64.Meterpreter	Trojan.MSIL.Rozena	Trojan-Downloader.MSIL.Tiny
Immunet	Win.Malware.Metasploit-10022275-0	Win.Trojan.MSShellcode-6360728-0	-	-
Kaspersky	Trojan.Win64.Packed.gen	Trojan.Win64.Packed.gen	-	-
Max Secure	Trojan_W64_090622_Packed_gen_YR	Trojan_W64_090622_Packed_gen_YR	-	-
McAfee	Trojan-FJIN!D8D12B41E269 trojan !!!	Trojan-FJIN!C681A28426B4 trojan !!!	-	Trojan-Veil-FLRH!FC0B92454BC0 trojan !!!
Microsoft Defender	Trojan:Win64/Meterpreter!pz	Trojan:Win64/Meterpreter!pz	Trojan:Win32/Rozena.D!bit	Trojan:MSIL/Tiny.EM!MTB
NANO	-	-	-	Multiple Threats Detected (Trojan.Win32.TrjGen.dbjaze, Trojan.Win32.TrjGen.dkksls)
NOD32	Win64/Rozena.M trojan	Win64/Rozena.J trojan	MSIL/Rozena.C trojan	MSIL/TrojanDownloader.Tiny.BQ trojan
Norman	Win32:MsfShell-V	Win32:ShikataGaNai-B	Win32:TheFatRat-A	Win32:Evo-gen
SecureAge APEX	Malicious	Malicious	-	Malicious
Seqrite	Trojan.Dynamer.S4605	Trojan.Dynamer.S4605	-	-
Sophos	Mal/Swrort-J	Mal/Swrort-J	Troj/Tiny-DJ	Troj/Rozena-D
Vba32	-	-	Trojan.MSIL.PShell.gen	-
VirITExplorer	Trojan.Win32.Generic.BZPS	Trojan.Win32.Generic.BZPS	-	Backdoor.Win32.Meterpreter.BG
VirusFighter	ATK/Meter-A	ATK/Swrort-J	ATK/FatRat-H	ATK/Dloadr-EFR
ZoneAlarm	HEUR:Trojan.Win64.Packed.gen	HEUR:Trojan.Win64.Packed.gen	HEUR:Trojan-Downloader.Win32.Generic	HEUR:Trojan.Win32.Generic

, we analyzed the samples by generating executable files using msfvenom and shikata_ga_nai with five iterations, PwnWind in FatRat, and the Veil framework to produce Windows malware for reverse TCP shell control. The results show that a large number of AV providers effectively identified the malware samples produced by various evasion methods. Although there were minor variations in the detection rates across the tools, all the significant AV tools detected attempts from msfvenom, PwnKind, shikata_ga_nai, and Veil, showing their limited efficacy in evading detection. Despite their higher likelihood of detection by AV software, these tactics remain prevalent today [51].

5.6.2. Proposed Method

A detailed report is contained in Appendix B.3. The data shown in

Table 4. Scan results for Windows x64 staged reverse TCP shellcode before and after encryption.

Vendors	Original	Logistic Map	Henon Map	Lorenz System	Chua’s Circuit System	Rössler System
AdAware	Generic.ShellCode.Marte.4.EFC3787B	-	-	-	-	-
Arcabit	Generic.ShellCode.Marte.4.EFC3787B	-	-	-	-	-
Avast	Win32:MsfShell-V	-	-	-	-	-
AVG	Win32:MsfShell-V	-	-	-	-	-
Avira	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641
ClamAV	Win.Malware.Metasploit-10022275-0	-	-	-	-	-
Crowdstrike Falcon	Threat detected.	-	Threat detected.	Threat detected.	-	Threat detected.
Emsisoft	Generic.ShellCode.Marte.4.EFC3787B	-	-	-	-	-
G Data	Generic.ShellCode.Marte.4.EFC3787B	-	-	-	-	-
IKARUS	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt
Immunet	Win.Malware.Metasploit-10022275-0	-	-	-	-	-
Microsoft Defender	Trojan:Win64/ Meterpreter.B	Trojan:Win64/ Meterpreter.B	Trojan:Win64/ Meterpreter.B	-	-	-
Norman	Win32:MsfShell-V	-	-	-	-	-
ZoneAlarm	HEUR:Trojan.Win32.Generic	-	-	-	-	-

and

Table 5. Scan results for Linux x64 staged reverse TCP shellcode before and after encryption.

Vendors	Original	Logistic Map	Henon Map	Lorenz System	Chua’s Circuit System	Rössler System
AdAware	Gen:Variant.MSILHeracles.120280	-	-	-	-	-
Avast	ELF:Agent-DDL	-	-	-	-	-
AVG	ELF:Agent-DDL	-	-	-	-	-
eScan	Gen:Variant.MSILHeracles.120280[ZP]	-	-	-	-	-
Norman	ELF:Agent-DDL	-	-	-	-	-

indicate that using encryption alone, without further evasion techniques, enhances the efficacy of avoiding detection by all AV vendors.

According to

Table 6. Worst case for scan results for Windows x64 staged reverse TCP shellcode before and after encryption.

Vendors	Original	Logistic Map
AdAware	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.7E57359F
Arcabit	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.7E57359F
Avast	Win32:MsfShell-V	Win32:MsfShell-V
AVG	Win32:MsfShell-V	Win32:MsfShell-V
Avira	HEUR/AGEN.1216641	HEUR/AGEN.121664
ClamAV	Win.Malware.Metasploit-10022275-0	Win.Malware.Metasploit-10022275-0
Crowdstrike Falcon	Threat detected.	-
Emsisoft	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.7E57359
G Data	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.7E57359F
IKARUS	Trojan.MSIL.Crypt	Trojan.MSIL.Cryp
Immunet	Win.Malware.Metasploit-10022275-0	Win.Malware.Metasploit-10022275-0-
Microsoft Defender	Trojan:Win64/Meterpreter.B	Trojan:Win64/Meterpreter.B
Norman	Win32:MsfShell-V	Win32:MsfShell-V
ZoneAlarm	HEUR:Trojan.Win32.Generic	-

, these samples are the least favorable due to the encryption keys being consistently fixed at certain values.

According to

Table 7. Worst case for scan results for Windows x64 staged reverse TCP shellcode before and after encryption.

Vendors	Original	Logistic Map with 0x00 Keys
AdAware	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.1CC7DB19
Arcabit	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.1CC7DB19
Avast	Win32:MsfShell-V	Win32:MsfShell-V
AVG	Win32:MsfShell-V	Win32:MsfShell-V
Avira	HEUR/AGEN.1216641	HEUR/AGEN.1216641
ClamAV	Win.Malware.Metasploit-10022275-0	Win.Malware.Metasploit-10022275-0
Crowdstrike Falcon	Threat detected.	-
Emsisoft	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.1CC7DB19
G Data	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.1CC7DB19
IKARUS	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt
Immunet	Win.Malware.Metasploit-10022275-0	Win.Malware.Metasploit-10022275-0
Microsoft Defender	Trojan:Win64/Meterpreter.B	Trojan:Win64/Meterpreter.B
Norman	Win32:MsfShell-V	Win32:MsfShell-V
ZoneAlarm	HEUR:Trojan.Win32.Generic	-

, when the Logistic Map is applied with a 0x00 key for encryption, the resulting samples are even more vulnerable to detection by AV software. The consistent use of a zero key significantly reduces the obfuscation strength, leading to high detection rates similar to those observed with unencrypted samples.

This study highlights the efficacy of the proposed chaotic encryption methods in evading antivirus detection. We have expanded the results to present detailed detection outcomes across various AV engines, emphasizing the successes and limitations of each chaotic encryption system for enhanced clarity in the results. The logistic map encryption with a zero key exemplifies a worst-case scenario, highlighting its increased vulnerability to AV detection resulting from inadequate obfuscation. The findings demonstrate the comparative efficacy of each chaotic encryption method with the corresponding detection rates from various antivirus vendors. This enhances information security in penetration testing by providing more effective techniques for evading antivirus systems. The analysis and enhanced obfuscation techniques provide important insights for cybersecurity professionals, especially those involved in penetration testing, as they aim to comprehend and address potential detection mechanisms.

6. Conclusions

In the current research, we explored the use of chaotic systems for shellcode encryption as a novel method of preventing AV detection. Traditional AV systems, which rely primarily on signature-based and heuristic methods, have been increasingly challenged by sophisticated methods of evasion such as polymorphic and metamorphic malware, obfuscation, and packing. Our research focuses on using chaotic maps as an obfuscation strategy for shellcode encryption, taking advantage of the inherent unpredictability and sensitivity to the beginning factors of chaotic systems.

The experimental findings show that chaotic systems, such as the logistic map and Henon map, have promising potential to generate random encryption patterns that can evade static AV detection techniques. By adding chaos theory into malware evasion tactics, we demonstrated how a chaotic system may successfully conceal shellcode, making detection by modern AV systems harder. This method additionally emphasizes the lightweight character of the chaotic system, making it suitable for resource-constrained situations.

Overall, this study contributes to the developing subject of malware evasion by providing a fresh perspective on the exploitation of nonlinear dynamic systems such as chaos theory. The results support the potential of chaotic systems to improve malware stealthiness, but further investigations in varying real-world scenarios are needed.

7. Future Work

Although this research has shown encouraging results with chaotic systems for shellcode encryption, various possibilities for further study remain.

7.1. Dynamic Malware Analysis

Chaotic encryption should be examined in order to prevent behavioral detection because AV systems use dynamic analysis in sandboxes. Chaos systems’ persistence against contemporary AV solutions may be tested under more clearly defined and varied sandbox limitations.

7.2. Real-World Deployment

By circumventing sophisticated AV systems, red teams might imitate advanced threats and assess organizational defenses [52]. Security experts might test their networks and AV systems’ resistance to elaborate obfuscation by adding chaotic encryption to Cobalt Strike or Metasploit. This method will help to improve cybersecurity offensive and defense methods.