An Anomaly Detection Method for Oilfield Industrial Control Systems Fine-Tuned Using the Llama3 Model

Abstract

The device anomaly detection in an industrial control system (ICS) is essential for identifying devices with abnormal operating states or unauthorized access, aiming to protect the ICS from unauthorized access, malware, operational errors, and hardware failures. This paper addresses the issues of numerous manufacturers, complex models, and incomplete information by proposing a fingerprint extraction method based on ICS protocol communication models, applied to an anomaly detection model fine-tuned using the Llama3 model. By considering both hardware and software characteristics of ICS devices, the paper designs a fingerprint vector that can be extracted in both active and passive network communication environments. Experimental data include real ICS network traffic from an oilfield station and extensive ICS device traffic data obtained through network scanning tools. The results demonstrate that the proposed method outperforms existing methods in terms of accuracy and applicability, especially in differentiating devices from various manufacturers and models, significantly enhancing anomaly detection performance. The innovation lies in using large language models for feature extraction and the anomaly detection of device fingerprints, eliminating dependency on specific ICS scenarios and protocols while substantially improving detection accuracy and applicability.

1. Introduction

An ICS consists of automation and process control components designed for real-time data collection and monitoring, facilitating automated operations in industrial infrastructure [1]. These systems—including SCADA, DCS, PLC, and PCS—are widely used across industries such as power generation, water treatment, oil refining, nuclear energy, and manufacturing. In multiple experimental trials, an industrial control equipment anomaly detection model, based on fine-tuned Llama 3, was employed for ICS device fingerprint recognition. The recognition accuracy consistently exceeded 98%.

Originally deployed in isolated environments prioritizing functionality over security, ICSs have increasingly become connected to enterprise networks for remote control and supervision [2]. This connectivity elevates exposure to cyberattacks, posing significant risks to human safety, environmental security, and economic stability. Ensuring the security and stability of ICSs is thus paramount.

ICS communication methods encompass both wired and wireless communication [3]. Wireless communication offers flexibility in equipment layout, reduces wiring needs and deployment costs, enables rapid installation, and allows for remote monitoring and management. Wireless Sensor Networks (WSNs) facilitate detailed monitoring and control of production processes, especially in widely distributed or hard-to-reach facilities. In hazardous environments, wireless solutions mitigate personnel risks and enable real-time data communication for tracking mobile equipment.

However, wireless communication is vulnerable to attacks due to data visibility to any device within a signal range [4]. Common attacks on ICSs include Distributed Denial of Service (DDoS), unauthorized device access, man-in-the-middle attacks, and malicious code injection [5,6,7,8]. These attacks can lead to data leaks, service interruptions, and compromise system integrity.

To address these challenges, we propose the following strategies:

1. We develop an ICS device fingerprint that comprehensively reflects software and hardware characteristics, obtainable in both active and passive network communication environments.

2. By scanning numerous ICS devices exposed on public networks, we collect communication traffic from different manufacturers and models to establish a database of abnormal ICS device fingerprints.

3. We capture and extract fingerprints from target industrial control network traffic to form a database of normal ICS device fingerprints.

4. By applying an anomaly detection model based on the fine-tuned Llama3 model, we maximize the correlation mining between ICS device fingerprint features, improving the model’s applicability and accuracy in actual production sites.

The rest of this paper is organized as follows. In Section 2, we first decompose the anomaly detection of industrial control devices and detail existing problems, including general methods of fingerprint feature extraction and common anomaly detection models. In Section 3, We introduce our method for fingerprint extraction based on the ICS protocol communication model and an anomaly detection model fine-tuned using the Llama3 model. In Section 4, we experimentally validate the proposed methods using real industrial wireless network data collected from an oil production site and analyze the threat characteristics in industrial wireless networks. Section 5 gives the discussions, the conclusions and future research.

2. Related Works

Anomaly detection in industrial control devices involves monitoring their operational states and behaviors to promptly identify irregularities or unauthorized access, ensuring safety and normal functioning. As depicted in Figure 1, the process typically includes data collection, fingerprint extraction, model training, and anomaly detection. The quality of ICS device fingerprints and the design of the anomaly detection model significantly impact detection performance.

Effective fingerprint extraction is crucial for accurately distinguishing between normal and abnormal states. ICS device fingerprints uniquely represent operational characteristics—such as parameters, performance data, and response times—precisely describing normal behavioral patterns. Incomplete or inaccurate fingerprints can lead to misjudgments, resulting in false positives or negatives. Moreover, ICSs operate in dynamic environments, requiring fingerprints to reflect device behaviors under varying conditions. Failure to capture these variations can reduce detection accuracy and increase false alarm rates.

The design of anomaly detection models directly affects their ability to differentiate between normal and abnormal states. An effective model must extract key features from ICS device fingerprints to identify abnormal behaviors. Poorly designed models with inappropriate feature selection or oversimplification may yield suboptimal performance. Models need good generalization capabilities to maintain high accuracy on unseen data and adapt to changes like equipment aging or new devices. Balancing the detection of true anomalies and reducing false positives is essential to maintain reliability without disrupting normal operations.

ICS device fingerprinting is a technology used to identify and track industrial control devices, encompassing hardware, software, and behavioral information to determine device model, version, configuration, and operational state. Fingerprint extraction methods are mainly categorized into passive and active detection.

2.1. Industrial Control Device Fingerprint Extraction Methods

Industrial control device fingerprinting is a technology for identifying and tracking industrial control devices through unique identifiers analogous to human fingerprints. These fingerprints encompass hardware, software, and behavioral information, enabling the identification of a device’s model, version, configuration, and operational state. Currently, this technology is applied in various fields, including device identification and tracking, security monitoring, risk assessment, fault diagnosis, and asset management.

2.1.1. Fingerprint Extraction Method for Industrial Control Equipment

At present, the extraction of ICS device fingerprints primarily relies on asset detection technology. The detection of assets within an ICS network involves tracking and understanding the status of connected industrial control devices, including network topology recognition, device role determination, and device information extraction. However, due to the sensitivity of ICSs to detection activities and the ongoing development of the Industrial Internet of Things (IIoT), ICSs now exhibit new characteristics such as a wide variety of devices, increasingly complex processes, and expanding control scales [9]. These advancements necessitate higher standards for the effectiveness and safety of asset detection methods. Fingerprint extraction methods are mainly categorized into two types: passive detection and active detection.

(1) Passive Detection

Passive asset detection utilizes network sniffing tools to capture data packets without interfering with the target, obtaining communication and hierarchical information [10]. Research focuses on basic information extraction, fingerprint recognition, and device information identification. Tools like NetworkMiner, GRASSMARLIN, and Ettercap assist in network topology recognition and device role determination.

Fingerprint recognition establishes a database to identify devices with abnormal status. Radhakrishnan et al. [11] leveraged hardware heterogeneity and clock skew for device recognition, while Shen et al. [12] used enhanced hybrid fingerprints for intrusion detection. Aboah et al. [13] monitored communication patterns and memory addresses of PLCs to identify potential cyberattacks, utilizing timing and process behavior analysis.

Device information identification aims to extract manufacturer, type, and version information, crucial for vulnerability analysis. Yang et al. [14] combined Time to Live (TTL) values, IP ID variation, and MAC address prefixes to improve device recognition accuracy, proposing a framework for identifying ICS assets. Genge et al. [15] utilized passive fingerprinting techniques with network features like MAC addresses.

(2) Active Detection

Active detection involves active network communication to identify devices, offering completeness and real-time data acquisition. Combining active and passive methods maximizes information while minimizing system impact. Advances in scanning tools like Nmap, Zmap, and Masscan have improved detection efficiency [16]. Li et al. [17] used a probabilistic model based on Naive Bayes classifiers to identify industrial control honeypots by analyzing common protocols. Bezawada et al. [18] proposed behavior-based fingerprinting for IoT devices, achieving over 99% accuracy in identifying device types. Skowron et al. [19] analyzed IoT device traffic using machine learning to achieve device identification across different network environments.

2.1.2. Content of ICS Device Fingerprint Extraction

Different fingerprints serve various purposes: hardware fingerprints identify device models, software fingerprints distinguish manufacturers, and behavioral fingerprints detect network attacks. Selecting appropriate features based on task requirements is essential.

(1) Hardware Fingerprint

Hardware fingerprints involve identifying device characteristics through hardware information analysis. Obtaining superficial information like CPU model or MAC address is challenging in industrial settings. Instead, analyzing timing characteristics from network traffic helps extract deeper hardware fingerprints. As illustrated in Figure 2, devices process network data packets through hardware modules like network interface cards, main memory, CPU, cache, and data bus. Differences in these components affect the timing characteristics of data packets. By analyzing these timings, fingerprint features can be extracted. A representative feature is the Cross-Layer Response Time (CLRT), referring to the time from sending the request to receiving the response across layers.

(2) Software Fingerprint

Software fingerprints are derived from operating systems, drivers, and applications. These characteristics include operating system instruction sets, software protocol stack features, driver interfaces, and system service types and versions. While some require internal access, software protocol stack features are accessible externally. As shown in Figure 3, differences in protocol stack implementations by manufacturers manifest in fields like the TCP/IP stack, with features like TTL, field increment value, and TCP receive window size.

(3) Behavioral Fingerprint

Behavioral fingerprints describe regular communication patterns of devices, including periodicity, traffic features, content features, and connection modes. These features depend on both the devices and the specific network environment and protocols. Methods for extracting these features operate at the TCP/IP level and the industrial control protocol level.

At the TCP/IP level, Ponomarev and Atkison [20] used network telemetry to extract communication characteristics for identifying unauthorized devices. Fan et al. [21] proposed a PU learning-based intrusion detection method addressing high-dimensional and correlated data in ICSs. At the protocol level, Gao et al. [22] proposed a signature-based intrusion detection system analyzing key fields of the Modbus protocol to detect cyberattacks using predefined signatures. Khan et al. [23] proposed a hybrid model leveraging predictable communication patterns between field devices. However, these methods often depend heavily on the modeling environment, affecting applicability and accuracy across different scenarios.

2.2. Anomaly Detection Models for Industrial Control Devices

Advancements in hardware and machine learning algorithms have enabled training complex models like deep learning for anomaly detection. Researchers have applied these algorithms to train ICS device fingerprint models for monitoring device security.

Shen et al. [12] enhanced ICS network security using ICS device fingerprinting and machine learning algorithms like Support Vector Machines (SVMs), Naive Bayes, and Decision Trees, achieving 99% identification accuracy. This approach combines ICS device fingerprinting with machine learning to improve system defense capabilities.

Sandhya et al. [24] used Deep Convolutional Neural Networks (DCNNs) to analyze network traffic, converting inter-arrival times into images processed by models like ResNet-50, improving device identification accuracy and defending against signature attacks. This method leverages machine learning characteristics to enhance network security through pattern recognition in large datasets.

Koball et al. [25] introduced an unsupervised learning approach using K-Nearest Neighbors and feature clustering to classify IoT device types without labeled data, achieving comparable accuracy to supervised methods. The automated feature selection and threshold creation framework enhances device identification accuracy, demonstrating scalability in dynamic environments.

Hao et al. [26] proposed the IoTTFID model, an incremental learning system identifying evolving IoT device types by extracting “traffic fingerprints” from communication traffic. These are converted into input vectors for neural network training, enabling effective identification. The model adopts an incremental learning strategy, continually expanding recognition capabilities without forgetting previous information, addressing the “catastrophic forgetting” problem and improving scalability. The IoTTFID model demonstrated high accuracy and low resource consumption across multiple datasets, proving its practicality in network security and device management.

Sheng et al. [27] proposed a method for classifying unknown attack traffic in SCADA systems using a novel Density-Based Heuristic Clustering (DBHC) technique. This method automatically discovers cluster centers and dynamically creates new clusters, introducing a self-growing attack traffic classification model for real-time detection without relying on pre-trained samples. It improved classification accuracy and optimized data processing through effective traffic characterization and nonlinear features, enhancing adaptability and detection performance for unknown attack types. This study leveraged unsupervised learning advantages in dynamic environments, providing a new machine learning solution for ICS network security.

Shan et al. [28] developed a deep learning-based honeypot system to enhance interaction with attackers and detect malicious traffic. Contributions include a sequence-to-sequence time series prediction model for simulating long-term physical process changes, a Modbus honeypot framework for fine-grained parameter simulation, and a malicious traffic detection model with an innovative loss function to improve detection effectiveness for known and unknown threats. These methods increased system interactivity and significantly enhanced threat identification capabilities.

Overall, integrating advanced fingerprint extraction methods and sophisticated anomaly detection models is pivotal for enhancing the security and reliability of ICS. Continuous developments in machine learning offer promising avenues for more accurate and adaptable anomaly detection solutions in dynamic industrial environments.

2.3. The Selection of Large Language Models

With the advancement of generative AI, large language models (LLMs) have progressively become a focal point of research among numerous technology companies. Many institutions have introduced open-source LLMs; however, their performance varies significantly. It is necessary to initially screen suitable LLMs for device anomaly detection based on their fundamental capabilities and then identify the most appropriate model through experimentation.

In the study by Marah et al. [29], various performance aspects of foundational LLMs are discussed. According to their findings, we first exclude non-open-source models like GPT-3.5 and those requiring GPU memory exceeding 8 GB, such as Phi-3-medium-14b and Mixtral-8 ×7b. Secondly, to accelerate experimental speed using Unsloth, we exclude Phi-3-small-7b, which is unsupported by Unsloth. Lastly, given the complexity of industrial control device anomaly detection, the MMLU benchmark serves as a critical indicator for assessing a model’s suitability for anomaly detection. Therefore, we select the three models with the highest MMLU test scores: Phi-3-mimi-3.8b, Gemma-7b, and Llama-3-8b.

3. Methods

This chapter addresses the shortcomings of existing anomaly detection methods for industrial control devices, such as the generality of ICS device fingerprints and the accuracy of anomaly detection models. We propose an anomaly detection method for industrial control devices based on the fine-tuned Llama3 model. As shown in Figure 4, this method consists of three main components: industrial control network traffic preprocessing, ICS device fingerprint extraction, and the anomaly detection model. The main innovation of this method lies in the introduction of ICS device fingerprints that comprehensively consider the software and hardware characteristics of the devices and can be obtained in both active and passive network communication environments. Additionally, the method features a powerful anomaly detection model based on the fine-tuned Llama3 model. This approach not only eliminates dependence on specific industrial control scenarios and protocols but also significantly improves the accuracy of detecting anomalous industrial control devices.

3.1. Fingerprint Extraction Method for Industrial Control Devices Based on the Industrial Control Protocol Communication Mode

As shown in Figure 5, industrial control devices typically communicate with each other using an industrial control protocol to facilitate the upload of process data and the issuance of control commands. This chapter focuses on industrial control protocols that are based on the TCP/IP protocol and utilize a Client/Server communication model. Protocols such as Modbus/TCP, EtherNet/IP, and S7comm are widely used in various industries, including oil, chemical, and water resources, and thus have broad representativeness. This chapter aims to propose a fingerprint extraction method that can comprehensively reflect the software and hardware characteristics of industrial control devices and can be obtained in both active and passive network communication environments, thereby improving the distinguishability among different industrial control devices.

Based on the summary of the industrial control device communication process in Figure 5, this chapter proposes the corresponding industrial control protocol communication model ICS_CM to formally describe the aforementioned communication process. Specifically, it is represented as follows:

$$ICS\_CM=(C_E,C_D,C_T)$$

(1)

In this model, $C_E$ represents the message sequence of the connection establishment phase, expressed as

$$C_E=〈P_{syn}^{src},P_{syn\_ack}^{dst},P_{ack}^{src}〉$$

(2)

In this context, $P_{syn}^{src}$ represents the SYN packet in the connection establishment phase, $P_{syn\_ack}^{dst}$ represents the SYN_ACK packet in the connection establishment phase, and $P_{ack}^{src}$ represents the ACK packet in the connection establishment phase. The superscripts src and dst, respectively, indicate that the sender of the packet is the host requesting to establish the TCP connection, i.e., the Client, and the host agreeing to establish the TCP connection, i.e., the Server. The subscripts indicate the packet type.

$C_D$ represents the message sequence of the data transmission phase, which consists of one or more data transmissions, expressed as

$$C_D=〈C_{d_1},C_{d_2},\cdots ,C_{d_n}〉$$

(3)

where $C_{d_i}$ represents the i-th data transmission, with $i=1,\cdots ,n$ and n denotes the total number of data transmissions. $C_{d_i}$ is expressed as

$$C_{d_i}=〈P_{i_{-}req}^{src},P_{i_{-}ack}^{dst},P_{i_{-}res}^{dst}〉$$

(4)

where $P_{i_{-}req}^{src}$ represents the request packet of the i-th data transmission, $P_{i_{-}ack}^{dst}$ represents the TCP protocol ACK response packet of the i-th data transmission, and $P_{i_{-}res}^{dst}$ represents the industrial control protocol data response packet of the i-th data transmission.

$C_T$ represents the message sequence of the connection termination phase, which consists of one or more message subsequences, expressed as

$$C_T=〈C_{t_1}^{src},C_{t_2}^{dst},\cdots 〉$$

(5)

where $C_{t_i}^{src}$ represents the sequence of packets initiated by the Client to terminate the TCP connection, and $C_{t_{i+1}}^{dst}$ represents the corresponding sequence of packets returned by the Server. $C_{t_i}^{src}$ and $C_{t_{i+1}}^{dst}$ are expressed as follows:

$$\begin{array}{cc}& C_{t_i}^{src}=〈P_{t_i\_1}^{src},P_{t_i\_2}^{src},\cdots 〉\hfill \\ & C_{t_{i+1}}^{dst}=〈P_{t_{i+1}\_1}^{dst},P_{t_{i+1}\_2}^{dst},\cdots 〉\hfill \end{array}$$

(6)

The types and quantities of packets contained in $C_T$, $C_{t_i}^{src}$, and $C_{t_{i+1}}^{dst}$ are determined by the specific implementation of the industrial control device protocol stack.

Based on the aforementioned industrial control protocol communication model (ICS_CM), this chapter designs the ICS device fingerprint vector (DF) from the perspective of the differences in the hardware and software implementations of industrial control devices, specifically expressed as

$$\begin{array}{c}\hfill DF=\{ITTL,IPDF,IDD,IWS,MSS,WSC,\\ \hfill SAP,ILRT,TON,TSCON,TCF,RTD\}\end{array}$$

(7)

The explanations of the relevant features of the ICSDFs are shown in

Table 1. List of fingerprint features of ICS devices.

Fingerprint Feature Identifier	Fingerprint Feature Description
$ITTL$	indicates the initial value of the TTL field in the IP protocol header of packet $P_{syn\_ack}^{dst}$
$IPDF$	indicates the value of the DF flag in the IP protocol header of packet $P_{syn\_ack}^{dst}$
$IDD$	indicates the increment value of the ID field in the IP protocol header between packet $P_{syn}^{src}$ and packet $P_{ack}^{src}$
$IWS$	indicates the value of the window size field in the TCP protocol header of packet $P_{syn\_ack}^{dst}$
$MSS$	indicates the initial value of the TTL field in the IP protocol header of packet $P_{syn\_ack}^{dst}$
$WSC$	indicates the value of the window scale field in the TCP protocol header of packet $P_{syn\_ack}^{dst}$
$SAP$	indicates the value of the SACK permitted field in the TCP protocol header of packet $P_{syn\_ack}^{dst}$
$ILRT$	indicates the time interval between packet $P_{i_{-}ack}^{dst}$ and packet $P_{i_{-}res}^{dst}$
$TON$	indicates whether packet $P_{syn\_ack}^{dst}$ includes the TCP timestamp option
$TSCON$	indicates whether the TSecr field value in the TCP timestamp option of packet $P_{syn\_ack}^{dst}$ matches the TSval field value in the TCP timestamp option of packet $P_{syn}^{src}$
$TCF$	indicates the update frequency of the TSval field value in the TCP timestamp option on the Server side
$RTD$	indicates the difference between the time interval of packet $P_{syn}^{src}$ and packet $P_{syn\_ack}^{dst}$ and the time interval of packet $P_{i_{-}req}^{src}$ and packet $P_{i_{-}res}^{dst}$

. Examples of industrial control equipment fingerprints are shown in the

Table 2. Example of industrial control system device fingerprint vector.

$\mathit{ITTL}$	$\mathit{IDD}$	$\mathit{IWS}$	$\mathit{MSS}$	$\mathit{WSC}$	$\mathit{ILRT}$	$\mathit{RTD}$	Others	Vendor	Device Type
64	25,085	4096	1456	0	0	−0.037305	…	Rockwell Automation/ Allen-Bradley (Milwaukee, Wisconsin, United States.)	Communication Adapter
128	1	2000	1418	−1	0	0.013798	…	Rockwell Automation/Allen-Bradley	Programmable Logic Controller
32	1	6144	1024	−1	0	0.000798	…	Omron Corporation (Kyoto, Japan.)
64	0	10,000	1460	0	0.00014	0.000043	…	swat
64	2	4096	1450	0	0	0.000480	…	honeypot

.

The aforementioned features of the DF can be broadly classified into two categories based on the characteristics they reflect: hardware features and software features. The software features mainly refer to the differential characteristics caused by the implementation of the industrial control device’s operating system or protocol stack, including $ITTL$, $IPDF$, $IDD$, $IWS$, $MSS$, $WSC$, $SAP$, $TON$, $TSCON$, and $TCF$. Hardware features are composed of $ILRT$ and $RTD$. As shown in Figure 6, $ILRT$ is the time interval represented by $T_3$. Since the $P_{i_{-}ack}^{dst}$ packet is replied by the transport layer of the industrial control device’s protocol stack, and the $P_{i_{-}res}^{dst}$ packet is replied by the application layer of the protocol stack, the time interval between the two reflects the hardware processing performance of the industrial control device. However, in some cases, the accessed industrial control device may not reply with a $P_{i_{-}ack}^{dst}$ packet, but instead directly reply with a $P_{i_{-}res}^{dst}$ packet. In response to the aforementioned situation, this chapter proposes the $RTD$ feature, which has a similar principle to $ILRT$. Although $RTD$ is less accurate than $ILRT$, it is easier to obtain. Specifically, $T_3$ is the difference between $T_4$ and $T_2$. When $T_2$ does not exist, the same function can be substituted with $T_1$, meaning that $RTD$ is the difference between $T_4$ and $T_1$. Therefore, $ILRT$ and $RTD$ together characterize the hardware features of industrial control devices.

It is worth noting that although the DFs proposed in this chapter utilize the communication patterns of industrial control protocols based on the TCP/IP protocol and the Client/Server communication model, they do not rely on any specific industrial control protocol specification, nor do they require a specific method for acquiring industrial control network traffic. In other words, as long as the industrial control network protocol is of the aforementioned type, the proposed DFs can be effectively obtained in both active and passive network communication environments. This greatly enhances the applicability of the proposed industrial control device anomaly detection method and facilitates the construction of training data for subsequent industrial control device anomaly detection models, thereby improving the detection accuracy of the models.

3.2. Industrial Control Device Anomaly Detection Model Based on Fine-Tuned Llama3 Model

The specific process of fine-tuning large models is shown in Figure 7. There are performance differences between different foundational large language models, and their effects often vary greatly after fine-tuning with the exact same data. Additionally, different models require varying amounts of computational power for fine-tuning and are suitable for different applications. Therefore, selecting an appropriate foundational large language model is the first step in fine-tuning the model to accomplish tasks in a specific domain.

First, the selected foundational large language model must be secure and compliant. Considering that the anomaly detection task must meet data security and privacy compliance requirements, it is advisable to choose an open-source foundational large language model that supports local deployment for fine-tuning. Second, while foundational large language models typically excel in handling natural language-related tasks, they generally have limited ability to learn discrete, purely numerical sample features in the context of industrial control device anomaly detection tasks. By preprocessing the training samples to convert the discrete, purely numerical sample features of the anomaly detection task into common question-and-answer pairs in natural language processing tasks, the fine-tuned model can better accomplish the industrial control device anomaly detection task. Finally, the training process of large language models often requires substantial computational resources and expensive hardware support. Even during the fine-tuning process, the hardware resources required and the training speed can vary significantly depending on the pre-model and the fine-tuning method. Therefore, it is also essential to consider how to reduce the demand for hardware resources and improve training speed while ensuring the accuracy of anomaly detection.

This study, based on publicly available information and evaluation data from authoritative institutions, selected gemma-7b-bnb-4bit, llama-3-8b-bnb-4bit, and Phi-3-mini-4k-instruct for fine-tuning and testing. The selection reasons are as follows: (1) All three models are open-source foundational large language models that support local fine-tuning and deployment without internet connection, meeting security and compliance requirements. (2) They can all be fine-tuned using question-and-answer pairs to accomplish the industrial control device anomaly detection task. Finally, (3) they can all be fine-tuned using Unsloth, with GPU resource requirements not exceeding 8 GB after using Unsloth. Moreover, compared to not using Unsloth acceleration, fine-tuning speed with Unsloth is approximately doubled.

Due to the specificity of the industrial control device anomaly detection task, the detection accuracy of anomalous devices is virtually zero for various foundational large language models before fine-tuning. After fine-tuning, different models exhibit varying abilities to detect anomalous devices. Experimental results identified llama-3-8b-bnb-4bit as the most suitable foundational large language model for the industrial control device anomaly detection task after fine-tuning, with the highest detection accuracy for anomalous devices when fine-tuned using the same method.

The llama-3-8b-bnb-4bit model adopts the latest optimization algorithms, significantly enhancing training efficiency. It can rapidly converge to high-quality model parameters while reducing computational resources and time, markedly lowering training costs. The model exhibits substantial improvements in inference tasks (such as complex logical analysis and question-answering systems) and natural language generation tasks (such as text continuation and summarization). The generated text is more coherent and logically consistent, demonstrating stability and consistency across multiple tasks. The model architecture has been deeply optimized to improve computational efficiency. Through more rational hierarchical design and parameter adjustments, under identical conditions, this model’s performance significantly surpasses previous versions, especially excelling in processing large-scale data.

During experiments, it was found that the number of training epochs significantly affects the detection accuracy of anomalous devices. If the number of training epochs is too small, the model cannot accurately grasp the characteristics of anomalous devices, leading to misidentification. Therefore, a sufficiently large number of training epochs is needed to ensure that the model learns effective features. However, increasing the number of training epochs can easily lead to overfitting due to the unique characteristics of industrial control device anomaly detection samples, still resulting in reduced detection accuracy. To address this issue, this study proposes using an annealing algorithm instead of the traditional linear algorithm. With a linear algorithm, if the model is trained for too many epochs, it has enough time to learn all details in the training data, including noise and outliers, leading to overfitting. In contrast, the annealing algorithm, with its probabilistic acceptance mechanism and temperature control strategy, helps the model to avoid local optima and reduces the risk of overfitting. This approach allows for selecting a larger number of training epochs for the model to fully learn the data features while avoiding overfitting, thus improving the accuracy of anomaly detection.

The selection of training epochs in the model fine-tuning process depends on multiple factors such as dataset size and model complexity. In practice, a small number of epochs is usually set initially and gradually increased based on performance on the validation set until the requirements are met, requiring numerous experiments to find the optimal number of training epochs, which is time-consuming and labor-intensive. Through experiments, this study observed that the relationship between the sample size and the optimal number of training epochs resembles a Sigmoid function trend. By uncovering this relationship, it would be possible to provide a rough estimate of the number of fine-tuning epochs for different sample sizes, offering a basis for setting training epochs and saving experimental time. Based on the above observation, this study designed the following formula for predicting the number of training epochs for model fine-tuning:

$$\mathit{f}\left(\mathit{x}\right)=\left[\alpha +{\displaystyle \frac{\beta }{1+{\mathrm{e}}^{4-\frac{1n^x}{3}}}}\right]$$

(8)

where $\alpha$ represents the adjustment constant of the fine-tuning epoch function, $\beta$ represents the linear coefficient of the fine-tuning epoch function, and x represents the number of training set samples.

4. Experiments and Results

This section will construct a real environmental experimental dataset based on the actual industrial control environment of an oilfield station to validate the proposed method. First, this section conducts a comprehensive analysis of the proposed method from two aspects: the effectiveness of DF and the performance of the anomaly detection model, testing the effectiveness of the proposed improvements. Second, this section compares and analyzes the proposed method with the more advanced existing methods in the field of industrial control device anomaly detection, further verifying the superior performance of the proposed method.

4.1. Experimental Hardware Configuration and Software Versions

Considering that the industrial enterprises discussed in this paper typically do not have Servers equipped with high GPU memory, the hardware requirements for this experiment are modest. The specific hardware configurations are shown in

Table 3. Hardware configuration.

Model	CPU	Memory	GPU	Hard Drive
X3650M5	2×E5-2620v4 (8 C, 85 W, 2.1 GHz)	4 × 16 = 64 GB DDR4	GPU: NVIDIA GeForce RTX 2080 SUPER. Max memory: 7.787 G	8 × 2.5″ slots, 2 × 2 TB SATA interface, M5210 Raid 0,1

.

The software versions are as follows:

Unsloth: Fast Llama patching release 2024.5 Pytorch: 2.3.0. CUDA = 7.5. CUDA Toolkit = 12.1. Bfloat16 = FALSE. Xformers = 0.0.26.post1. FA = False.

4.2. Description of Experimental Data

The experimental data used in this section consist of two parts: a real industrial control network traffic dataset from an oilfield station and a large dataset of network traffic scans of industrial control devices exposed on the public internet, obtained through network scanning tools. Due to security considerations, it is not feasible to perform network attacks directly on the real ICS of the oilfield station to obtain anomalous device data. Therefore, the network traffic scans of the numerous industrial control devices exposed on the public internet are used as anomalous device data, while the network traffic data from the oilfield station is used as normal device data. Since the proposed method does not rely on specific features of the industrial control network for anomaly detection, the differences in the network environments between the two datasets do not affect the validity of the experimental results.

As shown in Figure 8, the network architecture of the oilfield station includes both OT hardware such as Allen-Bradley PLC and Schneider RIO, and IT software and hardware such as SCADA, HMI, and Historian, supporting both wired and wireless communication methods. IT and OT components coexist on a single network, which follows the ICS Purdue model and uses EtherNet/IP as the main communication protocol. SCADA supervises the production process of the oilfield station through periodic monitoring and can manually control the entire station via pre-written PLC programs. SCADA also works with the Historian, which is primarily used to record and store process data.

The oilfield station includes segmented communication networks, both wired and wireless communication, distributed dynamic control, interconnection between PLCs, and full access to control logic within PLCs and HMIs. The station uses wireless access points (WAPs) to achieve wireless communication. The distribution of industrial control devices in the station is shown in

Table 4. List of wired/wireless devices at oilfield stations.

End Device	Communication Mode	IP Address
SCADA	wired	192.168.1.100
	wireless	192.168.1.102
PLC1	wired	192.168.1.10
	wireless	192.168.1.102
PLC2	wired	192.168.1.20
	wireless	192.168.1.22
PLC3	wired	192.168.1.30
	wireless	192.168.1.32
PLC4	wired	192.168.1.40
	wireless	192.168.1.42
PLC5	wired	192.168.1.50
	wireless	192.168.1.52
PLC6	wired	192.168.1.60
	wireless	192.168.1.62
Engineering Station	wired	192.168.1.201
HMI	wired	192.168.1.202
Historian	wired	192.168.1.200

.

The network traffic data from the oilfield station, used as normal device data in this section, cover communication traffic involving six PLCs and various devices such as SCADA and engineering stations. A continuous 30-minute segment of communication traffic (44,956,192 packets) was captured and randomly split into 70% training data, 20% validation data, and 10% test data, with no overlap between these subsets. The specific data distribution is shown in Figure 9.

Regarding anomalous device data, this section uses network scanning tools to simulate network communication in the station environment for 1012 industrial control devices and honeypots exposed on the public internet over 10 rounds, collecting a total of 131,215 network communication packets. Since DFs are independent of the network environment, the aforementioned data can be used for relevant validation in this section. The specific distribution of anomalous device data is shown in Figure 10. Because Allen-Bradley PLCs mainly use the EtherNet/IP industrial control protocol, their proportion is relatively high. Honeypot data refer to industrial control honeypots exposed on the public internet that support the EtherNet/IP protocol, which simulate the communication methods of industrial control devices to deceive attackers and capture attack traffic. The data categorized as “Others” refer to the network traffic data of industrial control devices from other brands. The software and hardware characteristics of these three types of devices exhibit different variations, which is beneficial for validating the methods proposed in this chapter.

This section uses two representative evaluation metrics, Accuracy and F1-Score, to compare and analyze the effectiveness of the relevant methods. The definitions of Accuracy and F1-Score are as follows:

$$\begin{array}{cc}& Accuracy={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}\hfill \\ & FI-Score=2\ast {\displaystyle \frac{Precision\ast Recall}{Precision+Recall}}\hfill \\ & Precision={\displaystyle \frac{TP}{TP+FP}},Recall={\displaystyle \frac{TP}{TP+FN}}\hfill \end{array}$$

(9)

where $TP$,$FP$,$TN$, and $FN$ represent true positive, false positive, true negative, and false negative data, respectively.

4.3. Experimental Results

To comprehensively evaluate the effectiveness of the proposed method in this chapter, this section conducts a comparative analysis of the evaluation metrics for the proposed DFs and the anomaly detection model based on the fine-tuned Llama3 model. This analysis verifies the effectiveness of the proposed improvements. Finally, the proposed method is compared with the more advanced existing methods in the field of industrial control device anomaly detection, demonstrating the performance level of the proposed method.

4.3.1. Analysis of the Effectiveness of DFs

For the first time, this chapter proposes DFs that can be obtained in both active and passive network communication environments, from the perspective of the software and hardware characteristics of industrial control devices. To validate the effectiveness of these fingerprints, this experiment divides the existing 12 fingerprint features into hardware fingerprints and software fingerprints based on the fingerprint extraction rules from the existing literature, and analyzes the effectiveness of different types of fingerprints from four different perspectives: Allen-Bradley, Honeypot, Others, and All. The hardware fingerprints consist of two features, and, while the software fingerprints consist of the remaining ten features.

As shown in Figure 11, the DFs proposed in this chapter, which comprehensively consider the software and hardware characteristics of industrial control devices, achieve the best experimental results on various types of anomalous data. The PLCs in the Allen-Bradley dataset are produced by the same manufacturer as those used in the oilfield station, and different models of PLCs typically exhibit certain differences in processing performance and communication performance, as well as differences in configuration logic to meet different control needs. Therefore, the hardware characteristics of PLCs from different models and scenarios are more easily distinguishable. In contrast, PLCs from the same manufacturer usually implement a unified operating system or protocol stack, resulting in smaller software differences. Consequently, hardware fingerprints perform better than software fingerprints on the Allen-Bradley dataset. Although software fingerprints may be less effective in distinguishing industrial control devices from the same manufacturer, they are more effective in differentiating industrial control devices from different manufacturers, such as the Honeypot and Others datasets in this section, as Figure 12 clearly demonstrates. Overall, hardware fingerprints consist of fewer features and are more challenging to obtain, while software fingerprints can encompass richer features and are easier to acquire, making them more advantageous for device anomaly detection.

4.3.2. Performance Analysis of the Anomaly Detection Model

To evaluate the effectiveness of the proposed industrial control device anomaly detection model based on the fine-tuned Llama3 model, this experiment compared the changes in the Accuracy metric before and after fine-tuning using gemma-7b, llama-3-8b, and Phi-3-mini-4k as the foundational large language models. The performance differences of these three foundational models after fine-tuning for industrial control device anomaly detection were also analyzed.

As shown in Figure 13, all three original models had zero capability for industrial control device anomaly detection and could not complete the task. However, the ability of the three models to detect anomalies in industrial control devices varied significantly after fine-tuning. The Phi-3-mini-4k model exhibited the poorest performance after fine-tuning, with an accuracy of only 68.4%. The gemma-7b model demonstrated better performance, achieving an accuracy of 88.0%. The llama-3-8b model had the best anomaly detection capability, with an accuracy of 99.8%. Figure 14 further illustrates that the fine-tuned lla-ma-3-8b model, which showed the strongest performance in industrial control device anomaly detection, maintained excellent Accuracy across various types of anomalous data.

The performance improvement is mainly attributed to the unique characteristics of industrial control device anomaly detection data. The original foundational large language models could not determine the anomaly status of industrial control devices based on the input. After fine-tuning the models in the same way, the ability to learn features of anomalous industrial control devices varied significantly due to the different foundational parameters of each model. Consequently, the anomaly detection capabilities of different models for industrial control devices showed substantial differences. In conclusion, selecting an appropriate foundational large language model is the first step in ensuring that the fine-tuned model can effectively perform anomaly detection tasks at a high level.

During the fine-tuning process, if the number of training epochs is too small, the model cannot sufficiently learn the features of the samples, resulting in poor performance in identifying anomalies in industrial control devices. As the number of training epochs increases, the model learns the sample features more thoroughly, improving its ability to identify anomalous devices. However, once the number of training epochs exceeds a certain value, the model may overfit the sample features, including noise and outliers, leading to poorer performance on unseen new data and a significant decline in prediction accuracy.

As shown in Figure 15, using the llama-3-8b-bnb-4bit model with the optimal number of training epochs, changing the algorithm from linear to cosine increased the model’s detection accuracy from 99.7387% to 99.8258%. With the linear algorithm, the fine-tuned model already achieved 100% accuracy in identifying anomalous devices for Allen-Bradley and Honeypot datasets, leaving no room for improvement. After switching to the cosine algorithm, the detection accuracy for devices in the Others dataset increased from 99.3671% to 99.5781%. This improvement is due to the cosine algorithm’s ability to enhance the model’s generalization capability, ultimately boosting the model’s overall accuracy.

Finding the appropriate number of training epochs often requires multiple experiments, which is very time-consuming and labor-intensive. Based on Equations (2)–(8), this chapter uses a sample size of 1200–5200 for training data and a sample size of 6039 for testing data. Using the least squares method, the model fine-tuning training epoch prediction function, with sample size as the independent variable, is fitted as follows:

$$\mathit{f}\left(\mathit{x}\right)=\left[34.94231916+{\displaystyle \frac{259.1611542}{1+e^{4-\frac{{ln}^x}{3}}}}\right]$$

(10)

As shown in

Table 5. List of wired/wireless devices at oilfield stations.

Number of Samples	Number of Training Epochs	Fine-Tuned Model Accuracy	The Number of Training Epochs Derived from the Function
1200	68	84.1%	78
	73	87.1%
	78	98.7%
	83	99.9%
	88	99.8%
2000	73	87.9%	84
	78	94.8%
	83	96.5%
	88	99.8%
	93	99.8%
2800	78	91.1%	88
	83	95.1%
	88	97.7%
	93	99.7%
	98	99.7%
3600	83	99.3%	92
	88	99.4%
	93	99.2%
	98	99.4%
	103	99.5%
4400	88	99.0%	95
	93	99.9%
	98	99.8%
	103	99.9%
	108	99.9%
5200	93	94.3%	98
	98	95.8%
	103	97.4%
	108	98.2%
	113	98.7%
6039	98	93.8%	100
	103	98.0%
	108	99.0%
	113	99.2%
	118	99.8%

, the number of training epochs derived from the proposed function is 78 when the sample size is 1200. By selecting a test step size of 5 and a test window of 20, we aim to find the optimal number of training epochs for this sample size. In experiments where the training sample size was continuously increased from 1200 to 5200, we consistently found a training epoch count within ±10 of the function’s output that achieved an accuracy of not less than 96%.

For a test sample size of 6039, we conducted a set of validation experiments where the number of training epochs was the only variable. We trained models for 18, 38, 58, and 78 epochs, respectively, and used these models for equipment anomaly detection. The detection results are shown in Figure 16. The experiments demonstrate that using this method allows the function to find the optimal number of training epochs within the window range while avoiding overfitting.

Therefore, the introduction of this function signifies an improvement from arbitrary trial-and-error in determining the number of training epochs to a function that outputs this number based on the sample size. This provides practitioners with a reference for finding the optimal number of training epochs and significantly reduces the time required to achieve this.

4.3.3. Performance Analysis of Different Anomaly Detection Methods

Finally, this section compares the proposed method with the advanced anomaly detection methods NeuPot and DBHC in the field of industrial control device anomaly detection. The analysis is conducted from the perspectives of different types of anomalous data and various evaluation metrics. The experimental results are shown in the following figures.

As shown in Figure 17 and Figure 18, the proposed method performs comparably to the NeuPot method on the Others dataset and outperforms the other compared methods on the other datasets and the overall data. The experimental results indicate that all three methods perform better on the Others dataset than on the other datasets. This is primarily because the industrial control devices in the Others dataset have the most significant hardware and software differences compared to the PLCs in the oilfield station, making them easier to identify. Conversely, all three methods achieve the lowest F1-Score on the Honeypot dataset. This is mainly because honeypots typically simulate the communication methods of industrial control devices, resulting in smaller software differences and thus reduced performance.

Overall, the proposed industrial control device anomaly detection model based on the fine-tuned Llama3 model can better learn the deep correlations between DF features, forming more accurate feature representations for identifying anomalous devices, thereby effectively improving the performance of the classification model in detecting anomalies.

Compared with existing anomaly detection models, the industrial control equipment anomaly detection model based on fine-tuning the Llama3 model has the advantage of leveraging the contextual understanding and reasoning capabilities of the original Llama3 model to identify the underlying causes of abnormal patterns, thereby providing more precise diagnostic information. It dynamically adjusts anomaly detection thresholds based on historical data and current environmental changes, making the system more intelligent. When an anomaly is detected, the model can automatically generate detailed natural language reports based on the detection results, enabling technical personnel to quickly grasp the situation.

An illustrative example of the proposed method is presented as follows. First, traffic data collected from oilfield industrial control systems (ICSs) are processed to extract device fingerprints, following the approach detailed in the Methods section. These fingerprints are then used as input for an anomaly detection model, which is specifically tailored to oilfield ICS equipment and developed through fine-tuning the Llama3 model. Finally, the anomaly detection results are fed back into the model. Leveraging the capabilities of the base Llama3 model, the specialized ICS anomaly detection model can autonomously generate comprehensive natural language reports on the detected anomalies.

5. Discussion and Conclusions

Industrial control device anomaly detection is the primary technical means for identifying abnormal operational states or unauthorized access to devices, effectively detecting various network attacks targeting industrial control devices, such as DDoS attacks, unauthorized device access, and man-in-the-middle attacks. Existing methods typically use limited or scenario-specific DFs combined with machine learning or deep learning algorithms to detect anomalous devices. These methods often exhibit significant accuracy fluctuations across different industrial control scenarios, leading to poor generalizability.

To address these issues, this chapter proposes DFs that comprehensively consider the software and hardware characteristics of devices and can be obtained in both active and passive network communication environments, as well as an anomaly detection model based on the fine-tuned Llama3 model. This approach effectively enhances the performance and applicability of industrial control device anomaly detection. Experimental results demonstrate that under appropriate conditions, the anomaly detection model based on the fine-tuned Llama 3 achieves a stable accuracy of over 98% in recognizing anomalies in industrial control equipment.

Finally, based on the network data of an oilfield station’s ICS, this chapter constructs a representative experimental dataset for industrial control device anomaly detection. This dataset is used to comprehensively evaluate the proposed method. The Experimental section of this chapter not only assesses the effectiveness of the proposed DFs and the fine-tuned Llama3 model-based anomaly detection model, but also compares their performance with advanced existing methods in the field of industrial control device anomaly detection, demonstrating the superior performance of the proposed method.

Meanwhile, with the continued development of industries such as power, water utilities, wastewater treatment, chemical, pharmaceutical, food, and discrete manufacturing (e.g., automotive and aerospace), the ICS in these sectors are increasingly required to connect to enterprise networks, leading to direct exposure to public networks and a heightened risk of cyberattacks. These systems were not originally designed with information and network security in mind, and their hardware and software lack the necessary resources to accommodate upgrades essential for ensuring cybersecurity. As shown in Figure 19, this method can be applied to nearly all ICSs.

Due to its characteristic of capturing information via side-channel techniques to generate DFs, it imposes no additional load on the system. Furthermore, by selecting appropriate hardware configurations and traffic capture intervals, the anomaly detection speed can be significantly improved, reaching sub-second levels. The anomaly detection model, based on the fine-tuned Llama 3, also demonstrates a considerably higher capability in detecting equipment anomalies compared to the most advanced existing methods in the field of ICS anomaly detection. It is anticipated that this method will greatly contribute to the advancement of anomaly detection in ICSs.

In future research, our method can be improved in the following ways:

1. Application and promotion of this method on data from other industries. All data in this experiment are sourced from an oilfield ICS. However, the author believes that DFs of ICSs are universal. We will endeavor to obtain traffic information from industries including, but not limited to, electric power, water supply, sewage treatment, chemical engineering, pharmaceuticals, food, and discrete manufacturing to test the universality of this method.

2. Large language models have significant potential in the domains of feature extraction and anomaly detection. We will attempt to explore the applications of large models in areas including, but not limited to, mechanical equipment life prediction and feature extraction and anomaly detection based on acoustic waves, optical waves, and electromagnetic waves.