LogEDL: Log Anomaly Detection via Evidential Deep Learning

Abstract

With advancements in digital technologies such as 5G communications, big data, and cloud computing, the components of network operation systems have become increasingly complex, significantly complicating system monitoring and maintenance. Correspondingly, automated log anomaly detection has become a crucial means to ensure stable network operation and protect networks from malicious attacks or failures. Conventional machine learning and deep learning methods assume consistent distributions between the training and testing data, adhering to a closed-set recognition paradigm. Nevertheless, in realistic scenarios, systems may encounter new anomalies that were not present in the training data, especially in log anomaly detection. Inspired by evidential learning, we propose a novel anomaly detector called LogEDL, which supervises the training of the model through an evidential loss function. Unlike traditional loss functions, the evidential loss function not only focuses on correct classification but also quantifies the uncertainty of predictions. This enhances the robustness and accuracy of the model in handling anomaly detection tasks while achieving functionality similar to open-set recognition. To evaluate the proposed LogEDL method, we conduct extensive experiments on three datasets, i.e., HDFS, BGL, and Thunderbird, to detect anomalous log sequences. The experimental results demonstrate that our proposed LogEDL achieves state-of-the-art performance in anomaly detection.

1. Introduction

Ensuring the stable operation of large modern systems is critical to business continuity, and a log of runtime status and events reflects the reliability of the system. Logs [1,2,3], as semi-structured text, typically consist of fixed templates and dynamic parameters that carry rich semantic information. In the event of a system failure, the most straightforward approach for maintenance personnel is to perform a log analysis. However, manually identifying anomalies based on massive log data has become impractical [4,5]. Due to the rapid growth in log size, log analysis by experienced experts has become increasingly challenging [6]. As a result, automatic detection of system anomalies from logs has become critical. To address this problem, automated log analysis approaches are proposed that aim to automatically analyze logs using machine learning (ML) or deep learning (DL) techniques. In the field of log anomaly detection, traditional approaches for processing log sequences include word embedding methods and RNN-based feature embedding methods [7]. However, these methods have difficulty in modeling long-term dependencies in processing lengthy sequences. In recent years, there has been a significant amount of research leveraging the Transformer architecture [8] to tackle natural language processing tasks, because it can model relationships between long sequences using attention mechanisms. Currently, pre-trained language models based on Transformer, such as BERT [9], RoBERT [10], GPT-3 [11], BART [12], and T5 [13] have demonstrated remarkable performance in various NLP tasks. Among them, BERT is pre-trained with two self-supervised learning methods: Masked language modeling (MLM) and next sentence prediction (NSP). MLM involves replacing certain tokens in the input sentences with [MASK] or other tokens and predicting their corresponding positions. NSP concatenates two sentences with the token [SEP] in between and predicts whether the two sentences are semantically related based on the [CLS] token positioned at the beginning of the input sequence [9]. While BERT has demonstrated excellent performance in various natural language processing tasks, its application to log anomaly detection, such as MIM tasks, still has certain limitations.

Traditional machine learning and deep learning methods typically assume that training and test data come from the same distribution, meaning the categories seen during training are entirely consistent with those encountered during testing. This approach is known as a closed-set recognition paradigm [14]. In this paradigm, the model classifies and recognizes data by learning features from the training data. However, its performance can be limited in practical applications, especially when encountering new, unseen categories or anomalies. For instance, in the task of log anomaly detection, data often follow a long-tail distribution [15], with anomalies being rare and dynamically changing. In a closed-set recognition paradigm, the model may not effectively discover new categories or anomalies in the logs. This limitation arises because the model is not equipped to handle cases where the test data include classes or anomalies not present in the training data.

To address this challenge, the introduction of an open-set recognition mode is particularly important [16]. In the open-set recognition mode, the model can more flexibly handle different categories or data distributions, meaning it can face different log anomaly situations during both the training and testing phases. In addition to open-set recognition mode, there is another important approach called evidential deep learning (EDL) [17,18] for tasks such as anomaly detection. Evidential learning is a method for handling uncertainty, and its core idea is to better model and infer data by introducing uncertainty into the model. Unlike traditional closed-set recognition mode, evidential learning not only considers the features of known data but also quantifies the model’s confidence in each prediction. In the context of anomaly detection, evidential learning can help the model identify which data points have higher confidence and which may represent unseen anomalies. In this way, evidential learning can not only identify known anomaly categories but also provide inference and modeling for unknown anomaly situations, thereby improving the robustness and accuracy of anomaly detection systems. In summary, evidential learning is an important method for handling uncertainty and unknown data, similar to the functionality of open-set recognition mode, and can provide a more comprehensive and reliable solution for tasks such as anomaly detection.

In this paper, we pay close attention to the issue of uncertainty estimation in log data and design an evidential learning loss function to supervise the network. Specifically, we replace the output parameters of the network with the parameter set of Dirichlet density. The Dirichlet distribution is a multivariate probability distribution that effectively describes the uncertainty of data. Through this substitution, our model can not only learn the distribution characteristics of normal log sequences but also make anomaly judgments by evaluating uncertainty when facing abnormal logs. To better reflect real-world scenarios, we only use normal log data to train the model, and then, evaluate both normal and abnormal log data separately during testing.

The main contributions of this paper are as follows:

• Analyzing the similarity between the log anomaly detection task and open-set recognition, and introducing the relationship and differences between open-set recognition and evidential learning, clarifying the applicability of evidential learning to log anomaly detection methods.
• Proposing an evidential learning network, LogEDL, suitable for log anomaly detection, enabling the model to identify unknown logs and quantitatively assess the uncertainty of unknown data, thereby assisting in identifying anomalous data.
• Applying evidential learning algorithms to log anomaly detection tasks on public datasets, demonstrating the effectiveness and generalization ability of the proposed method.

2. Related Work

In this section, we first introduce the application of the self-training paradigm in log anomaly detection. Subsequently, we summarize the methods of semi-supervised log anomaly detection based on deep learning introduced in recent years.

2.1. Paradigm of Self-Training

Self-supervised pre-training is a significant research topic aimed at learning generalizable and shareable knowledge from large-scale data, which further benefits downstream tasks. The pretext tasks of self-supervised learning can be broadly divided into contrastive learning and masking modeling. In recent years, with the maturation of contrastive learning and masking modeling techniques, several self-supervised pre-training methods designed for log sequences have emerged.

In log sequences, the masking modeling paradigm is commonly applied, where the model is optimized by reconstructing the masked content from the unmasked portions. LogBERT [19] enhances the self-supervised training approach based on BERT [9] by introducing two novel self-supervised training tasks. To adapt to log sequences, masked log key prediction (MLKP) is proposed to enable the network to learn patterns of normal log sequences. The NSP of BERT is transformed into the volume of hypersphere minimization (VHM). LAnoBERT [20] utilizes masked language modeling based on BERT to train the model and employs the masked language modeling loss function for each log key during testing, enabling unsupervised anomaly detection. The LogFiT [21] model utilizes the RoBERTa model to handle log datasets containing up to 512 tokens per sample, while the Longformer model is employed for datasets where the token count per sample exceeds 512. Similarly, it employs BERT-based masked language modeling to learn the language patterns of normal log data. The encoder of BERT and its self-supervised paradigm enable it to generate contextually sensitive log keys for the input log sequences and reconstruct the masked log sequences. However, directly masking log sequences disrupts the inherent structure of log sequences, making reconstruction overly challenging and unable to guide representation learning. In contrast to the direct reconstruction approach used in previous works, MMSM proposes a novel masking modeling task, which involves reconstructing the original log sequence from multiple randomly masked log sequences.

2.2. Semi-Supervised Anomaly Detection

DeepLog models the error between the predicted vector and the actual vector in the validation set as a Gaussian distribution. During testing, if the error falls within the high-confidence interval of the Gaussian distribution, the parameter value vector of the log entry is considered normal. Otherwise, it is deemed anomalous. LogBert establishes a candidate set consisting of g normal log keys. If the log key is in the candidate set, it is considered normal. Otherwise, it is deemed an anomalous log key. Subsequently, when a log sequence consists of more than r anomalous log keys, the sequence is labeled as anomalous. LAnoBERT utilizes computed loss errors and prediction probabilities for anomaly detection. When the probability of predicting a log occurrence is low, it is difficult to find the corresponding context in the normal log context. Thus, it is identified as an anomalous log sequence. LogFiT is trained only on normal log data using masked sentence prediction. When new log data are used, the $top-k$ prediction accuracy of the model can serve as a threshold to judge whether the new log data deviate from the normal log data. The aforementioned methods distinguish anomalous log sequences based on data distribution or empirical thresholds. Enabling the neural networks to recognize log sequences belonging to anomalous data distributions is achieved by making the network autonomously determine whether a log sequence is anomalous. Placing a Dirichlet distribution on class probabilities, we treat the neural network predictions as subjective opinions and learn a function from the data that collects evidence leading to these opinions of deterministic neural networks. Leveraging evidential neural networks, the network can autonomously identify normal log sequences and anomalous log sequences.

3. Preliminary

3.1. Background of Open-World Learning

In real-world recognition/classification tasks, it is usually difficult to collect training data for all classes when training a recognizer or classifier due to various objective factors. The dataset is typically closed-set in deep learning, with the training and test data having the same annotations and feature space. Based on this assumption, traditional recognition and classification algorithms have achieved remarkable success in various machine learning tasks. However, real-world scenarios are often open-set and non-smooth, such as log fault detection, where the emergence of invisible situations significantly weakens the robustness of these existing approaches. To address this challenge, this requires a model that can both recognize known normal log sequences and reject unknown log faults.

Previously, researchers have proposed open-set recognition (OSR), which cannot fully learn all data knowledge during training, but the model can recognize unknown category data during testing. This requires the classifier to be able to accurately recognize not only the visible categories but also the unseen ones. Due to the small number of abnormal log sequences, previous researchers have usually used semi-supervised log anomaly detection methods, where only normal log sequences are targeted during training and abnormal log sequences need to be found during testing. A general framework for open-set recognition learning algorithms can help us to continuously recognize unknown category data.

3.1.1. Open-World Learning Algorithms

Mohsen Jafarzadeh et al. [16] developed a generalized framework for open-world learning algorithms containing six main elements: feature extraction, known classifications, novelty detection, novelty discovery, novelty management, and modified incremental learning. The implementation of AIOps is the same process as the open-world learning algorithm, except for the self-healing. One of the most important processes in AIOps is log anomaly detection, which belongs to novelty detection in the open-set recognition framework. The complexity of open-world learning lies in the fact that the dataset is a function of time. Specifically, as time progresses, initially unknown data can be labeled to become known.

3.1.2. Open-World Learning Processes

Figure 1 shows the open-world learning process. The open-world stream is defined as $S_k^O=\left\{x_k\in Q_k|Q_k\subseteq {\mathcal{W}}_k\right\}$, the flow of which is a time sequence. Each member is a single or batch of data points $x_k$ drawn from the query set $Q_k$ at time step $k\ge 1$. ${\mathcal{W}}_k$ corresponds to the data of the whole world. The whole process of training the model at time step k is ${Agent}_k$. During open-world learning, the model trains a feature extractor and a decoder on a closed-set dataset. In the open-set world stream $S^O$, the sample $x_k$ undergoes feature extraction to obtain the feature representation $f_k$ through the feature extractor $E_k$. Subsequently, the decoder $D_k$ is used to compute the probability $P_k$ of the data belonging to unknown, known, and discovered classes, $P_k={\left[p_u,p_{K_1},p_{K_2},\dots {,p}_{K_{n-1}},p_{K_n},p_{D_1},p_{D_2},\dots ,p_{D_{m_k-1}},p_{D_{m_k}}\right]}^T$ where n is the number of known categories in the training set, and $m_k$ is the number of discovery categories. If the unknown probability $p_u$ is less than a threshold $\tau$, the buffer $B_k$ is equal to the memory bank $M_k$. Otherwise, the buffer $B_k$ is equal to the cascade of features $f_k$ and memory banks $M_k$. Each instance of the buffer $B_k$ is given a supervised (human or another agent) or unsupervised label by novelty discovery using the function $C_k$. Finally, the ${Agent}_k$ is updated by adapting to new data based on the buffer $B_k$ and supervised/unsupervised labels. In the subsequent time step $k+1$, the agent ${Agent}_{k+1}$ is utilized for improved incremental learning, continuing this cycle until all classes are discovered.

3.2. Background of Evidential Deep Learning

Existing deep learning-based models typically use a softmax layer at the top of deep neural networks (DNNs) for classification, using maximum likelihood estimation (MLE) to find the best network parameters. However, these softmax-based DNNs are unable to estimate predictions for classes outside of the closed set, because the softmax scores are inherently point estimates of the predictive distribution. For out-of-distribution categories, the outputs of the softmax layer tend to be overconfident in false predictions. Out-of-distribution anomaly detection can be implemented in a variety of ways, with EDL [17,18] networks being a natural approach that allows the network to effectively say “no”.

EDL overcomes the limitations of softmax-based DNNs by introducing Dempster’s evidential framework, Shafer theory (DST) [22], and subjective logic (SL). EDL combines multi-class classification and uncertainty modeling to provide interpretable methods. Specifically, EDL interprets the standard output of the classification network as a distribution over any possible belief state categories. Assuming that category probabilities follow a prior Dirichlet distribution, the set of distribution parameters is replaced with the parameters of the Dirichlet density for novelty detection.

4. Methodology

In this section, the main networks used in the proposed method and the architecture of the proposed model are explained. Section 4.1 gives the description and importance of masked language modeling (MLM), Section 4.2 presents the uncertainty in log sequence detection, Section 4.3 introduces log preprocessing, Section 4.4 offers the problem definition, and Section 4.5 introduces the LogEDL model.

4.1. Masked Language Model

The operating mechanism of LogEDL proposed in this study is shown in Figure 2. Since LogEDL performs anomaly detection through the pre-training method of MLM of BERT and the EDL method, a detailed description of MLM is given in this section, and a detailed description of the EDL method in the log anomaly detection process is given in Section 4.2.

The reasons for using MLM for system log anomaly detection are as follows. Due to the existence of a large amount of log data, there are enough data to effectively pre-train BERT using the MLM approach, resulting in a large number of contextual and structural features of the log sequences. MLM is inspired by the completion task, which replaces certain tokens of the input sentence with [MASK], and then, predicts the token in the [MASK] tokens. Bert is a large-scale bidirectional (fully contextual) trained model, which means that the model takes into account the contextual information of the entire input sequence when generating the representation of each word, unlike the pre-training of left- or right-directed language models. The LogEDL model uses the MLM-trained model for anomaly detection, which is expected to improve the generalization performance of the logs for effective anomaly detection.

Second, the task of labeling log sequences is very difficult, and the number of normal log sequences is much larger than the abnormal log data. Anomaly detection is a method based on unsupervised learning that uses only normal data for training to detect anomalies. While MLM is a self-training task, it does not need to label the data, and it is suitable for cases where the normal data are much larger than the abnormal data, which meets the purpose of anomaly detection. Therefore, it is more suitable than supervised binary classification-based methods for situations where there is a large amount of normal data.

Finally, from a practical operations perspective, experts detect anomalies by comparing the actual log keys with the generated log keys. While MLM methods can focus on patterns in context, they tend to have lower prediction probabilities for abnormal patterns [20]. However, MLM training methods cannot yet express anomalies.

4.2. Uncertainty in Log Sequence Detection

In this paper, we apply EDL combined with the MLM method to construct anomaly log sequence detection.

First, valuable semantic information is mainly contained in log sequence variations such as trends, periods, and peaks and valleys, which may correspond to the system state. For log sequences, there is a correlation between the logs at each location in the log sequence and the nearby logs. Anomalous log sequences are often the result of unexpected logs appearing at log locations where they should be.

Second, recent models using the MLM approach in BERT cannot detect anomalies autonomously. However, researchers have been able to detect anomalies through some phenomena such as low prediction probability of anomalous logs and large loss values [20]. Because the logs on the mask can be in multiple situations, normal logs can be mistaken for anomalous logs by simple presentation methods.

We propose a log anomaly detection system, LogEDL, that exploits the fact that EDL can be characterized by high uncertainty values for unseen patterns to overcome the limitations of existing methods. Uncertainty modeling is performed for each [mask] location, and the uncertainty of the [mask] location responds to the likelihood of an anomaly occurring there. In the training phase, by applying EDL to the MLM learning of normal logs, we are essentially trying to collect evidence of the classes reflected by the context at each mask. In the testing phase, the assumption of contextual uncertainty at each mask allows the model to “know the unknown”. Therefore, uncertainty is utilized to detect anomalies in the logs.

4.3. Preprocessing

There are a number of log parsing methods, such as Spell [23], IPLoM [24], Drain [25], and so on. According to previous studies [26], weak log parsing methods can negatively affect the performance of subsequent anomaly detection tasks. Drain has become a common method used by researchers with its superior accuracy and efficiency [27,28]. Therefore, in this work, we use Drain for log message parsing. As shown in Figure 2, the HDFS dataset is used as an example. Each raw log message consists of a constant part (also known as “log keys” or “log events”) and a variable part (also known as “parameters”). The logs are first parsed into log keys using the method of Drain, then each log is mapped to its key to obtain structured logs, and finally, a vocabulary of log keys is constructed, and a log sequence is obtained by sorting the log data in time order according to the block of the log data.

4.4. Problem Definition

Log anomaly detection in this study can be defined as follows. Given an unstructured sequence of log messages, we aim to detect whether this sequence is normal or abnormal. We can then define the log sequence as an ordered sequence of log keys $S=\left\{k_1,\dots ,k_t,\dots ,k_T\right\}$, where $k_t\in V$ represents the log key at position t, and V denotes the set of unique log keys extracted from the log messages. The goal of this task is to predict whether a new log sequence S is abnormal or not based on a training dataset $Datase{t}_{train}={\left\{S_i\right\}}_{i=1}^N$ that contains only normal log sequences. To achieve this goal, LogEDL models normal log sequences and further derives an anomaly detection criterion to identify anomalous sequences. In the training phase, normal log key sequence keys are replaced by [MASK] tokens with a certain probability for network optimization; in the testing phase, the replacement is performed with the same probability to determine whether a log sequence is normal or not for the number of uncertainty log keys.

4.5. LogEDL

In this section, we present LogEDL, a semi-supervised log representation framework for anomaly detection. An overview of the proposed LogEDL framework is shown in Figure 3. It consists of three main components, Transformer Encoder, ENN head, and uncertainty. Transformer Encoder is used after the log sequence is tokenized to extract the contextual information of the log sequence and transform it into semantic vectors. In the second step, ENN head is used to convert the semantic vectors into evidence of the logs, which is the core of LogEDL, and finally, uncertainty is applied to make the judgment of anomaly detection based on the computed uncertainty. For this purpose, we build an evidential deep learning anomaly detection model. It mainly solves the problem of how to extract evidence of separable features with contextual information, which means that normal and abnormal sequences have boundaries in the feature space. Finally, the uncertainty of the extracted evidence is computed for anomalous log sequence detection. We describe each component in detail in the following subsections.

4.5.1. Transformer Encoder

We treat log sequences as sentences in natural language and use a Transformer Encoder to extract the semantic vectors of log keys and capture the input log sequence contextual semantic information through a multi-head attention mechanism. The Transformer Encoder consists of multiple transformer layers. Each transformer layer consists of a multi-head self-attention and a position-wise feed-forward sub-layer using residual connection around the two sub-layers followed by layer normalization [8]. The sequence of logs $S=\left\{k_1,\dots ,k_t,\dots ,k_T\right\}$, where T is the sequence length, then represents each log key $k_t$ as an input representation $x_t$, where the representation $x_t$ is obtained by using the tokenizer, resulting in $X=\left\{x_1,\dots ,x_t,\dots ,x_T\right\}$. Secondly it is used as input to the Transformer Encoder to obtain a d-dimensional semantic vector $h_t$, where d is set to 256. Finally, we obtain the sequence of semantic vectors, written as $H=\left\{h_1,\cdots ,h_t,\cdots ,h_T\right\}$. To summarize, the Transformer Encoder represents the log keys with semantic vectors, which preserves the semantic information of the events; similar log keys also have similar semantic vectors, mitigating the effect of unstable logs and noise.

4.5.2. ENN Head

After the Transformer Encoder, each log key is represented as a semantic vector. To detect anomalous sequences, we propose an evidential neural network head (ENN head) based on an uncertainty mechanism, as shown in Figure 4. This module includes two fully connected (FC) layers, which are used to learn the evidence of contextual patterns across the entire sequence. Given a sequence of semantic vectors, $H=\left\{h_1,\cdots ,h_t,\cdots ,h_T\right\}$, the evidence $E=\left\{e_1,\cdots ,e_t,\cdots ,e_T\right\}$ for this sequence is learned in two layers of FC after the ENN head. Note that we have the last layer of hidden states as output, denoted as

$$e_t={\sigma (w}_2\sigma \left(w_1{h}_t+b_1\right)+b_2)$$

(1)

where $w_1,w_2,b_1$, and $b_2$ are the trainable parameters of the input $h_t$, and $\sigma \left(·\right)$ is the ReLU function.

When we obtain the evidence $e_t$, it can efficiently learn the information about the mask position of the input log sequence about the context and use the last step of the hidden state of the output as a representation of the context pattern of this log key position. To achieve this goal, we propose MMSM to model normal sequences, followed by log sequence-wise contrastive learning for representation learning, and further, utilize an evidential neural network module to derive an anomaly detection criterion for identifying anomalous sequences.

4.5.3. Uncertainty

After obtaining the evidence, the Dirichlet prior distribution is utilized to compute the uncertainty, which is then used for subsequent anomaly detection. The Dirichlet distribution is a probability density function (pdf) that defines the possible values of a probability mass function (pmf) p. It is characterized by K parameters $\alpha =\left[{\alpha }_1,\dots ,{\alpha }_k,\dots ,{\alpha }_K\right]$ and is expressed as follows:

$$Dir\left(p|\alpha \right)=\left\{\begin{array}{ll}{\displaystyle \frac{1}{B\left(\alpha \right)}}\prod _{k=1}^K{p}_k^{{\alpha }_k-1},& forp\in P_K\\ 0,& otherwise\end{array}\right.$$

(2)

where $P_K$ is the K-dimensional unit simplex, $P_K=\left\{p\mid {\sum }_{i=1}^K{p}_i=1\mathrm{a}\mathrm{n}\mathrm{d}0\le p_1,\cdots ,p_K\le 1\right\}$${\alpha }_k$ represents the concentration parameter of the Dirichlet distribution, and K denotes the number of log key classes.

According to belief theory [29], subjective logic provides a belief mass $b_k$ for each singleton $k=1,\cdots ,K$, along with an overall uncertainty mass u. These $K+1$ mass values are non-negative and sum to 1, i.e.,

$${\displaystyle u+\sum _{k=1}^K{b}_k=1}$$

(3)

where $u\ge 0$ and $b_k\ge 0$, for $k=1,\dots ,K$.

A belief mass $b_k$ for a singleton k is computed using evidence for a singleton. Let $e_k\ge 0$ be the evidence derived for the $k\mathrm{th}$ singleton, then the belief $b_k$ and the uncertainty u are computed as

$$b_k={\displaystyle \frac{e_k}{\mathcal{S}}}\mathrm{and}u={\displaystyle \frac{K}{\mathcal{S}}}$$

(4)

where $\mathcal{S}={\sum }_{k=1}^K{\alpha }_k$ is referred to as the Dirichlet strength, and a belief mass assignment, i.e., subjective opinion, corresponds to a Dirichlet distribution with parameters ${\alpha }_k=e_k+1$.

The probability is given by

$$p_k={\displaystyle \frac{{\alpha }_k}{\mathcal{S}}}$$

(5)

We optimized the network by setting a suitable loss function, enabling it to effectively learn normal log sequences. By leveraging uncertainty, we identified unknown anomalous logs, thus determining whether a log sequence is anomalous. We use the K-simplex (illustrated as a triangular heatmap in Figure 5) to jointly represent the token classification probability and uncertainty at the masked positions. The uncertainty detection at [MASK] is particularly suitable for log anomaly detection. Using a three-class classification as an example and assuming the first class is the correct label, a well-calibrated model should give accurate and certain (AC) predictions (Figure 5a), indicating that the context for the [MASK] token is very certain. Accurate and uncertain (AU) predictions (Figure 5b) indicate that the model is somewhat confused by the context and cannot classify it well. Inaccurate and certain (IC) predictions (Figure 5c) suggest that the [MASK] token could have multiple possible values, which aligns well with the contextual variability in log sequences. Inaccurate and uncertain (IU) predictions (Figure 5d) indicate that the context for the [MASK] token is completely unfamiliar, suggesting a high likelihood of the log sequence being anomalous.

4.5.4. Anomaly Detection

After training, we can deploy LogEDL for detecting anomalous log sequences. The idea behind applying LogEDL for log anomaly detection is that “there are differences in the context between normal system logs and anomalous system logs”. In other words, when a language model trained solely on normal log data encounters the context of anomalous logs during testing, it is expected to exhibit higher uncertainty.

In this study, the defined uncertainty refers to the logit values produced by the model when predicting the [MASK] token, indicating the network’s inability to make a judgment on this contextual pattern. Furthermore, when the uncertainty is high, it suggests that the corresponding context is difficult to find in normal contexts, thus identifying it as an anomaly.

Therefore, we can derive the anomaly score of a log sequence based on the predicted logits for the [MASK] token. To achieve this, given a test log sequence, we first randomly replace a certain proportion of log keys with the [MASK] token, similar to the training process, and use the randomly masked log sequence as input to LogEDL. The uncertainty of the tth log key refers to the degree of unknowns represented by $u_t$ when the log key at the [MASK] position is considered as the answer. For each [MASK] token, we compute $u_t$, which reflects the uncertainty at this masked position. For N [MASK] tokens, we select the $top-n$$u_t$ values and compute the final anomaly score to detect anomalies. The formula for calculating the anomaly score of a test log sequence is shown in Equation (6).

$$\mathrm{abnormal}={\displaystyle \frac{1}{n}}\sum _{t\in Top-nindices}{u}_t$$

(6)

4.5.5. Evidential Uncertainty Loss

To train the LogEDL model, we follow the MLM paradigm to capture patterns in normal log sequences. The neural network architecture of LogEDL is very similar to that of a classical neural network. The only difference is that we replace the softmax layer with an activation layer, such as ReLU, to ensure non-negative outputs, which serve as the evidence vector for predicting the Dirichlet distribution.

Given a sample $k_i$, let $f\left(k_i\right|\Theta )$ represent the evidence vector $e_i$ predicted by the network for classification, where $\Theta$ denotes the network parameters. The corresponding Dirichlet distribution has parameters ${\alpha }_i=f\left(k_i\right|\Theta )+1$. Once these distribution parameters are computed, their mean, $k_i/\mathcal{S}$, can be used as an estimate of the class probabilities. Let the ground truth $y_i$ be the one-hot vector encoding the observation $k_i$, with $y_{ij}=1$ and $y_{ij}=0$ for all $i\ne j$. Here, ${\alpha }_i$ is the parameter of the Dirichlet density for the predictor. Assuming that class probability follows a prior Dirichlet distribution, the evidential mean squared error loss (EMSE) to be minimized for learning eventually reduces to the following form.

$$\begin{array}{ll}\hfill {\mathcal{L}}_{EMSE}^i\left(\Theta \right)& {\displaystyle =\int {∥y_i-{\mathrm{p}}_i∥}_2^2\frac{1}{B\left({\alpha }_i\right)}\prod _{i=1}^K{p}_{ij}^{{\alpha }_{ij-1}}d{\mathrm{p}}_i}\\ \hfill & {\displaystyle =\sum _{j=1}^K\mathbb{E}[y_{ij}^2-2y_{ij}{p}_{ij}+p_{ij}^2]}\\ \hfill & {\displaystyle =\sum _{j=1}^K(y_{ij}^2-2y_{ij}\mathbb{E}\left[p_{ij}\right]+\mathbb{E}\left[p_{ij}^2\right])}\\ \hfill & {\displaystyle =\sum _{j=1}^K{(y_{ij}-{\alpha }_{ij}/{\mathcal{S}}_i)}^2+\frac{{\alpha }_{ij}({\mathcal{S}}_i-{\alpha }_{ij})}{{\mathcal{S}}_i^2({\mathcal{S}}_i+1)}}\\ \hfill & {\displaystyle =\sum _{j=1}^K{(y_{ij}-p_{ij})}^2+\frac{p_{ij}(1-p_{ij})}{({\mathcal{S}}_i+1)}}\end{array}$$

(7)

By decomposing the first and second moments, the loss function aims to achieve a joint objective of minimizing both the prediction error and the variance of the Dirichlet distribution generated by the neural network for each sample in the training set. In this way, it prioritizes data fitting over variance estimation. Given that the patterns of normal log sequences and anomalous log sequences are different, we expect that once LogEDL can correctly predict the masked log keys, it will be able to distinguish between normal and anomalous log sequences.

We can obtain the negative log marginal likelihood by integrating out the class probability; the evidential negative log-likelihood loss (ENL) takes

$$\begin{array}{ll}\hfill {\mathcal{L}}_{ENL}^i\left(\Theta \right)& {\displaystyle =-\log (\int \prod _{j=1}^K{p}_{ij}^{y_{ij}}\frac{1}{B\left({\alpha }_i\right)}\prod _{i=1}^K{p}_{ij}^{{\alpha }_{ij-1}}d{\mathrm{p}}_i)}\\ \hfill & {\displaystyle =\sum _{j=1}^K{y}_{ij}(\psi \left(S_i\right)-\psi \left({\alpha }_{ij}\right))}\end{array}$$

(8)

The same approach can be applied also to the cross-entropy loss, the evidential cross-entropy loss (ECE) reads

$$\begin{array}{ll}\hfill {\mathcal{L}}_{ECE}^i\left(\Theta \right)& {\displaystyle =\int \left[{\displaystyle \sum _{j=1}^K-y_{ij}\log \left(p_{ij}\right)}\right]\frac{1}{B\left({\alpha }_i\right)}\prod _{i=1}^K{p}_{ij}^{{\alpha }_{ij-1}}d{\mathrm{p}}_i}\\ \hfill & {\displaystyle =\sum _{j=1}^K{y}_{ij}(\psi \left(S_i\right)-\psi \left({\alpha }_{ij}\right))}\end{array}$$

(9)

5. Experiment and Analysis

5.1. Experiment Setting

5.1.1. Dataset

We benchmark our log anomaly detection method on three log datasets: HDFS [30], BGL [27], and Thunderbird [31]. These datasets are sourced from the large-scale log dataset LogHub, published by Jieming Zhu et al. [32]. The statistics of these datasets are shown in

Table 1. Statistics of evaluation datasets.

Dataset	Log Messages	Normal	Anomaly
HDFS	11,175,629	10,887,339	288,290
BGL	4,747,963	4,399,503	348,460
Thunderbird	20,000,000	1,241,438	758,562

. These datasets have been preprocessed as described earlier, with log data grouped into log sequences. For the HDFS dataset, data are partitioned using Block ID, while the BGL and Thunderbird datasets are partitioned using a time window. The preprocessing results are shown in

Table 2. Preprocessing on HDFS, BGL, and Thunderbird datasets.

Dataset	Log Sequences	Log Keys	Train Dataset	Test Dataset		Average Length
			Normal	Normal	Anomaly
HDFS	742,527	47 (18)	167,466	558,223	16,838	19
BGL	37,315	334 (175)	13,718	20,579	3018	562
Thunderbird	122,540	1165 (866)	46,293	30,862	45,385	326

.

The HDFS dataset was generated by running benchmark workloads in a private cloud environment and was manually labeled using hand-crafted rules to identify anomalies. Log data were divided into log sequences based on block IDs. Each log sequence associated with a block ID was assigned a ground truth label of either normal or anomalous. The HDFS dataset contains 11,175,629 log messages, grouped into 742,527 log sequences, with 725,689 normal sequences and 16,838 anomalous sequences. The average sequence length is 19. A total of 47 log keys were extracted from the dataset, with 18 log keys appearing in normal sequences. The model was trained on this dataset for 200 epochs with a masking ratio of 0.65.

The BGL dataset comes from the BlueGene/L supercomputer system at the Lawrence Livermore National Laboratory (LLNL) in the United States. The logs include both alert and non-alert messages, with alert messages considered anomalies. The BGL dataset contains 4,747,963 log messages, grouped into 37,315 log sequences, with 10,131 normal sequences and 2630 anomalous sequences. For BGL, the data were partitioned using a sliding time window of 5 min to generate log sequences, with an average sequence length of 562. A total of 334 log keys were extracted from the dataset, with 175 log keys appearing in normal sequences. The model was trained on this dataset for 200 epochs with a masking ratio of 0.50.

The Thunderbird dataset is another large log dataset collected from a supercomputer system, specifically the Thunderbird microprocessor. The logs contain both alert and non-alert messages, identified by alert category labels. In the first column of the logs, “-” indicates a non-alert message, while other entries indicate alert messages. This labeling information can be used for alert detection and prediction research. The Thunderbird dataset contains 20,000,000 log messages, grouped into 122,540 log sequences, with 77,155 normal sequences and 45,385 anomalous sequences. For the Thunderbird microprocessor dataset, we used a sliding time window of 1 min to generate log sequences, with an average sequence length of 326. A total of 1165 log keys were extracted from the dataset, with 866 log keys appearing in normal sequences. The model was trained on this dataset for 200 epochs with a masking ratio of 0.50.

5.1.2. Implementation Details

The model was implemented using the open-source deep learning framework PyTorch (v1.9.0, created by the Meta Platforms, Menlo Park, CA, USA), and it was run on a server equipped with an AMD EPYC 7773X 64-core processor (manufactured by AMD, Santa Clara, CA, USA) and an Nvidia GeForce RTX 3090 GPU (NVIDIA, Santa Clara, CA, USA). The network was optimized using the Adam optimizer with a learning rate of 1 × 10⁻⁴. A validation set was used to select the best model for testing.

5.1.3. Evaluation Metrics

In this study, similar to previous research, we use the F1 score, precision, and recall as evaluation metrics to validate the effectiveness of the LogEDL model for anomaly detection. precision represents the percentage of true anomalies among all detected anomalies, while recall represents the percentage of detected anomalies in the anomalous dataset. The F1 score is calculated as the harmonic mean of these two metrics, as shown below:

$$precision={\displaystyle \frac{TP}{TP+FP}}$$

(10)

$$recall={\displaystyle \frac{TP}{TP+FN}}$$

(11)

$$F1score=2·{\displaystyle \frac{precision·recall}{precision+recall}}$$

(12)

where $TP$ is true positive, $FP$ is false positive, and $FN$ is false negative. The F1 score ranges from 0 to 1, with higher values indicating better performance of the classifier. A higher F1 score signifies that the model has a good balance between precision and recall, effectively capturing the overall performance in anomaly detection. This makes the F1 score a comprehensive metric for evaluating the model’s effectiveness in identifying anomalies.

5.1.4. Baselines

To evaluate the performance of LogEDL, we compare it with the following log anomaly detection methods:

Principal component analysis (PCA) [30]: A traditional machine learning method for dimensionality reduction-based anomaly detection, combined with item weighting techniques from the data retrieval field. Suitable for simpler data scenarios.

Isolation Forest (iForest) [33]: An unsupervised learning algorithm for anomaly detection that represents features as tree structures.

One-class SVM (OCSVM) [34]: A well-known model for one-class classification, widely used for anomaly detection in log data where only normal data are observed.

LogCluster [4]: Based on similarity clustering methods, LogCluster automatically identifies similar log patterns and clusters them to achieve automatic anomaly recognition.

DeepLog [7]: An anomaly detection model based on log templates which uses LSTM to model log data, automatically learns patterns of anomalous behavior, and compares predicted logs with real logs for anomaly detection.

LogAnomaly [35]: Integrates log templates by analyzing synonyms in logs; encodes log templates into semantic vectors and inputs them into an LSTM model for anomaly detection.

LogBERT [19]: A method based on BERT, LogBERT constructs two self-supervised training tasks to learn patterns of normal log sequences and detect anomalies deviating from the normal model.

5.2. Results and Analysis

5.2.1. Comparison Methods

Our method LogEDL is compared with traditional machine learning and deep learning approaches. The comparative experiments are shown in

Table 3. Performance comparison of LogEDL and baseline methods.

Method	HDFS			BGL			Thunderbird
	Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1
PCA	5.89	100.00	11.12	9.07	98.23	16.61	37.35	100.00	54.39
iForest	53.60	69.41	60.49	99.70	18.11	30.65	34.45	1.68	3.20
OCSVM	2.54	100.00	4.95	1.06	12.24	1.96	18.89	39.11	25.48
LogCluster	99.26	37.08	53.99	95.46	64.01	76.63	98.28	42.78	59.61
DeepLog	88.44	69.49	77.34	89.74	82.78	86.12	87.34	99.61	93.08
LogAnomaly	94.15	40.47	56.19	73.12	76.09	74.08	86.72	99.63	92.73
LogBERT	87.02	78.10	82.32	89.40	92.32	90.83	96.75	96.52	96.64
LogBERT (EVT)	29.24	69.01	41.07	28.11	73.05	40.60	30.78	74.24	43.51
LogEDL (ours)	90.06	92.80	91.41	99.84	97.26	98.53	97.80	98.08	97.91

; traditional anomaly detection methods, such as PCA, Isolation Forest, and OCSVM, demonstrate suboptimal performance across the three datasets. Although these methods sometimes achieve high precision or recall, for instance, PCA’s recall reaches $100\%$ on all datasets, their F1 score remain exceedingly low. This indicates an inability to balance precision and recall effectively, likely due to their failure to capture the temporal sequence information inherent in log data, resulting in inadequate anomaly detection capabilities.

LogCluster, specifically designed for log anomaly detection, shows improvements over traditional methods, particularly on the HDFS and BGL datasets. However, its performance on the Thunderbird dataset remains unsatisfactory, with F1 score failing to reach high levels.

DeepLog and LogAnomaly, two deep learning-based methods, significantly outperform the traditional methods and LogCluster, showcasing the advantages of deep learning in capturing log sequence patterns. Both methods achieve high F1 score across the three datasets, especially excelling on the HDFS and Thunderbird datasets.

LogBERT, a method based on pre-trained models and self-supervised learning tasks, demonstrates superior performance across all three datasets, with F1 score markedly higher than those of traditional methods and other deep learning approaches. LogBERT successfully models normal log sequences and accurately identifies anomalous sequences through self-supervised pre-training tasks.

Extreme value theory (EVT) [36], also known as the Fisher–Tippett Theorem, is used as a simple evidence theory for anomaly detection based on the LogBERT model. However, since it relies on empirical thresholds, the results are not ideal.

Our proposed method, LogEDL, surpasses all other methods across the three datasets, exhibiting exceptional performance. On the HDFS dataset, LogEDL achieves an F1 score of 91.41, significantly higher than LogBERT’s 82.32. On the BGL dataset, LogEDL’s F1 score of 98.53 substantially exceeds LogBERT’s 90.83. On the Thunderbird dataset, LogEDL’s F1 score of 97.91 outperforms LogBERT. This demonstrates that LogEDL maintains high precision and recall while achieving the best F1 score.

The comparative analysis indicates that LogEDL outperforms other methods in log anomaly detection tasks. Traditional methods (PCA, Isolation Forest, OCSVM) exhibit significant shortcomings in balancing precision and recall, resulting in low F1 score. The specialized method LogCluster shows some improvements but still lags behind deep learning methods. DeepLog and LogAnomaly demonstrate the strengths of deep learning, while LogBERT further enhances performance through self-supervised learning. Ultimately, LogEDL consistently achieves the best overall performance across the three datasets, proving its efficiency and reliability in log anomaly detection tasks.

In real-time log anomaly detection, computational efficiency is crucial. In

Table 4. A comprehensive analysis of the computational efficiency of LogEDL (HDFS); all models tested with a sequence length of 512 and batch size of 1.

Method	Flops	Parameters	Time (s)
PCA	-	-	$2.6\times {10}^{-6}$
iForest	-	-	$2.3\times {10}^{-6}$
OCSVM	-	-	$1.2\times {10}^{-2}$
LogCluster	-	-	$3.2\times {10}^{-5}$
DeepLog	1.0 M	0.1 M	$3.8\times {10}^{-4}$
LogAnomaly	2.0 M	0.1 M	$1.3\times {10}^{-3}$
LogBERT	628.6 M	2.1 M	$3.7\times {10}^{-3}$
LogEDL (ours)	1076.8 M	2.1 M	$3.7\times {10}^{-3}$

, we analyzed computational efficiency using the HDFS dataset. The results show that PCA and iForest have the shortest computation times, followed by LogCluster, DeepLog, LogBERT, and LogEDL, with OCSVM being the slowest. DeepLog and LogAnomaly have relatively low computation and parameter counts with moderate computation times, making them efficient methods. Although LogEDL has the highest computation and parameter counts, its computation time is similar to LogBERT. For practical online deployment, our method’s inference time remains long, necessitating future research on model acceleration and quantization.

5.2.2. Ablation Studies

The ablation experiments are shown in

Table 5. Ablation experiments on different evidence loss functions.

Method	HDFS			BGL			Thunderbird
	Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1
LogBERT	87.02	78.10	82.32	89.40	92.32	90.83	96.75	96.52	96.64
LogEDL (w. ENL)	89.14	84.59	86.81	97.43	92.54	94.92	96.70	96.96	96.83
LogEDL (w. EMSE)	89.93	84.62	87.19	97.46	92.34	94.83	96.99	97.80	97.39
LogEDL (w. ECE)	90.06	92.80	91.41	99.84	97.26	98.53	97.80	98.08	97.91

. Across all three datasets, the LogEDL model optimized with the ECE loss function demonstrates the best performance. ECE outperforms other methods regarding precision, recall, and F1 score. Notably, on the BGL dataset, its precision reaches an almost perfect $99.84\%$. Compared to LogBERT, the LogEDL model shows improvements in all cases, especially when using the $ECE$ loss function, resulting in significant enhancements. Therefore, for similar log anomaly detection tasks, it is recommended to use the LogEDL model optimized with the evidence cross-entropy loss function to achieve the best performance.

6. Discussion

We have demonstrated the feasibility and effectiveness of using uncertainty to identify anomalous logs in LogEDL, showcasing the significant potential of evidential neural networks in practical log anomaly detection. Nevertheless, several open research questions related to log anomaly detection remain to be further explored and elucidated.

In the real world, log patterns continuously evolve with software updates or system configuration changes. LogEDL assumes that log templates are stable and unchanging, which hinders the model’s ability to detect anomalies due to changes in log patterns. However, with the advancement of large language models (LLMs), current changes in log patterns can be easily transformed into our desired format. We propose developing a log template management system that combines LLMs and machine learning technologies to automatically detect and update log templates. This system will ensure the accuracy of log parsing and anomaly detection. For logs that differ significantly from the existing log template set, an online learning or incremental learning framework should be utilized to update the log collection.

The interpretability of deep learning models is crucial. The LogEDL model quantifies the degree of anomaly in logs using uncertainty, yet this study does not delve deeply into the relationship between the sequence of logs labeled as anomalies and the underlying causes of these anomalies. By employing masked language modeling (MLM), the LogEDL model effectively learns the contextual relationships of normal log sequences and identifies logs that do not conform to contextual semantics as anomalous. The process of log templating can result in the loss of some critical information inherent in the original logs, which hinders the interpretability of log anomalies. In the future, deep learning techniques should be utilized to extract latent semantic vectors from logs to better understand the reasons behind certain log sequences being marked as anomalies. Integrating interpretable AI techniques will enhance the interpretability of the anomaly detection process.

The scalability of LogEDL in handling large volumes of log data has yet to be thoroughly investigated. In practical applications, logs are generated in the thousands, necessitating the anomaly detection system to scale accordingly. These models must be able to operate efficiently with varying scales of log data. In the future, message queues could be employed to manage sudden surges of log sequences, custom-adapting to existing monitoring tools and databases.

Future research could explore the development of more computationally efficient, highly generalizable, and scalable models that require minimal normal data for training, thereby quickly adapting to changes in real-world applications. Additionally, utilizing large models and incremental learning strategies could be beneficial. With this approach, the model could be initially labeled using a large model, and as new data are parsed, the model could be updated incrementally. This could help the system adapt to evolving logs without requiring extensive computational resources.

7. Conclusions

This paper proposes a log sequence anomaly detection framework based on evidential deep learning, named LogEDL. This method goes beyond traditional log sequence anomaly detection approaches by introducing a new task that leverages uncertainty to identify anomalies. Specifically, LogEDL not only prioritizes precise classification but also quantifies the uncertainty of prediction outcomes, thereby augmenting the model’s robustness and accuracy in the context of anomalous log detection tasks. This functionality is similar to open-set recognition patterns. The experimental results show that compared to state-of-the-art anomaly detection methods, LogEDL can maximally learn various contextual patterns of normal log sequences, thereby achieving advanced performance in anomaly detection tasks. In the future, we plan to further extend LogEDL to large-scale and diverse log datasets to establish a foundational model for log sequence analysis.