5.1. Comparison Experiments
We make a comparison between IPLog and the six methods introduced in baselines. From the results in
Table 3, we can see that our method outperforms almost all baseline methods in the evaluation metrics GA and PA. Specifically, in terms of group accuracy (GA), IPLog can achieve more than 0.95 accuracy on all datasets. In addition, IPLog achieves nearly 1.0 group accuracy on all three datasets except Hadoop, which outperforms existing log parsing methods. IPLog’s performance is even more outstanding in terms of parsing accuracy (PA), which reaches more than 0.99 on both HDFS and Apache datasets, significantly outperforming the baseline method. For the Hadoop and Zookeeper datasets, IPLog still achieves the best performance, although it does not achieve accuracy above 0.9. IPLog’s remarkable parsing accuracy underscores its ability to effectively discern the templates and corresponding parameters within log messages. The experimental results conclusively validate IPLog’s capability to accurately identify log templates and parameters, thereby streamlining the categorization of logs into cohesive templates.
In order to further study the parsing effect of IPLog on specific logs, we randomly selected 10 log messages in the HDFS dataset and studied and compared the parsing results of the seven methods for these 10 log messages. As shown in
Figure 5, the green log templates represent the correctly parsed templates, e.g., for the LFA method, only the second log message “PacketResponder 2 for block blk_5340239390217577926 terminating” is correctly parsed as the log template “PacketResponder <*> for block <*> terminating”, and the other nine log messages are parsed incorrectly. By comparison, we find that IPLog performs the best, successfully parsing all 10 log messages into correct log templates. For example, for complex logs like “Receiving block blk_-251869386118078153 src:/10.251.203.246:38466 dest:/10.251.203.246:50010”, IPLog can accurately parse it into the log template “Receiving block <*> src: <*> dest: <*>”. Except for LogPPT, which is also based on a neural network and can also parse it correctly, the other methods all parse it incorrectly. Overall, Drain, the best parser among the non-neural network methods, only parsed five of the ten log messages correctly, which also shows that it is difficult to adapt traditional parsing methods to complex system logs. In the neural network-based comparison, compared with LogPPT, IPLog parses the parameters more accurately. In the log message “Starting thread to transfer block blk_4565119633233737252 to 10.251.107.50:50010,10.251.107.19:50010”, LogPPT incorrectly replaces the last two parameters “10.251.107.50:50010,10.251.107.19:50010” with a single <*>, which is obviously not conducive to log template differentiation.
Comparison of the parsing results of 10 randomly selected log messages in the HDFS dataset with different parsing methods illustrates that IPLog can identify log templates and parameters more accurately, confirming the powerful ability that IPLog shows in log parsing tasks.
IPLog can maintain a high accuracy primarily because it learns semantic information from logs, thus achieving precise predictions of templates and parameters, a capability not found in rule-based methods. For example, in
Figure 5, the log “Receiving block blk_-251869386118078153 src:/10.251.203.246:38466 dest:/10.251.203.246:50010” pertains to a data transfer process in HDFS. In this specific log, the “/” serves not merely as a character but also as a part of the network address, thereby ensuring the integrity and correctness of the parameters. However, apart from IPLog and LogPPT, which determine through learned semantic information that “/” is part of a network address, other methods merely split the log by characters without understanding the role of “/” in representing addresses, leading to incorrect field extraction.
To further study the grouping effect of IPLog on specific log templates, we randomly selected seven log templates from the Hadoop dataset and studied and compared the grouping results of the seven methods on these templates. The results are shown in
Table 4, where the numbers in the table indicate how many groups each method actually grouped the current template into after parsing was completed. The correct grouping situation is that each template should correspond to one group. Through this comparative analysis, we can evaluate the accuracy and effectiveness of each method in grouping templates. As can be seen from the table, both IPLog and Drain perform best by correctly grouping all the chosen templates into their respective original groups. The other methods, however, show large differences in their grouping results, especially the SLCT, Spell, and LenMa methods, which group into a large number of groups on multiple templates, suggesting that these methods have significant deficiencies in grouping log templates together.
IPLog processes all system logs using a fixed set of hyperparameters, without the need for readjustments for each dataset or the setting of domain-specific regular expressions for different systems. However, similarity-based clustering, frequent pattern mining, and heuristic methods need to be adjusted according to different systems. Here, we demonstrate the regular expressions and the “support” parameter settings for the SLCT algorithm tailored to different systems. As shown in
Table 5, a suitable regular expression is required to match IP addresses in all four types of system logs. For example, in HDFS logs, an additional match for block ID “blk_-?\d+” is necessary, while in Zookeeper logs, a leading “/” must be handled. These requirements vary by system. Additionally, the “support” parameter is passed to the SLCT method, specifying the threshold for template extraction support needed during the log parsing process. This parameter determines how many times a template must appear in the log data to be considered valid or representative. Adjusting the “support” parameter significantly affects the results of log parsing, including the number, type, and specifics of the templates.
Table 5 shows the settings of the “support” parameter for each system under optimal conditions, reflecting their differences. These differences hinder the methods’ generality.
Importantly, as systems are updated and log templates and formats change, rule-based log parsing methods like SLCT struggle to match new log formats. To further validate our idea, we simulated the process of log changes: relatively simple and few logs from loghub-2k to represent the log output before system upgrades, and a large number of complex logs from loghub2.0 to represent the log output after system updates. As
Table 6 shows, IPLog experienced a minimal decrease in GA and PA metrics, only 0.30% and 0.76%, and was virtually unaffected. However, other methods, except for LFA, showed a more significant decrease in these metrics. For example, SLCT saw the largest drop in GA and PA metrics, at 25.08% and 22.61%, respectively. The neural network-based LogPPT experienced decreases of 7.10% and 9.63% in GA and PA metrics. The experimental results confirmed that these methods struggle to adapt to changes in log formats, resulting in severe performance degradation.
In the experimental results, we observed that only LFA showed an increase in PA performance. A detailed analysis of LFA’s parsing results showed that it accurately parsed only a few log templates, such as “PacketResponder <*> for block <*> terminating”. As the proportion of such templates increased in loghub2.0, LFA’s PA metric correspondingly rose. From the comparison between IPLog and six other methods across loghub-2k and loghub2.0, it is inferred that in real production environments, when log templates change, IPLog is almost unaffected, whereas the other six methods experience a significant decline in performance, struggling to adapt to these changes.
5.2. Ablation Experiments
This section critically evaluates the performance of the fundamental components and parameters of our innovative approach. We first remove the manual feedback strategy, and as can be seen from the results in
Table 7, after removing the manual feedback, the average GA of IPLog on the four datasets decreases by 17.2% (from 0.987 to 0.817), with little change in PA. We analyzed this result and designed the manual feedback merge query to correct mainly the errors and omissions of templates in the grouping process, so the improvement for GA is significant. For templates that are grouped incorrectly, it is possible for the incorrect template to become the original correct template during the merge process, but since the number of templates in this category is small, the impact on PA is not significant.
In addition, in order to verify the importance of prompt tuning for IPLog, we first replaced the prompt tuning with fine-tuning, and the results in
Table 7 show a relative decrease in both GA and PA. In addition, during the postprocessing merging process, we found that the increase in parsing errors led to a significantly higher number of requests during merging, which undoubtedly increased the operational difficulty of postprocessing. In order to ensure the fairness of the comparison, we additionally dropped the manual feedback strategy at the same time on the basis of removing the prompt tuning, and the results in the table clearly show that IPLog based on fine-tuning and dropping the manual feedback decreased by 19.3% (from 0.817 to 0.659) in the metric GA and 2.9% (from 0.912 to 0.885) in the metric PA. The results of the ablation experiments in
Table 7 confirm that the manual feedback strategy and prompt tuning can effectively improve the performance of IPLog.
Our adaptive system sampling algorithm can effectively select the best training sample data. In order to confirm the effectiveness of our sampling algorithm, we compare the effect of two other sampling algorithms with the results: random sampling and is DPP sampling. Note that for the fairness of the comparison, we uniformly cancelled the postprocessing operation in our experiments. Random sampling is a basic data sampling method that constructs subsets by randomly selecting samples from historical log datasets. The method does not take into account the intrinsic structure or characteristics of the data, and all samples have the same probability of being selected. DPP sampling is a sampling method based on the determinantal point process (DPP), aiming to select samples with diversity and representativeness. The DPP captures the similarity between samples by defining a kernel matrix, thus tending to select samples that are not similar to each other.
From the results in
Figure 6, it is clear that the adaptive system sampling algorithm performs well in selecting samples, resulting in optimal model results. This result demonstrates the effectiveness of our sampling algorithm in maximally generalizing the features of the entire history log while using a small amount of data. Specifically, these sample data are not only optimized in terms of quantity but also balanced in terms of representativeness and diversity, thus ensuring the efficiency and accuracy of the model in subsequent training and analysis.