Knowledge-Guided Cyber Threat Intelligence Summarization via Term-Oriented Input Construction

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Summary:

Cyber Threat Intelligence (CTI) documents are lengthy, unstructured, and densely populated with technical jargon. Due to the high density of domain-specific terms, weak structural organization, and dispersed key information, it is difficult for general summarization methods to generate good summaries for CTI.

 

The authors propose using unsupervised term extraction to identify salient terms and also utilize supervised term generation to bridge the gap between raw CTI content and target domain terms. The extracted terms and generated terminology are prepended to the original CTI.

 

Based on experiments, overall, the proposed approach combining term extraction and terminology performs better than both the other approaches of just using raw CTI (DirectLM) and just using the extracted terms (AutoTerm). They also determined that adding more high-quality domain terms gives diminishing results after reaching a certain threshold. The terms added to the beginning give better performance than adding them at the end.

 

Comments:

 

Overall the paper is written well and is easy to follow. The authors also provide sufficient experimentation to support their proposed approach.

 

The proposed approach seems general enough to be applicable to other domains; hence, it is domain agnostic.

 

With experiments, the authors successfully show that added context about term definitions benefits the summarization of CTI reports.

 

Improvements:

 

However, the paper claims that added terminology reduces hallucinations and provides more structured cues. These claims seem unjustified, so they should be removed, or experiments should be added to provide evidence for these claims.

 

Also, in testing for the positioning of terminology, only the start and end of the CTI are tested. The authors should also add tests for testing terminology added just before the introduction of a new term in the CTI.

 

Finally, the authors also do not provide a clear definition and difference between CTS and APS tasks, which are crucial for the utilized dataset.

Author Response

See attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper presents a knowledge-guided framework for cyber threat intelligence (CTI) summarization, utilizing a term-oriented input construction approach that integrates structured domain-specific terminology to enhance summarization quality. Despite its innovative premise, there are several concerns that warrant rejection:

1. Insufficient Novelty: The core strategy of incorporating domain-specific terms into input prompts is a relatively incremental extension of existing terminology-guided approaches. The paper does not sufficiently differentiate its methodology from prior work in related fields, such as biomedical or legal document summarization, where similar entity- or concept-guided techniques have been explored.

2. Limited Technical Depth: The hybrid term construction pipeline and input injection paradigm, while described, lack rigorous technical details and innovation. The paper does not provide a comprehensive analysis of how the hybrid extraction mechanisms outperform standard methods or ablation studies confirming the individual contributions of each component.

3. Evaluation Limitations: The experimental evaluation relies primarily on performance metrics across multiple models and settings but lacks convincing assessments against strong baselines, such as existing domain-specific summarization systems or state-of-the-art benchmarks. The datasets and metrics used are not described in sufficient detail to ascertain the significance of the reported improvements.

4. Generalization and Practical Impact: While the authors claim strong zero-shot and supervised generalization, there is limited discussion concerning real-world deployment challenges, such as the quality and consistency of domain-specific term extraction in dynamic threat environments. The approach's scalability and adaptability to evolving threats remain underexplored.

5. Clarity and Reproducibility: The manuscript suffers from unclear descriptions of some methodological steps, particularly regarding the post-processing rules and the fusion of unsupervised and supervised term extraction. This hampers reproducibility and the reproducibility of claimed advancements.

Comments on the Quality of English Language

The quality of the English language should be improved.

Author Response

See attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

This paper proposes a novel framework that applies term-oriented input construction to improve the accuracy and efficiency of Cyber Threat Intelligence (CTI) document summarization. The authors introduce a hybrid domain term extraction pipeline that combines automatic keyword extraction with supervised term generation, and a knowledge-injected input strategy that places the generated terms at the beginning of the original document. This approach aims to enhance the model's understanding of threat semantics and improve summary consistency and informativeness. Experimental results show consistent performance improvements across various models (e.g., BART, T5, GPT-3.5) and settings (zero-shot, fine-tuning), demonstrating the generalizability and scalability of the proposed method.

(1) [Line 66–79]: The list of contributions is clear and well-structured, but terms such as "summary fidelity" and "hallucination" require clearer definitions.

(2) [Figure 1]: The overall architecture is well summarized, but providing an example or visualization of the “Term refinement” stage would help improve understanding.

(3) [Line 159–164]: The implementation details or examples of “Character Cleaning” and “Redundancy Removal” are not provided, which somewhat limits reproducibility.

(4) [Table 1, 2]: While the performance comparisons are systematically presented, the absence of statistical significance indicators (e.g., p-values, standard deviations) may weaken the robustness of the conclusions.

(5) [Line 295–316]: The ablation study effectively illustrates the core impact of the proposed approach, but including a comparison with a “term-only input” (summary generated from terms alone) would enable stronger conclusions.

(6) [Line 369–396]: The analysis of few-shot learning effects is valuable, but including more granular sampling (e.g., 1%, 5%) rather than 10% intervals would better support the authors’ claim regarding “low-resource applicability.”

Author Response

See attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have done an excellent job of answering my concerns. The added experiments for hallucinations and terminology placement provide proper justification for the claims made in the paper.

Author Response

We sincerely thank the reviewer for the positive feedback and for recognizing our efforts in addressing the concerns. We are glad that the additional experiments on hallucination reduction and terminology placement have provided sufficient justification for our claims. We deeply appreciate the constructive comments, which have significantly improved the clarity and rigor of our work.

Reviewer 2 Report

Comments and Suggestions for Authors

This paper presents a novel knowledge-guided approach for CTI summarization that explicitly incorporates domain-specific terminology to improve factual accuracy and relevance. While the proposed method demonstrates promising results across multiple models and scenarios, it lacks a comprehensive discussion on certain critical aspects of cyber threats. Notably, the paper does not address complex threat vectors such as side-channel attacks or discuss strategies for defending against them. For example, below are several examples you can refer to: (1) FOAP: Fine-Grained Open-World Android App Fingerprinting (USENIX Security’22), (2) Eavesdropping Mobile App Activity via Radio-Frequency Energy Harvesting (USENIX Security’23), (3) Packet-Level Open-World App Fingerprinting on Wireless Traffic (NDSS’22), (4) Uncovering User Interactions on Smartphones via Contactless Wireless Charging Side Channels (S&P’23), (5) Manipulating Voice Assistants Eavesdropping via Inherent Vulnerability Unveiling in Mobile Systems (TMC’24), (6) MagSign: Harnessing Dynamic Magnetism for User Authentication on IoT Devices (TMC’23)

Additionally, the discussion on threat landscape evolution and how the summarization approach adapts to emerging tactics is limited. To strengthen the contribution, the authors should expand the discussion on diverse cyber threats, including side-channel attacks and defense mechanisms, as well as provide insights into how their method can handle such sophisticated threats in real-world scenarios. Major revisions are needed to address these gaps and enhance the paper's relevance to the cybersecurity community.

Author Response

Please see the attached reply

Author Response File: Author Response.pdf

Round 3

Reviewer 2 Report

Comments and Suggestions for Authors

I appreciate the authors for their efforts in the submission and revisions. All my concerns are perfectly addressed.