LLM-Powered Proactive Cyber-Defense Framework Using Cyber-Threat Indicators Collected from X Platform

Abstract

Security organizations increasingly rely on cyber threat intelligence (CTI) sharing to enhance their resilience against cyberattacks. Indicators of Compromise (IoCs) play a critical operational role in CTI by providing malicious artifacts that support threat detection, incident response, and facilitate proactive defense. However, the rapid growth of social media as CTI sources, characterized by short-text content, poses significant challenges to automated IoC extraction, contextual interpretation, operational integration, and reliable verification. To address these challenges, this study proposes a comprehensive framework that integrates Large Language Models (LLMs) across multiple stages of the CTI pipeline. The framework leverages LLM-driven data augmentation, a hybrid classification model, and contextual summarization to enhance short-text understanding while supporting expert-in-the-loop validation for operational reliability. Extensive experimental evaluations demonstrate that LLM-driven data augmentation substantially improves model robustness and generalization while reducing false-positive alerts, achieving a precision of $98.87$%. Quantitative diversity analysis and expert-based human evaluation further confirm the linguistic quality and correctness of the generated augmented samples. In addition, IoC reports are validated using both reference-based and reference-free evaluation metrics that show strong alignment and high semantic adequacy. Moreover, a technology acceptance model was integrated with cybersecurity domain constructs to assess the acceptance factors of the proposed framework. Regression analysis showed that perceived usefulness, behavioral intention, trust in automation, and risk were the strongest predictors of actual use. These predictors are commonly interpreted as indicators of technology acceptance.

1. Introduction

Cyberattacks pose a significant risk to critical infrastructure, leading to disruptions in digital services that affect multiple dimensions of human activity [1]. Cyber threat intelligence (CTI) has emerged as a key component of proactive defense strategies, leveraging insights collected from diverse Open-Source Intelligence (OSINT) sources. Among the operational elements of CTI, Indicators of Compromise (IoCs) represent one of the most widely used, providing actionable forensic artifacts associated with cyberattacks and traces of malicious activities. These artifacts typically include malicious IP addresses, hashes of malicious files, phishing URLs, and Botnet domains [1]. The effective use of such indicators enables organizations to proactively mitigate threats by identifying early signs of compromise, updating firewall rules, and refining intrusion detection system signatures [2].

OSINT typically comprises unstructured data, which can be processed and analyzed using advanced natural language processing and machine learning (ML) techniques. Prior research efforts have extensively applied these approaches to filter and analyze data streams from diverse OSINT sources for IoC detection and extraction. In most studies, the extracted IoCs are presented to security teams as raw unstructured lists such as blacklists, leaving their effective interpretation, sifting, and operationalization largely dependent on human expertise and manual analysis [2,3,4,5]. Moreover, the atomic nature of these extracted indicators, combined with the lack of contextual, behavioral, and strategic information, significantly limits their actionable value in real-world security operations. A limited number of studies have attempted to bridge this gap by structuring identified indicators into standardized threat intelligence representations [6,7]. However, these approaches primarily focus on the syntactic organization and indexing of indicators, rather than enriching them with contextual reasoning or analyst-oriented explanations that could directly support security decision-making and proactive defense strategies. Recent advances in large language models (LLMs) have demonstrated significant potential to automate and enhance cybersecurity operations by enabling deeper analytical understanding and reasoning over unstructured threat data [2]. However, to date, no prior research has comprehensively investigated the use of LLM contextual reasoning capabilities to support human interpretation of collected CTI or to transform raw threat artifacts into directly actionable defensive insights. Although LLMs can substantially enhance threat-hunting and analytical workflows, their deployment in cybersecurity settings demands high efficiency and accuracy, as even minor errors may result in false alerts or missed threats, with serious implications for security decision-making and incident response effectiveness [8,9]. Therefore, achieving an appropriate balance between detection accuracy and computational efficiency remains a critical design consideration in developing CTI extraction models from social media intelligence streams.

This study addresses the identified research gap by proposing a comprehensive LLM-driven framework that integrates Bidirectional Encoder Representations from Transformers (BERT) and Chat Generative Pre-trained Transformers (ChatGPT) to enhance cyber-threat detection, extraction, and reporting. The proposed approach is evaluated using real-world CTI data collected from the X platform, examining the capabilities of LLMs across several key dimensions: (i) generating synthetic data to mitigate data scarcity and class imbalance; (ii) accurately and efficiently detecting IoC-related posts through a hybrid BERT–BiLSTM classification model; and (iii) automatically producing analyst-ready IoC alert reports, including extracted indicators, concise tweet summaries, recommended mitigation, and Structured Threat Information eXpression (STIX)-compliant [10] threat intelligence suitable for integration into existing CTI platforms.

The proposed LLM-driven solution is systematically evaluated in terms of model performance, output quality, and user acceptance. An ablation study is conducted to quantify the individual contributions of each architectural component of the hybrid model to overall classification effectiveness. LLM-generated reports are evaluated using a combination of reference-based and reference-free intrinsic evaluation metrics. In addition, user acceptance of the proposed framework is assessed through the Technology Acceptance Model (TAM), extended with security-specific constructs such as perceived trust in automation and perceived risk, providing a theoretically grounded perspective on factors influencing the adoption of LLM-driven CTI systems. Finally, the extracted IoCs are cross-referenced with established threat intelligence repositories to characterize their nature and assess the attributes of CTI content disseminated on the X platform.

The remainder of this paper is organized as follows: Section 2 presents the foundational concepts and contextual background necessary to understand the proposed solution. Section 3 reviews the relevant related work. The methodology is described in Section 4. The experimental setup and results are presented in Section 5. Section 6 examines the practical implications and limitations of the proposed approach. Finally, this paper concludes with key findings and outlines potential directions for future research.

2. Background and Technical Foundations

This section outlines the key characteristics of various OSINT sources and provides an overview of LLM adaptation techniques, along with the metrics and methods used to evaluate LLM generated outputs.

2.1. Overview of OSINT Sources

Different OSINT sources contribute to CTI through diverse data collection mechanisms, analytical approaches, and dissemination practices, resulting in variations in intelligence quality and timeliness [11]. Prior research broadly examined four primary categories of OSINT sources: (i) threat-sharing platforms, (ii) threat reports, (iii) social media platforms, and (iv) dark web sources. Threat-sharing and analysis platforms aggregate CTI data from multiple scanning engines, web crawlers, and automated detection tools. Prominent examples include IPVoid [12], Hybrid Analysis [13], and VirusTotal [14], which typically disseminate intelligence in structured formats that facilitate integration with security control systems and automated threat analysis workflows. However, these platforms often impose access restrictions, encouraging users to subscribe for comprehensive data access. Empirical studies indicate that, despite their broad coverage, even the most accurate threat-sharing platforms fail to detect a substantial portion of threats in a timely manner; approximately 30% of phishing URLs remain undetected four weeks after their initial deployment [15]. Consequently, such platforms cannot guarantee real-time IoC detection, as newly emerging indicators are frequently reported with considerable delays. To mitigate these limitations, multiple threat-sharing platforms can be integrated to enhance coverage, which ultimately increases operational complexity and overhead.

Threat reports are comprehensive analytical documents produced by research institutions, government agencies [16], and leading security vendors such as Verizon and McAfee [17]. These reports provide in-depth analyses of emerging threats, prevalent attack patterns, and recommended defensive strategies. However, access to complete reports is restricted, with only partial content being made publicly available and full access typically requires paid subscriptions. Although these reports are published in multiple formats, the extensive level of analysis involved often results in delayed dissemination, which limits threat reports’ effectiveness for real-time IoC detection and rapid response.

Social media platforms such as security blogs, X, and open-source repositories like GitHub, have become increasingly valuable sources of OSINT, providing near real-time IoCs shared by experts and researchers. Unlike traditional threat-sharing platforms, these channels are freely accessible. Prior studies examined the role of X as an OSINT source [1,2,5], showing that it often captures emerging and credible attack artifacts significantly earlier than conventional sources. Notably, only 10% of malicious URLs disclosed on X were presented in publicly available blacklist databases at the time of reporting. This increased to 26% after one week [5]. These findings highlight the potential of the X platform to support proactive cyber-defense systems [5]. In addition, existing studies indicate that most IoCs shared on X are intentionally defanged to prevent accidental activation of malicious content [2]. For instance, URLs are commonly obfuscated by replacing “http” with “hxxp” or by enclosing a period in parentheses. Such obfuscation practices must be accounted for when designing automated IoC extraction systems that leverage social media as OSINT sources.

The dark web constitutes a restricted subnet of the deep web that is not indexed by conventional search engines and is therefore inaccessible through standard web queries. Among the dark web platforms, hacking forums are particularly prominent sources of CTI, as they often disseminate IoCs associated with recent cyberattacks [18]. However, collecting data from these forums presents significant technical challenges. Access typically requires user registration and, in some cases, additional vetting or trust-building procedures. Furthermore, automated data collection tools must closely emulate human browsing behavior, including request patterns and connection frequency, to avoid triggering security mechanisms that may lead to account suspension or blocking [19]. Beyond these technical barriers, the use of dark web data also raises important ethical and legal concerns, including risks related to privacy violations, potential legal liabilities, and the possibility of unintentionally disrupting forum operations [20]. Consequently, despite the potential value of dark web sources for CTI collection, these practical, ethical, and legal challenges limit their accessibility and large-scale utilization in automated intelligence extraction systems.

In light of the foregoing, the characteristics of the X platform make it a suitable and effective data source for this study. X enables near real-time access to CTI and, when used appropriately, does not introduce significant ethical or privacy concerns, thereby ensuring compliance with established research ethics and aligning with practices adopted in prior OSINT-based cybersecurity research [2,3,5]. However, several challenges must be addressed when designing automated CTI systems based on this platform, including managing the trade-off between detection accuracy and computational efficiency, reliably distinguishing tweets that report IoCs from those that provide benign or contextual information, and adapting to platform-specific linguistic patterns and conventions.

2.2. Pre-Trained LLM and Adaptation Strategies

LLMs are trained on a massive natural language corpora using deep neural architectures and are capable of producing content that exhibits human-like fluency, coherence, and reasoning abilities. However, the default outputs of such models are often excessively generic and may not adequately meet specialized application domain requirements. LLM adaptation strategies address this limitation by tailoring model behavior to task-specific data and objectives, enabling outputs that better conform to the linguistic, structural, and domain-specific conventions of the target application. Several LLM adaptation strategies have been widely adopted in the literature, including:

• Fine-tuning: This strategy involves further training a pre-trained LLM on task-specific data, allowing the model to adjust its internal parameters while retaining general language knowledge  [2]. Fine-tuning can range from full-parameter updates for maximal domain alignment to parameter-efficient approaches which update only a small subset of parameters, reducing computational overhead. Another common approach involves attaching task-specific components, such as Bidirectional Long Short-Term Memory (BiLSTM) and Fully Connected (FC) layers, in addition to the LLM representations. In such configurations, the underlying LLM may be fully fine-tuned, partially frozen, or used as a fixed feature encoder, offering flexible trade-offs between performance, generalization, and computational efficiency.
• Prompt engineering: This technique entails crafting input prompts to guide the model toward producing accurate, relevant, and contextually appropriate outputs without modifying model parameters [9,21]. Prompt engineering enables rapid adaptation across tasks with minimal computational cost, making it well suited to resource-constrained environments. Common techniques include zero-shot, one-shot, and few-shot prompting, as well as chain-of-thought, role-based prompts, and attribute-based prompts that impose user-defined constraints (e.g., tone, length, or output format) to improve controllability and task specificity.
• In-context learning: This technique exploits the capacity of models to infer task structure and desired behavior directly from a small number of examples embedded within the input context. This technique enables the models to generalize from an exemplar without task descriptions or parameter updates, eliminating additional training costs.
• Reinforcement learning with human feedback: This method improves alignment between model outputs and individual expectations by incorporating reinforcement signals derived from human evaluators preferences [22]. This method penalizes undesirable behaviors and rewards responses that align with domain objectives, improving output quality and reliability.
• Retrieval-augmented generation (RAG): This strategy augments LLM with external knowledge sources, such as vector databases or curated document repositories, enabling the retrieval of relevant information at inference time [9]. By grounding generation on trusted external sources, RAG improves factual accuracy, reduces hallucinations, and enhances domain relevance without additional model training.

These strategies enable the effective adaptation of LLMs across a wide range of downstream applications, offering scalability and flexibility in complex real-world scenarios. Nevertheless, their deployment introduces several challenges, including increased computational complexity and the risk of hallucination, whereby models generate factually incorrect or fabricated content with high linguistic confidence, undermining output reliability [23]. Addressing these challenges is essential to ensure that LLM-driven solutions remain trustworthy, robust, and aligned with the selected domain objectives.

2.3. Metrics and Methods for Evaluating LLM Outputs

Assessing the quality and utility of LLM-generated outputs is essential to ensuring their relevance to downstream applications. Evaluation metrics are commonly categorized into human assessment and automatic matrices that can be further divided into intrinsic and extrinsic metrics, which provide complementary perspectives on model performance. Human assessment requires experts to assess subjective evaluation criteria. Several measures can be employed to quantify this agreement including Cohen’s Kappa coefficient, which accounts for chance agreement and provides a robust estimate of consistency between evaluators. Higher Kappa values indicate stronger agreement, thereby reinforcing the reliability and methodological soundness of human assessment results.

Intrinsic metrics focus on assessing the linguistic, semantic, and structural quality of the generated content and can be further divided into reference-based and reference-free metrics. Reference-based metrics compare LLM-generated outputs against high-quality reference texts to quantify lexical overlap, semantic similarity, and entailment. Lexical overlap is typically measured using n-gram-related metrics such as Bilingual Evaluation Understudy (BLEU) [24] and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [25], while semantic similarity is assessed using embedding-based methods, including Sentence-BERT (SBERT), BERTScore, and MoverScore [23]. Entailment-based metrics evaluate whether LLM-generated texts logically entail, contradict, or diverge from the reference text, with representative methods such as Natural Language Inference (NLI) scores, Summarization Consistency (SummaC) [26], and the factual Consistency Checking model (FactCC) [27].

Reference-free metrics assess generated content without relying on ground-truth references, producing scores based on fluency, factual accuracy, and diversity. Fluency is often measured using perplexity [28], whereas readability metrics [29] capture the clarity and interpretability of generated texts. Factuality metrics evaluate the extent to which generated outputs preserve semantic consistency and alignment with trusted information sources, thereby mitigating risks of hallucinations and misinformation. Reason-then-Score (RTS) leverages LLMs knowledge to verify LLM-generated texts [23]. Diversity metrics, including Self-BLEU and distinct-n, quantify lexical and semantic variation to mitigate redundancy across generating samples.

Extrinsic evaluation examines LLM effectiveness in real-world settings by measuring task-specific performance and operational utility. In the context of synthetic data generation, extrinsic evaluation metrics measure the extent to which generated samples improve downstream task performance, including enhancing classification accuracy, reducing false positives or false negatives in accordance with design objectives, enriching CTI, and strengthening the robustness of detection models. Moreover, extrinsic evaluation captures human-centered factors, including user acceptance, usefulness, usability, and satisfaction, providing a practical assessment of the impact of LLM-driven solutions.

3. Related Work

This section reviews existing ML and LLM-based approaches for IoC extraction and utilization. It highlights the employed models, utilized data sources, reported performance metrics, and the limitations of prior studies that directly motivated this research.

3.1. ML-Enabled CTI Solutions

ML techniques have been widely adopted in cybersecurity to support a broad range of tasks, including detection, analysis, and mitigation of cyberattacks. Technical blogs and security articles have been widely used as intelligence sources for capturing detailed contextual information on emerging IoCs [4,6,30,31]. In [30], a neural sequence labeling approach that combines a BiLSTM network with a Conditional Random Fields (CRF) layer was proposed for IoC extraction from English cybersecurity articles. By integrating an attention mechanism and token-level features, the model achieved an F1-score of $88.8$% on long and structurally complex texts. Building on this work, subsequent studies introduced multi-head self-attention mechanisms to better capture broad contextual dependencies in CTI text [31]. This approach slightly outperforms earlier sequence labelling models, achieving average F1-scores of $89.0$% on English articles and $81.8$% on Chinese articles. The iACE system presented in [6] employed graph mining techniques to automatically extract IoCs and their associated contextual information from unstructured technical blog posts. iACE was based on the observation that IoCs frequently co-occur with domain-specific contextual terms and follow characteristic grammatical patterns. Using these features alongside Regular Expressions (Regex), the system trained a Linear Regression (LR) classifier to identify sentences containing IoC tokens, achieving a precision of $92.0$% and an F1-score of $89.0$%. Despite its effectiveness, iACE exhibited limitations in scenarios where IoC-related contextual terms were absent or distant from the IoC tokens. Furthermore, the system was not evaluated on social media data, which typically consists of shorter, noisy text, often lacking consistent contextual patterns and frequently containing defanged IoCs.

The ChainSmith framework [4] further advanced IoC extraction by integrating Word2Vec embeddings with Named Entity Recognition (NER) and Latent Dirichlet Allocation (LDA). It models malware delivery campaigns across four stages: “baiting”, “exploitation”, “installation”, and “command & control” enabling context-aware IoC extraction and supporting actionable CTI. The framework employed four binary neural classifiers to detect campaign stages and potential IoCs. When evaluated in security articles, ChainSmith achieved a precision of $91.9$% for IoC detection and an average precision of $78.2$% for campaign-stage classification. In [8], expert-annotated cybersecurity documents were used to train a CRF model, where the low-level indicators were initially extracted with Regexs. The hybrid rule-based and statistical approach was used to generate structured threat reports in STIX format, facilitating automated intelligence sharing and interoperability across CTI platforms. However, the study was evaluated on a relatively small dataset comprising only 50 of cybersecurity reports and security blogs, which limited the diversity and generalizability of the learned patterns. Consequently, the proposed approach exhibited modest detection performance, achieving precision of $70.0$%, thereby underscoring the impact of data scarcity on supervised IoC extraction. It also highlighted the need for more scalable and data-efficient approaches.

As threat intelligence dissemination increasingly shifts toward open channels, social media platforms have emerged as both a valuable and challenging source for real-time IoCs identification and discrimination. Several studies have investigated the utility of platforms such as X [5,32], dark web forums [33], and technical blogs [6]. In [32], a Convolutional Neural Network (CNN) model was developed to classify tweets and identify actionable IoCs, achieving an F1-score of $98.80$%. The study also compared the speed of IoC dissemination on X with that of traditional threat-sharing platforms, highlighting the growing role of X as a rapid and reliable CTI source. TIMiner [7] employed a CNN-based classifier to first categorize cyber threats into domains such as “General”, “Power Grid”, and “Healthcare”, followed by IoC extraction from social media posts. By leveraging Word2Vec embeddings and syntactic dependency features, TIMiner captured both semantic and structural relationships. It achieved over 84% accuracy for domain classification and 94% accuracy for IoC extraction. The IoCMiner system [5] integrated graph-theoretic analysis, Random Forest (RF) classification, and text mining techniques to filter non-CTI content from X data streams. Tweets were collected from credible accounts identified using a reputation model based on follower and friend metrics. From the retained IoC-related tweets, IoCMiner applied Regex-based extraction and achieved 97% accuracy. The MTL-CTI framework [3] combined Recurrent Neural Network (RNN) and CRF layers to generate NER outputs that facilitate IoC extraction from social media, achieving an F1-score of $97.3$%.

In [34], a framework was proposed to enable ML models’ sharing within security communities, rather than limiting intelligence exchange to detection outputs alone. Unlike traditional IoCs, which represent static and labeled artifacts such as IP addresses or file hashes, shared ML models offer adaptive detection capabilities that can generalize to previously unseen threats, thereby improving the robustness of adversarial evasion. However, distributed ML models can also expand the attack surface, particularly without adequate security controls. To mitigate these risks, recent research explored cryptographic protection mechanisms for secure model sharing. Despite their effectiveness, such techniques often incur significant computational overhead, increase key management complexity, and introduce trade-offs between model utility and privacy preservation [34].

In our previous study [2], we investigated the effectiveness of traditional ML models for detecting cybersecurity-related content in social media streams using handcrafted lexical, syntactic, and sentiment-based features. The study primarily focused on benchmarking conventional ML algorithms against emerging LLM-based approaches for CTI detection. The best-performing traditional algorithm was Support Vector Machine (SVM), utilizing unigram, bigram, IoC-centric, and sentiment-based features, which achieved the highest accuracy of $94.27$%. However, the proposed approach focused primarily on classification performance and did not address the transformation of extracted IoCs into contextualized and actionable threat intelligence that can directly support security analysts.

Given the foregoing, prior studies have largely relied on lexical features and shallow semantic representations, which constrain their capacity for deep contextual reasoning. Furthermore, extracted IoCs are typically delivered to analysts as unstructured artifacts, with limited enrichment or transformation into high-level intelligence products that support decision-making or automated defense workflows. Therefore, collected IoCs require substantial manual investigation. When coupled with the average daily volume of 4484 alerts and nearly three hours devoted to manual triage by Security Operations Center (SOC) analysts, this burden intensifies alert fatigue, delays incident response, and increases the likelihood of overlooking critical threats [35]. Incorporating rich contextual information and structured intelligence representations can significantly reduce manual effort, improve analytical efficiency, and mitigate error-prone decision-making.

3.2. LLM-Enabled CTI Solutions

The rapid advancement of LLMs has substantially reshaped automated CTI analysis by improving detection accuracy while reducing the extensive effort required for feature engineering [2]. In [9], ChatGPT-4 LLM is prompted to extract IoCs from 71 unique tweets, after which prompt engineering and RAG are used to automatically generate syntactically valid Elasticsearch and Kibana queries for threat hunting within an Elasticsearch–Kibana stack. While the approach demonstrates the operational potential of LLM, the evaluation reported a 45% error rate due to missed critical IoCs, and the RAG configuration exhibited a hallucination rate of $35.52$%, raising concerns regarding the reliability of this system in critical security settings.

The LANCE framework in [16] introduces hybrid expert-in-the-loop pipeline for IoC extraction that combines rule-based Regex with LLM-driven contextual classification. The system first applies handcrafted Regex patterns to extract candidate artifacts from 50 CTI threat reports, after which ChatGPT-4o is employed in a zero-shot prompting setting to classify each candidate as an IoC or non-IoC based on its surrounding textual context. LANCE further provides an analyst-centric visualization interface that uses color-coded labels and explanation cues to enhance the interpretability and analyst trust of extracted IoCs. Overall, the framework extracted 1791 labeled IoCs, achieving a precision of $96.7$% and an F1-score of $97.6$%. However, LANCE was evaluated exclusively on structured CTI threat reports curated by security experts, which typically exhibit formal language and well-defined contextual boundaries. Therefore, its effectiveness on social media OSINT streams characterized by high noise, informal language, abbreviations, implicit references, and a large volume of benign or ambiguous technical artifacts remains invalidated.

Recent studies have also explored fine-tuning BERT-based models for automated detection of IoC-related content [2,36]. In [36], the TSTEM system applies a two-stage pipeline in which a fine-tuned BERT model first identifies IoC-relevant content, followed by NER to extract specific indicators. Although the system achieved strong overall performance, the reported F1-score of 84% suggests limitations in handling class imbalances and a bias toward the majority classes. Improved performance was observed in the IoC-Collector framework [2], which introduced practical capabilities such as real-time detection, analyst-driven customization, and transparent IoC prioritization. By fine-tuning BERT with an extended tokenizer and fully trainable layers, IoC-Collector achieved $99.36$% accuracy. Nevertheless, this performance comes at the cost of substantial computational overhead, which may hinder scalability in high-volume OSINT environments and increase deployment costs in resource-constrained operational settings. Moreover, the system presents the collected IoCs as atomic artifacts prioritized primarily by confidence scores, without detailing the contextual information provided by the original OSINT sources reporting these indicators or explaining how they relate to the surrounding threat narrative. False-positive alerts generated by such systems remain a critical operational concern. In addition, the framework does not provide mitigation or response recommendations, limiting its usefulness for downstream analyst decision-making and operational response.

Given the foregoing, although existing LLM-based and transformer-driven approaches demonstrate strong predictive performance, they largely neglect the critical trade-off between detection accuracy and computational efficiency required for real-world deployment. Moreover, most prior studies treated LLMs primarily as black-box predictors, underutilizing their inherent reasoning and contextualization capabilities to support analyst decision-making and actionable intelligence generation. As a result, many of these approaches remain insufficiently suited to time-sensitive and high-volume cybersecurity operations.

4. Methodology

This section presents the methodological pipeline for the proposed framework (Figure 1). The pipeline consists of several sequential stages, including data acquisition and preprocessing, model training and IoCs reporting, and evaluation criteria. Each stage comprises multiple sub-steps designed to systematically process the data and support the overall analytical workflow. These stages include data collection, data labeling, data augmentation, preprocessing, model training, IoC extraction, and standardized reporting of the extracted artifacts. In addition, this section describes the design of the custom user interface developed to facilitate analyst interaction with the system. Finally, it outlines the evaluation criteria used to assess the effectiveness and operational suitability of the proposed framework.

4.1. Data Collection

The data collection phase constitutes the initial step of the proposed pipeline and is designed to capture a diverse and representative set of cybersecurity-related tweets. Tweets were retrieved using a comprehensive keyword-based search strategy, developed through extensive domain analysis and iterative experimentation. Two complementary keyword categories were employed. The first category comprised domain-specific terms associated with well-known malware families and IoC-related entities (e.g., LockBit, Botnet, Emotet), while the second included general cybersecurity terms intended to capture broader discussions of cyber threats and incidents.

In addition to keyword-based retrieval, tweets authored by security vendors and expert accounts were collected to ensure high-credible intelligence and practitioner-driven perspectives. A curated list of keywords and expert accounts was tracked using the X Search API [37].

Table 1. Description of dataset features.

Feature	Description	Type
Tweet ID	Tweets unique identifier	Numerical
Tweet text	Textual content of the tweet	Text
Keywords	Keywords used for tweet retrieval	Categorical
User Account	Author account name	Text

summarizes the dataset features, along with their corresponding descriptions and data types.

Column 2 of

Table 2. Data statistics of tweets for each class label across different keyword groups.

Keywords Type	No. of Tweets	Class Label
		Tweets Report IoCs	Tweets Not Report IoCs
General	5791	2302	3489
Domain	3813	2980	833
Expert account	4790	457	4333
Total	$\mathrm{14,394}$	5739	8655

reports the descriptive statistics of the collected dataset. In total, $\mathrm{14,394}$ tweets were collected. Of these, 5791 tweets were retrieved using general keywords, 3813 tweets were obtained using domain-specific terms, and 4790 tweets originated from security vendors and expert accounts.

4.2. Data Labeling

The data-labeling phase aimed to assign ground-truth labels to the collected tweets and was conducted by three cybersecurity domain experts. The process commenced with the development of comprehensive annotation guidelines instructing annotators to classify each tweet as either “reporting IoC” or “not reporting IoC”.

To ensure labeling consistency, the three experts independently annotated a shared subset comprising 20% of the dataset. Inter-annotator agreement was then measured using Fleiss’ $\kappa$, yielding $\kappa =0.89$, which indicates near-perfect agreement according to established interpretation guidelines. This result reflects the clarity of the annotation criteria and the relatively objective nature of identifying explicit IoC artifacts in tweets. Following the independent annotation stage, disagreements were reviewed and resolved through an adjudication process involving discussion among experts until a final consensus label was reached. The finalized consensus labels were subsequently used to construct the ground-truth dataset employed in the experimental evaluation.

Using this procedure, 5739 tweets were labeled as “reporting IoC”, while 8655 tweets were labeled as “not reporting IoC”. Columns 3 and 4 of Table 2 summarize the distribution of tweets across the two class labels for each keyword group. Tweets retrieved using domain-specific keywords predominantly belong to the “reporting IoC” class (2980 tweets), whereas the majority of tweets collected by general cybersecurity keywords were classified as “not reporting IoC” (3489 tweets). In contrast, 457 tweets obtained from expert accounts were identified as “reporting IoC”. These observations indicate that domain-specific keywords are substantially more effective at capturing tweets that disclose IoC artifacts, consistent with prior work [2].

4.3. Data Augmentation and Enrichment

Given the inherent restrictions of the X API in terms of rate constraints and restricted access to historical data, the collected tweet corpus is often sparse and exhibits a significant class imbalance. As a result, certain IoC categories, including IPv4/IPv6 addresses, file hashes (e.g., SHA-1, SHA-256, and SHA-512), and diverse URL obfuscation or formatting patterns, may be underrepresented or entirely absent from the dataset. Such skewed distributions can bias model learning and adversely affect generalization to real-world threat intelligence streams.

To mitigate these issues, a data augmentation technique was employed to expand both the diversity of indicator representations and the range of syntactic and lexical variations. This enhanced the model robustness to heterogeneous and unstructured threat data. In addition to augmenting IoC-related tweets, synthetic non-IoC samples were generated to emulate short-form cybersecurity discussions commonly observed on social media. These discourse-oriented samples capture informal language, contextual narratives, and topic variability across multiple cybersecurity domains, as summarized in

Table 3. Cybersecurity topics used in synthetic non-IoC tweets generation.

Topics

Phishing awareness training, ransomware trends, privileged access security, social engineering prevention, employee cybersecurity awareness, LLM jailbreaks, patch management, cloud misconfigurations, insider threats, insider data leaks, AI in cybersecurity, IoT device security, malware detection techniques, DDoS attack mitigation, email spoofing, and threat intelligence sharing.

. Incorporating such contextual non-IoC content enhances the model’s ability to discriminate actionable threat disclosures from benign technical discussions, ultimately reducing false positives and increasing classification reliability in noisy OSINT environments.

Few-shot and structured prompt-engineering strategies were employed to synthesize tweets for both target classes, emulating authentic cybersecurity discourse on the X platform. As illustrated in Figure 2, the prompt used to generate synthetic tweets that report IoCs was deliberately designed to balance linguistic realism with controlled indicator injection. This design ensures that the generated content reflects natural social media communication patterns while preserving precise control over the inclusion and formatting of IoC artifacts. The prompt architecture comprises six core components: (i) a role definition that aligns the model identity with cybersecurity expertise, thereby constraining generation to domain-appropriate terminology, (ii) contextual information describing the social media environment to capture platform-specific linguistic norms, (iii) a task specification that clearly defines the generation objective and required inputs, (iv) explicit constraints governing text generation, including the use of defanged IoCs, (v) a predefined output format to ensure structural consistency and facilitate automated parsing, and (vi) few-shot exemplars that guide stylistic coherence and content consistency.

To ensure comprehensive coverage of threat indicators, multiple seed prompts explicitly referencing diverse IoC categories were incorporated through the IoCs_Example set. This controlled seeding strategy promotes representational diversity across IoC types, formats, and contextual usages. Therefore, the synthesized corpus may capture a broader and more representative spectrum of real-world IoC heterogeneity, improving class balance, reducing overfitting to dominant indicator patterns, and enhancing the generalization capability of downstream detection models when deployed on noisy and large-scale OSINT streams.

For the generation of synthetic cybersecurity-related tweets that do not contain explicit IoCs, the LLM was guided using carefully engineered prompts to ensure that the generated content remained grounded in relevant cybersecurity contexts while explicitly excluding sensitive or actionable indicators. As illustrated in Figure 3, the prompt’s design emphasizes contextual relevance and enforces strict generation constraints, including the production of a single tweet per response, a maximum length of 280 characters, and explicit prohibitions against defanged IoCs. To maintain domain relevance and narrative diversity, the prompts are based on a predefined pool of common cybersecurity topics, as summarized in Table 3. For each generation batch, a topic is randomly sampled from this pool and provided as input to the model. This ensures broad and balanced coverage of cybersecurity narratives across multiple domains.

The OpenAI Python SDK version 1.101.0 and Chat Completions API were used to format and submit prompts to the ChatGPT-4o model. This model was selected due to its demonstrated emerging reasoning and generation capabilities relative to other pre-trained LLMs [16,23]. The prompt templates were dynamically instantiated under controlled diversity settings, as summarized in

Table 4. Configuration of the LLM for synthetic tweets generation.

Parameter	Value
Model	ChatGPT-4o
Sample size	5000 tweets
Batch size	10 per API call
Temperature	$0.7$, $0.8$, $0.9$ (to encourage diversity)
Top-p	$1.0$
Max token	120

. Temperature values of $0.7$, $0.8$, and $0.9$ were randomly sampled for each generation batch to balance creativity with output consistency. The top-p parameter was fixed at $1.0$. In total, 5000 synthetic tweets were generated in batches of ten per API call.

The generated tweets were stored in a structured CSV format using pandas (v 2.2.3), together with associated metadata (https://github.com/NawalMutairiKSU/SynthticTweetsIoC (accessed on 19 February 2026)). For the 2500 IoC-related tweets, the recorded metadata included: (i) tweet text, (ii) IoC type (IP, URL, domain, or hash), (iii) defanged IoC value, and (iv) the associated threat category (e.g., malware or phishing). For the remaining cybersecurity-related and non-IoC tweets, metadata comprised: (i) tweet text, and (ii) the corresponding cybersecurity topic. Figure 4 presents random examples of generated tweets showing the tweet content that reflects these design principles.

4.4. Data Preprocessing

The collected dataset is preprocessed to ensure textual consistency, reduce noise, and enhance model robustness and classification accuracy. The preprocessing pipeline consists of the following steps:

• Duplicate removal: To mitigate redundancy and potential sampling bias, pairwise Jaccard similarity is computed between collected tweets. Tweets exhibiting a similarity score greater than $0.70$ are considered near-duplicates and removed. Tweets generated during the data augmentation phase are excluded from this process to preserve synthetic sample diversity. Following this step, $\mathrm{11,103}$ unique tweets are retained. When combined with the synthetic samples introduced during data augmentation, the final dataset comprises $\mathrm{16,103}$ tweets, including 7978 not reporting IoC samples and 8125 reporting IoC samples.
• Text cleaning: Raw tweets often contain noise such as special characters, emoticons, emojis. These elements are removed using Regex, ensuring tweet semantic integrity. Moreover, non-English tweets were filtered solely to ensure compatibility with the English-based LLM used in this study; this filtering was not based on the geographic origin of the tweets.
• Normalization: In the domain of cybersecurity, identical concepts are frequently expressed in heterogeneous lexical forms. For instance, the term “two-factor authentication” may appear as “2FA”, while “command and control” is often abbreviated to “C2” or “C&C.” To address this lexical variability, we adopted the domain-specific normalization lexicon proposed by [38]. This normalization strategy enforces a consistent representation of semantically equivalent terms, reducing vocabulary fragmentation and improving downstream NLP and ML reliability.
• Stop words and punctuation filtering: Stop words, which primarily serve grammatical functions and contribute limited semantic value in isolation, are removed, along with punctuation. The English stop words list from the NLTK (http://www.nltk.org/ (accessed on 10 January 2026)) “stopwords” corpus is used for this purpose.

4.5. Model Training

BERT was selected as the base encoder for the proposed framework due to its demonstrated effectiveness in prior cybersecurity applications, including phishing email detection [39], malicious URL classification [40], and threat report analysis [41]. Following dataset preparation, tweets were tokenized using the BERT tokenizer, which converts each input sequence into input IDs, token type IDs, and attention masks, as illustrated in Figure 5. These inputs are then processed by the BERT embedding layer, where token, segment, and positional embeddings are combined and passed through the Transformer encoder to generate contextualized representations with a dimensionality of 768. To preserve the pre-trained linguistic and contextual knowledge captured during large-scale pre-training, all BERT transformer encoder layers were frozen during training. The resulting representations were used as feature inputs for a hybrid downstream classification architecture.

The BERT encoder produces two complementary outputs, each serving distinct analytical purposes. The first is the embedding corresponding to the special [CLS] token, commonly referred to as the pooler output, which captures a global representation of the entire tweet and is widely used for sentence and document level classification tasks. The second is the sequence output, which retains contextualized embeddings for all tokens in the input sequence, enabling fine-grained modeling of token-level semantic relationships. In this study, both outputs are jointly leveraged to exploit their complementary representational properties. Each output is first passed through a FC layer with a ReLU activation function to project the representation from 768 into a unified 256-dimensional feature space. The sequence output is then processed by a BiLSTM layer to model long-range contextual dependencies, followed by a dropout layer with a rate of $0.2$, an attention mechanism, and a ReLU activation function. The resulting representation is concatenated with the transformed pooler output to form a comprehensive feature embedding, which captures both global and token-level contextual information. This concatenated representation is passed through additional FC layers, with neuron configurations specified in Figure 5, each employing ReLU activation. Dropout at a rate of $0.2$ is applied between layers to reduce overfitting. The final output layer uses a softmax activation function to produce probability scores for the two target classes.

The proposed hybrid BERT–BiLSTM model was implemented using the PyTorch version 2.9.0 framework, following the architecture and dimensional transformations illustrated in Figure 5. The pre-trained bert-base-uncased model was employed as the encoder backbone, consisting of 12 stacked transformer encoder layers, each with 12 self-attention heads and a hidden dimension of 768. Input tweets were tokenized using the BERT tokenizer and truncated or padded to a fixed maximum sequence length of 256 tokens. This ensured uniform input dimensions while preserving sufficient contextual information for IoC detection. The BiLSTM layer comprises 256 input units and 512 output units, with the forget gate parameter set to $0.1$. Model optimization was performed with the AdamW optimizer with an initial learning rate of $2\times {10}^{-5}$. To enhance training stability and convergence, the learning rate was dynamically adjusted using a ReduceLROnPlateau scheduler based on validation losses. Cross-entropy loss was adopted as the objective function. The model was trained with a batch size of 32 for a maximum of 10 epochs. Early stopping with a patience of three epochs was applied to halt training when validation performance ceased to improve, further preventing overfitting. To ensure robustness and generalizability, model performance was evaluated using a 10-fold stratified cross-validation. Synthetic samples were incorporated exclusively into the training split for each fold to avoid data leakage. Final performance metrics were obtained by averaging the results across all folds. Following training, the model was employed to identify and aggregate tweets predicted to contain IoCs into a dedicated analysis pool. This was used for downstream inspection and report generation. Classification performance was assessed using accuracy, precision, recall, specificity, and F1-score, providing a comprehensive evaluation of detection capability.

4.6. IoC Extraction

IoCs were extracted from the candidate tweets aggregated in the analysis pool using a rule-based pattern-matching approach implemented with Regex. The extraction process primarily targeted commonly reported IoC types on the X platform, including malicious URLs, IP addresses, and multiple categories of file hashes. As discussed in Section 2, IoCs are frequently defanged, a deliberate obfuscation technique intended to prevent the inadvertent activation or misuse of malicious artifacts. To accurately identify and reconstruct such obfuscated indicators, the Python IoCExtractor (v 1.16.1) library was employed, leveraging its robust capability to detect, interpret, and normalize a wide range of defanging patterns [42]. This normalization step restores IoCs to their canonical, operational form, enabling precise extraction, and supporting reliable cross-verification on external threat-sharing platforms.

4.7. IoC Alert Report Generation

To reduce the manual triage and resource burden on security operations teams, this study leverages LLM reasoning and summarization capabilities to shift the analyst role from manual indicator extraction to informed validation. Specifically, the LLM enriches security alerts with concise contextual explanations and synthesizes critical IoC-related intelligence from tweets, transforming raw social media signals into analyst-ready alerts. Compared with prior approaches that primarily produce unstructured lists of indicators [2,5], the generated alerts provide structured context that supports faster, evidence-based decision-making, and recommends mitigation. To further enhance operational usability and interoperability, the framework incorporates the STIX file format into the alerting process. STIX enables a standardized representation of extracted IoCs and their associated contextual metadata. This facilitates integration with existing CTI platforms, SOC tools, and threat sharing.

Zero-shot, role-based, and structured prompt engineering strategies were employed to guide the ChatGPT-4o LLM in generating concise and standardized threat summaries aligned with established CTI reporting conventions. The detailed prompt used for threat alert generation is illustrated in Figure 6. The prompt is designed around four core components: (i) a role definition that instructs the model to operate as a cybersecurity analyst, (ii) a contextual description that frames the generation task, (iii) explicit task specifications that define analytical objectives, and (iv) output constraints that enforce the expected structure and reporting style. Each prompt instance includes both the raw tweet text and the pre-extracted IoCs, ensuring that the LLM is grounded in source evidence and relevant indicators. The output is constrained to a structured JSON format with predefined keys, enabling automated parsing and direct integration into downstream workflows. The sections include data and time of report, summary of tweets, IoCs details (significance, type, and value) and recommendation mitigation.

The prompt-based generation mechanism enables the model to transform raw and heterogeneous, unstructured threat data into standardized intelligence summaries suitable for SOC consumption. Figure 7 presents the JSON schema used to structure the LLM-generated IoC alert reports, illustrating how summary, IoC details, and recommended mitigation actions are organized into clearly defined fields. The content of these reports is derived from LLM reasoning and summarization capabilities rather than explicit fine-tuning on labeled cybersecurity datasets.

To ensure reliability and analyst trust, the proposed framework adopts an expert-in-the-loop workflow that supports expert validation and contextual assessment. Experts systematically review extracted IoCs alongside the associated contextual information derived from tweet content. When an IoC is deemed accurate and actionable, the expert can trigger an automated STIX generation process via a dedicated interface control, using a prompt shown in Figure 8. This action invokes a prompt-driven LLM to translate the validated indicators and their contextual metadata into a standard-compliant STIX 2.1 representation. The resulting STIX objects remain fully editable, enabling experts to correct inaccuracies, refine semantic relationships, and enrich metadata prior to dissemination. This expert-in-the-loop design ensures high-quality, trustworthy CTI while maintaining alignment with standardized CTI sharing frameworks.

4.8. Custom User Interface

To support practical deployment and efficient analyst interaction, a web-based user interface that supports the English language was implemented using Python and Flask version 3.0.3 to visualize and manage LLM-generated IoC alert reports. The interface presents the JSON output as an indexed file list and renders each report in a structured, human-readable layout while preserving the hierarchical schema shown in Figure 7. This design enables rapid alert inspection and minimizes cognitive overhead, allowing experts to prioritize ambiguous or high-risk cases. Experts can review and validate the extracted IoCs, either converting confirmed indicators into standardized STIX representations for downstream sharing or discarding non-actionable artifacts. This interactive design reinforces an expert-in-the-loop workflow, ensuring analytical efficiency and quality control for operational CTI.

4.9. Evaluation Criteria

The proposed framework was systematically evaluated across three complementary dimensions: performance, output quality, and technology acceptance. The specific metrics and measurement instruments adopted for each dimension are described in the following subsections.

4.9.1. Performance

This criterion assesses the effectiveness and robustness of the LLM-enabled components within the proposed framework. It focuses on two aspects: (i) the classification performance of the hybrid detection model, and (ii) the contribution of LLM-generated synthetic data to improving predictive accuracy and class balance. To this end, an ablation study was conducted to systematically quantify the impact of key architectural components by isolating performance gains attributed to BERT representations.

Table 5. Model variants used in the ablation study to evaluate the contribution of model components.

Model ID	Model Name	Description/Modification
M1	Hybrid BERT–BiLSTM (Full Model)	The complete model that integrates BERT sequence embeddings and the CLS pooler output, followed by BiLSTM, attention layer, and FC layers.
M2	Hybrid BERT (Sequence output)–BiLSTM	Same as the full hybrid model but removes the CLS pooler output branch, using only sequence embeddings processed through BiLSTM, attention layer, and FC layers.
M3	BERT (Pooler output)	Only the CLS pooler output representation is retained and passed to FC layers, without incorporating sequence embeddings, BiLSTM components, or an intention layer.

summarizes the three model variants evaluated in the ablation analysis.

To further examine the effectiveness of synthetic data augmentation, classification experiments were repeated for each model variant listed in Table 5, both with and without synthetic samples. Model performance was evaluated using standard classification metrics, including accuracy, precision, recall, specificity, and F1-score.

4.9.2. Output Quality

The synthetic tweets generated for data augmentation were assessed through a two-stage quality evaluation designed to determine whether the samples preserved linguistic realism, contextual relevance, and technical correctness. The first stage employed automated metrics including a self-Bilingual Evaluation Understudy (Self-BLEU) [24] that quantifies textual diversity and redundancy within the constructed corpus. High Self-BLEU scores indicate strong lexical similarity among the generated samples, reflecting low diversity, whereas lower scores signify significant lexical variation and novelty. Applying Self-BLEU aims to ensure that the augmentation process does not introduce repetitive or homogeneous content, and thereby maintains the variability required for effective and robust model training.

The second stage employed a qualitative, human-based evaluation conducted by three cybersecurity domain experts fluent in English and experienced in annotation tasks. Annotators independently assessed each generated tweet, without the aid of artificial intelligence (AI) tools, according to the following criteria:

• Relevance: alignment of the generated tweet with the intent and constraints specified in the prompt.
• Coherence: logical flow and readability of the content.
• Fluency: Grammatical correctness and the naturalness of the language.
• Correctness: factual and contextual accuracy, including the appropriate defanging of any referenced IoCs where applicable.

All criteria were rated using a five-point Likert scale (1 = strongly disagree to 5 = strongly agree). Inter-rater reliability was measured using Fleiss’s Kappa coefficient ($\kappa$) to quantify the level of agreement among the annotators. High Kappa $\kappa$ values indicate consistent and reliable judgments, reinforcing the validity of human evaluation. Synthetic samples achieving $\kappa \ge 0.80$ and exhibiting consistent confidence across all evaluation dimensions were retained for subsequent experiments.

The generated summaries incorporated into the IoC alert reports were systematically evaluated to assess their quality, consistency, and reliability. This evaluation framework combined both reference-based metrics, using the original tweets as ground truth, and reference-free metrics to provide a comprehensive assessment. Specifically, the reference-based evaluation included SBERT-based semantic similarity, natural language inference (entailment), and FactCC for factual consistency verification. The reference-free evaluation comprised perplexity to assess fluency, readability metrics to measure linguistic clarity, and RTS in which the language model first produces structured reasoning about summary quality before assigning a quantitative score.

The generated STIX file and mitigation recommendations integrated into the IoC alert report were evaluated using an expert-in-the-loop validation mechanism. Cybersecurity specialists were expected to review the LLM-generated file structure and contents and mitigation recommendations to verify their correctness, contextual relevance, and operational feasibility. This human verification step helps mitigate potential risks associated with automated text generation, such as inaccurate or misleading recommendations. Consequently, the proposed system is intended to assist security analysts by providing structured situational awareness while ensuring that critical defensive actions remain subject to human judgment and expert oversight.

4.9.3. Technology Acceptance

Several theoretical models have been proposed to explain and predict user acceptance of new technologies [43]. Among these, TAM is one of the most widely validated frameworks, providing a robust theoretical basis for understanding the determinants of users’ behavioral intentions and actual use of new technologies. In its original model, TAM explains technology adoption through two core constructs: Perceived Usefulness (PU) and Perceived Ease of Use (PEoU). These constructs influence the user’s Behavioral Intention (BI) to use a system, which leads to Actual Use (AU). To increase the explanation power of TAM, External Variables (EVs) may be incorporated to account for domain-specific contextual factors [44].

In this study, TAM is extended by integrating two constructs that are particularly salient in cybersecurity contexts: Perceived Risk (PR) and Perceived Trust (PT). In the context of this study, PR reflects the concerns of users regarding the potential for LLM-powered cybersecurity tools to generate false positives, false negatives, or incorrect alert content. PT refers to the extent to which users consider such tools reliable, unbiased, and aligned with established cybersecurity expertise and practices. Both constructs are modeled as direct determinants of BI, as they capture critical cognitive and affective factors that shape users’ confidence in the LLM-powered tool and their willingness to rely on it in the context of cybersecurity. In addition, PT is modeled as a determinant of both PU and PR, reflecting its central role in influencing users’ evaluations of system effectiveness and potential uncertainty. By incorporating PR and PT into the TAM framework, this study provides a more comprehensive understanding of user acceptance drivers for LLM-powered cybersecurity tools. The proposed conceptual model, illustrated in Figure 9, aligns with prior TAM extensions in AI-driven domains, including recent studies on AI-enabled training and decision-support systems [45].

Based on the proposed conceptual framework the following hypotheses are formulated:

H1. 

PEoU has a positive and significant effect on PU.

H2. 

PEoU has a positive and significant effect on BI.

H3. 

PU has a positive and significant effect on BI.

H4. 

PT in LLM positively influences the PU of LLM-driven cybersecurity tools.

H5. 

PT in LLM positively influences BI to use LLM-driven cybersecurity tools.

H6. 

PT in LLMs negatively affects the PR associated with using LLM-driven cybersecurity tools.

H7. 

PR associated with LLM-driven cybersecurity tools negatively influences BI to use them.

H8. 

BI has a positive and significant effect on AU.

The proposed hypotheses were empirically tested using multiple linear regression analysis. For each hypothesis, the direct effect of the independent variable on the corresponding dependent variable is evaluated. Standardized regression coefficients ($\beta$) were examined to compare the relative explanatory power of the predictors, while statistical significance and model adequacy were assessed using p-values and explained variance ($R^2$).

5. Results

A comprehensive experimental evaluation was performed in a controlled environment using a Google Colab Pro environment equipped with an NVIDIA Tesla T4 GPU and a high-RAM configuration. The following subsections detail the setup and evaluation of: (i) synthetic data quality, (ii) performance of a hybrid classification model, (iii) quality of IoC alert reports, (iv) technology acceptance, (v) characteristics of the extracted IoCs, and (vi) comparative analysis with state-of-the-art solutions: ML-enabled and LLM-enabled systems.

5.1. Evaluate Synthetic Data

To quantify the diversity of the generated samples, the Self-BLEU metric was employed. The synthetic tweets achieved a Self-BLEU score of $0.393$, compared to $0.497$ for the collected tweets, indicating substantially lower intra-dataset similarity. This reflects higher lexical and structural diversity among the generated samples, minimizing redundancy. Such diversity is desirable in data augmentation settings, as it mitigates the risk of overfitting to repetitive patterns and promotes improved generalization to previously unseen data. The synthetic corpus introduced 2627 unique tokens not present in the original dataset, demonstrating a notable expansion of the effective vocabulary. Newly observed cybersecurity terms include “malcode”, “surfing”, “infiltrate”, “superuser”, “data leakage”, and “divulging”, to name a few. This increase in lexical coverage enriches the training data with broader linguistic variations and domain-relevant semantic cues, thereby enhancing the capacity of the model to learn more discriminative and context-aware representations.

To further assess the content quality, each generated tweet was evaluated by three cybersecurity experts along four dimensions: (i) relevance, (ii) coherence, (iii) fluency, and (iv) correctness using a five-point Likert scale.

Table 6. Human evaluation of synthetic tweets.

Metric	Relevance	Correctness	Coherence	Fluency	Overall
Mean score	$4.71$	$4.67$	$4.60$	$4.58$	$4.64$
Agreement ($\kappa$)	$0.883$	$0.856$	$0.827$	$0.843$	$0.853$

summarizes the average scores across all evaluation dimensions. The results indicate consistently high quality across evaluation dimensions. Strong relevance scores confirm that the LLM effectively captured the intent and thematic context specified in the prompts, producing outputs that align well with the targeted IoC categories. High coherence and fluency scores demonstrate that the generated texts are logically structured, grammatically correct, and natural in expression, enhancing their realism and readability. Furthermore, the high correctness ratings underscore the factual reliability of the generated content and the consistent defanging of all the reported IoCs. This is a critical requirement in cybersecurity contexts, as it imitates the nature of real-world threat intelligence shared on the X platform. Each criterion demonstrated strong inter-rater agreement, with $\kappa$ values exceeding $0.827$ across all evaluation dimensions, indicating a high level of consistency and reinforcing the reliability and objectivity of the human assessment process. Consequently, all synthetic tweets were retained for subsequent analysis.

5.2. Evaluate ML Models

The results of the ablation study conducted on both the original dataset and the augmented dataset incorporating synthetically generated samples are summarized in

Table 7. Ablation study results using original and synthetic data illustrating the contribution of individual components within the proposed hybrid BERT–LSTM model.

Model	Using Original and Synthetic Data
	Accuracy	Precision	Recall	Specification	F1-Score
M1	$0.9861$	$0.9887$	$0.9838$	$0.9885$	$0.9863$
M2	$0.9849$	$0.9885$	$0.9815$	$0.9883$	$0.9850$
M3	$0.9422$	$0.9439$	$0.9419$	$0.9425$	$0.9429$
	Using Original Data
	Accuracy	Precision	Recall	Specification	F1-Score
M1	0.9825	0.9793	0.9863	0.9786	0.9828
M2	0.9836	0.9807	0.9870	0.9801	0.9839
M3	0.9369	0.9292	0.9476	0.9259	0.9383

. The incorporation of synthetic data into the training set resulted in higher classification accuracy than that of models trained exclusively on the original dataset. The complete hybrid model (M1) achieved the highest performance, attaining an accuracy of $0.9861$ and an F1-score of $0.9863$ with the augmented dataset, compared to $0.9825$ accuracy and $0.9828$ F1-score when trained on the original data alone, highlighting the effectiveness of the augmentation strategy. Moreover, the findings confirm that the joint integration of LLM-based representations, BiLSTM sequence modeling, and the attention layer yields a richer and more discriminative representation of IoC-related textual patterns.

To further examine the contribution of each individual architectural component, Model M2 excludes the CLS token branch. It retains the BERT sequence output, BiLSTM, attention layer, and FC layer. Compared to training on the original dataset alone, incorporating synthetic data led to a modest improvement in accuracy, precision, specificity, and F1-scores. The results further indicate that while the BiLSTM effectively captures sequential dependencies, the global semantic information encoded in the CLS token contributes to overall discriminative performance.

The BERT-only baseline model (M3) exhibited noticeable degradation in performance across all metrics compared to other models. This performance gap underscored the importance of temporal sequence modeling introduced by the BiLSTM layer. This enables the model to capture contextual dependencies and dynamic token interactions critical for accurate tweet classification. In the absence of sequential modeling, reliance on static CLS-based representations limits the ability of models to capture nuanced patterns where semantic meaning is influenced by token order and multi-word structures.

Figure 10 presents the confusion matrix for the evaluated models. As mentioned earlier, all configurations of models trained with synthetic data achieved higher overall accuracy. This was primarily through a reduction in false-positive predictions, albeit with a slight increase in false negatives. In the context of IoC detection, minimizing false positives is particularly important to reduce analyst fatigue and avoid unnecessary investigation overhead. The observed trade-off reflects a more conservative decision boundary that prioritizes precision and alert reliability, aligning well with real-world deployment scenarios where high-confidence threat alerts are essential.

5.3. Evaluate IoC Alert Report Summary

As illustrated in Figure 11, the web-based interface presents LLM-generated IoC alert summaries in a structured and analyst-friendly format, enabling rapid access to critical intelligence elements. The corresponding source tweet is shown in Figure 12, demonstrating the raw input from which the alert was derived. In this example, two IoCs are identified: a malicious IP address and a malicious URL. The generated summary effectively condenses the source content while preserving essential IoC details and providing potential mitigation recommendations. These elements facilitate efficient threat assessment and support informed decision-making by security analysts.

Figure 13 presents the STIX 2.1 file corresponding to the first indicator (the IP address shown in Figure 11), automatically generated by the proposed framework. The file content remains editable, allowing analysts to review, refine, and correct any potential inaccuracies prior to export. This expert-in-the-loop capability enhances reliability and trust while maintaining automation efficiency. The structured representation further ensures interoperability and seamless integration with threat intelligence platforms and security orchestration tools.

To evaluate the quality and reliability of the generated summaries, both reference-based and reference-free evaluation metrics were applied. The results are summarized in

Table 8. Evaluation metrics for generated summaries.

Category	Metrics	Value
Reference-based	SBERT	$0.7138$
	Entailment	$0.9736$
	FactCC	$0.5548$
Reference-free	Perplexity	$96.322$
	Readability	$34.52$
	RTS	$0.9200$

. Semantic similarity between the generated summaries and the original tweets was measured using SBERT, implemented via the sentence-transformers Python library, yielding a similarity score of $0.7138$. This result indicates strong semantic alignment, suggesting that core meaning and IoC-related information are largely preserved despite abstraction and paraphrasing during summarization.

Entailment was further assessed using an NLI-based approach with the NLI-Roberta-Base model. The high score ($0.9736$) demonstrates that the generated summaries are strongly supported by the source tweets, accurately reflecting the original content without introducing contradictory or unsupported claims. In addition, FactCC was employed to verify the factual consistency between the summaries and their corresponding source tweets, resulting in a score of $0.5548$. Although lower than the NLI-based metric, this value indicates moderate factual alignment and reflects minor discrepancies introduced through information compression and paraphrasing. Together, these metrics demonstrate that the generated summaries maintain semantic, entailment, and factual fidelity, supporting their reliability for threat intelligence reporting.

Reference-free metrics were used to assess linguistic quality. The perplexity score of $96.322$ indicates coherent and linguistically plausible outputs, while the readability score of $34.52$ reflects the technical complexity of cybersecurity content while remaining suitable for professional audiences. Moreover, the high RTS score ($0.92$) confirms strong overall alignment with LLM knowledge. Collectively, these results demonstrate that the proposed framework produces semantically faithful, factually reliable, and operationally useful threat alert summaries suitable for integration into SOC workflows while highlighting opportunities for further refinement in factual precision.

5.4. Technology Acceptance Evaluation

To evaluate technology acceptance, 78 participants with diverse gender distributions and varying levels of cybersecurity experience were recruited. Participants were granted access to the developed web-based application, which visualizes generated IoC alert reports and supports STIX generation. They were instructed to explore extracted IoCs, review the corresponding reports, examine the recommended mitigations, and generate STIX artifacts when deemed appropriate for enhancing proactive defense. Informed consent was obtained prior to participation, and upon task completion, participants completed a 24-item questionnaire.

Table 9. Demographic characteristics of the study participants.

Variable	Category	Number of Participants (%)
Year of Experience	Less than 3 years	8 ($10.26$%)
	3–9 years	35 ($44.87$%)
	10+ years	35 ($44.87$%)
Age Group (Years)	21–26	9 ($11.53$%)
	27–35	32 ($41.03$%)
	36+	37 ($47.44$%)
Gender	Male	40 ($51.28$%)
	Female	38 ($48.72$%)

summarizes the demographic characteristics of the participants, indicating a relatively balanced gender distribution (40 male, 38 female) and a majority of participants with more than three years of cybersecurity experience. Regarding age, 9 participants were between 21 and 26 years, 32 were aged 27–35, and 37 were aged 36 and above.

The evaluation was guided by the TAM framework described in Section 4.9.3. Data was collected using an online survey implemented via Google Forms after participants interacted with the web-based interface for collecting and reporting IoC artifacts. The questionnaire measured six constructs: PU, PEoU, BI, AU, PR, and PT. Measurement instruments for PU and PEoU were adapted from [46], BI and AU from [2], PR from [47], and PT from [48]. Each item was rated on a five-point Likert scale ranging from 1 (“very low”) to 5 (“very high”). IBM SPSS version 30.0 statistic was used to perform descriptive and inferential analyses. Instrument reliability and internal consistency were evaluated using Cronbach’s Alpha. As shown in

Table 10. Cronbach’s Alpha, Skewness, and Kurtosis values for model constructs.

Test/Variable	PEoU	PU	PT	PR	BI	AU
Cronbach’s Alpha	$0.809$	$0.727$	$0.926$	$0.794$	$0.733$	$0.839$
Skewness	$-1.419$	$-0.650$	$-0.859$	$0.767$	$-0.867$	$-0.458$
Kurtosis	$0.614$	$-1.588$	$-0.379$	$-1.011$	$-0.581$	$-1.597$

, the reliability coefficients for all constructs ranged from $0.727$ to $0.926$, exceeding the commonly accepted threshold of $0.70$, thereby demonstrating satisfactory internal consistency and measurement reliability [44]. The overall Cronbach’s Alpha for the entire instruments was $0.769$, providing additional evidence of the robustness of the measurement scales. To assess the normality of the data, Skewness and Kurtosis statistics were computed for each construct. As shown in Table 10, the Skewness values ranged from $-1.419$ to $0.767$, while the Kurtosis values ranged from $-1.597$ to $0.614$. All values fell within the commonly accepted $\pm 2$ range [49], indicating no substantial deviation from normality. These findings confirm that the data was approximately normally distributed, satisfying the assumptions necessary for parametric analyses, including regression analysis.

The eight hypotheses outlined in Section 4.9.3 were evaluated using linear regression analysis. Each hypothesis examines the direct effect of an independent construct on its corresponding dependent construct. Hypothesis evaluation was based on the coefficient of determination ($R^2$), standardized regression coefficients ($\beta$), and associated p-values, with a significance threshold of $p<0.05$ consistent with established practices in technology acceptance research [44,45]. The regression results, summarized in

Table 11. Summary of hypothesis-testing results using linear regression analysis.

	H1	H2	H3	H4	H5	H6	H7	H8
	PEoU→PU	PEoU→BI	PU→BI	PT→PU	PT→BI	PT→PR	PR→BI	BI→AU
$R^2$	$0.026$	$0.025$	$0.665$	$0.201$	$0.075$	$0.148$	$0.554$	$0.135$
$\beta$	$0.161$	$0.183$	$0.943$	$0.265$	$0.187$	$-0.246$	$-0.795$	$0.365$
p	$0.160$	$0.166$	<0.05	<0.05	$0.015$	<0.05	<0.05	<0.05
Decision	Rejected	Rejected	Supported	Supported	Supported	Supported	Supported	Supported

, provide strong overall support for the proposed theoretical model, with six of the eight hypotheses confirmed. However, the magnitude, direction, and explanatory power of confirmed relationships vary across constructs. This is evidenced by the observed variation in $\beta$ coefficients and $R^2$ values.

PEoU had a positive but statistically insignificant effect on PU ($\beta$ = $0.161$, p = $0.160$). Although the direction of influence aligned with the theoretical expectation of TAM, the effect was not significant at the $0.05$ level. The model explained only $2.6$% of the variance in PU ($R^2$ = $0.026$), indicating that PEoU contributed minimally to how users perceived the usefulness of LLM-based cybersecurity tools. This finding partially challenges a core assumption of TAM, suggesting that in complex and sensitive domains such as cybersecurity, ease of use alone may be insufficient to shape PU. Instead, users’ evaluations of utility may be more strongly influenced by factors such as trustworthiness, reliability, and interpretability of LLM-generated outputs. Therefore, while interface simplicity and reduced cognitive effort remain desirable design attributes, they appear to play a secondary role relative to domain-specific performance and assurance-related factors in determining PU within this context.

Similarly, the relationship between PEoU and BI was positive but statistically insignificant ($\beta$ = $0.183$, p = $0.166$). The model explained only $2.5$% of the variance in BI ($R^2$ = $0.025$), suggesting that the direct impact of ease of use on the BI of users was minimal. Although the direction of the relationship is consistent with the TAM assumptions, the findings suggest that ease of use alone is insufficient to substantially motivate users’ intentions to adopt or continue using LLM-driven cybersecurity tools. This outcome may be attributed to the specialized and professional nature of such systems, where users typically trained as practitioners tend to prioritize functional performance, analytical capability, and decision support over interface simplicity. These findings are consistent with prior research [45], which indicates that the relative importance of ease of use diminishes as technologies mature and as users’ experience and expectations evolve. In the context of LLM-driven cybersecurity systems, users may therefore tolerate higher levels of interface complexity provided that the technology delivers superior utility and actionable insights.

In contrast, PU emerged as a strong and statistically significant predictor of BI ($\beta$ = $0.943$, $p<0.05$), explaining $66.5$% of its variance. This substantial explanatory power underscores the dominant role of PU within the TAM framework, confirming that users’ intentions to adopt or continue using the LLM-driven system are primarily driven by their belief in its effectiveness and practical value. The magnitude of this relationship suggests that when users perceive the system as highly beneficial in enhancing their cybersecurity decision-making and operational efficiency, their intention to use it substantially increases. This result provides strong empirical support for the theoretical proposition that PU constitutes the most influential determinant of BI, particularly in professional, task-oriented, and sensitive domains where technological utility and performance outcomes are paramount.

PT demonstrated significant positive effects on PU and BI and a negative statistically significant effect on PR. In the context of PU ($\beta$ = $0.265$, $R^2$ = $0.201$), the findings indicate that users who trust the LLM-driven cybersecurity system are more likely to perceive it as useful and beneficial in supporting their security-related tasks. In the context of BI ($\beta$ = $0.187$ and $R^2$ = $0.075$), we find that users’ trust in the LLM-driven cybersecurity framework enhances their confidence in adopting it. When users perceive the system as trustworthy, demonstrating accuracy, ethical compliance, and reliability, they are more inclined to rely on its outputs and integrate it into their cybersecurity operations. This finding aligns with prior studies which position PT as a pivotal antecedent of BI, particularly in risk-sensitive domains [46,48]. Regarding PR, PT exhibited a negative and statistically significant effect ($\beta$ = $-0.246$, $R^2$ = $0.148$, $p<0.05$), indicating that high levels of trust significantly reduced users’ PR associated with the system. This implies that trust alleviates concerns related to potential errors, misuses, or unintended consequences of LLM-driven cybersecurity solutions. As users develop increased confidence in system reliability, their perception of operational and security risks diminishes, further facilitating acceptance and sustained use. Therefore, these results reaffirm the central role of trust as both a cognitive and affective determinant of technology acceptance in the cybersecurity domain.

PR exhibited the strongest negative influence on BI. These results confirm the theoretical relevance of the risk construct in explaining technology acceptance within cybersecurity contexts. PR exerted the strongest negative influence on BI ($\beta$ = $-0.795$ and $R^2$ = $0.554$), indicating that users’ BI to adopt LLM-driven cybersecurity technologies is highly sensitive to PR. When users perceive higher levels of security, potential bias, or incorrect risk in reporting IoCs, their willingness to adopt or engage with such systems declines substantially. This result underscores the critical importance of mitigating PR through enhancing reliability, transparency, and robustness in order to foster greater user trust and strengthen adoption intentions.

BI demonstrated a statistically significant predictive effect on AU, accounting for approximately $13.5$% of the variance. This finding aligns with the TAM assumptions and is consistent with prior empirical research, reinforcing the critical role of intention as a proximal determinant of actual system usage.

5.5. Characteristics of Extracted IoCs

The extracted IoCs were analyzed on two primary dimensions: volume and credibility. Volume refers to the total number of extracted indicators and the distribution across IoC types (e.g., IP addresses, URLs, and file hashes). Credibility refers to the proportion of indicators verified as malicious using established CTI sources. Credibility assessment was performed by cross-referencing extracted IoCs against authoritative CTI sources. VirusTotal [14] and IPVoid [12] were used to validate IP addresses and URLs. VirusTotal and Hybrid Analysis [13] were employed to verify the file hashes. It is critical to emphasize that this evaluation aimed to assess the credibility of the extracted IoCs rather than the real-time detection capabilities of the verification platforms themselves. Detailed results are presented in the following subsections.

5.5.1. File Hashes

Table 12. Statistics of collected, unique, and verified IoCs.

Collected Artifacts	Total	Unique	Verified (%) 1
Hashes	938	879	825 ($93.86$%)
URLs	5630	4747	4533 ($95.49$%)
IPs	2455	1665	1178 ($70.75$%)
Overall	9023	7291	6536 ($89.64$%)

¹ The numbers represent the total count of IoCs verified using the two verification methods.

summarizes the number of file hashes extracted by the proposed framework. A total of 938 file hashes were collected, of which 879 were unique. These comprised 215 MD5, 28 SHA1, and 636 SHA256 hashes; no SHA-512 hashes were identified. The absence of SHA-512 hashes is likely attributed to the 280-character limit imposed on traditional X accounts, which restricts the inclusion of longer indicators.

For verification, we queried VirusTotal and Hybrid Analysis and calculated the proportion of hashes flagged as malicious. As shown in Figure 14a, $93.86$% of the collected hashes were confirmed as malicious. Of these, $35.84$% were detected exclusively by VirusTotal, $0.23$% solely by Hybrid Analysis, and $57.79$% by both platforms. The comparatively low number of unique detections by Hybrid Analysis can be attributed to its reliance on dynamic sandbox execution, as opposed to VirusTotal, which aggregates data from multiple antivirus engines. A total of 54 hashes ($6.14$%) remained unverified and were further examined using Kaspersky Threat Intelligence [50]. This additional analysis confirmed four hashes as malicious, while the remainder (about 50 hashes) could not be verified. These findings reinforce prior research advocating the use of multiple CTI sources to enhance IoC validation accuracy and reliability, and they reflect the heterogeneous nature of indicators shared on X by diverse threat actors and security communities.

The extracted file hashes were further analyzed to identify associated file types, as shown in Figure 14b. The majority correspond to Dynamic Link Library (DLL) ($32.65$%) and executable (EXE) files ($19.80$%). In addition to traditional executables, a wide range of malicious file types was observed, including Android application packages (APK), compressed archives (ZIP/RAR), Microsoft Office documents (e.g., XLSX, DOC, and ONE), Windows shortcut files (LNK), Windows installer packages (MSI), Apple disk images (DMG), Rich Text Format (RTF) files, plain text files (TXT), and web-related files (HTML, ASP). For 67 files ($7.62$%), the file type could not be determined and therefore was classified as “Unknown.” These findings indicate that collected file hashes span a broad spectrum of malicious artifacts. This indicates that threat actors employ diverse delivery mechanisms across heterogeneous platforms and operating environments.

5.5.2. URLs

As reported in Table 12, the framework extracted 5630 URLs, of which 4747 were unique. VirusTotal and IPVoid were queried to assess their maliciousness. The verification results, illustrated in Figure 15a, show that 4533 URLs ($95.49$%) were confirmed as malicious. Among these, 3887 URLs ($81.88$%) were validated by both platforms, 525 ($11.06$%) exclusively by IPVoid, and 121 ($2.55$%) solely by VirusTotal. The remaining 214 URLs ($4.51$%) were not verified by either source and were subjected to further analysis.

Unverified URLs were examined using MalwareURL and manual inspection to determine their validity. The results are shown in Figure 15b. The majority of unverified URLs 119 ($55.61$%) were linked to threat-sharing platforms such as VirusTotal, Alienvault OTX, and MalwareBazaar. In total, 13 ($6.07$%) URLs were linked to social media such as Instagram, Youtube, and security blogs. Seven URLs were legitimate website links and ten were linked to applications. In total, $10.76$% were tweet shortcuts and 11 ($5.14$%) were Github links. Eighteen links were verified as malicious or phishing domains by MalwareURL and 13 ($6.07$%) remained unverifiable even when domain level was used to verify the maliciousness. The prevalence of benign and unverifiable URLs in IoC-related tweets is largely driven by tweets that reference reputable scanning services or threat-intelligence platforms when discussing threats, rather than directly embedding malicious links. This observation underscores the limitations of document-level IoC extraction and underscores the need for fine-grained, sentence- or token-level classification to accurately distinguish actionable indicators from references co-occurring in cybersecurity discourse.

5.5.3. IP Addresses

Table 12 summarizes the number of potentially malicious IP addresses extracted by the proposed framework. A total of 1665 unique IP addresses were identified, all of which were IPv4. VirusTotal and IPVoid were queried via their respective IP reputation APIs to assess maliciousness. As shown in Figure 16a, $14.29$% of IPs were confirmed as malicious by both platforms, $6.25$% exclusively by IPVoid, and $50.21$% solely by VirusTotal. A total of 487 IPs ($29.25$%) were not flagged by either source and were further analyzed using the AlienVault OTX API. This investigation revealed that ten IPs belonged to private address ranges and were therefore not publicly routable or assessable by external CTI services. The remaining IPs were publicly routable and not identified as malicious.

The malicious IPs were further investigated and the geographical distribution of the countries associated with these IPs was obtained. The majority were associated with the United States ($29.10$%) and China ($22.67$%), with additional significant activity observed in the Russian Federation ($7.80$%), Hong Kong ($4.90$%), India ($3.91$%), Netherlands ($2.90$%), Japan ($3.63$%), Germany ($2.72$%), the UK ($1.63$%), Korea ($2.36$%), and Viet Nam ($1.72$%). This wide geographical distribution underscores the global and transnational nature of cyber threats. Importantly, the geographical attribution of IP addresses should be interpreted as indicative of hosting or infrastructure location rather than definitive evidence of the attacker’s origin.

5.6. Comparison with State-of-Art ML-Enabled Solutions

The performance of the proposed hybrid model was benchmarked against several established ML–based approaches presented in the literature, including iACE [6], TIMiner [7], ChainSmith [4], IoCMiner [5], MTL-CTI [3], and ML based approaches reported in IoC-Collector [2].

Table 13. System-level comparison summary for ML-enabled studies in the context of IoCs. NR indicates the information is not reported in original study; ✔ indicates presence (yes), and ✕ indicates absence (no).

Properties		iACE	TIMiner	ChainSmith	IoCMiner	MTL-CTI	IoC-Collector	LLM-Powered
		[6]	[7]	[4]	[5]	[3]	[2]
Algorithm	Data source	Technical blog	Social media	Security articles	X	Social media	X	X
	ML Classifier	LR	CNN	LDA	RF	RNN	SVM	hybrid
	Feature	NER	Word2Vec & syntactic dependencies	Embedding & NER	BoW	Embedding& NER	Unigram/bigram ∪ IoC-centric ∪ sentiment scores	Contextualized BERT embeddings
Task	Mining	✔	✔	✔	✔	✔	✔	✔
	Extracting	✔	✔	✔	✔	✔	✔	✔
	Operational integration	OpenIoC items	Threat-Index	✕	✕	✕	✕	STIX
	Advanced feature	✕	✕	✕	✕	✕	Prioritization	Alerting with contextual summary
Perf.	Accuracy	NR	$94.00$%	NR	$97.00$%	NR	$94.27$%	$98.61$%
	Precision	$92.00$%	NR	$91.90$%	NR	NR	$95.97$%	$98.87$%
	Recall	$87.00$%	9200	$97.80$%	NR	NR	$96.42$%	$98.38$%
	F1-score	$89.00$%	$93.00$%	NR	NR	$97.30$%	$96.20$%	$98.63$%
Acceptance Evaluation		✕	✕	✕	✕	✕	✕	TAM

provides a comprehensive comparison of these frameworks. It summarizes the data sources used, ML classifier applied, extracted features, targeted tasks, reported performance metrics, and observed functional trends. In addition to technical characteristics, the comparison incorporates insights from user-centric evaluations where available, highlighting both methodological robustness and practical usability considerations.

Most existing studies rely on conventional word-level representations as input features, including Word2Vec [7], dependency-based word embeddings [4], or embeddings learnt end-to-end within deep neural architectures such as CNNs and BiLSTMs [3]. Other approaches employ traditional BoW, Unigram, bigram, and sentiment-based features in conjunction with classical machine learning classifiers [2,5]. While these methods demonstrate effectiveness in mining and extracting IoC artifacts, comparatively limited attention has been devoted to translating extracted indicators into actionable threat alerts or the integrating them into operational security workflows [6,7].

The work in [6] uses LR and NER to introduce the iACE system for mining and extracting IoCs from technical blogs. The iACE framework provides a structured representation of extracted IoCs by converting them into the OpenIOC format; however, the approach does not enrich the indicators with the contextual metadata required for assessing reliability, severity, or operational relevance. Moreover, although iACE achieves an F1-score of 89%, indicating relatively strong extraction capability, the performance still suggests the presence of both false positives and false negatives that may affect operational reliability in real-world CTI settings. TIMiner [7] mines, extracts, and indexes collected IoCs to estimate threat severity, reporting an improved F1-score of $93.0$%, which reflects enhanced extraction accuracy compared to earlier approaches. However, despite its strong extraction results and severity estimation, TIMiner focuses on post-collection indexing and analysis. It does not address real-time alert generation, automated threat summarization, or integration of indexed indicators into incident response workflows.

ChainSmith [4] employs a two-stage ML approach, first detecting IoCs from unstructured security articles and subsequently classifying the collected indicators according to associated attack campaigns. This hierarchical design enables not only artifact extraction but also high-level contextual grouping. However, the framework does not include automated severity assessment, real-time alert generation, or a standardized CTI export format to facilitate direct integration with operational security platforms. The frameworks in [3,5] present preliminary designs to mine and subsequently extract IoCs from social media. Although they demonstrate promising extraction capabilities ($97.30$% F1-score and 97% accuracy, respectively), their scope remains largely limited to artifact detection without comprehensive contextual enrichment, severity assessment, or structured threat intelligence generation. The IoC-Collector framework [2] ranks extracted IoCs using statistical features derived from social characteristics and tweet-level attributes. While informative, this ranking strategy lacks semantic and contextual threat interpretation, limiting its utility for credibility assessment and operational decision-making in security operations.

In contrast, the proposed framework leverages LLM-based contextual embeddings to capture rich semantic representations and long-range contextual dependencies, yielding substantial performance improvements over existing approaches and achieving an accuracy of $98.61\%$ and an F1-score of $98.63\%$. Beyond extraction performance, the framework supports core stages of a CTI pipeline, including IoC mining, automated alert generation, and support integration into operational security environments. Furthermore, this study uniquely incorporates a systematic user acceptance evaluation grounded in the TAM, examining PT, PU, PEoU, and BI to adopt LLM-enabled CTI systems in real-world SOC settings.

5.7. Comparison with LLM-Enabled Solutions

Table 14. System-level comparison summary for studies that use LLMs in the context of IoCs. ✕ indicates absence (no).

Property	LANCE [16]	TSTEM [36]	AIThreatAttack with RAG [9]	IoC-Collector [2]	LLM-Powered
Data Source (Size)	CTI reports (50)	Online sources ($\mathrm{26,000}$ training 1000 testing and 600 validation)	X (71 unique tweets)	X (3148 unique tweets)	X ($\mathrm{11,103}$ unique tweets)
LLM	ChatGPT-4o	BERT	ChatGPT-4	BERT	BERT
Adoption Strategy	Prompt zero-shot	Fine-tuned BERT	RAG & prompt	Fine-tuned BERT with classification head and extended tokenizer	Fine-tuned with hybrid classification head
Tasks	Mining & Extraction	Mining & extraction	Extraction, ElasticSearch, & Kibana query generation	Mining, extraction, prioritization, & verifying	Mining, extraction, reporting, & verifying
Extracted Artifacts	Total: 1791 Unique: 1791	Total: 163 Unique: 79	Total: 336 Unique: 336	Total: 1855 Unique: 1529	Total: 9023 Unique: 7291
Advanced LLM-enabled functionalities (if any)	Color-coded visual cues for IoC interpretation	None	None	None	Alerting with contextual summary
Performance	Precision: $96.7$% F1: $97.6$%	Accuracy: $98.00$% Precision: $83.00$% Recall $85.00$% F1: $84.00$%	45% IoCs were missed & $35.52$% hallucination rate	Accuracy: $99.36$% Precision: $98.27$% Recall: $99.25$% F1: $98.76$%	Accuracy: $98.61$% Precision: $98.87$% Recall: $98.38$% F1: $98.63$%
Efficiency	NR	High computational cost	NR	Training: $57.57$ min. Testing: $33.70$ min.	Training: $42.1$ min. Testing: $0.27$ min.
Acceptance Evaluation	✕	✕	✕	✕	TAM

presents a system-level comparative overview of recent studies which employed LLMs for IoC-related tasks, highlighting substantial differences in dataset scale, LLM model, adoption strategies, task scope, number of extracted artifacts, advanced LLM-enabled functionalities, and operational efficiency. Furthermore, the table indicates whether these studies include an evaluation of user acceptance of the proposed technology. As shown in the table, ChatGPT and BERT-based architectures are among the most commonly adopted models. As in the case of ML-based approaches, LLM-driven solutions focus on mining and extracting IoCs from heterogeneous threat intelligence sources. A limited number of studies extend beyond extraction by providing analyst-oriented interfaces. For example, the LANCE framework in [16] introduces color-coded labeling schemes that leverage prompt-based LLM inference to classify extracted artifacts as malicious or benign depending on their surrounding textual context are presented. Although this framework reports high performance, with reported F1-scores reaching up to $97.6\%$, their reliance on relatively small datasets (50 reports) and the lack of large-scale empirical validation constrain their applicability in operational CTI environments. The total number of extracted artifacts was 1791, reflecting the ability of this framework to identify and consolidate relevant IoCs from CTI reports.

The prompt strategy demonstrated measurable improvements in computational efficiency and generation reliability. The AIThreatAttack framework [9] employed a prompt-based RAG approach to generate Elasticsearch and Kibana queries for 336 extracted IoCs; however, evaluation revealed that approximately 45% of IoCs were missed, and the system exhibited a hallucination rate of $35.52$%, raising significant concerns about prompt reliability in extraction and query generation tasks. The extracted IoCs are therefore presented as a list without additional metadata that allows analysts to review and validate the extracted and generated queries to minimize the risk of erroneous or incomplete threat data. These findings highlight the importance of prompt design and grounding mechanisms to maintain accuracy and trustworthiness in operational CTI workflows.

Fine-tuning strategies have demonstrated strong predictive performance, achieving 98% accuracy in [36] and $99.36$% accuracy in [2]. However, these approaches exhibit several practical limitations. In [36], the framework extracted only a limited number of indicators, identifying 79 unique IoCs, and presented the results as a simple list without leveraging LLM capabilities for advanced analytical functionalities. In addition, the framework incurred relatively high computational costs. The study in [2] further improved predictive performance by introducing an extended tokenizer augmented with IoC-specific tokens, along with a source-credibility-based prioritization mechanism. Although this approach increased the number of extracted indicators to 1529 IoCs, fine-tuning strategies still introduce significant efficiency challenges. Specifically, the fine-tuning process in IoC-Collector requires $57.57$ min for training and $33.70$ min for testing, excluding the additional time required for data preprocessing, augmentation, and IoC artifact prioritization. Furthermore, such approaches typically require large-scale, well-annotated datasets to maintain robustness and generalization across evolving cyber-threat landscapes. However, both frameworks primarily focus on IoC detection and extraction, without fully exploiting the contextual reasoning capabilities of LLMs to generate analyst-oriented CTI, such as structured alert summaries, contextual threat interpretation, and actionable mitigation recommendations.

In the proposed solution, we address these limitations by balancing predictive effectiveness with computational and operational efficiency, employing a hybrid model to detect and extract IoCs while leveraging LLMs to generate structured reports from tweet sources. The preprocessing stage requires approximately $0.000526$ s per tweet, while generating a single synthetic tweet requires approximately $1.570$ s. To estimate the operational cost of the LLM component, the report generation module allows for a maximum output of approximately 300–1000 tokens per alert, depending on the complexity of the detected threat description. In this setting, $5.798$ s is required for one report generation and $4.46$ s for STIX threat intelligence generation. However, the synthetic data generation process is performed only once during the offline training phase to augment the dataset and improve model robustness. Consequently, this step does not contribute to the runtime latency of the deployed system, where only the lightweight preprocessing and detection modules operate in real time, resulting in an operational latency of approximately $10.259$ s per alert. This additional processing enables the generation of structured threat intelligence artifacts, providing richer and more actionable outputs for security analysts compared to detection-only approaches. The framework emphasizes high-precision detection ($98.87$%), minimizing false-positive alerts and reducing analyst workload. This design choice is particularly critical in operational security environments, where excessive false alarms can contribute to alert fatigue and erode trust in automated systems. By producing fewer but more reliable IoC alert reports, the proposed approach enhances practical usability, supports timely and confident decision-making, and ultimately improves the acceptance and operational viability of LLM-driven CTI systems. Moreover, this study examines the acceptance of the technology using TAM, assessing PU, PEoU and cybersecurity-specific constructs among security analysts factors that have not been systematically studied in prior frameworks. Interestingly, the proposed framework extracted 7291 unique IoCs, surpassing the total reported by prior frameworks and demonstrating its enhanced coverage and extraction capability.

6. Discussion

This section discusses the practical application of the proposed framework for real-world cybersecurity operations and CTI workflows while also highlighting key limitations that may influence its applicability and generalizability in operational environments.

6.1. Practical Applications

This study offers a practical and deployable framework for enhancing cybersecurity operations and CTI workflows. The framework can be configured to continuously monitor heterogeneous open-source platforms relevant to domains frequently targeted by phishing campaigns, such as the financial services, e-commerce, and government sectors. By leveraging sector-specific keywords and contextual cues, the system enables early detection of phishing-related IoCs, including fraudulent domains and spoofed URLs. This customized monitoring capability supports timely alerting and facilitates proactive mitigation. This allows SOC teams to respond to emerging campaigns before they impact a broad population of end users.

The proposed framework facilitates automated CTI acquisition and utilization. It encompasses data collection, intelligent analysis through classification and extraction of artifacts, and actionable alerting. Unlike existing approaches that primarily focus on artifact extraction, the alerting mechanism integrates LLM-generated contextual summaries and reasoning capabilities to enhance situational awareness and offer recommendations for mitigation. These context-aware alerts substantially reduce manual analysis effort and accelerate incident triage by supporting an expert-in-the-loop workflow, in which analysts can rapidly validate, prioritize, and act upon high-confidence intelligence while minimizing cognitive overload and response latency. The improved precision achieved by the proposed model further reduces false positives, mitigating alert fatigue and strengthening analyst trust in automated intelligence systems.

From an integration perspective, the framework allows for the automated generation of STIX files, a capability that remains largely underexplored in existing CTI research. The generated STIX files can be revised and incorporated into established SOC infrastructures, including security information and event management systems and external threat intelligence feeds. This structured output enables downstream automation, such as rule generation, correlation with internal logs, and enrichment within incident response workflows. This supports a more proactive and intelligence-driven security posture, facilitating continuous threat hunting and early-stage attack detection.

The computationally efficient design of the framework facilitates deployment in resource-constrained environments, such as small and medium-sized organizations that lack extensive computational infrastructure or large annotated datasets. By balancing predictive accuracy with operational efficiency, the proposed framework lowers the barrier to adopting advanced LLM-based CTI capabilities. Collectively, these practical benefits demonstrate the potential of LLM-enabled IoC collection and alerting systems to enhance operational efficiency, reduce response times, and improve the overall effectiveness of cybersecurity incident management.

6.2. Limitations

Although the proposed framework demonstrates strong performance and practical effectiveness in IoC extraction, several limitations must be acknowledged. The proposed framework relies heavily on the quality, coverage, and timeliness of open-source data, which can vary substantially across platforms and over time. Inconsistent data availability or reporting practices may affect the completeness and representativeness of extracted indicators. The ability of a framework to continuously collect intelligence from X depends on API availability, access policies, and associated rate limits. Changes in platform terms of service or restrictions on API usage may disrupt data continuity and reduce intelligence acquisition reliability. Therefore, integrating a broader range of heterogeneous data sources, such as dark web forums, malware repositories, and specialized security blogs, would enhance the framework’s ability to capture a more diverse spectrum of IoCs, thereby improving the robustness and resilience of the generated threat intelligence.

Nevertheless, the current implementation relies on tweet-level classification to identify posts that potentially contain IoC artifacts. While effective for rapid intelligence collection, this coarse-grained approach may include benign references such as informational URLs. Future work will therefore explore fine-grained IoC classification and token/span-level extraction techniques to more precisely identify actionable indicators and apply contextual filtering mechanisms. Such enhancements would further reduce indicator pollution, improve extracted artifacts quality, and minimize downstream analysis workload.

The proposed framework relies on an English LLM that excludes non-English content during preprocessing. While this design choice simplifies language modeling and improves extraction accuracy for English artifacts, it restricts the framework’s visibility to threat intelligence originating from non-English-speaking regions. To mitigate this limitation, the framework can incorporate multilingual LLMs or automated translation pipelines to enable the processing of multilingual content without sacrificing analytical consistency. Such capabilities would improve global coverage, reduce linguistic bias, and support early detection of emerging threats across diverse geopolitical contexts.

Although the framework supports automatic STIX file generation, it does not fully eliminate expert analyst oversight. SOC analysts remain responsible for validating extracted indicators, assessing their operational relevance, and making final-response decisions. Therefore, the framework should be viewed as an augmentation for human expertise rather than a replacement. Its operational effectiveness may vary depending on analyst skill levels and organizational processes.

LLM-generated alert mitigation recommendations require further validation by systematic analyst-based evaluation. While the current framework incorporates expert review to ensure the usefulness of generated outputs, future work should conduct structured assessments involving cybersecurity analysts to evaluate factual accuracy, actionability, and potential hallucinations or error rates. Such an evaluation would provide a more comprehensive understanding of the practical reliability of LLM-generated recommendations in real-world cyber-defense settings.

Despite the comprehensive integration of the TAM with cybersecurity-specific constructs to examine the acceptance of LLM-based tools, this study does not account for the potential influence of demographic factors. Variables such as age, gender, educational background, professional role, years of cybersecurity experience, and prior exposure to artificial intelligence technologies may shape PU, PEoU, PT, and PR. The exclusion of these factors limits the ability to assess possible moderating effects across different user segments and may restrict the generalizability of the findings. Future research should incorporate demographic variables and employ multi-group analysis or moderation modeling to determine whether acceptance mechanisms differ across expertise levels, organizational contexts, or AI familiarity.

7. Conclusions and Future Work

This study presents a hybrid LLM-driven framework for automated CTI mining, IoC detection, structured reporting, and operational integration. By combining BiLSTM, attention mechanisms, and BERT with frozen layers, the proposed approach balances predictive effectiveness with computational efficiency. The framework successfully extracted 7291 unique IoCs and generated structured, analyst-friendly IoC reports while minimizing false-positive alerts. This design prioritizes reliability in operational security environments, where alert fatigue and trust erosion remain critical challenges.

Compared with prior ML- and LLM-enabled approaches, the proposed solution extends beyond artifact extraction by incorporating structured reporting and interoperability through standardized formats. The emphasis on high-precision detection ensures fewer but more reliable alerts are produced, improving analyst confidence and decision-making efficiency. Furthermore, this study contributes to a user-centric perspective by evaluating technology acceptance using TAM, assessing PU, PEoU, and cybersecurity-specific constructs. The findings indicate strong acceptance potential, underscoring the importance of aligning technical performance with human factors in operational SOC contexts.

Despite the contributions to the current framework, several avenues for future research remain. First, expanding the evaluation to larger and more diverse real-world datasets across multiple threat domains would further validate robustness and generalization capabilities. Second, efforts should focus on improving recall while maintaining high precision to achieve more balanced detection performance without increasing false-positive alerts. Building on these foundations, future works will also aim to extend the system toward fully closed-loop proactive cyber defense by integrating automated IoC collection and extraction with real-time incident-response mechanisms, such as SIEM and SOAR platforms, enabling immediate mitigation actions. Furthermore, deep semantic enrichment, such as automated attack pattern mapping (e.g., MITRE ATT & CK alignment), threat actor attribution, and confidence scoring, will enhance the contextual value of collected intelligence. Incorporating streaming architectures for near real-time processing and systematically evaluating operational impact through metrics like time-to-detection, threat containment speed, and improvement over existing CTI ingestion workflows will support a comprehensive, proactive cyber-defense capability, bridging intelligence collection and automated response while facilitating actionable insights for security analysts.