Collaboration of Large and Small Models for Event Type Discovery in Completely Open Domains

Abstract

Event type discovery in open domains aims to automate the induction of event types from completely unlabeled text data. Conventionally, small models utilize clustering techniques to address this task. Nonetheless, the fully unsupervised nature of these methods results in suboptimal performance by small models in this context. Recently, large language models (LLMs) have demonstrated excellent capabilities in contextual understanding, providing additional relevant information for specific task scenarios, albeit with challenges in precision and cost effectiveness. In this paper, we use LLM to guide the clustering of event texts and distill this process into a fine-tuning task for training smaller pre-trained language models. This approach enables effective event type discovery even in scenarios lacking annotated data. The study unfolds in three stages: in action acquisition, leveraging LLMs to extract type-relevant information from each event text, ensuring that the event representations are particular to task-specific details; in clustering refinement and dual-fine-tune, LLMs refine results from both task-agnostic and task-specific perspectives, with the refinement process designed as fine-tuning tasks under different viewpoints to adjust encoders; and in type generation, post-clustering, LLMs generate meaningful event type labels for each cluster. Experiments show that our method outperforms current state-of-the-art approaches and excels in event type discovery tasks even in completely open-domain with no labeled data.

1. Introduction

Events as a distinct category of information represent fundamental units through which humans interpret and experience the world. Typically, events are characterized by the occurrence of actions or state transitions. Although event information extraction methods have achieved considerable success, they are significantly dependent on human-guided real-world event ontology and meticulously structured supervisory schema. Due to the open and dynamic nature of real-world information, obtaining such supervision is challenging. Consequently, there is an increasing need to automate the extraction of novel events from open information sources and to infer new types of events, including those that have not previously appeared.

To address the challenges presented by open domains, several approaches have been proposed, including event type induction [1,2] and event schema induction [3,4]. These methods aim to enable systems to autonomously identify new event types and extract instances corresponding to these types. However, most of these approaches have struggled to adapt fully to completely open domains. Until recently, they often relied on models accessing prior knowledge, such as trigger word mentions [2] or partially known event types [5], to facilitate the discovery of new types within semi-open contexts.

In completely open domains, where only the event text itself is available, type discovery naturally assumes the form of a clustering task. Current mainstream methods utilize pre-trained language models to encode text into high-dimensional vector spaces, followed by clustering algorithms [6]. However, when applied to raw event text, vectors generated by pre-trained language models such as BERT [7] predominantly capture surface-level similarities, failing to encapsulate critical information pertinent to event types within the text [8]. Additionally, the event types derived from these methods often lack practical significance and are typically represented in a highly abstract manner [5].

Recently, large language models have demonstrated exceptional capabilities in information extraction [9]. In completely open domains, these models are adept at comprehensively understanding context and delivering task-specific key information according to user preferences and instructions [10,11,12], thereby facilitating event type discovery. Despite their impressive summarization and discovery abilities, the immense scale of LLMs presents significant challenges for real-world deployment [13,14], and their outputs are frequently imprecise and difficult to fine-tune.

In this paper, we introduce a novel framework that integrates both large and small models to address the task of event type discovery. The main purpose is to utilize the generative and summarization capabilities of LLM to guide smaller models through three main stages of event type discovery, thereby enabling the smaller models to effectively identify event types even in entirely open and unannotated domains. Our approach is distinguished by its performance in three key stages: First, for the action acquisition stage, it employs the summarization capabilities of LLMs to extract type-relevant information from event texts, facilitating the generation of event representations focused on type information by the smaller models. Second, for the clustering refinement and dual-fine-tune stage, it involves using an LLM to refine clustering results, with the refinement process structured as fine-tuning tasks from both task-agnostic and task-specific perspectives, thus equipping the smaller models with task-specific preferences to enhance adaptive adjustments. Finally, for the type generation stage, it extracts samples from each cluster and utilizes an LLM to summarize these samples into a set of event types with practical significance.

Our approach utilizes an LLM to inject highly relevant information about event types as task-specific perspectives, thereby offering supervisory signals for event types within entirely open domains without the necessity of any additional prior information. Concurrently, it transfers knowledge from the LLM into fine-tuning tasks by executing dual fine-tuning on the encoder from both task-specific and task-agnostic perspectives (reflecting global information about events). This strategy circumvents the need for intricate task-specific adjustments directly to the LLM.

In summary, the main contribution of this article is at least three-fold:

• Our approach functions on an event corpus within completely open domains, without the need for prior knowledge or manual annotation. By using straightforward instructions, it allows LLMs to discern information pertinent to event types and reconstruct event types with substantive semantic content.
• We utilize an LLM to optimize clustering results by integrating user intent and task perspectives into the encoder. Adjustments are made to the encoder from both task-agnostic and task-specific perspectives, thereby achieving a significant enhancement in clustering performance.
• We assess the performance of our model using two extensively utilized datasets. The results demonstrate that, in fully open domains, our approach significantly outperforms the current state-of-the-art models.

2. Related Work

In this section, we first review the existing research on event type discovery. Subsequently, based on the employed methodologies, we provide an overview of the advancements in research on LLMs and text clustering.

2.1. Event Type Discovery

The primary objective of the event type discovery task is to uncover potential event types within text that may not have been predefined or labeled. Formally, this task bears similarity to the verb clustering task presented in QasemiZadeh et al. [15], which focuses on mapping contextually embedded verbs to semantic frames. Given the open, zero-shot nature of the task, most methods employ semi-supervised, unsupervised, or self-supervised learning techniques, aiming to accomplish the task without relying on additional prior knowledge.

Huang and Ji [1] defined a semi-supervised model, SS-VQ-VAE, which leverages a vector quantization variational autoencoder model to utilize known types. ETypeClus [2] models events using (predicate, object) structures, defining these structures as P-O pairs. It then reconstructs a clustering algorithm based on these P-O pairs. However, due to the constraints imposed by the P-O pair structure, this method is restricted to scenarios where event types are triggered by predicate verbs, which limits its generalizability. Tang et al. [16] designed an Event Schema Harvester (ESHer), which automatically generates high-quality event schemas through contextual concept generation, credibility-aware schema structuring, and graph-based schema aggregation. Although this method demonstrates good performance, it relies on predefined known event types. Recently, Li et al. [5] introduced a novel open type discovery approach grounded in transfer learning principles. This approach identifies event types by leveraging knowledge from existing types and represents these event types through abstract semantic models generated by the system.

The aforementioned methods exhibit two principal limitations in the event type discovery task: (1) They incorporate additional information to varying degrees, such as relying on ground-truth trigger word annotations or leveraging known types, which diminishes the openness of the discovery scenario. (2) The resulting event types are typically represented in abstract formats, such as keywords or predicate–object pairs, rendering them challenging to apply effectively in downstream tasks.

2.2. LLMs for Text Clustering

LLMs exhibit remarkable comprehension capabilities, enabling users to perform tasks with minimal instructions. The key advantage of this approach is that LLMs can effectively align data with task perspectives without necessitating additional auxiliary information. This feature offers substantial support for text clustering tasks that lack supervisory signals.

In recent years, a growing body of research has utilized LLMs to enhance text clustering, demonstrating significantly improved performance. Viswanathan et al. [14] utilized LLMs to provide guidance at various stages of clustering, enhancing input features, applying clustering constraints, and refining results. This facilitates a balance between cost and accuracy, leading to improved clustering outcomes. Zhang et al. [13] introduced CLUSTERLLM, a framework that employs large models to guide text clustering. The adjustment of clustering through LLMs is framed as a fine-tuning task applied to the encoder, which effectively integrates user intent and task perspectives into the encoder, thereby optimizing the clustering results. Wan et al. [17] leveraged LLMs within a two-stage framework for extensive text mining. This approach employs LLMs to automate the complete pipeline of label generation and assignment, efficiently managing the entire process from clustering to label allocation for any specified use case. Given that text clustering tasks often fail to provide interpretable labels for clusters, recent research has sought to mitigate these challenges by utilizing LLMs for topic modeling [18,19].

In our approach, we build upon the principles of the previously mentioned methods, leveraging LLMs to integrate essential information from the perspective of event types in open-domain contexts. This integration significantly improves the effectiveness of event type discovery.

3. Problem Definition

Given an unlabeled event text corpus $\mathcal{S}=\{S_1,\dots S_N\}$ which is a collection of event sentences, where $S_i=[w_1,\dots w_i,\dots w_n]\in \mathcal{S}$ represents a sequence of tokens in an event sentence, our goal is to infer a set $\mathcal{T}=\{T_1,\dots T_K\}$ from S, where $T_i\in \mathcal{T}$ represents an event type, and K is the number of discovered event types. To achieve this, the unlabeled corpus S is used as input, and the output is a clustering assignment $\mathcal{Y}={\left\{y_i\right\}}_1^K$, which maps the input text to cluster indices. Based on the clustering results, a meaningful label is assigned to each cluster, thus resulting in $\mathcal{T}$.

4. Methodology

The overall framework of the proposed method is depicted in Figure 1. This method employs the distinct functionalities of LLMs to facilitate the discovery of event types through three stages. We detail the working mechanisms of each stage in the following sections.

4.1. Stage I: Action Acquisition

In this stage, our primary goal is to identify information that exhibits a strong correlation with event types and to facilitate the model’s ability to capture these critical details during the clustering process. We define "action" as the pivotal information that reflects event types and employ LLM to extract these actions from events. Subsequently, we develop a salience score to evaluate and select actions from among the candidates based on their strong relevance to the event types.

4.1.1. Action Extraction

Previous research [2] has demonstrated that the majority of event types are triggered by phrases containing predicate verbs and their objects, which are regarded as highly pertinent to identifying event types. However, this structural approach imposes limitations on the information framework by necessitating the inclusion of predicate verbs in events. In reality, according to conventional definitions of events, nearly all events can be described as occurrences of actions. These actions are not exclusively triggered by predicate verbs; alternative forms such as gerunds, infinitives, and prepositional phrases can also signify the occurrence of actions. In other words, this aspect of the structure in event texts, once lemmatized, aligns with verbs denoting actions. Based on this analysis, this paper defines all elements in event texts that reflect the occurrence of actions as actions within the event. Actions may manifest in various components or forms within events, but they consistently denote the occurrence of actions, and events of similar types generally involve analogous actions.

While we have identified actions that reflect event types, we are unable to directly guide the model to extract this information solely based on the event texts. Consequently, we employ an LLM to capture the actions present within the events. The advanced comprehension and generalization abilities of the LLM facilitate a nuanced understanding of our objectives and enable the extraction of actions from the events through straightforward instructions. Specifically, we utilize GPT-4o for the task of extracting actions from each event. We provide the LLM with a prompt comprising the event text, extraction steps, and the desired output format. The LLM then generates the candidate actions as output.

4.1.2. Action Selection

Upon acquiring multiple candidate actions for an event, we find that these actions vary in their contribution to the discovery of event types. Some actions exhibit a strong correlation with event type information, while others show weak or negligible relevance, potentially obscuring or even undermining the clarity of event type information. Consequently, our objective in this step is to select those actions from the candidate pool that are most strongly associated with the event type to enhance the clarity of event type information.

In an entirely open and unannotated context, we assess the salience of actions using two key criteria: (1) Events of the same type are typically triggered by similar actions, so significant actions should be frequently observed among the candidate actions. (2) Significant actions associated with a specific event type should be infrequent in events of other types, ensuring that such actions are not overly common throughout the event corpus. Given a set of candidate actions $\mathcal{A}={\left\{A_i\right\}}_1^N$, where $A_i={\left\{a_j\right\}}_1^M$ is the set of candidate actions for the i-th event, and $a_j=[w_i,\dots ,w_k]$ is the token list for the j-th candidate action in $A_i$, based on the TF-IDF concept, the salience of each candidate action $a_j$ can be calculated as follows:

$$Salience\left(a_j\right)=\sum _{k=1}^K(1+log{\left(freq\left(w_k\right)\right)}^2)log({\displaystyle \frac{\left|\mathcal{S}\right|}{freq\_\mathcal{S}\left(w_k\right)}}),$$

(1)

where $freq\left(w_k\right)$ is the frequency of the word $w_k$ in the candidate actions, $\left|\mathcal{S}\right|$ is the number of event sentences in the corpus, and $freq\_\mathcal{S}\left(w_k\right)$ is the sentence frequency of the word $w_k$ in the corpus. By analyzing the salience of actions, we can select the action from the candidates that most effectively represents the event type information.

4.2. Stage II: Clustering Refinement and Dual-Fine-Tune

In this stage, we begin by employing two identical encoders to separately encode the text and actions, then we concatenate these vector representations to form the input for clustering. Following the initial clustering results, we aim to pinpoint samples with the highest levels of uncertainty and reassign them to alternative clusters. We leverage an LLM to refine the clustering outcomes, documenting the optimization process as triplets for the subsequent fine-tuning of the encoders. Specifically, we adjust the encoders considering both the original event text and the action perspectives to achieve optimal encoding results from different perspectives.

4.2.1. Entropy-Based Triplet Formation

Based on the results from Section 4.1.2, we initialize two encoders, $f_{\mathcal{S}}$ and $f_{⊣}$, to encode the text and action, respectively, resulting in the encoded vector sets ${\mathcal{H}}_{\mathcal{S}}={\{h_i=f_{\mathcal{S}}\left(s_i\right)\}}_{i=1}^N$ and ${\mathcal{H}}_{\mathcal{A}}={\{h_i=f_{\mathcal{A}}\left(a_i\right)\}}_{i=1}^N$. We then obtain the merged vector set $\mathcal{H}=\{h_i=h_s+h_a|h_s\in {\mathcal{H}}_{\mathcal{S}},h_a\in {\mathcal{H}}_{\mathcal{A}},i=s=a\}$ by directly summing the corresponding text and action encodings, which serves as input for clustering. We choose BERT as the encoder, and the initial two encoders have the same structure and parameters.

After obtaining the encoded set $\mathcal{H}$, we perform clustering on $\mathcal{H}$. Based on the clustering results, we compute the cluster centers $c_k$ by calculating the average of the embeddings assigned to each cluster k:

$$c_k={\displaystyle \frac{1}{N_k}}\sum _{i\in C_k}{h}_i,$$

(2)

where $c_k$ is the cluster center for cluster k, $N_k$ is the number of embeddings in cluster k, $C_k$ is the set of indices for embeddings in cluster k, and $h_i$ is an embedding in cluster $C_k$. Our goal is to identify the instances with the highest uncertainty within the clusters, which are considered the “most ambiguous” due to their highest uncertainty in membership between different clusters. Referring to previous work [13], we similarly identify the most ambiguous instances based on entropy as anchors. First, we use the Student’s t-distribution to obtain the probability distribution of each instance being assigned to each cluster:

$$p_{ik}={\displaystyle \frac{(1+||h_i-c_k{\left|\right|}^2{/\alpha )}^{-\frac{\alpha +1}{2}}}{{\sum }_{k^{\prime }}(1+||h_i-c_{k^{\prime }}{\left|\right|}^2{/\alpha )}^{-\frac{\alpha +1}{2}}}}.$$

(3)

We set the degrees of freedom in the formula to $\alpha =1$. This distribution allows an instance to belong to multiple clusters simultaneously, with varying probabilities for each cluster. According to the probability distribution $p_{ik}$, the cluster with the maximum probability distribution $p_{ik}$ is considered to be the K-nearest neighbors (KNN) cluster of instance i:

$$KNN=max(ϵK,2).$$

(4)

Here, $ϵ$ is set to a small value, and the specific setting should be adjusted according to the size of the dataset. According to the definition, the entropy E of a discrete random variable X is calculated using the formula

$$E\left(X\right)=-\sum _{i}P\left(x_i\right)logP\left(x_i\right),$$

(5)

where $x_i$ represents the possible values of X, and $P\left(x_i\right)$ is the probability of $x_i$. From Formulas (2)–(4), we can obtain the information entropy of instance i:

$$e_i=-\sum _{k=1}^{KNN}{\overline{p}}_{ik}log\left({\overline{p}}_{ik}\right),$$

(6)

Here, ${\overline{p}}_{ik}$ is the re-normalized probability, representing the normalized probability of instance i within the $KNN$ nearest clusters. At this point, we have obtained the set $\mathcal{E}={\left\{e_i\right\}}_{i=1}^Q$ of entropy values for each instance. By sorting the entropy values in set $\mathcal{E}$, we can select samples with higher entropy as our anchor points. These selected anchor points are the ambiguous samples in the clustering results, where the cluster membership is uncertain. After selecting the anchor point instances, we choose the two closest clusters from the clusters to which the anchor points belong, and then select one instance from each of these two clusters $X_1$ and $X_2$ as the two candidate instances to compare with the anchor point instance. In our experiments, we observe that configuring the number of triples Q to 1024 yields a substantial improvement on an event dataset comprising approximately 5000 instances.

4.2.2. Cluster Correction Based on LLM

According to Section 4.2.1, we obtain a triplet $(X_a,X_1,X_2)$, where $X_a$ is the instance in the clustering results that needs correction, and $X_1$ and $X_2$ are samples randomly selected from two different clusters, representing possible correction directions for $X_a$. We first specify the user perspective focusing on event types, then have the LLM make a selection to identify which of $(X_1,X_2)$ better aligns with $X_a$ in terms of event type. Based on the choice of the LLM, the selected sample becomes the positive candidate $X_{a^{+}}$, and the other becomes the negative candidate $X_{a^{-}}$. Thus, we obtain a new triplet $(X_a,X_{a^{+}},X_{a^{-}})$, which reflects the preferences of the LLM from the event type perspective and captures the user’s intent and task view. This will assist in the next step to fine-tune the encoder, ensuring that the generated embeddings focus on event type information, thereby achieving improved performance in clustering.

4.2.3. Fine-Tuning Embedder from Different Perspectives

To enhance clustering results that accurately reflect distinct event types, a fundamental approach is to ensure that the vector embeddings used in clustering are strongly aligned with the event types. Specifically, this involves making the embeddings rich in information relevant to event types. Building on this principle, we propose developing a task-specific fine-tuning procedure for the embedding model to create an embedding space that more effectively captures event type distinctions. For each instance, which has associated positive and negative samples, we define the fine-tuning task as a self-supervised contrastive learning problem. This task aims to enhance embedding quality by maximizing the similarity between anchor and positive examples, while minimizing similarity with negative examples. In line with prior research [20], for the triplet $(X_a,X_{a^{+}},X_{a^{-}})$, we seek to optimize the following objective:

$$L=-{\displaystyle \frac{1}{N}}\sum _{i=1}^{N}log{\displaystyle \frac{exp(f{\left(X_a\right)}^{\top }f\left(X_{a^{+}}\right)/\tau )}{{\sum }_{X^{-}\in \mathcal{B}}exp(f{\left(X_a\right)}^{\top }f\left(X^{-}\right)/\tau )}}.$$

(7)

Here, $\mathcal{B}$ represents all samples in a batch, including the positive $X_{a^{+}}$ corresponding to $X_a$ in the triplet and all negative examples $X^{-}$ within the batch. $\tau$ is the temperature coefficient used to adjust the distribution of similarity scores, controlling the model’s ability to distinguish between negative samples. For fine-tuning tasks under two different perspectives, the encoder f and the content of the triplets in the objective function change accordingly. In this paper, for the task-specific event type perspective, the instances in the triplets are composed of actions as described in Section 4.2.1, and the encoder is denoted as $f=f_{\mathcal{A}}$. In contrast, for the task-agnostic global information perspective, the instances in the triplets are composed of the original event texts, and the encoder is denoted as $f=f_S$.

The fine-tuned encoder can be used with our sampling method to identify triplets that contain more information and iteratively improve performance. We obtain new clustering assignments by running clustering algorithms on the adjusted embeddings, and then perform the next round of sampling and fine-tuning until the model converges on the task.

4.3. Stage III: Type Generation

In this stage, we leverage the final clustering outcomes to determine event types. Initially, we identify the Top K samples that are closest to the centroid within each cluster of the final clustering results. These samples are subsequently fed into the LLM, which, utilizing the semantic information from these K samples, produces a thematic summary and categorizes the information into specific event types.

Referring to Equation (2), we initially compute the centroid $c_k$ for each cluster. Based on these centroids, we compute the Euclidean distance between each sample within each cluster and its respective centroid. Using this distance metric, we select the top K samples that are closest to the centroid, thereby exemplifying the core characteristics and features of their respective clusters. Once identified, these samples are fed into the LLM alongside manually curated example types. This process aims to generate event type labels for each cluster. Subsequently, these labels undergo refinement by consolidating redundant themes and eliminating less frequent types. The manually curated example types, derived from a subset of samples within the corpus, do not feature in the final set of obtained event types. Their primary role is to guide the LLM in transforming textual information into types, thereby reflecting a semantic mapping from text to types.

5. Experiments

5.1. Datasets

We select two publicly available event datasets that both contain English event description texts from the general domain. ACE-2005 (https://catalog.ldc.upenn.edu/LDC2006T06, accessed on 1 June 2024) is a classic event dataset widely used in event research. ERE-EN [21] is another dataset from the Entities, Relations, and Events annotation task created under the Deep Exploration and Filtering of Test (DEFT) program because it covers more recent articles. We adopt the same data processing strategy as in the previous work [22]. The ACE dataset contains 3805 sentences with 33 event types and the ERE dataset has 4581 sentences with 38 types. We test the performance of our approach on event type discovery and event mention clustering. The statistics of the processed dataset are shown in

Table 1. The statistical information of the ACE-2005 and ERE-EN datasets, where T represents the number of event types, and N represents the number of samples.

Dataset	ACE-2005		ERE-EN
	$\left|\mathit{T}\right|$	$\left|\mathit{N}\right|$	$\left|\mathit{T}\right|$	$\left|\mathit{N}\right|$
Total	33	3805	38	4581
Mean	115.30		120.55
Stdev	206.32		169.58

.

5.2. Implementation

We implemented all algorithms using PyTorch [23] and Transformer [24] library for our experimental setup. In cases involving the use of LLM, we consistently employed GPT-4o within our methodology. The encoder bert-large-uncased was chosen as our initialization and fine-tuned specifically for our tasks. In Section 4.2.1 and Section 4.2.3, the number of clusters was set to match the number of ground-truth event types. We defined the number of triplets $Q=1024$ and the temperature coefficient $\tau =0.01$ in the optimization objective. We initialized the fine-tuning learning rate at 2 $\times {10}^{-6}$ with AdamW [25] serving as the optimizer. The maximum gradient step length was set to 1280, utilizing a batch size of 4, and training was conducted over 20 epochs. For the baseline model configurations, we kept their original parameter settings unchanged. All experiments in this article were conducted on a Linux server equipped with two RTX 3090 GPUs.

5.3. Evaluation Metrics

We evaluate our metrics from two aspects: event type discovery results and clustering results. For event type discovery, we follow the approach of [2,3] to manually inspect whether the discovered clusters can be mapped to the ground-truth event types. We assess the clustering results using several standard metrics.

We denote the ground-truth type label as $Y^{*}$, the predicted clusters label as Y, and the total number of event mentions as N.

Adjusted Rand Index (ARI) [26] measures the similarity between two clustering assignments based on the number of pairs that are consistently or inconsistently clustered. The formula is defined as follows:

$$ARI={\displaystyle \frac{RI-\mathbb{E}\left(RI\right)}{maxRI-\mathbb{E}\left(RI\right)}},$$

(8)

$$RI={\displaystyle \frac{TP+TN}{N}},$$

(9)

In this context, $TP$ refers to the number of element pairs within the same clusters in Y and $Y^{*}$, while $TN$ refers to the different. Meanwhile, $\mathbb{E}\left(RI\right)$ denotes the expected value of $RI$ under random clustering.

Normalized mutual information (NMI) quantifies the normalized mutual information between two clustering assignments. The formula is defined as follows:

$$NMI={\displaystyle \frac{2\times MI(Y^{*};Y)}{H\left(Y^{*}\right)+H\left(Y\right)}},$$

(10)

where $MI$ denotes the mutual information between the two clustering assignments, and H represents entropy.

Accuracy (ACC) measures the clustering quality by finding the permutation function that maps predicted cluster IDs to the ground-truth IDs with the highest accuracy. Let $y_i$ and $y_i^{*}$ denote the clustering label and the true label of the i-th instance, respectively. The calculation of ACC is defined as follows:

$$ACC=\underset{\sigma \in Exchange\left(k\right)}{max}{\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{I}(y_i^{*}=\sigma \left(y_i\right)),$$

(11)

where $Exchange(·)$ refers to the permutation function. In this paper, we use the Hungarian algorithm as the permutation function, and $\mathbb{I}$ represents the indicator function.

For all four metrics, higher values indicate better model performance.

5.4. Baselines

We compare our method with the following approaches in a fully open zero-shot setting:

• SS-VQ-VAE [1]: A semi-supervised zero-shot event detection model. It employs a variational autoencoder to learn discrete event features. SS-VQ-VAE is trained based on seen event types and annotations, and it can be applied to produce clusters on unseen types.
• Triframes [27]: A graph-based clustering algorithm that constructs a k-nearest neighbor (k-NN) event mention graph and utilizes the fuzzy graph clustering method WATSET to produce clusters.
• JCSC [3]: A joint constrained spectral clustering approach that iteratively refines clustering outcomes through the use of constraint functions, ensuring that predicates and objects with mutual dependencies form coherent clusters.
• ETYPECLUS [2]: A novel event type information P-O (predicate–object) pair is introduced, and a clustering algorithm is redefined based on these P-O pairs. Furthermore, a joint embedding and clustering algorithm in latent space is proposed.
• ESHer [16]: An event schema harvester. It generates high-quality event schemas by employing context-aware conceptualization, credibility-informed schema structuring, and graph-based schema clustering techniques.
• TABS [5]: An open event type discovery with type abstraction. The notion of type abstraction is introduced through the development of a collaborative training framework that generalizes information from known types to previously unknown types.

We evaluate these baseline models under identical open conditions, where only raw event text is provided and access to any ground-truth information is restricted. Performance assessments are conducted for all models within this setup.

6. Results and Analysis

6.1. Main Results

Event Clustering Result. The primary results of the event clustering are presented in

Table 2. Event clustering results shown as percentages. Average and standard deviation from 5 random experiments (ESHer run once due to high cost). Triframes cannot be assessed by accuracy, as they assume that ground truth clusters match generated clusters.

Dataset	Model	ACC	NMI	ARI
ACE-2005	SS-VQ-VAE (Huang and Ji [1])	21.19 ± 0.93	25.66 ± 0.72	10.24 ± 0.37
	Triframes (Ustalov et al. [27])	-	26.83 ± 0.33	13.54 ± 0.43
	JCSC (Huang et al. [3])	25.39 ± 0.66	37.5 ± 0.56	29.04 ± 0.82
	ETYPECLUS (Shen et al. [2])	39.62 ± 0.73	48.35 ± 0.41	33.29 ± 0.05
	ESHer (Tang et al. [16])	40.71 ± 0.00	48.29 ± 0.00	27.83 ± 0.00
	TABS (Li et al. [5])	42.23 ± 1.32	44.19 ± 0.47	34.41 ± 0.46
	proposed	53.25 ± 1.11	59.75 ± 0.65	43.53 ± 0.99
ERE-EN	SS-VQ-VAE (Huang and Ji [1])	14.38 ± 0.77	21.33 ± 0.04	13.79 ± 0.48
	Triframes (Ustalov et al. [27])	-	13.46 ± 0.11	8.31 ± 0.72
	JCSC (Huang et al. [3])	23.44 ± 0.89	29.05 ± 0.21	12.22 ± 0.39
	ETYPECLUS (Shen et al. [2])	31.56 ± 0.95	32.04 ± 0.19	18.09 ± 0.55
	ESHer (Tang et al. [16])	35.53 ± 0.00	47.2 ± 0.00	20.03 ± 0.00
	TABS (Li et al. [5])	30.73 ± 1.77	36.81 ± 0.44	21.34 ± 0.96
	proposed	49.62 ± 1.16	50.29 ± 0.38	34.53 ± 0.69

. We evaluate our method against baseline models using two datasets. The results demonstrate that our approach attains superior performance across all three clustering evaluation metrics when applied to a completely open event corpus. Compared to other baseline methods, our model’s advantage lies in integrating user intent and task perspectives through LLMs throughout the clustering process, which enables the model to concentrate on event-related information at each stage. In this context, ESHer exhibits suboptimal performance since this method also employs LLMs to generate abstract event pattern graphs during the initial phase. The application of LLMs in event clustering effectively mitigates the challenges of insufficient auxiliary information and supervision signals in open domains. LLMs offer targeted support under specific task perspectives, allowing the model to directly address event types, and thus outperforming other methods.

Event Type Discovery. Our approach was applied to each of the two datasets to identify candidate event clusters. Following the methodology of Huang et al. [3], we manually verified whether the discovered clusters align with ground-truth event types. In ACE-2005, we successfully identified 27 out of 33 event types, while in ERE-EN, we identified 20 out of 38 event types. We present a selection of recovered event types in the

Table 3. Examples of event types and corresponding sentences identified by our method in the ACE-2005 and ERE-EN datasets. The first three types are from ACE-2005, and the last three are from ERE-EN.

Type Discovery (Ground-Truth Type)	Number of Samples (Ground-Truth Number)	Action	Example Sentences
Election (Personnel:Elect)	72 (131)	·win election ·re-elect president	·You become honorable by winning an election of some sort or having a high post, being an ambassador or a governor or some thing. ·my focus is on re-electing president bush and dick cheney next year, the convention is going to be here in the city of new york.
Injury (Life:Injure)	65 (80)	·cause injury ·suffered injury	·Her parents said they don’t know yet what caused their daughter’s injuries. ·it took the child three weeks to die in the hot from – hospital from the injuries it suffered at the hands of its mother and she’s free.
Nomination (Personnel:Nomimate)	7 (10)	·candidate for the nomination ·name a new head	·Senator Christopher Dodd of Connecticut made the announcement today that he would not be the 10th candidate for the nomination. ·and the pope will reportedly name a new head of the troubled boston archdiocese this week.
Meet (Contact:Meet)	130 (198)	·chair meeting ·have appointment	·Typically, the lieutenant governor would chair meetings of the council, a fractious and often rebellious panel that dates to the Colonial era. ·I had my first doctor appointment at 10 weeks, then at 13 and today at 18.
Departure (Personnel:End-Position)	79 (140)	·get fired ·be release	·Now he’s gotten in trouble for it, and I hope that if his intentions were what I think they were, he gets fired. ·Martin-Artajo, a Spaniard who worked in the bank’s London office, was released soon after his surrender and arrest.
Divorce (Life:Divorce)	13 (21)	·divorce ·be divorced	·The couple divorced, remarried and divorced again. ·My perspective is coming from the fact that I’m divorced and my XH tried to milk me for everything I’m worth, even though he cheated on me.

, including example events and their associated actions. From the results, it is evident that our method accurately identifies ground-truth event type labels, demonstrating a strong correlation between actions and event types, thereby effectively capturing event type information.

6.2. Ablation Analysis

To explore the influence of different stages on the final performance, we conducted ablation experiments primarily targeting the initial two stages. Specifically, we separately removed the action extraction and action selection steps to evaluate their impact on performance. Moreover, during the fine-tuning phase, we performed experiments with only one perspective maintained at a time—either the task-specific or the task-agnostic perspective—to assess how information from different perspectives affects performance. The results of these ablation experiments are presented in

Table 4. Ablation experiment results. “w/o” indicates removed model parts. Bold values show the best results among models.

Model	ACE-2005			ERE-EN
	ACC	NMI	ARI	ACC	NMI	ARI
Ours	53.25	59.75	43.53	49.62	50.29	34.53
w/o action extraction	39.53	49.25	35.76	38.93	42.8	21.37
w/o action selection	47.82	54.31	38.29	41.23	46.62	26.18
only action-fine-tune	48.94	55.06	40.67	45.12	48.23	30.62
only sentence-fine-tune	44.95	52.83	38.14	42.07	43.31	28.74

.

The results indicate that the removal of action information has a significant impact on the final performance. Action information, which reflects the task perspective and is highly correlated with event types, is crucial for capturing and utilizing key information in an open domain. Its absence substantially reduces the method’s effectiveness. Furthermore, retaining all action information may also adversely affect performance. This is because an event can involve multiple actions, each with varying degrees of relevance to the event type; including actions with weak relevance can introduce noise and degrade performance.

In the fine-tuning task, experimental results demonstrate that action-only fine-tuning yields better performance compared to text-only fine-tuning. This indicates that fine-tuning the model from a task-specific perspective enhances its ability to focus on event type information. Post-fine-tuning, the encoder is more attuned to the type information within events, though this may result in the loss of some global information that also reflects event types. Fine-tuning exclusively from a global perspective leads to a more pronounced decline in performance because the encoder attends to all semantic information in the event, thereby neglecting the local information pertinent to event types, which does not align with the objectives of our task.

6.3. Clustering Algorithms

We also evaluated the performance of different clustering algorithms by replacing them within the method, as detailed in the

Table 5. The results of applying different clustering algorithms. We use five random seeds, and report the average and standard deviation as the final outcomes.

Clustering Algorithm	ACE			ERE-EN
	ACC	NMI	ARI	ACC	NMI	ARI
AggClus	37.53 ± 0.69	46.92 ± 0.81	31.73 ± 0.72	30.22 ± 0.65	39.71 ± 0.58	22.04 ± 0.99
DBSCAN	39.21 ± 1.05	51.15 ± 0.76	37.29 ± 0.83	31.03 ± 1.09	43.12 ± 0.79	28.52 ± 0.94
GMM	31.49 ± 0.71	42.06 ± 0.53	29.52 ± 0.39	27.39 ± 0.94	30.26 ± 0.87	24.18 ± 0.47
K-Means (ours)	53.25 ± 1.11	59.75 ± 0.65	43.53 ± 0.98	49.62 ± 1.16	50.29 ± 0.38	34.53 ± 0.69

. Specifically, we compared K-Means with three alternative algorithms: Agglomerative Clustering (AggClus), DBSCAN, and Gaussian Mixture Model (GMM). Our findings indicate that K-Means significantly outperforms the other three algorithms. The BERT-encoded text vectors generally exhibit a relatively homogeneous distribution in high-dimensional space, enabling K-Means to utilize Euclidean distance effectively for clustering. The K-Means optimization technique is well suited for handling such high-dimensional data and can exploit the semantic information inherent in the encoded vectors to produce more precise clusters. Conversely, DBSCAN is adept at identifying clusters of varying shapes and demonstrates some robustness to noise. Given that BERT vectors typically possess certain density characteristics, DBSCAN can effectively detect dense cluster regions. However, its performance is highly sensitive to parameter settings, which can lead to considerable variability in results. Agglomerative clustering, while capable of generating hierarchical structures, may suffer from computational complexity issues when applied to high-dimensional text vectors. The computational overhead and distance calculations involved in hierarchical clustering can result in less accurate outcomes compared to other algorithms, particularly with large datasets. Additionally, BERT vectors often deviate from the Gaussian distribution assumption, which affects GMM performance adversely. The distribution of text data generally does not conform to Gaussian models, especially in high-dimensional spaces, leading to the poorest performance of GMM, as it struggles to accurately capture the underlying distribution of text vectors.

6.4. Qualitative Analysis

Figure 2 illustrates the results of five iterations of the proposed method. Throughout the iterative process, we sample triplets from the previously fine-tuned embedding space and proceed with further fine-tuning of the model, using the preceding checkpoint as the initialization point. The performance variations following the fine-tuning of the embedder are presented from two distinct perspectives. Our analysis reveals that performance generally improves and reaches a plateau by the fifth iteration. Specifically, for task-specific perspectives, the model exhibits faster convergence and more significant enhancements. In contrast, for task-agnostic perspectives, the convergence rate is notably slower.

We further assess the performance of the method by visualizing the clustering results as illustrated in Figure 3. This involves comparing the initial clustering outcomes with those obtained after iterative refinement on two datasets. The figure reveals that the application of our method leads to a notable enhancement in clustering quality. There is a marked improvement in the separation between different types, and the representation of event types within the clusters becomes significantly more precise.

7. Limitation

In this paper, we employ a synergistic approach that integrates both large and small models to identify event types within open-domain contexts. While this methodology offers several advantages, it also presents notable limitations that impact its effectiveness and generalizability. One significant challenge is the tuning of LLMs, which can be inherently unpredictable and difficult to fine-tune. The variability in LLM outputs can introduce biases and propagate errors throughout the event type discovery process, affecting the consistency and reliability of the results. Additionally, the LLM outputs may reflect biases present in the training data, which can skew the event type discovery and limit the model’s ability to generalize across diverse domains.

Furthermore, the construction of fine-tuning tasks involves sampling anchor points, which currently lacks iterative refinement. This static sampling strategy can lead to a decrease in the relevance and informational value of anchor points as the clustering process evolves. As a result, the effectiveness of clustering may diminish over time, impacting the overall performance of the method. Addressing these limitations requires ongoing refinement of the tuning process and sampling strategies to enhance the robustness and fairness of our approach.

8. Conclusions

In this paper, we introduce a synergistic approach that integrates large and small models to address the challenge of event type discovery in open-domain contexts. This approach effectively addresses the critical issues encountered in existing methodologies, such as the lack of scenario openness and the weak alignment between clustering outcomes and task objectives. The LLM offers perspective information that is aligned with the task goals throughout the event type discovery process, and formulates fine-tuning tasks for the small model to leverage this information. This approach ensures that intermediate information generated at each stage is highly pertinent to the event types, thereby enhancing overall performance. Empirical results indicate that our method can precisely reconstruct event types in completely open domains without relying on any prior information.