EANT: Distant Supervision for Relation Extraction with Entity Attributes via Negative Training

Abstract

Distant supervision for relation extraction (DSRE) automatically acquires large-scale annotated data by aligning the corpus with the knowledge base, which dramatically reduces the cost of manual annotation. However, this technique is plagued by noisy data, which seriously affects the model’s performance. In this paper, we introduce negative training to filter them out. Specifically, we train the model with the complementary label based on the idea that “the sentence does not express the target relation”. The trained model can discriminate the noisy data from the training set. In addition, we believe that additional entity attributes (such as description, alias, and types) can provide more information for sentence representation. On this basis, we propose a DSRE model with entity attributes via negative training called EANT. While filtering noisy sentences, EANT also relabels some false negative sentences and converts them into useful training data. Our experimental results on the widely used New York Times dataset show that EANT can significantly improve the relation extraction performance over the state-of-the-art baselines.

1. Introduction

Knowledge graphs (KGs) describe real-world concepts and their relations in graphs and play an essential role in natural language processing tasks, such as recommendation systems [1,2], search engines [3], and Q&A systems [4,5]. Currently, open-source available KGs such as FreeBase [6], DBPedia [7], and Wikidata [8] have been widely used. Although they already contain hundreds of millions of relation facts, they still cannot cover the nearly infinite number of facts in the real world. To improve the completeness of KGs, relation extraction (RE) is proposed to automatically extract the relation between two given entities from unstructured text.

Most existing supervised RE methods require large amounts of annotated data, which is often expensive and time-consuming. To address this problem, Mintz et al. [9] proposed a distant supervision (DS) approach to generate training data automatically. It heuristically aligns the corpus with the KGs and assumes that if two entities have a relation in the KGs, then sentences that contain both entities may express the relation. However, it is clear that this assumption is too strong and inevitably introduces noise problems, such as false positives (FP) and false negatives (FN). The former is because the sentence does not actually express the target relation but is incorrectly marked as positive; the latter is due to the sentence actually expressing the target relation but being marked as negative because of the missing corresponding facts in the KGs. These two types of noise seriously affect the RE performance of the model.

The current noise reduction methods can be broadly classified into four categories: (1) feature-based methods that manually construct suitable features and filter noise using machine learning; (2) representation-enhancing methods that improve the contribution of valid sentences in the dataset using attention mechanisms [10], extra information [11], pre-trained models [12], etc.; (3) noise-filtering methods that filter out noisy sentences using reinforcement learning [13,14], adversarial learning [15], pattern matching [16,17], etc.; (4) relabeling methods that assign correct relation labels to sentences using soft labeling [18], re-labeling [19], noise modeling [20], etc.

However, none of these approaches take full advantage of the rich information contained in entity attributes (e.g., description, alias, and types). Such information can provide richer background knowledge of entities for sentence representation, and incorporating them can improve the performance of the model. As shown in Figure 1, given the sentence “Larry Page and Sergey Brin founded Google in 1998 on the back of PageRank algorithm.”, the task of RE is to discover the relation “Founder_of” between the entity “Larry Page” and “Google”. From this example, we can clearly see the role of entity attribute information: the description of “Larry Page” mentions that “Larry Page is the founding CEO of Google”, which undoubtedly provides valid evidence for the relation “Founder_of”. Additionally, the alias of “Google” as “Google Inc” can provide more information of “Google is a company” to the entity “Google”. By extracting these features, we can effectively determine similar relations. Finally, types information can reduce the probability that sentences are classified as impossible relations. For example, if “Larry Page” is an instance of “person” and the tail entity “Google” is an instance of “company”, then it definitely not be an instance of the relation “Place_of_birth”.

Moreover, since the DSRE dataset is automatically generated from aligned KGs, previous methods often lack explicit supervised information to capture the noise. Therefore, we propose to introduce negative training (NT) [21] to train the noise selector to achieve the separation of noisy data. Different from positive training (PT), negative training is based on the idea that “the sentence does not express the target relation”, and the complementary labels are selected to train the model. A complementary label indicates a label other than the original label. As shown in Figure 1, the correct label for the sentence should be “Founder_of”. However, under the strong assumption of DS, the sentence may be marked as other labels, resulting in incorrect information provided to PT. In negative training, the sentence is assigned a complementary label other than “Founder_of”, such as “Place_of_birth”, to train the model that the sentence is not “Place_of_birth”. Since the probability of selecting a real label as a complementary label is low, it is assumed that the label provides the correct information to NT in this case. In other words, the relation expressed in the sentence may not be “Founder_of”, but it is certainly not “Place_of_birth”. The resulting model is able to separate the noisy data from the training data.

In this paper, we propose a DSRE model with Entity Attributes via Negative Training, called EANT. Furthermore, we argue that some of the noisy data simply lack correct labels but still contain useful information. If reliable labels can be assigned to these sentences, they will be transformed into useful training data, which will help improve the model’s performance. Therefore, we relabel the noisy sentences according to their confidence levels.

The contributions of this paper can be summarized as follows:

• We incorporate additional entity attribute information to provide more information for sentence representation. To reflect the importance of different attributes, we use knowledge graph embedding to assign weights to them.
• We introduce negative training into the DSRE task and train the model with the complementary label. By negative training, the model is able to widen the confidence gap between clean and noisy data to achieve noise filtering.
• The experimental results on the New York Times dataset show that the proposed approach can significantly improve the relation extraction performance over the state-of-the-art baselines.

2. Related Work

Mintz et al. [9] proposed distant supervision to address the lack of large-scale annotated data in relation extraction supervision methods. The method aligns the KGs with the corpus and labels sentences containing entity pairs from the KGs as the relation in the KGs. Although this approach generates annotated data quickly without manual effort, the generated data contains much noise, which degrades the performance of the RE model. To solve the noise problem, Riedel et al. [22] proposed the at-least-once assumption, where sentences with the same entity pairs were formed into a bag. It is the bag that expresses the relation rather than an individual sentence. Meanwhile, they designed lexical features, syntactic features, named entity labeling features, and predicted relations using machine learning methods, improving the model’s results.

Although these feature-based approaches are well designed, this is not enough. The NLP tools they used had inevitable errors, which led to the propagation or accumulation of errors. With the successful application of deep learning methods, scholars have started to extract features using deep neural networks.

These approaches can be broadly classified into the following three categories: the first category of approaches makes full use of the non-noisy sentences in the dataset to extract rich information to generate more accurate representations for the sentences. Lin et al. [10] used sentence-level selective attention to assign different weights to each sentence in the dataset to distinguish their impact. Ji et al. [11] used entity description information as supplementary background knowledge to strengthen the learning of entity representation. Jat et al. [23] argued that only a few words in a sentence were relevant to the expressed relation and assigning higher weights to these words can help to improve the RE performance.

The second category of approaches attempts to filter out noisy sentences from the original dataset to eliminate their influence. Feng et al. [13] and Qin et al. [14] trained an instance selector using reinforcement learning with a policy function to decide whether to keep an instance or not. At the same time, the performance of the relation classifier was readjusted as a reward or penalty for reinforcement learning. Qin et al. [15] chose generative adversarial networks to train generators and discriminators that achieved similar noise filtering effects. Jia et al. [16] extracted the expression pattern of the relation from the sentences and removed the sentences that did not match pattern as noise.

The third category of approaches is trained with noisy data and aims to generate relation labels for instances that better match the true expression of the sentence. Liu et al. [18] proposed a soft-label noise reduction method that dynamically assigned a new label to a sentence during the training period based on the semantic information of the sentence. Shang et al. [19] used unsupervised deep clustering to generate reliable labels for noisy sentences. Luo et al. [20] described the noise patterns by dynamically generating a transition matrix, which indicated the possibility that the DS-labeled relation was confused for each instance.

In this paper, we combine the features of the above three approaches and incorporate the attributes of entities as additional information into the model to obtain a more accurate sentence representation. Meanwhile, we use the NT method to train a noise filter to filter noisy sentences from the dataset. After filtering out the noisy sentences, we assign possible labels according to their confidence levels to utilize the information in them.

3. Methodology

In this section, we will introduce our proposed DSRE model, EANT, in detail. The structure of the model is shown in Figure 2. It contains three modules: a sentence encoder, an entity attribute encoder and a negative training process. The sentence encoder is used to convert sentences into corresponding distributed representations. The entity attribute encoder learns entity representations incorporating entity attribute information. A noise filter is trained by negative training, and it is decided whether the filtered noise sentence needs to be relabeled according to its confidence.

3.1. Problem Definition

Before we formally start to introduce the model, we first give the problem definition. We denote the dataset obtained by DS as ${\mathcal{D}}^{*}$ = {${\mathcal{S}}^{*}$, $\mathcal{R}$}, where ${\mathcal{S}}^{*}$ = {$<s_1,r_1^{*}>,\dots ,<s_n,r_n^{*}>$} is the set of sentences $s_i$ expressing the relation $r_i^{*}$. $s_i$ is a sentence associated with two entities $<e_1,e_2>$, and $r_i^{*}\in \mathcal{R}$ is the noisy relation label of sentence $s_i$ generated by DS. Following the at-least-once assumption, sentences with the same entity pair are packaged into a bag $\mathcal{B}=\left\{\left\{s_1,s_2,...,s_m\right\},r_i^{*}\right\}$, indicating that this bag of sentences expresses the target relation $r_i^{*}$. The goal is to select sentences from ${\mathcal{S}}^{*}$ that correctly express the labeled relations to form a clean dataset $\mathcal{D}$ = {$\mathcal{S}$, $\mathcal{R}$}, where $\mathcal{S}={\mathcal{S}}^{*}-{\mathcal{S}}_{noise}$. We then train the relation classifier on the dataset $\mathcal{D}$ to predict the relation of each bag.

3.2. Sentence Encoder

For the input sentence $s=\{w_1,w_2,...,w_n\}$, we first transform it into a low-dimensional vector consisting of word embeddings and position embeddings. The word embeddings map each token $w_i$ in the sentence to a k-dimensional vector of real values to form distributed representations of the words. This distributed representation of words has been shown to greatly capture syntactic and semantic information as well as similarity properties [24]. It can be obtained by looking up the embedded representation matrix, e.g., Word2vec (https://code.google.com/p/word2vec/, accessed on 1 September 2022) and GloVe [25]. The position embeddings are a p-dimensional vector proposed by Zeng et al. [26] to represent the relative distance of each word to the head and tail entity and are widely used in RE tasks. These two embeddings are then concatenated together to form the initial representation of the sentence $\mathbf{s}\in {\mathbb{R}}^{n\times (d_k+2d_p)}$.

BiLSTM has been proved to extract complete, sequential information of every word in a sentence. Therefore, we use BiLSTM to learn the semantic features of sentences. The hidden state of the BiLSTM is denoted as:

$$\mathbf{H}={\left({\mathbf{h}}_{\mathbf{1}},{\mathbf{h}}_{\mathbf{2}},\dots ,{\mathbf{h}}_{\mathbf{N}}\right)}^T,$$

(1)

where ${\mathbf{h}}_{\mathbf{i}}=[\overrightarrow{h_i}:\overleftarrow{h_i}]$ is a concatenation of the forward hidden state $\overrightarrow{h_i}$ and the backward hidden state $\overleftarrow{h_i}$ for the i-th word. N is the sentence length. T is the transpose operation.

We then input $\mathbf{H}$ to the attention layer to distinguish the importance of different words. The sentence representation $\mathbf{s}$ obtained by the sentence encoder is denoted as:

$$\begin{array}{c}\mathbf{M}=tanh\left(\mathbf{H}\right),\\ \mathbf{a}=softmax\left({\mathbf{w}}^T\mathbf{M}\right),\\ \mathbf{s}=\mathbf{H}{\mathbf{a}}^T,\end{array}$$

(2)

where ${\mathbf{w}}^T$ is a trained parameter vector.

3.3. Entity Attributes Encoder

Entity attributes can provide rich background knowledge for an entity, which can help improve the performance of RE. We extract entity descriptions, entity aliases, and entity types as shown in Figure 1 from the public database of Freebase [6] and Wikidata [8]. We first query the word embeddings of these attributes and input them to an attention-based BiLSTM encoder to obtain a vector representation ${\mathbf{s}}_{\mathbf{a}}$ of each attribute, as $\mathbf{s}$ in Equation (2).

Furthermore, we note that not all attribute information is beneficial for RE, which may contain irrelevant information and introduce some additional noisy information. Therefore, we propose to use knowledge graph embedding TransE [27] to discriminate useful attributes. TransE treats relations as a translation from head entity $e_h$ to tail entity $e_t$, i.e., $e_h+r\approx e_t$. Therefore, the relation can be approximated in the following way: $\mathbf{r}={\mathbf{e}}_{\mathbf{t}}-{\mathbf{e}}_{\mathbf{h}}$. We calculate the weight of each attribute by multiplying ${\mathbf{s}}_{\mathbf{a}}$ with $\mathbf{r}$. The final representation of the entity attributes information is denoted as follows:

$$\begin{array}{cc}\hfill {\alpha }_i& =\frac{exp\left({\mathbf{s}}_{\mathbf{a}}·\mathbf{r}\right)}{{\sum }_{i=1}exp\left({\mathbf{s}}_{\mathbf{a}}·\mathbf{r}\right)},\hfill \\ \hfill {\mathbf{d}}_e& =\sum _{i=1}^l{\alpha }_i{\mathbf{s}}_{\mathbf{a}},\hfill \end{array}$$

(3)

where ${\alpha }_i$ is the attention weight of attribute ${\mathbf{s}}_{\mathbf{a}}$. The entity attribute embedding ${\mathbf{d}}_e$ is calculated by the weighted average of all attributes.

Finally, we concatenate the sentence encoder output with the entity attribute encoder output to obtain the new sentence representation. It can be expressed as:

$$\overline{\mathbf{s}}=[\mathbf{s}:{\mathbf{d}}_{\mathbf{h}}:{\mathbf{d}}_{\mathbf{t}}].$$

(4)

To calculate the confidence level for each relation, we feed the sentence representation into a softmax classifier after a linear transformation, as follows:

$$\mathrm{p}\left(r\right|s,\theta )=Softmax\left({\mathbf{M}}_s\overline{\mathbf{s}}+{\mathbf{b}}_s\right),$$

(5)

where ${\mathbf{M}}_s$ is the transformation matrix and ${\mathbf{b}}_s$ is the bias.

3.4. Negative Training

Our goal is to enable the model to be trained robustly on noisy datasets, so we need to distinguish between noisy data and clean data as much as possible. In other words, we aim to widen the gap between the confidence of clean data and noisy data. In positive training, which is based on the idea that “the sentence does express the target relation”, the cross-entropy loss function is often defined as follows:

$${\mathcal{L}}_{PT}=-\sum _{i=1}^{\left|\mathcal{R}\right|}{r}_{i}log\mathrm{p}\left(r_i\right|s,\theta ),$$

(6)

where $\left|\mathcal{R}\right|$ denotes the number of relations in relation set $\mathcal{R}$, and $r_i$ is an $\left|\mathcal{R}\right|$-dimensional one-hot vector. As the loss decreases, the probability of a sentence to a given label will be close to 1.

In contrast, the negative training is based on the idea that “the sentence does not express the target relation”, and the complementary label is chosen to train the model. Specifically, for an input s with label $r_i\in \mathcal{R}$, we randomly sample a complementary label $r^{*}$ from $\mathcal{R}$, but not $r_i$. Since we want the input to be far away from this complementary label, that is, to make its probability far away from 1. Therefore, we propose the loss function as follows:

$${\mathcal{L}}_{NT}=-\sum _{i=1}^{\left|\mathcal{R}\right|}{r}^{*}log(1-\mathrm{p}\left(r^{*}\right|s,\theta )).$$

(7)

This allows the probability value of the complementary labels to be optimized to zero, so that the probability value of other classes increases, achieving the purpose of NL.

3.5. Noise Filtering and Relabeling

After negative training, the model is able to distinguish between noisy data and clean data. We can simply set thresholds to filter out the noise in them. However due to the serious long-tail problem in the DSRE task, it is necessary to set a corresponding threshold for each relation to prevent the tail relation from being filtered out. Here, we propose a simple method for setting filtering thresholds:

$$T{h}_d=Th·\underset{i=1}{\overset{N}{max}}\left\{p_r^i\right\},$$

(8)

where $Th$ is a global threshold, and $p_r^i$ denotes the probability value of an instance with relation r. N is the number of all instances with relation r.

As mentioned above, some of these filtered sentences may simply lack the correct labels but still contain useful information on their own. When these sentences are assigned reliable labels, they will become useful training data. Similarly, we devise a simple strategy to relabel these sentences with the following equation:

$$r_i^{\prime }=\underset{r}{argmax}\left\{p_r^i>{Th}_r\right\},$$

(9)

where ${Th}_r$ is the relabel threshold. Note that we only relabel those false-negative sentences. Finally, we perform positive training on the clean dataset to obtain the final relation classifier.

4. Experiments

4.1. Dataset and Evaluation Metrics

We evaluate the performance of the proposed EANT model on a widely used distant supervised dataset, New York Times (NYT). It was generated by aligning Freebase with the New York Times corpus. The dataset contains a total of 52 relations and an NA label, which indicates no relation between entities. The detailed statistics of this dataset can be found in

Table 1. Statistics of NYT dataset.

NYT	Train	Test
#Sentences	522,611	172,448
#Entity pairs	281,270	96,678
#Triplets	18,252	1950
#Relations	53	53

.

In this work, we perform a hold-out evaluation [10] to demonstrate the effectiveness of the model. The hold-out approach compares the relations predicted by the model with those in Freebase. Specifically, we plot the precision-recall (PR) curve to show the trade-off between precision and recall and report the precision at Top N (P@N) metric to consider the accuracy values at different cutoffs. The area enclosed by the PR curve and the two axes intuitively represents the performance of the model, i.e., the larger the area of the PR curve, the better the performance of the model. P@N is the accuracy performance of the top N predictions with the highest prediction scores after sorting the results in descending order. Therefore, a higher value of P@N indicates better performance of the model.

4.2. Baselines

We compare with several typical baselines for DSRE tasks, as follows:

• Mintz: Mintz et al. [9] proposed a multi-class logistic classifier optimized using L-BFGS with Gaussian regression model for DSRE.
• MultiR: Hoffmann et al. [28] proposed an undirected graphical model based on factor graphs and used the SampleRank framework for multi-instance learning.
• MIMLRE: Surdeanu et al. [29] jointly modeled multi-instance and multi-label learning with a graphical model.
• PCNN: Zeng et al. [30] proposed to introduce a piecewise max-pooling mechanism based on traditional convolutional neural networks (CNN) to enhance the representation of sentences.
• PCNN+ATT: Lin et al. [10] proposed a PCNN-based model and distinguished the importance of different sentences using sentence-level attention.
• PCNN+ATT+Soft: Liu et al. [18] proposed a PCNN-based model which dynamically generated a soft label to replace the hard label as the gold label for each bag during the training process.
• BGWA: Jat et al. [23] proposed a word-level attention model based on bi-directional gated recurrent unit (Bi-GRU) to identify key phrases in the sentence.
• RESIDE: Vashishth et al. [31] proposed a graph convolution network (GCN)-based model which employed relevant side information and syntactic information for DSRE.
• A2DSRE: Shi et al. [32] proposed a DSRE framework based on adaptive dependency-path and additional KG supervision.

4.3. Experimental Settings

In the experiment, we adjusted other hyperparameters on the training data using the cross-validation method to minimize the loss and determine the hyperparameters of the model via a grid search. After adjustment, we chose the following settings: we used a 50-dimensional glove vector (https://github.com/stanfordnlp/GloVe, accessed on 1 September 2022) as the word embedding for initialization. The number of BiLSTM network layers was set to 1, and the size of the hidden vector was set to 128. We use the stochastic gradient descent (SGD) algorithm for optimization, and the learning rate was set to 0.05.

Table 2. Parameter settings.

Parameter	Value
Word dimension, $d_w$	50
Position dimension, $d_p$	5
Hidden layer dimension, $d_h$	128
Learning rate, $\alpha$	0.05
Batch size, b	128
Global filter threshold, $Th$	0.25
Relabel threshold, $T{h}_r$	0.7
Dropout rate, $p_{drop}$	0.5

shows all the parameters used in our experiments.

4.4. Results

As shown in Figure 3, we plot the P-R curve comparison results of our model against several baselines on the NYT dataset. From the figure, we can observe that (1) the neural models outperform human feature-based methods, which illustrates the limitations of manually designed features and the advanced nature of neural network methods; (2) compared with PCNN and PCNN+ATT, the PCNN+ATT+Soft model achieves better performance, which indicates that assigning suitable labels to noisy sentences can make full use of the information contained in them; (3) RESIDE and A2DSRE achieve sub-optimal results, which indicates that utilizing additional information in the knowledge graph can effectively improve the performance of model relation extraction; (4) our approach achieves the best performance compared to the other baseline models. This demonstrates the effectiveness of incorporating entity attribute information and negative training. The entity attribute information can provide richer information for entity representation, and the negative training can effectively filter out the noisy data in the dataset and improve the quality of the training data.

Following previous work [10], we also evaluated our method on the P@N metric, which is the precision of the top N sentences ranked by prediction confidence. Here, we report P@100, P@200, P@300 and their averages for each model, and the comparison results are shown in

Table 3. P@N of EANT and baselines.

Method	P@100	P@200	P@300	P@Mean
Mintz [9]	54.0	50.5	45.3	49.9
MultiR [28]	75.0	65.0	62.0	67.3
MIMLRE [29]	70.0	64.5	60.3	64.9
PCNN [30]	72.3	69.7	64.1	68.7
PCNN+ATT [10]	76.2	73.1	67.4	72.2
PCNN+ATT+Soft [18]	87.0	84.5	77.0	82.8
BGWA [23]	82.0	75.0	72.0	76.3
RESIDE [31]	84.0	78.5	75.6	79.4
A2DSRE [32]	87.0	79.0	77.6	81.2
EANT(ours)	87.6	83.2	78.6	83.1

. The best value is shown in bold, while the second best value is underlined. From the table, we can observe that our model achieves the best performance on most of the metrics. Specially, compared with the latest baseline model (i.e., A2DSRE), EANT achieves a 2.34% improvement on the P@Mean metric. From this, we can conclude that our model can extract richer sentence representations from sentences and effectively reduce the interference of noisy data.

4.5. Ablation Study

The proposed EANT model mainly consists of the sentence encoder, entity attribute encoder and the negative training process. In order to verify the effectiveness of different modules of our model, we designed two variants of the model to perform the ablation experiments.

Figure 4 depicts the PR curves of the different variants of the model. EANT-woEA is the method without the entity attribute encoder, which only uses the sentence encoder to obtain the sentence representation. EANT-woNT is the method without negative training, which obtains the final relation classifier by positive training only. From the figure, we can observe that removing any one module will result in a loss of model performance. This illustrates the positive effect of incorporating entity attribute information and negative training on model performance improvement.

4.6. Case Study

As discussed in Section 3, EANT can filter noise and re-label some of them by negative training.

Table 4. Some examples of the model-refined NYT dataset.

Sentence	Sentence Label	Refined Label
Critic’s notebook correction: 18 March 2005, friday a critic’s notebook article on tuesday about gertrude stein misstated the title of an opera she wrote with virgil thomson.	/people/person /profession	NA
China should throw its influence as a global trading giant behind efforts to revive talks aimed at lowering trade barriers, susan c. schwab, the united states trade representative, said in Beijing.	/location/location /contains	NA
It is the museum of contemporary art San diego, which has sites in La Jolla and San diego, not the San diego museum.	/location/neighborhood /neighborhood_of	NA
In the decades he has spent in executive suites, Michael Armstrong was identified with some of the best-known totems of American corporate: I.B.M., hughes, at&t and Comcast.	NA	/business/company /locations
But the full extent of the death toll began to be known only on thursday after a reporter from china central television, or cctv, filed a story from Pingshi, Hunan.	NA	/location/location /contains

shows some examples of the model-refined DSRE dataset. The first column indicates the sentence in the dataset, the second column denotes the original label, and the third column shows the decision made by the model for that sentence. The first sentence is labeled as relation “/people/person/profession” in the original dataset, but we can see that it does not express this relation. The model calculates the confidence value of the sentence and identifies that it is below the threshold $T{h}_d$ of the relation “/people/person/profession”, so the sentence is refined as “NA”. Similarly, the fourth sentence’s confidence value is higher than the threshold $T{h}_r$ of the relation “/business/company/locations”, so EANT relabeled such false-negative instance with the highest confidence relation label. From these examples, we can see the effectiveness of the model in distinguishing noisy data.

5. Conclusions

In this paper, we propose a new distant supervised relation extraction method, EANT, to alleviate the noise interference problem. EANT enhances the sentence representation by incorporating entity attribute information and filters out the noisy sentences in the dataset by negative training. The negative training selects complementary labels to train the model instead of the original labels. The obtained model can widen the gap between the confidence of clean and noisy data to achieve the noise filtering effect. Subsequently, for some potentially valid noisy sentences, we assign possible labels to them according to their confidence so we can make full use of them. The experimental results on the New York Times dataset show that our approach effectively improves the performance of the model for the relation extraction task.

In the future, we plan to explore pre-trained models to enhance sentence representation and capture useful semantic and syntactic properties of text through deep language models. This information is then used to adjust the entity attribute filtering strategy to reduce the impact of noisy entity attribute information.