A Hierarchical Heterogeneous Graph Attention Network for Emotion-Cause Pair Extraction

Abstract

Recently, graph neural networks (GNN), due to their compelling representation learning ability, have been exploited to deal with emotion-cause pair extraction (ECPE). However, current GNN-based ECPE methods mostly concentrate on modeling the local dependency relation between homogeneous nodes at the semantic granularity of clauses or clause pairs, while they fail to take full advantage of the rich semantic information in the document. To solve this problem, we propose a novel hierarchical heterogeneous graph attention network to model global semantic relations among nodes. Especially, our method introduces all types of semantic elements involved in the ECPE, not just clauses or clause pairs. Specifically, we first model the dependency between clauses and words, in which word nodes are also exploited as an intermediary for the association between clause nodes. Secondly, a pair-level subgraph is constructed to explore the correlation between the pair nodes and their different neighboring nodes. Representation learning of clauses and clause pairs is achieved by two-level heterogeneous graph attention networks. Experiments on the benchmark datasets show that our proposed model achieves a significant improvement over 13 compared methods.

1. Introduction

As a research hotspot in natural language processing (NLP), emotion-cause extraction (ECE), aimed at extracting the causes corresponding to the emotions specified in a given document, has been widely utilized in public opinion analysis, human–machine dialogue systems, and so on. Originally, taking events as the causes, Lee et al. [1] regarded ECE as a word-level sequence annotating task. Afterwards, some studies redefined the granularity of annotation in ECE to the clause level to make full use of context information [2,3]. Although annotating emotions in advance contributes to cause extraction, it is very labor-consuming, which limits the real application of the ECE approach. To solve this problem, Xia and Ding [4] put forward a new emotion analysis task called emotion-cause pair extraction (ECPE), which extracts emotion clauses and their corresponding cause clauses in pairs. ECPE does not rely on labeling emotions, so it is preferable, but more challenging, than ECE. Furthermore, they also proposed a two-stage pipelined framework to handle this new task, in which the emotions and causes are first extracted and then paired. Since this two-stage approach may result in cross-stage propagation of errors, a lot of end-to-end approaches are presented and achieve improvements over two-stage approaches. In the end-to-end ECPE approaches, the crucial issue is to learn good representations of semantic elements. GNN [5,6] can learn node representations based on node features and the graph structure; therefore, it is a powerful deep representation learning method and has been widely utilized in many application fields. Inspired by this, a few researchers attempted to apply GNN to the ECPE task. They mostly construct a homogeneous graph with the semantic information of a document and employed GNNs to learn these semantic representations. For example, Wei et al. [7] and Chen et al. [8] model the inter-clause and inter-pair relations, respectively.

Nevertheless, existing GNN-based ECPE approaches only concentrate on one semantic level, ignoring the rich semantic relations between different kinds of semantic elements. Hence, the captured semantic information is local, rather than global. In fact, in the ECPE task, a document involves different semantic granularity of words, clauses, clause pairs, and so on; hence, the constructed text graph should come with multiple types of nodes, also well-known as a heterogeneous graph. Furthermore, all the associations between these nodes can provide clues for extracting causality. Therefore, it is conductive for the joint extraction of emotion clauses, cause clauses, and emotion-cause pairs to take all semantic elements into account and model the global semantic relations between them.

In this study, we propose an end-to-end hierarchical heterogeneous graph attention model (HHGAT). Different from the existing methods that only consider clause or pair nodes, we introduce word nodes into our heterogeneous graph, together with clause and pair nodes, to cover all semantic elements. In particular, the introduced word nodes can not only extract fine-grained clause features by modeling the dependency between clauses and words, but also act as an intermediate node connecting clause nodes to enrich the correlation between clause nodes. Moreover, a fully connected pair-level subgraph is established to capture the relations between a pair node and its neighboring nodes on different semantic paths. Depending on such a hierarchy of “word-clause-pair”, we realize a model of the global semantics in a document.

2. Related Work

Emotion analysis is active in the field of NLP. In many application scenarios, it is more important to understand the emotional cause than the emotion itself. Here, we focus on two challenging tasks, namely ECE and ECPE.

2.1. ECE

Different from traditional emotion classification, the purpose of ECE is to extract the causes of specific emotions. Lee et al. [1] first defined the ECE task and introduced a method based on linguistic rules (RB). Subsequently, for different linguistic patterns, a variety of RB methods are proposed [9,10,11]. In addition, Russo et al. [12] designed a novel method combining RB and common-sense knowledge. However, the performance of these RB methods is usually unsatisfactory. Considering that it is impossible for rules to cover all language phenomena, some machine learning (ML)-based ECE methods are proposed. Gui et al. [13] designed two ML-based methods, combined with 25 rules. Ghazi et al. [14] employed conditional random field (CRF) to tag emotional causes. Moreover, Gui et al. [2] constructed a new clause-level corpus and utilized support vector machine (SVM) to deal with the ECE task. To benefit from the representation learning ability of deep learning (DL), some DL-based methods achieved excellent performance on ECE. Gui et al. [15] presented a new method based on convolutional neural network (CNN). Cheng et al. [16] used long short-term memory networks (LSTM) to model the clauses. To obtain better context representations, a series of hierarchical models [17,18,19,20,21,22,23,24] were explored. Inspired by multitask learning, Chen et al. [25] and Hu et al. [26] focused on the joint extraction of emotion and cause. In addition, Ding et al. [27] and Xu et al. [28] reformulated ECE into a ranking problem. Considering the importance of emotion-independent features, Xiao et al. [29] presented a multi-view attention network. Recently, Hu et al. [30] proposed a graph convolution network (GCN) integrating semantics and structure information, which is the state-of-the-art ECE method.

2.2. ECPE

2.2.1. Pipelined ECPE

ECE requires the manual annotation of emotion clauses before cause extraction, which is labor-consuming. To solve this problem, Xia and Ding [4] proposed a new task called ECPE, and they introduced three two-stage pipelined models, namely Indep, Inter-CE, and Inter-EC. For Inter-EC [4], Shan and Zhu [31] designed a new cause extraction component based on transformer [32] to improve this model. Yu et al. [33] applied the self-distillation method to train a mutually auxiliary multitask model. Jia et al. [34] realized mutual promotion of emotion extraction and cause extraction by recursively modeling clauses. To improve the pairing stage of two-stage pipelined methods, Sun et al. [35] presented a dual-questioning attention network. Moreover, Shi et al. [36] simultaneously enhanced both stages of the pipelined method.

2.2.2. End-to-End ECPE

Although the pipelined approach has been proved to be effective for ECPE, it leads to cross-stage error propagation. To solve this problem, a series of end-to-end ECPE approaches are proposed.

Wu et al. [37] jointly trained the three subtasks in ECPE via a unified framework and had clause features shared to exploit the interaction between subtasks. To make full use of the implicit connection between emotion detection and emotion-cause pair extraction, Tang et al. [38] tackled these two tasks in a joint framework. Concentrating on the interaction between emotion-cause pairs, Ding et al. [39] presented a 2D transformer and its two variants. Fan et al. [40] introduced a scope controller to concentrate the predicted distribution of emotion-cause pair. Ding et al. [41] restricted ECPE to the emotion-centered cause extraction in the sliding window and proposed a multi-label learning method. Cheng et al. [42] took advantage of two symmetrical subnetworks to conduct a local search [43,44] around emotion or cause, respectively. Singh et al. [45] adopted the prediction results of emotion extraction to promote the cause extraction. Considering the importance of order information, Fan et al. [46] captured the sequential features of clauses through three LSTMs: forward LSTM, backward LSTM, and BiLSTM. Yang et al. [47] utilized the consistency of emotion type between the emotion clause and clause pair. Chen et al. [48] achieved the mutual promotion of emotion extraction and cause extraction through iterative learning. Furthermore, some studies [49,50,51,52] coincidentally reformulated ECPE as a sequence labeling problem.

Recently, some graph structure-based approaches are proposed. Song et al. [53] treated ECPE as a link prediction task of directed graph; however, they did not adopt a GNN that is more suitable for graph structure modeling. Despite Fan et al. [54] introduced a novel approach that regards ECPE as an action prediction task in directed graph construction; their model is not based on GNN, either. In addition, Wei et al. [7] exploited a graph attention network (GAT) to enhance inter-clause relation modeling and deal with the ECPE task from a ranking perspective. Chen et al. [8] developed an approach based on a graph convolutional network to capture the relevance among local neighboring candidate pairs. However, the above graph-based approaches ignored the relationship between heterogeneous nodes, so they failed to model global semantics.

3. Methodology

3.1. Task Definition

In this section, the ECPE task is formalized as follows. Let $d=[c_1,\cdots c_i\cdots ,c_m]$ be a document that contains $m$ clauses, where $c_i=[w_{i,1},\cdots w_{i,j}\cdots ,w_{i,n}]$ is the i-th clause and further decomposed into a sequence of $n$ words. The aim of ECPE is to extract the emotion-cause pairs from $d$:

$$P={\left\{p_k\right\}}_{k=1}^{\left|P\right|}={\{(c_k^e,c_k^c)\}}_{k=1}^{\left|P\right|},$$

(1)

where $c_i^e$ is the emotion clause in the k-th emotion-cause pair, $c_j^c$ corresponds to the cause clause, and $P$ represents the candidate pair set.

3.2. Overview

In this work, we first represent a document with a “word-clause-pair” heterogeneous graph, as illustrated in Figure 1. Then, we present a hierarchical heterogeneous graph attention network to model the “word-clause-pair” hierarchical structure and identify the emotion-cause pairs according to the learned node representation. As shown in Figure 2, our proposed model mainly includes three components: (1) the node initialization layer, which utilizes word-level BiLSTM, followed by a self-attention module or pre-trained BERT to obtain the initial semantic representations of word and clause nodes; (2) the clause node encoding layer employs a node-level heterogeneous graph attention network to integrate the inner-clause contextual features into the clause representations by capturing the dependencies between clause nodes and word nodes they contains; (3) the pair node encoding layer is a heterogeneous graph attention network based on meta-path, which first applies a node-level attention and then a meta-path level attention. Finally, three multilayer perceptrons (MLP) are adopted to predict the emotion clauses, cause clauses, and emotion-cause pairs, respectively.

3.3. Heterogeneous Graph Construction

We denote our hierarchical heterogeneous graph as $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}={\mathcal{V}}^w\cup {\mathcal{V}}^c\cup {\mathcal{V}}^p$ represents a node set that consists of three types of nodes, and $\mathcal{E}$ stands for the edges between all nodes. ${\mathcal{V}}^w={\cup }_{i=1}^m{\left\{w_{i,j}\right\}}_{j=1}^n$, ${\mathcal{V}}^c={\left\{c_i\right\}}_{i=1}^m$, and ${\mathcal{V}}^p={\cup }_{i=1}^m{\left\{p_{i,j}\right\}}_{j=1}^m$ indicate the sets of words, clauses, and pair nodes, respectively. As shown in Figure 2, a word-to-clause edge distinctly indicates which clause a word is contained in. The two clause nodes connected with the same pair node together form a candidate emotion-cause pair. Moreover, the association between two pair nodes is represented by a pair-to-pair edge.

On the one hand, most current methods employ two clause-level subtasks (i.e., emotion extraction and cause extraction) in a unified framework to facilitate the detection of emotion-cause pairs. On the other hand, good clause representation is conducive to the feature construction of clause pairs. Hence, in order to learn the semantic representations of clause and pair nodes in detail, we divide our heterogeneous graph into two subgraphs, i.e., word-clause ${\mathcal{G}}^{wc}=({\mathcal{V}}^w\cup {\mathcal{V}}^c,{\mathcal{E}}^{wc})$ and pair-level ${\mathcal{G}}^p=({\mathcal{V}}^p,{\mathcal{E}}^p)$ subgraphs. Here, ${\mathcal{E}}^{wc}$ denotes the word-to-clause edge set, and ${\mathcal{E}}^p$ represents the pair-to-pair edge set. Furthermore, ${\mathcal{G}}^{wc}$ and ${\mathcal{G}}^p$ are further divided into a series of more fine-grained subgraphs, i.e., ${\cup }_{i=1}^m{\mathcal{G}}_i^{wc}$ and ${\cup }_{i=1}^m{\mathcal{G}}_i^p$, respectively, to facilitate the formalized description of our algorithm.

3.4. Hierarchical Heterogeneous Graph Attention Network

3.4.1. Node Initialization Layer

In this layer, a word embedding matrix $E_w\in {ℝ}^{d_w\times d_v}$ is first applied to transform each word $w_{i,j}$ into a vector $v_{i,j}$. Here, $d_w$ and $d_v$ are the vocabulary size and embedding dimension, respectively. Next, the contextual information for each word is captured through a BiLSTM module:

$$[h_{i,1}^w,\cdots h_{i,j}^w\cdots ,h_{i,n}^w]=\mathrm{BiLSTM}([v_{i,1},\cdots v_{i,j}\cdots ,v_{i,n}]),$$

(2)

where, $h_{i,j}^w$ represents the hidden state of the j-th word in the i-th clause. Then, an attention module is adopted to aggregate the word representations in the clause $c_i$:

$$h_i^s=\mathrm{Attention}([h_{i,1}^w,\cdots h_{i,j}^w\cdots ,h_{i,n}^w]),$$

(3)

where $h_i^s$ is the vectorization representation of the i-th clause.

Furthermore, inspired by the BERT [55], we implement another version of node initialization layer, which utilizes the pre-trained BERT model to replace above BiLSTM and attention modules. The tokens [CLS] and [SEP] are inserted at the beginning and end of a given clause $c_i$, respectively, to obtain a sequence $c_i=[w_{\mathrm{CLS}},w_{i,1},\cdots w_{i,j}\cdots ,w_{i,n},w_{\mathrm{SEP}}]$. It is worth noting that $w_{i,j}$ represents the j-th token, rather than j-th word of the clause $c_i$, in the BERT version. Afterwards, the sequences corresponding to all clauses in the document are concatenated to form a whole sequence, and then input it to BERT. Through stacked transformer modules, we can obtain the output vectors ${\cup }_{i=1}^m\{h_{i,1}^w,\cdots h_{i,j}^w\cdots ,h_{i,n}^w\}$ and ${\left\{h_i^s\right\}}_{i=1}^m$, which are the initialization representations of word and clause nodes, respectively. Here, $h_i^s$ is the output of $w_{\mathrm{CLS}}$ corresponding to the clause $c_i$.

3.4.2. Clause Node Encoding Layer

Inner-clause relationships plays an important role in semantic understanding. In addition, a word can be also treated as a specific relation between the clauses containing it. Therefore, to further learn the semantic representation of a clause node, we extract each clause node and its connected word nodes from the hierarchical graph to build a fine-grained word-clause subgraph. Given a constructed subgraph ${\mathcal{G}}_i^{wc}$, with the clause node $c_i$ and word nodes ${\left\{w_{i,j}\right\}}_{j=1}^n$, we apply a heterogeneous graph attention network to update the representation of the clause node.

Since two types of nodes exist in the heterogeneous subgraph, different types of nodes may belong to different feature spaces. Consequently, type-specific transformation matrices $W_s$ and $W_w$ are adopted to respectively project the features of clause and word nodes, with possibly different dimensions into the same feature space. The projection process can be shown in the following:

$${\tilde{h}}_i^s=W_s\cdot h_i^s,{\tilde{h}}_{i,j}^w=W_w\cdot h_{i,j}^w,$$

(4)

where $h_i^s$ is the initialization representation of clause node $c_i$, and $h_{i,j}^w$ denotes the initialization representation of word node $w_{i,j}$.

The node-level attention mechanism is then applied to learn the importance of different neighboring nodes to each target node. For a word-clause subgraph ${\mathcal{G}}_i^{wc}$, the clause node $c_i\in {\mathcal{V}}^c$ is the target node, while the corresponding neighboring nodes come from the word node set ${\left\{w_{i,j}\right\}}_{j=1}^n$. Specifically, importance scores are computed through a linear layer parameterized by $w_1{}^{\top }$, and then they are normalized to obtain weight coefficients via the softmax function. Next, according to these weight coefficients, the node aggregation over the subgraph is conducted by a weighted summation. In addition, we also apply a residual connection when updating the semantic representation of the clause node $c_i$. The specific process is as follows:

$$e_{i,j}=\mathrm{LeakyReLU}(w_1{}^{\top }\cdot \tanh ({\tilde{h}}_i^s\parallel {\tilde{h}}_{i,j}^w)),$$

(5)

$$a_{i,j}=\frac{\exp (e_{i,j})}{{\displaystyle {\sum }_{k=1}^n\exp (e_{i,k})}},$$

(6)

$${\widehat{h}}_i^s=\mathrm{Re}\mathrm{LU}({\displaystyle {\sum }_{j=1}^n{a}_{i,j}\cdot {\tilde{h}}_{i,j}^w}+b_w)+h_i^s,$$

(7)

where $w_1$ is trainable weight matrix, $b_w$ is the bias parameter, $\parallel$ denotes the concatenation operation, and $\top$ represents the transpose of matrix. As a result, the clause representation ${\widehat{h}}_i^s$ integrating word semantics is generated.

Once obtaining updated node representation ${\widehat{h}}_i^s$, it is fed into the emotion clause classifier to determine whether the clause corresponding to $c_i$ is an emotion clause or not, and the classifier is implemented by a linear layer (parameterized by $w_e^{}$ and $b_e$) with the sigmoid function:

$${\widehat{y}}_i^e=\mathrm{s}\mathrm{igmoid}(w_e^{\top }\cdot {\widehat{h}}_i^s+b_e) ,$$

(8)

where ${\widehat{y}}_i^e$ is the predicted probability that the clause node $c_i$ is an emotion clause. The calculation process of obtaining the cause probability ${\widehat{y}}_i^c$ is similar to that of ${\widehat{y}}_i^e$, except that the parameters are replaced by $w_c$ and $b_c$.

3.4.3. Pair Node Encoding Layer

It can be observed that there are only simple subordinate relationships between the clause and pair nodes, rather than complex semantic relationships. Hence, we just need to consider pair nodes and the correlation between them when performing subgraph segmentation in this section. Furthermore, in a fine-grained pair-level subgraph ${\mathcal{G}}_i^p$, the neighboring nodes of a node $p_{i,j}$ are restricted to those nodes with the same emotion candidate as this one. Therefore, a pair-level, fully connected subgraph is formalized as ${\mathcal{G}}_i^p=({\left\{p_{i,j}\right\}}_{j=1}^m,{\mathcal{E}}_i^p)$. Moreover, a meta-path ${\mathsf{\Phi }}_t$ is described as a kind of path in the forms of $p_{i,k}\to \cdots p_{i,j-1}\to p_{i,j}$ and $p_{i,j}\leftarrow p_{i,j+1}\cdots \leftarrow p_{i,k}$, where $t=|k-j|$ represents the number of hops from a source node $p_{i,k}$ to the target node $p_{i,j}$. According to the statistical results of [8], the proportion that the distance between an emotion clause and the corresponding cause clause less than or equal to 2 is 95.8%. Taking into account this, we introduce four kinds of meta-paths: ${\mathsf{\Phi }}_0$, ${\mathsf{\Phi }}_1$, ${\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$. Different from the other three types of paths, ${\mathsf{\Phi }}_3$ indicates the length of the path from the source node to the target node is $\ge 3$.

Given a pair-level subgraph ${\mathcal{G}}_i^p$, the initial representation $h_{i,j}^p$ of a node $p_{i,j}=(c_i^e,c_j^c)$ in ${\mathcal{G}}_i^p$ is obtained by concatenating three vectors:

$$h_{i,j}^p={\widehat{h}}_i^s\parallel {\widehat{h}}_j^s\parallel h_{i,j}^{rep},$$

(9)

where ${\widehat{h}}_i^s$ and ${\widehat{h}}_j^s$ represent the semantic representations of candidate emotion clause $c_i^e$ and candidate cause clause $c_j^c$, respectively. $h_{i,j}^{rep}$ indicates the relative position embedding, which is randomly initialized by the sampling of a uniform distribution. Considering that the meta-path-based neighbors play different roles in the representation of each node, we apply a meta-path-based graph attention network, which aggregates the features of neighboring nodes from different-typed paths to update the representation of this node. Specifically, two aggregation operations need to be performed.

Firstly, node-level attention is leveraged to aggregate the path-specific node representations. Specifically, for all pair nodes in the subgraph ${\mathcal{G}}_i^p$, a shared linear transformation, followed by the tanh function, is employed. Given a target node $p_{i,j}$ and meta-path ${\mathsf{\Phi }}_t$, the weight coefficient $e_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}$ of a neighboring node $p_{i,k}$ that is connected to node $p_{i,j}$ through meta-path ${\mathsf{\Phi }}_t$ is calculated. $e_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}$ reflects the importance of node $p_{i,k}$ to node $p_{i,j}$. The weight coefficients of all ${\mathsf{\Phi }}_t$-based neighboring nodes are then normalized via the softmax function. By weighted summation, ${\mathsf{\Phi }}_t$-specific aggregate representation ${\tilde{h}}_{i,j}^{{\mathsf{\Phi }}_t}$ of the node $p_{i,j}$ is generated:

$${\tilde{h}}_{i,j}^p=W_p\cdot h_{i,j}^p,{\tilde{h}}_{i,k}^p=W_p\cdot h_{i,k}^p,$$

(10)

$${\tilde{e}}_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}=\mathrm{LeakyReLU}(w_{{\mathsf{\Phi }}_t}{}^{\top }\cdot \tanh ({\tilde{h}}_{i,j}^p\parallel {\tilde{h}}_{i,k}^p)),$$

(11)

$$e_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}=I_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}\cdot {\tilde{e}}_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t},I_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}=\left\{\begin{array}{l}1,p_{i,k}\in P_{i,j}^{{\mathsf{\Phi }}_t}\\ 0,p_{i,k}\notin P_{i,j}^{{\mathsf{\Phi }}_t}\end{array}\right.,$$

(12)

$$a_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}=\frac{\exp (e_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t})}{{\displaystyle {\sum }_{k^{\prime }=1}^m\exp (e_{(i,j),(i,k^{\prime })}^{{\mathsf{\Phi }}_t})}},$$

(13)

$${\tilde{h}}_{i,j}^{{\mathsf{\Phi }}_t}=\mathrm{Re}\mathrm{LU}({\displaystyle {\sum }_{k=1}^m{a}_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}\cdot {\tilde{h}}_{i,k}^p+b_{{\mathsf{\Phi }}_t}}),$$

(14)

where $W_p$ and $w_{{\mathsf{\Phi }}_t}$ are trainable weight matrices, $b_{{\mathsf{\Phi }}_t}$ denotes the bias, and $h_{i,j}^p$ represents the initial feature of node $p_{i,j}$. In addition, $I_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}$ is the node mask, which injects structural information into the model. Additionally, $I_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}=1$ means that $p_{i,k}$ belongs to the ${\mathsf{\Phi }}_t$-based neighboring node set $P_{i,j}^{{\mathsf{\Phi }}_t}$ of $p_{i,j}$.

Secondly, path-level attention is applied to measure the importance of different meta-paths to the target node. For this purpose, the path-specific aggregate representations obtained by previous node-level attention are transformed into the weight values through a linear transformation matrix. After that, the softmax function is employed to normalize these weight values, so as to obtain the weight coefficients of different paths. Using the learned weight coefficients, the aggregate representations from different meta-paths are fused with the initial node representation $h_{i,j}^p$. The final semantic representation ${\widehat{h}}_{i,j}^p$ of node $p_{i,j}$ is obtained by:

$$a_{i,j}^{{\mathsf{\Phi }}_t}=\frac{\exp (w_2{}^{\top }\cdot {\tilde{h}}_{i,j}^{{\mathsf{\Phi }}_t})}{{\displaystyle {\sum }_{t^{\prime }=0}^T\exp (w_2{}^{\top }\cdot {\tilde{h}}_{i,j}^{{\mathsf{\Phi }}_{t^{\prime }}})}},$$

(15)

$${\widehat{h}}_{i,j}^p={\displaystyle {\sum }_{t=0}^T{a}_{i,j}^{{\mathsf{\Phi }}_t}\cdot {\tilde{h}}_{i,j}^{{\mathsf{\Phi }}_t}}+h_{i,j}^p,$$

(16)

where $w_2{}^{\top }$ is a trainable transformation matrix, the meta-path ${\mathsf{\Phi }}_t$ belongs to the path set $\mathsf{\Phi }={\left\{{\mathsf{\Phi }}_t\right\}}_{t=0}^T$, and $T=\left|\mathsf{\Phi }\right|-1$. $a_{i,j}^{{\mathsf{\Phi }}_t}$ represents the weight coefficient of meta-path ${\mathsf{\Phi }}_t$ to node $p_{i,j}$. Here, it is worth noting that, if the target nodes are different, the weight distribution of the meta-paths is also different.

Then, a logistic regression layer (parameterized by $w_p^{\top }$ and $b_p$) is utilized to identify whether each pair node is a true emotion-cause pair node:

$${\widehat{y}}_{i,j}^p=\mathrm{s}\mathrm{igmoid}(w_p^{\top }\cdot {\widehat{h}}_{i,j}^p+b_p) .$$

(17)

3.5. Model Training and Optimization

The loss function of extracting emotion-cause pairs from a given document $d$ is formulated as follows:

$$L_p=-\frac{1}{m^2}\cdot {\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^m(y_{i,j}^p\cdot \log ({\widehat{y}}_{i,j}^p)+(1-y_{i,j}^p)\cdot \log (1-{\widehat{y}}_{i,j}^p))}},$$

(18)

where $y_{i,j}^p$ is the ground-truth of node $p_{i,j}$. To benefit from the other two subtasks, the loss terms of the emotion extraction and cause extraction are introduced. For simplicity, only the calculation process of loss term for the emotion extraction is provided in the following:

$$L_e=-\frac{1}{m}\cdot {\displaystyle {\sum }_{i=1}^m(y_i^e\cdot \log ({\widehat{y}}_i^e)+(1-y_i^e)\cdot \log (1-{\widehat{y}}_i^e))},$$

(19)

where $y_i^e$ is the emotion annotation of clause $c_i$. Therefore, the total loss of our model is

$$L_{total}=L_p+L_e+L_c.$$

(20)

Finally, the purpose of the model training is to minimize the total loss. The overall process is shown in Algorithm 1.

Algorithm 1: The overall process of HHGAT.

Input : The heterogeneous graph $\mathcal{G}=\left(\mathcal{V},\mathcal{E}\right)$, $\mathcal{V}={\mathcal{V}}^w\cup {\mathcal{V}}^c\cup {\mathcal{V}}^p$,             The initial feature $h_i^s$ of clause node $\forall c_i\in {\mathcal{V}}^c={\left\{c_i\right\}}_{i=1}^m$,             The initial feature $h_{i,j}^w$ of word node $\forall w_{i,j}\in {\mathcal{V}}^w={\cup }_{i=1}^m{\left\{w_{i,j}\right\}}_{j=1}^n$. Output: The clause node representations ${\left\{{\widehat{h}}_i^s\right\}}_{i=1}^m$,               The pair node representations ${\cup }_{i=1}^m{\left\{{\widehat{h}}_{i,j}^p\right\}}_{j=1}^m$. for word-clause subgraph ${\mathcal{G}}_i^{wc}\subset {\mathcal{G}}^{wc}$ do     Project feature space ${\tilde{h}}_i^s=W_s\cdot h_i^s$;     for word node $w_{i,j}\in {\left\{w_{i,j}\right\}}_{j=1}^n$ do             Project feature space ${\tilde{h}}_{i,j}^w=W_w\cdot h_{i,j}^w$;             Calculate the node-level weight coefficient $a_{i,j}$;     Update clause node feature ${\widehat{h}}_i^s=\mathrm{Re}\mathrm{LU}({\displaystyle {\sum }_{j=1}^n{a}_{i,j}\cdot {\tilde{h}}_{i,j}^w}+b_w)+h_i^s$; for pair-level subgraph ${\mathcal{G}}_i^p\subset {\mathcal{G}}_i^p$ do     for pair node $p_{i,j}\in {\mathcal{G}}_i^p$ do             Initialize the node representation $h_{i,j}^p={\widehat{h}}_i^s\parallel {\widehat{h}}_j^s\parallel h_{i,j}^{rep}$;             Project feature space ${\tilde{h}}_{i,j}^p=W_p\cdot h_{i,j}^p$;             for meta-path ${\mathsf{\Phi }}_t\in \mathsf{\Phi }$ do                for ${\mathsf{\Phi }}_t-\mathrm{based}$ neighboring node $p_{i,k}\in P_{i,j}^{{\mathsf{\Phi }}_t}$ do                    Calculate the node-level weight coefficient $a_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}$;                Aggregate node feature ${\tilde{h}}_{i,j}^{{\mathsf{\Phi }}_t}=\mathrm{Re}\mathrm{LU}({\displaystyle {\sum }_{k=1}^m{a}_{(i,j),(i,k)}^{{\mathsf{\Phi }}_t}\cdot {\tilde{h}}_{i,k}^p}+b_{{\mathsf{\Phi }}_t})$;                Calculate the weight coefficient $a_{i,j}^{{\mathsf{\Phi }}_t}$ of meta-path ${\mathsf{\Phi }}_t$;             Update pair node feature ${\widehat{h}}_{i,j}^p={\displaystyle {\sum }_{t=0}^T{a}_{i,j}^{{\mathsf{\Phi }}_t}\cdot {\tilde{h}}_{i,j}^{{\mathsf{\Phi }}_t}}+h_{i,j}^p$; Calculate the total loss $L_{total}=L_p+L_e+L_c$; Back propagation and update parameters; return ${\left\{{\widehat{h}}_i^s\right\}}_{i=1}^m$, ${\cup }_{i=1}^m{\left\{{\widehat{h}}_{i,j}^p\right\}}_{j=1}^m$.

4. Experiments

4.1. Dataset and Evaluation Metrics

To evaluate our method, we utilized the benchmark ECPE dataset released by Xia and Ding [4], which consists of 1945 Chinese news documents. In these documents, there are a total of 490,367 candidate pairs, of which, the real emotion-cause pairs account for less than 1%, and each document possibly contains more than one emotion corresponding to multiple causes. According to the data-split setting of previous work, the dataset was segmented into 10 equal parts, and they were chosen as the train and test sets in the proportion of 9 to 1. In order to achieve statistically credible verification, we applied 10-fold cross-validation and repeated the experiments 20 times to average the results. Furthermore, precision (P), recall (R), and F1-score (F1) were selected as the evaluation metrics for emotion, cause, and emotion-cause pair extraction.

4.2. Experimental Settings

In our experiments, to make a fair comparison, the word embedding trained in [4] is utilized in our method. The dimensions of word embedding, BiLSTM’s hidden state, and relative position embedding were set to 200, 100, and 50, respectively. In addition, for our BERT version model, the output dimension of pre-trained BERT is 768. The weight matrices and bias vectors involved in the two versions of our model were all randomly initialized by a continuous uniform distribution, $\mathrm{U}(-0.01,0.01)$. To avoid overfitting, we applied dropout, and the dropout rate was set to 0.1. Compared to some excellent global optimization algorithms [56,57,58], Adam [59] is more effective in deep learning. Therefore, in the training process of our model, we utilized the Adam optimizer to update all parameters with the learning rate of 0.005, mini-batch size of 32, and $L_2$ regularization coefficient of 1 × 10⁻⁵. Our models were performed on the NVIDIA GeForce RTX 2080 Ti GPUs.

4.3. Compared Methods

We compared our method with the following state-of-the-art methods. It is worth noting that the models above the dotted line in

Table 1. Comparison of experimental results on the emotion extraction, cause extraction, and ECPE.

Category	Method	Emotion Extraction			Cause Extraction			Emotion-Cause Pair Extraction
		P	R	F1	P	R	F1	P	R	F1
Pipelined	Inter-EC [4]	0.8364	0.8107	0.8230	0.7041	0.6083	0.6507	0.6721	0.5705	0.6128
	Inter-ECNC [31]	-	-	-	0.6863	0.6254	0.6544	0.6601	0.5734	0.6138
	DQAN [35]	-	-	-	0.7732	0.6370	0.6979	0.6733	0.6040	0.6362
End-to-end	E2EECPE [53]	0.8552	0.8024	0.8275	0.7048	0.6159	0.6571	0.6491	0.6195	0.6315
	MTNECP [37]	0.8662	0.8393	0.8520	0.7400	0.6378	0.6844	0.6828	0.5894	0.6321
	SLSN [42]	0.8406	0.7980	0.8181	0.6992	0.6588	0.6778	0.6836	0.6291	0.6545
	LAE-MANN [38]	0.8990	0.8000	0.8470	-	-	-	0.7110	0.6070	0.6550
	TDGC [54]	0.8716	0.8244	0.8474	0.7562	0.6471	0.6974	0.7374	0.6307	0.6799
	ECPE-2D [39]	0.8627	0.9221	0.8910	0.7336	0.6934	0.7123	0.7292	0.6544	0.6889
	PairGCN [8]	0.8857	0.7958	0.8375	0.7907	0.6928	0.7375	0.7692	0.6791	0.7202
	UTOS [52]	0.8815	0.8321	0.8556	0.7671	0.7320	0.7471	0.7389	0.7062	0.7203
	RANKCP [7]	0.9123	0.8999	0.9057	0.7461	0.7788	0.7615	0.7119	0.7630	0.7360
	RSN [48]	0.8614	0.8922	0.8755	0.7727	0.7398	0.7545	0.7601	0.7219	0.7393
Ours	w/o BERT	0.8361	0.8327	0.8337	0.7157	0.6519	0.6811	0.7143	0.6238	0.6654
	HHGAT	0.8655	0.9181	0.8906	0.7427	0.7988	0.7686	0.7458	0.7631	0.7525

did not adopt BERT.

• Inter-EC, which uses emotion extraction to facilitate cause extraction, archives the best performance among the three pipelined methods proposed in [4].
• Inter-ECNC [31], as a variant of Inter-EC, employs transformer to optimize the extraction of cause clauses.
• DQAN [35] is a dual-questioning attention network, separately questioning candidate emotions and causes.
• E2EECPE [53] is an end-to-end link prediction model of directed graph, which establishes the directional links from emotions to causes by a biaffine attention.
• MTNECP [37] is a feature-shared, multi-task model and improves cause extraction with the help of position-aware emotion information.
• SLSN [42] is a symmetrical network composed of two subnetworks. Each subnetwork also performs a local pairing search, while extracting each target clause.
• LAE-MANN [38] explores a hierarchical attention to model the correlation between each pair of clauses.
• TDGC [54] is a transition-based, end-to-end model that regards ECPE as the construction process of directed graph.
• ECPE-2D [39] designs a 2D transformer to model the interaction between candidate pairs.
• PairGCN [8] employs a GCN to learn the dependency relations between candidate pairs.
• UTOS [52] redefines the ECPE task as a unified sequence labeling task, in which each label indicates not only the clause type, but also pairing index.
• RANKCP [7] is a ranking model that introduces a GAT to learn the representations of clauses.
• RSN [48] explicitly realizes the pairwise interaction between the three subtasks through multiple rounds of inference.

4.4. Main Results

The comparative results are shown in Table 1. We can observe that HHGAT achieves the best performance. In general, the end-to-end models obviously perform better than the pipelined models (e.g., Inter-EC, Inter-ECNC, and DQAN) because the end-to-end manner can avoid the cross-stage propagation of errors. In addition, better performance is usually achieved by the models with pre-trained BERT than those without it. Significantly, in terms of the F1-score, the non-BERT version of HHGAT outperforms SLSN (i.e., it is the best-performing model, without employing pre-trained BERT, and is based on LSTM) by 1.09% on emotion-cause pair extraction, which verifies the effectiveness of HHGAT for emotion-cause pair extraction.

By adopting BERT to encode the initial representation of nodes, the performance of HHGAT is further improved. Although LAE-MANN also designs a hierarchical attention network, it is not graph structure oriented, so it is inferior to our graph attention network in modeling the structural features of text. As shown in Table 1, LAE-MANN underperforms HHGAT by 9.75% in the F1-score of emotion-cause pair extraction. Inspired by Inter-EC, which utilizes the prediction results of emotions to promote cause extraction, ECPE-2D, UTOS, and RSN explicitly establish the interaction between emotion and cause, in their respective ways, to improve their performance. However, even without using the measures used in the above three methods, our model still outperforms them. Compared to the best-performing model RSN, the F1-scores of our HHGAT are increased by 1.51%, 1.41%, and 1.32% on emotion, cause, and emotion-cause pair extraction, respectively. This demonstrates that, even if the interaction between emotion and cause is not explicitly constructed, HHGAT can achieve excellent performance because of powerful modeling ability of the graph neural network.

Furthermore, TDGC, PairGCN, and RANKCP all employ graph structures to represent documents. However, TDGC is not realized by the graph neural network, but by LSTM, so its performance is the worst among these graph structure-based methods. Despite PairGCN and RANKCP employ GCN and GAT to learn node representations, respectively, they are all homogeneous graph oriented. This leads them to only focus on learning the correlations between the same kind of semantic elements. Different from them, our heterogeneous graph contains more kinds of nodes and richer semantic information. Compared to these three-graph, structure-based methods, our method improves the F1 score of emotion-cause pair extraction by 7.26%, 3.23%, and 1.65%, respectively. In summary, experimental results indicate that our method, based on the heterogeneous graph, is effective.

4.5. Ablation Study

To further validate the components of our model, we conduct an ablation experiment, where G1 denotes the clause node encoding layer, G2 represents the pair node encoding layer, and H1 and H2 correspond to the heterogeneous design of G1 and G2, respectively. The ablation results are shown in

Table 2. Experimental results of structural ablation.

Method	Emotion Extraction			Cause Extraction			Emotion-Cause Pair Extraction
	P	R	F1	P	R	F1	P	R	F1
HHGAT	0.8655	0.9181	0.8906	0.7427	0.7988	0.7686	0.7458	0.7631	0.7525
w/o G2	0.8553	0.9164	0.8839	0.7365	0.7970	0.7644	0.7093	0.7692	0.7361
w/o G2&H1	0.8625	0.9116	0.8860	0.7300	0.7654	0.7464	0.6895	0.7296	0.7075
w/o G1	0.8618	0.9021	0.8808	0.7381	0.7583	0.7477	0.7113	0.7224	0.7164
w/o G1&H2	0.8375	0.9170	0.8748	0.7308	0.7615	0.7449	0.6884	0.7379	0.7111
w/o G1&G2	0.8596	0.9148	0.8858	0.7296	0.7456	0.7353	0.6831	0.7169	0.6974

.

Firstly, HHGAT removes G2, resulting in the absence of dependency relations between local neighboring candidate pairs. As a result, the F1-score of emotion-cause pair extraction is decreased by 1.64%. This demonstrates that it is not enough to rely solely on modeling the word-clause connections. Specially, without an explicit interaction between emotion and cause, local context from neighboring pair nodes plays an important role in pairing the emotions and their corresponding causes.

Secondly, HHGAT w/o G2&H1 means that it only applies a graph attention network to learn the inter-clause relationships. Compared with HHGAT, the F1-score on emotion-cause pair extraction drops by 4.5%. The significant degradation of performance is mainly caused by the following two aspects. On the one hand, as the basic elements in clauses, words can provide more fine-grained semantic information. On the other hand, word nodes can enrich the correlations among clause nodes.

Then, HHGAT w/o G1 underperforms HHGAT by 0.98%, 2.09%, and 3.61% in the F1 scores of the three subtasks, respectively, which shows that our hierarchical design is beneficial to the ECPE task. This is because there is a natural hierarchical relationship between different semantic elements in human language. In addition, in the joint learning of three subtasks, good clause representation is helpful for the extraction of emotion-cause pairs.

Next, we can observe that the performance of HHGAT w/o G1&H2 is further dropped, compared with HHGAT w/o G1, because HHGAT w/o G1&H2 does not consider that the semantic information aggregated from neighboring nodes on different meta-paths is different. Hence, to learn more comprehensive pair node representations, it is necessary to employ a graph attention network based on meta-path on the pair-level subgraphs.

Finally, HHGAT w/o G1&G2 uses a clause-level BiLSTM to replace our two-layer graph attention network, which means that it is not a GNN-based method. Consequently, HHGAT w/o G1&G2 achieves the worst performance in all ablation models (F1-score dropped by 5.51%). The above results further show that each module of our method is helpful for the ECPE task.

4.6. Evaluation on Emotion-Cause Extraction

To provide a wider comparison, we also evaluate our model on the benchmark ECE corpus [2], and the compared models are as follows:

• Multi-Kernel [2] proposes a convolution kernel-based learning method to train a multi-kernel SVM.
• Memnet [15] is a convolutional deep memory network, which regards ECE as an answer retrieval task.
• PAE-DGL [27], as a reordering model, integrates the relative position and global label, with text content.
• CANN [18] presents a co-attention network based on emotional context awareness.
• MBiAS [24] designs a multi-granularity bidirectional attention network in a machine comprehension frame.
• RTHN [21] introduces a hierarchical neural network composed of RNN and transformer.
• FSS-GCN [30] adopts a graph convolutional network to model the dependency information between clauses.

The comparative results are shown in Figure 3. It can be observed that our model achieves slightly higher F1 than RTHN (i.e., the best-performing one in the models that are not based on graph neural networks). This further verifies the effectiveness of our approach on emotion-cause extraction. Furthermore, the performance of our model and FSS-GCN (i.e., a graph structure-based model) is nearly matched, in terms of the F1-score. Different from FSS-GCN, in which only clause nodes are considered, the heterogeneous graph built by us contains more kinds of nodes, and the structure of our model is more complicated. However, it is worth noting that the compared methods listed in Figure 3 all need to annotate emotions before extracting causes. This is very labor-consuming. Therefore, when the performance is equivalent, our method is more suitable for real applications.

4.7. Case Study

4.7.1. Effect of Word-Clause Graph Attention

As shown in Figure 4, the information regarding the three clauses in one representative case (i.e., Document 41) is introduced, including the word identifier, clause identifier, and details of the clause. This document consists of eight clauses and contains one emotion-cause pair $(c_4,c_3)$, where $c_4$ and $c_3$ are the emotion and cause clauses, respectively. To examine the effect of word-clause graph attention, we visualize the weight vector $a_i=[a_{i,1},\dots ,a_{i,n}]$. The visualization results are shown in Figure 4—where the darker the color is, the higher the relevance is.

We can find that the dark color is mainly concentrated around the word “anxious” in the emotion clause $c_4$, which indicates that HHGAT can effectively capture the emotion keywords and ignore other non-emotion words. Moreover, in the cause clause $c_3$, the words “unable”, “to”, and “consider” are significantly darker, which semantically constitutes the cause for triggering the emotion “anxious”. This shows that our HHGAT is also able to focus on the cause keywords. In sharp contrast, the color of all words in clause $c_2$ is very similar, which causes attention to be dispersed because $c_2$ is neither an emotion clause nor a cause clause. Consequently, HHGAT is effective in learning the features of emotion and cause clauses.

4.7.2. Effect of Meta-Path-Based Attention

In this section, Document 41 is analyzed again to verify the effect of meta-path-based attention. To this end, we visualize the weight coefficients of different-typed meta-paths to each pair node, as shown in Figure 5. Since the document consists of eight clauses, we divide the visualization results into eight subgraphs, and each subgraph shows the attention visualization results of those pair nodes with the same candidate emotion clause. The color instructions are the same as that in the previous section.

From the visualization results in Figure 5, we can observe that the color distribution on these subgraphs is very similar. In each subgraph, the color of ${\mathsf{\Phi }}_0$ corresponding to the pair node containing the ground-truth cause is the darkest. Additionally, in each row, the path with the largest weight coefficient to the target node is mostly the one where the real cause lies. In addition, as the offset from the central node or path increases, the correlation usually becomes lower. This shows that our method can find pair nodes containing ground-truth causes, according to the meta-paths.

Next, we conduct an inter-graph analysis, comparing the maximum attention coefficients in those rows corresponding to the ground-truth causes. In addition to Document 41, we also select the documents numbered 43, 167, and 151 as representative cases, where their emotion-cause pairs are $p_{5,5}$, $p_{6,4}$, and $p_{5,4}$, respectively. The comparison results are shown in Figure 6. We can notice that the highest point on each fold line is consistent with the ground-truth emotion-cause pair, which indicates that our meta-path-based graph attention network can effectively identify the emotion-cause pairs. It is worth noting that the values of all points on the fold line denoting Document 43 are relatively close. This is because the clause $c_5$ in Document 43 is both an emotion and cause clause, and each pair node on the fold line includes the clause $c_5$. The above results further verify that our method is effective for ECPE.

4.7.3. Error Analysis

In this section, we collect all emotion-cause pairs that were erroneously predicted on the test set. Inspired by [52], we also classify these errors into four categories, i.e., emotion, cause, both, and missing errors. Depending on the statistical results in

Table 3. The statistics of error emotion-cause pairs.

Category	Emotion Error	Cause Error	Both Error	Missing Error
Proportion	3.3%	46.2%	30.8%	19.7%

, we can notice that the proportion of cause errors is the largest, followed by both errors. However, we can find that most of both errors are due to unlabeled emotions, which are usually irrelevant to the topic of the document. Furthermore, the proportion of missing errors is also relatively large. Therefore, we select two cases to analyze the cause and missing errors, respectively.

For the first case in

Table 4. Two error cases.

Case	Truth	Prediction
[ ... ]. [Xiao was holding Long’s 2-year-old son]7. [Fearing that Long would hurt the child]8, [he knocked Long on the head with a lid]9, [and then Long became angry]10. [ ... ].	[8, 8] [10, 9]	[8, 8] [10, 8]
[ ... ]. [“It feels like the sky is falling down”]3. [Xu Ping described how she felt when she learned that her husband was ill]4, [he knocked Long on the head with a lid]5. [ ... ].	[3, 5]	[ ]

The superscript number at the end of a clause indicates the clause number.

, our model correctly predicts the emotion-cause pair $p_{8,8}$, while it identifies Clause 8 as the cause clause in the emotion-cause pair $p_{10,9}$ by mistake. It may be the cause of the prediction error that Clause 8 triggers the occurrence of the event described in Clause 9. Therefore, the ability of our model in distinguishing the indirect causes from direct causes needs to be further strengthened. Furthermore, in the prediction result of Case 2, the ground-truth emotion-cause pair $p_{3,5}$ is missing. We observe that the clause “it feels like the sky is falling down” is a metaphor, so it expresses an implicit emotion. Obviously, there are no emotion keywords in implicit emotional expression, and the identification of such emotions needs to comprehensively consider language style, rhetoric, metaphor, and so on, so it is more difficult to identify implicit emotions.

5. Conclusions and Future Work

In this paper, we propose HHGAT to capture the global semantic information contained in the documents. Specifically, we first constructed a heterogeneous graph that considers all types of semantic elements involved in the ECPE and models the global semantic relations between these elements. Secondly, we proposed a hierarchical heterogeneous graph attention network to learn the representations of clauses and clause pairs with global semantic information. Thirdly, we conducted extensive experiments on the benchmark ECPE dataset. The experimental results show that our proposed method achieves a better performance than the 13 compared methods and out-performs the best competitor, RSN, by a 1.32% F1-score.

In addition, the essence of pairing emotions and causes is to calculate the similarity between them. Nevertheless, similarity is a fuzzy, and not clearly defined, concept. It is difficult for traditional graph neural networks to handle the fuzzy relationship. Therefore, we will introduce fuzzy graph theory [60,61,62] into graph neural networks in our future work, so as to effectively learn the fuzzy relation between clauses.