A Convolutional Sequence-to-Sequence Attention Fusion Framework for Commonsense Causal Reasoning

Abstract

Commonsense causal reasoning is the process of understanding the causal dependency between common events or actions. Traditionally, it was framed as a selection problem. However, we cannot obtain enough candidates and need more flexible causes (or effects) in many scenarios, such as causal-based QA problems. Thus, the ability to generate causes (or effects) is an important problem. In this paper, we propose a causal attention mechanism that leverages external knowledge from CausalNet, followed by a novel fusion mechanism that combines global causal dependency guidance from the causal attention with local causal dependency obtained through multi-layer soft attention within the CNN seq2seq architecture. Experimental results consistently demonstrate the superiority of the proposed framework, achieving BLEU-1 scores of 20.06 and 36.94, BLEU-2 scores of 9.98 and 27.78, and human-evaluated accuracy rates of 35% and 52% for two evaluation datasets, outperforming all other baselines across all metrics on both evaluation datasets.

1. Introduction

Commonsense causal reasoning entails inferring causal relations between everyday events or actions. For example, the statement “Amanda feels hot” can reasonably cause the effect “Amanda turns on the fan”, indicating a causal link between the two events. Previous works [1,2] have proposed various causal strength metrics to rank these candidates and identify the most plausible alternative for the premise. However, these ranking models for commonsense causality rely on human-labeled candidates, which is impractical for many natural language processing (NLP) generation scenarios, such as question answering and dialog completion. To address this limitation, we reframe the commonsense causal reasoning task as a generation problem. In this formulation, the model is presented with a cause (or effect) sentence as the premise and is tasked with generating reasonable effect (or cause) sentences as targets. We refer to the cause-to-effect inference process as “forward reasoning”. For instance, given the cause sentence “Amanda feels hot”, potential effect sentences could be “Amanda turns on the fan” or “Amanda takes off her coat”, among others. In contrast, “backward reasoning” treats the input sentence as an effect and attempts to infer the cause. An exemplary cause output for the effect “Amanda feels hot” might be “The air-conditioner stops working”.

Existing approaches for causal reasoning [1,2,3,4,5] have predominantly relied on a selection-based style. Although these methods can be adapted to support generation-based causal reasoning, the adaptation process involves automatically generating a candidate set for each premise and subsequently employing the selection-based methods to reason about causal relationships. However, these adapted methods suffer from two limitations. Firstly, incorporating an additional online step to construct an appropriate candidate set for each premise results in a considerable computational burden. The computational cost of this step can lead to inefficiencies in the reasoning process. Secondly, the adapted model is restricted to selecting targets exclusively from the predefined candidates, thereby diminishing its flexibility in handling diverse causal reasoning tasks. In recent years, sequence-to-sequence (seq2seq) learning has achieved tremendous success in various text generation applications, such as machine translation [6,7,8,9,10], summarization [11,12,13,14,15,16], language modeling [17,18,19], story generation [20,21], and dialogue [22,23,24]. Building upon this progress, this paper introduces a novel model based on the convolutional sequence-to-sequence framework [25], equipped with the causal attention fusion mechanism to empower generation-based causal reasoning tasks.

Within the encoder-decoder sequence-to-sequence architecture, the attention mechanism is intentionally designed to capture the semantic dependencies between the current decoding hidden state and each of the encoding hidden states. These semantic dependencies, commonly interpreted as alignments in machine translation and summarization, also play an important role in causal reasoning by serving as causal dependencies. However, the sparsity and ambiguity of commonsense causalities embedded in texts raise concerns about whether the seq2seq model could learn a robust causal dependency model for causality generation. To address this challenge, we first introduce syntactic causal patterns to automatically harvest causalities from the text corpus. Then, we present a causal attention fusion method to augment the seq2seq model with global causal dependency guidance for causality generation. Specifically, we propose a causal attention mechanism that leverages external causal knowledge from CausalNet [2] and a novel attention fusion mechanism that combines the global causal dependency guidance from the causal attention with the local causal dependency obtained from the soft attention mechanism at decoding time steps.

In summary, this paper makes the following contributions:

• We define a new problem, commonsense causality generation, which benefits many NLP applications such as question answering.
• We propose a method to automatically create a cause-effect pairs corpus which facilitates the training process for commonsense causality generation.
• We propose a novel causal attention fusion mechanism which introduces the global causal dependencies observed from an external knowledge source. Extensive experiments show that our approach outperforms multiple strong baselines by a substantial margin.

2. Related Work

Commonsense causal reasoning involves capturing and understanding the causal dependencies among events and actions. The most commonly used dataset for this task is the Choice of Plausible Alternatives (COPA), which includes a premise and two alternatives, along with a prompt specifying the relation between them. Previous studies can be broadly categorized into three lines: feature based methods, co-occurrence based methods, and neural-based methods. All of these approaches are applied to COPA to determine which alternative conveys a more plausible cause or effect based on the prompt from the premise sentence.

As for feature-based methods, Goodwind [3] proposes the COPACETIC system developed by the University of Texas at Dallas (UTD) for COPA. They take COPA as a classification problem and use features derived from varied datasets. For co-occurrence based methods, Roemmele [1] and Gordon [26] focus on lexical co-occurrence statistics gathered from story corpora. They use the Pointwise Mutual Information (PMI) statistic [27] to compute the frequency of two words co-occurring within the same context relative to their overall frequency. It is essential to note that this co-occurrence measure is order-sensitive. In contrast, Luo [2] proposes a framework that automatically harvests a network of causal-effect terms from a large web corpus referred to as the CausalNet. This framework identifies sequences matching lexical templates indicative of causality.

In neural-based methods, Roemmele [28] introduces a neural encoder-decoder model capable of learning to predict relations between adjacent sequences in stories as a means of modeling causality. Dasgupta [29] adopts meta-reinforcement learning to solve a range of problems, each containing causal structure, using an RNN-based agent trained with model-free reinforcement learning (RL). Additionally, Yeo [30] addresses the problem of multilingual causal reasoning in resource-poor languages, constructing a new causality network (PSG) of cause-effect terms, targeting machine-translated English without any language-specific knowledge of resource-poor languages.

Recently, neural-based sequence-to-sequence models have met a great success in machine translation and summarization [31,32,33,34,35], especially CNN seq2seq models [12,25,36,37], which are much faster than others. It shows that the CNN seq2seq models can obtain the latent relation between encoder input and decoder output. Inspired by this, we identify the causality generation problem.The proposed model employs a CNN seq2seq architecture combined with a causality attention mechanism. To facilitate training, we introduce a causality dataset consisting of cause-effect sequence pairs. The causality attention mechanism is a hybrid approach, incorporating both traditional attention and causal strength computed by CausalNet [2]. We further introduce an explicit switch probability $\lambda$ to adjust the traditional attention distribution and causal strength distribution. This fusion attention mechanism enables the model to capture causality between texts, thereby allowing the generation of causal sentences from input sentences.

To facilitate a comprehensive comparison among the aforementioned methods, we summarize the main objectives and pros and cons of each method in

Table 1. Comparison of methods for commonsense causal reasoning.

Method	Main Objective	Pros	Cons
Feature-based methods	Causal/Non-causal classification	Utilizes diverse feature datasets	Limited to predefined features
Co-occurrence based Methods	Causal strength computation	Captures statistical dependencies	Limited to lexical templates
Neural-based methods (previous)	Causality prediction	Provides a high level of accuracy for causal/non-causal classification	Limited to acquiring causal pairs; Struggles with capturing complex causalities
Neural-based methods (ours)	Causality generation	Enables the generation of causal sentences; captures complex causalities leveraging external knowledge sources	Limited to the word-based attention fusion mechanism

.

3. Our Approach

In this section, we first describe our methodology for cause-effect pairs extraction. Then we illustrate the causal attention mechanism and its fusion with the soft attention within the CNN seq2seq model for generating commonsense causality.

3.1. Cause-Effect Pairs Extraction

Sentences serve as fundamental semantic units that comprehensively describe events, encompassing both agents and actions. We build a new dataset of cause-effect sentence pairs from novels (described in Section 4). The extraction process involves identifying main and subordinate clauses linked by causal conjunctions, specifically “because” and “so”. To be more specific, we carry out the extraction as follows:

• Cue Sentences Collection. We begin by collecting sentences containing either “because” or “so” through regular expression matching. These sentences are then processed using the Stanford CoreNLP [38] tools for tokenization, POS tagging, constituent parsing, and dependency parsing.
• Negation Detection. During the analysis, if the causal conjunction node in the dependency tree has a negation word sibling, such as “not” or “n’t”, we exclude this sentence from further extraction.
• Clause Detection. To extract cause-effect sentence pairs, we devise syntactic pattern matching rules. Specifically, we ensure that the occurrence of“because" in the sentence introduces a subordinate conjunction word leading to a subordinate clause (tagged as ‘SBAR’), which in turn contains a declarative sentence(‘S’) with a subject(‘NP’) and predicate(‘VP’). Similarly, we design analogous syntactic rules for the ‘so that’ pattern. Additional illustration of these syntactic rules is provided in Figure 1.
• Spans Extraction. Next, we remove redundant punctuation and unwanted sentence constituents from both the main clause and the subordinate clause subtrees. The text spans within these subtrees are then extracted as cause-effect sentence pairs.

Input Transformations. We generate two data examples from each cause-effect sentence pair, one for forward causal reasoning and one for backward causal reasoning. During training, we prepend the special tokens $〈f〉$ or $〈b〉$ to the target span, indicating whether it corresponds to a forward or backward reasoning example. For testing, we initialize the first token of the generated sequence as $〈f〉$ or $〈b〉$ to perform either forward or backward reasoning accordingly. Furthermore, we perform co-reference resolution for the cue sentences, as we create both forward and backward reasoning examples. Finally, as shown in Figure 2, we generate two source-target pairs from the example sentence in Figure 1.

3.2. Background: CNN Seq2seq Model

In this section, we briefly introduce the convolutional sequence-to-sequence (CNN seq2seq, https://github.com/facebookresearch/fairseq, accessed on 22 November 2023) model for causality generation. Later, we enhance the model by incorporating our proposed mechanism of causal attention along with its corresponding fusion approach with this framework.

In the CNN seq2seq models, given the input elements $\mathbf{x}=(x_1,x_2,\dots ,x_m)$ and output elements $\mathbf{y}=(y_1,y_2,\dots ,y_n)$ ($m>n$), we obtain the input representations $\mathbf{X}=(X_1,\dots ,X_m)$ and output representations $\mathbf{Y}=(Y_1,\dots ,Y_n)$ by combining word embeddings with their corresponding position embedding vectors. The encoder and decoder of the CNN seq2seq framework are composed of stacked convolutional layers, each with L layers ($L=8$). Specifically, the output of the l-th convolutional layer in the encoder is denoted as ${\mathbf{z}}^l=\left(z_1^l,\dots ,z_m^l\right)$, while in the decoder, it is represented as ${\mathbf{h}}^l=\left(h_1^l,\dots ,h_n^l\right)$. Each convolutional layer in both the encoder and decoder employs architectural features such as Gated Linear Units (GLU) [39] and residual connections [40], facilitating effective progressive information propagation across layers. The output state at the i-th position of the l-th layer, denoted as $h_i^l$, can be expressed as follows:

$$h_i^l=GLU\left(W^l\left[h_{i-k/2}^{l-1},\dots ,h_{i+k/2}^{l-1}\right]+b_w^l\right)+h_i^{l-1},$$

(1)

where k is the kernel width.

While decoding, the cross-attention mechanism is incorporated into each decoding layer to introduce adaptable and dynamic weights that quantify the semantic relationships between individual input elements and the current output element. To compute attention scores (i.e., dynamic weights), the current decoding state, $h_i^l$, is combined with the embedded representation of previously generated elements, $Y_i$, resulting in the construction of a decoding state summary, $d_i^l$, formulated as follows:

$$d_i^l=W_d^l{h}_i^l+b_d^l+Y_i,$$

(2)

where $W_d^l$ and $b_d^l$ denote parameters of the l-th linear transformation layer. Within the l-th decoding layer, the attention score $a_{ij}^l$ is computed as the dot product between $d_i^l$ and the encoded output state $z_j^L$ linked to the j-th input element, which stems from the last layer of the encoder (i.e., L-th encoding layer):

$$a_{ij}^l=\frac{exp\left(d_i^l·z_j^L\right)}{{\sum }_{t=1}^{m}exp\left(d_i^l·z_t^L\right)},$$

(3)

where ${\sum }_{t=1}^m{a}_{it}^l=1$, and $a_{ij}^l$ assumes values ranging from 0 to 1. Additionally, the conditional input $c_i^l$ to the current decoder layer is a weighted sum of both encoder states $z_j^L$ and input element representations $X_j$, utilizing attention scores obtained from Equation (3):

$$c_i^l=\sum _{j=1}^m{a}_{ij}^l\left(z_j^L+X_j\right).$$

(4)

Moreover, $c_i^l$ is added to $h_i^l$ as the input for the next decoder layer. In the decoding phrase, to generate the next element $y_{i+1}$, a linear transformation is first applied, followed by a softmax transformation, to the uppermost decoder output state $h_i^L$, facilitating the computation of the probability distribution $p\left(y_{i+1}|y_1,\dots ,y_i,\mathbf{x}\right)$:

$$p\left(y_{i+1}|y_1,\dots ,y_i,\mathbf{x}\right)=softmax\left(W_o{h}_i^L+b_o\right)\in {\mathbb{R}}^T,$$

(5)

where $W_o$ and $b_o$ are the parameters of the linear transformation layer.

3.3. Causal Attention Fusion Mechanism

The CNN seq2seq model for causality generation is trained using the extracted cause-effect pairs described in Section 3.1. However, the scarcity and noise presented in the training pairs give rise to concerns regarding its capacity to effectively learn a robust causal dependency model for causality generation. To address this limitation, we propose a causal attention fusion mechanism that incorporates external causal knowledge sourced from CausalNet [2]. Our causal attention fusion mechanism aggregates causal information from the global and local causal dependency to refine the decoding at every time step. We illustrate the proposed attention fusion mechanism as shown in Figure 3.

To be more specific, CausalNet [2] consists of a comprehensive web-scale corpus network of word-based causal relationships, along with the causal strength metric $C{S}_{\lambda }$ that could quantify the causal relationships among short texts. In seq2seq models, the attention mechanism governs the semantic interconnections between input and output elements, dynamically assigning weights to measure the strength of these semantic links, denoted as attention scores. As for causality generation, such semantic links can be interpreted as indicative of causal dependencies. Backed by the CausalNet framework and its causal strength metric $C{S}_{\lambda }$, we compute global causal attention scores. These scores are then integrated with the original attention scores to infuse augmented semantic representations through the amalgamation of causal knowledge. This integration enhances the effectiveness of the causal generation model. Notably, given the multiplicative nature of the $C{S}_{\lambda }$ method, we compute causal attention scores by utilizing word-level causal relationship strength (referred to as $cs$) within a logarithmic context. To further illustrate our model design, we consider the example of forward reasoning, wherein the input signifies the cause, and the generated target signifies the effect. Here, we assume that $x_j$ is the j-th word in the input sequence (cause events), which is denoted as $x_j^{\left(c\right)}$, and $y_i$ is the i-th word in the target sequence (effect events), denoted as $y_i^{\left(e\right)}$. Thus, the logarithm causal strength between $x_j^{\left(c\right)}$ and $y_i^{\left(e\right)}$ is computed as follows:

$$cs(x_j^{\left(c\right)},y_i^{\left(e\right)})=log\left(C{S}_{\lambda }(x_j^{\left(c\right)},y_i^{\left(e\right)})\right)-log\xi .$$

(6)

Here, $\xi$ is the minimum causal strength for all causal pairs, defined as:

$$\xi =C{S}_{\lambda }({\widehat{u}}^{\left(c\right)},{\widehat{v}}^{\left(e\right)}),$$

(7)

$${\widehat{u}}^{\left(c\right)},{\widehat{v}}^{\left(e\right)}=\underset{\begin{array}{c}{u}^{\left(c\right)},v^{\left(e\right)}\\ C{S}_{\lambda }(u^{\left(c\right)},v^{\left(e\right)})>0\\ u\in \mathcal{V},v\in \mathcal{V}\end{array}}{arg\; min}C{S}_{\lambda }(u^{\left(c\right)},v^{\left(e\right)}),$$

(8)

where $\mathcal{V}$ is the vocabulary. Similarly, during backward reasoning, we consider the input sequence as the effect and generate the output sequence as the cause. As illustrated in Section 3.1, we transform the input data format to distinguish between forward and backward reasoning.

To compute the causal attention score $c{a}_{ij}^l$ utilizing the word-based causal strength $cs(x_j^{\left(c\right)},y_i^{\left(e\right)})$, we first introduce weights ${{\mathbf{w}}_{\mathbf{p}}}^l$ for each decoder layer to reserve the hierarchical semantic attending information. Then, we compute the weighted average of the causal strengths $cs(x_j^{\left(c\right)},y_{·}^{\left(e\right)})$ between the source word $x_j$ and all target candidate words $y_{·}\in \mathcal{V}$ as the causal attention score:

$$c{a}_{ij}^l={{\mathbf{w}}_{\mathbf{p}}}^l·\mathbf{cs}\left(x_j\right),$$

(9)

where $\mathbf{cs}\left(x_j\right)$ represents the causal strength vector. The i-th element of $\mathbf{cs}\left(x_j\right)$ is denoted as $cs({x_j}^{\left(c\right)},{{\mathcal{V}}_i}^{\left(e\right)})$, with i representing the index corresponding to vocabulary terms. We substitute $c{a}_{ij}^l$ in Equation (9) for the $a_{ij}^l$ in Equation (3). This forms the causal attention baseline model denoted as CAttn model in our experiments.

The original cross attention mechanism $a_{ij}^l$ is capable of learning local semantic dependencies and propagating them layer by layer. In contrast, the causal attention $c{a}_{ij}^l$ harvested from CausalNet [2] carries the causal dependency globally. Thus, we propose an attention fusion mechanism to fuse the causal attention with the stacked soft attention, equipping the sequence-to-sequence model with global causal information. Specifically, we combine the global causal dependency with the local causal dependency using a weighting scheme to make the model more expressive. We implement this by weighing the causal attention scores $c{a}_{ij}^l$ and the soft attention scores $s{a}_{ij}^l$ through ${\lambda }_{cs}^l$ called causal fusion weight. The fused attention score $f{a}_{ij}^l$ is computed as follows:

$$f{a}_{ij}^l={\lambda }_{cs}^l·c{a}_{ij}^l+(1-{\lambda }_{cs}^l)·s{a}_{ij}^l,$$

(10)

where ${\lambda }_{cs}$ represents the weight of causal attention score, controlling the influence of global causal information in the model. Equations (3) and (9) show the computation of $c{a}_{ij}^l$ and $s{a}_{ij}^l$ separately. Then, we design the causal fusion weight ${\lambda }_{cs}^l$ as the normalized trainable parameter to make the weighing scheme more flexible. Thus, ${\lambda }_{cs}^l$ is computed as:

$${\lambda }_{cs}^l=\sigma \left(w^l\right),$$

(11)

where $\sigma$ denotes the sigmoid function which normalizes ${\lambda }_{cs}^l$ between 0 and 1, and $w^l$ is a learnable parameter.

4. Experiments

In this section, we first give some statistics of our corpus, and evaluate the quality of our extracted cause-effect causal training pairs. We then introduce two evaluation datasets and a range of evaluation metrics we adopted in our experiments. Next, we compared our results on the commonsense causality generation task against a number of baselines.

4.1. Dataset Description

We first describe the text corpus we used for cause-effect pairs extraction, and then introduce two evaluation datasets for the following experiments.

4.1.1. Cause-Effect Pairs Corpus

Causal reasoning has been hindered by the lack of extensive labeled resources, owing to issues like the intricacies of data annotation, substantial costs, and inadequate standardization. Previous works [26,41] show that the data of story texts from novels and books benefit the commonsense reasoning tasks more than other sources. Thus, we use a large number of novels, which are a good source of commonsense causal information, as our text corpus for the cause-effect pairs extraction.

Specifically, we first crawl the raw novel corpus from Library Genesis (libgen.is), a search engine for articles and books that allows free access. To maintain the quality of our corpus, we collect a group of prize-winning lists and top-x lists for novels, for example the Nobel Prize in Literature and Time Magazine All Time 100 Novels, and limit our crawler to them. The resulting 19,000 novels are split into around 100 million sentences, which served as our data source after filtering and text cleaning. We perform the cause-effect pairs extraction process on the crawled corpus (described in Section 3.1). Each pair can be transformed into two examples following the input transformation shown in Figure 2. Finally, we obtain 539,930 cause-effect example pairs, then split them into training split (431,944 examples, 80%), validation split (53,993 examples, 10%), and test split (53,993 examples, 10%).

4.1.2. Evaluation Datasets

We conduct the experiments on two evaluation datasets for the generation-based causal reasoning. We use the test split in Section 4.1.1 as the first evaluation dataset denoted as NOVTest. NOVTest contains 53,993 examples and shares the same data distribution as the training data. The specific examples are illustrated in Figure 2.

To better understand the performance of our model against related research, we also use the Choice of Plausible Alternatives (COPA) as the second evaluation dataset, which is a widely-used causal reasoning benchmark. It consists of one thousand multiple-choice questions. Each question is composed of a premise and two alternatives, where the task is to select the more plausible alternative as a cause (or effect) of the premise. We perform the input transformation on each question, which puts the special token $〈f〉$ or $〈b〉$ in front of the premise or alternative to form a cause-effect sequence pair. We use the test split in [26] which contains 500 questions. After the input transformation, the built COPATest evaluation dataset consists of 1000 cause-effect pairs.

4.2. Baselines

There are five baselines in the evaluation experiments, which can be divided into two categories: the selection-based methods and the generation-based methods. In this section, we describe those baselines in details.

4.2.1. Selection-Based Methods

To perform the causality generation task, selection-based models first automatically generate a set of target candidates $C^t$ for the given source s, and then select the best answer from those candidates to reason about causalities. For instance, in the context of forward reasoning as an example, when a cause s is designated as source, we automatically match s with all the cause events extracted from the corpus and select 10 semantically most similar cause events, forming the set $C^s$. Within $C^s$, for each $s^{\prime }$, its corresponding effect event $t^{\prime }$ constitutes the target candidate set $C^t$, with each pair $(s^{\prime },t^{\prime })$ serving as a training instance. Then we compute the causal strength between s and each t in $C^t$ to select the best answer and generate it as the target. We design three selection-based baselines as follows, among which the main difference is the matching approach when generating the candidate set $C^s$ for s. The first method, OUniMat (Original Unigram Matches), computes the count of matches between s and the unigram words present in the cause events from the training data (normalized by event length) to select the most matching events to form $C^s$. A variant of this method, termed PUniMat (Processed Unigram Matches), involves preprocessing events by removing stopwords before matching. Additionally, we explore the use of bigram words as a substitute for unigram words in matching, referred to as BiMat (Bigram Matches).

4.2.2. Generation-Based Methods

While the Transformer Seq2Seq model does offer the advantage of capturing long-range dependencies and performs well in text generation tasks, our experiments revealed notable advantages of the CNN Seq2Seq model in terms of swift convergence and ease of training. Transformer Seq2Seq is data hungry and sensitive to noise, making it unsuitable for weakly supervised causal generation tasks. Therefore, generation-based baselines for causality generation are all based on the CNN encoder-decoder sequence-to-sequence model. We denote the model using multi-layer soft attention mechanisms as MultiSAttn (Multi-step Soft Attention) and the model incorporating causal attention mechanisms as CAttn (Causal Attention.). Finally, we propose the FAttn (Fused Attention) model equipped with a fusion mechanism to combine both attention types. These models can directly generate the causalities without the $C^t$ selection step in selection-based baselines.

4.3. Evaluation Metrics

We evaluate the performance of our methods by automatic metrics and human evaluation.

Automatic Metrics. We use two automatic metrics for evaluation: BLEU scores and Accuracy.

• BLEU scores [42] includes BLEU-1, BLEU-2, and BLEU-3 which combines modified n-gram precision and a sentence brevity penalty:

  $$BLEU=min(1,exp(1-\frac{r}{c}))·\sum _{n=1}^N\left(w_n{p}_n\right),$$

  (12)

  where r is the sum of the best match lengths for each generated and corresponding reference text in the corpus and c is the total length of the predicted corpus. $w_n=\frac{1}{N}$ is uniform weights. N is based on the type of BLEU, for example, N is 2 for BLEU-2 and $p_n$ is the modified n-gram precision. To compute $p_n$, one first counts the maximum number of times a word occurs in a reference text. Next, one clips the total count of each predicted word by its maximum reference count, adds these clipped counts up, and divides by the total number of predicted words.
• Accuracy (Acc) is the evaluation metric for COPA, computed as:

  $$Acc=\frac{n_{ct}}{N_{all}},$$

  (13)

  where $n_{ct}$ and $N_{all}$ denotes the number of the questions with correct selection and all questions in COPA, respectively. We generate cause (or effect) according to the premise in COPA and select the more “similar” alternative based on the similarity score, BLEU-1 metric, which is well-suited for evaluating the alignment between short sequences.

Human Evaluation. We use human-evaluated accuracy to supplement the automatic evaluation, considering the intricate many-to-many relationship inherent in the determination of cause and effect. To this end, we randomly sample generated cause-effect pairs for evaluation by three human annotators. For a given sample, we ask each annotator to manually inspect whether there is a correct causality between source sequence and generated sequence, and the final result depends on the consensus reached by at least two annotators. Then, we calculate the percentage of the generated cause-effect pairs with correct causality in each model. This estimates the ability of models to generate correct causes (or effects) based on provided premises.

4.4. Experiment Results and Analysis

In this section, we compare our proposed model with all competing models described above both quantitatively and qualitatively.

4.4.1. Comparison Results

We evaluate our model as well as baselines (Section 4.2) on two evaluation datasets (Section 4.1) using both automatic metrics and human evaluation metrics (Section 4.3). The comparison results are shown in

Table 2. Comparisons of results on NOVTest dataset.

Model	BLEU-1	BLEU-2	BLEU-3	Human Accuracy
OUniMat	17.71	1.17	0.0	0.26
PUniMat	17.54	-	-	0.22
BiMat	19.95	2.55	0.01	0.23
CAttn	14.41	4.74	1.21	0.18
MultiSAttn	18.72	9.67	4.76	0.21
FAttn	20.06	9.98	4.82	0.35

and

Table 3. Comparisons of results on COPATest dataset.

Model	BLEU-1	BLEU-2	BLEU-3	Accuracy
OUniMat	11.15	0.14	0.0	0.26
PUniMat	12.35	-	-	0.17
BiMat	13.42	0.41	0.0	0.26
CAttn	21.64	10.90	1.94	0.21
MultiSAttn	37.26	27.68	18.42	0.25
FAttn	36.94	27.78	18.73	0.52

.

To be more specific, Table 2 and Table 3 show the comparisons of BLEU scores and human-evaluated accuracies among all competing models across the NOVTest and COPATest datasets. Notably, the PUniMat model, owing to the removal of stop words, is incapable of computing bi-grams and tri-grams, thus preventing the evaluation of BLEU-2 and BLEU-3 scores. We can see that except for a slightly lower BLEU-1 score compared to the MultiSAttn model, the causal attention fusion approach, FAttn, consistently outperforms all other baselines across all metrics on both evaluation datasets. Backed by the CNN seq2seq architecture, FAttn achieves BLEU-1 scores of 20.06 and 36.94, BLEU-2 scores of 9.98 and 27.78, along with human-evaluated accuracy rates of 35% and 52% for the NOVTest and COPATest, respectively.

Furthermore, we compare the performance of neural-based models, namely CAttn, MultiSAttn, and FAttn on both NOVTest and COPATest datasets, as shown in Figure 4. We observe that the CAttn model underperforms in the generative-based approach due to its reliance on attention scores derived from global causal knowledge, which is confined to word-level causal associations. Therefore, at each decoding step, the causal strength vectors for different decoding layers remain the same, hindering the effective transfer of lower-level learned distributional features to higher levels and impeding the utilization of multi-layer convolutional networks for feature combination. In contrast, the MultiSAttn model, which employs a multi-step soft attention mechanism, effectively propagates semantic mapping information represented by attention across layers, achieving slightly better performance. Moreover, the FAttn model, utilizing the proposed causal attention fusion mechanism, combines the causal dependency information acquired through multi-step soft attention with global causal attention drawn from external knowledge sources, ultimately achieving superior results.

4.4.2. Case Studies

We have observed that, compared to the selection-based approach, the generation-based approach is more flexible and effective, performing better in end-to-end experiments. Next, we perform case studies to examine the impact of various attention mechanisms on the generation-based models. The case studies utilize running examples from the NOVTest and COPATest datasets, which are presented in

Table 4. The influence of attention mechanism for causality generation.

(a) Causality generation case from the NOVTest dataset.
Cause	the little boy forward was always in the shelter of her arms
Effect	lien pushed the little boy forward and inched herself along on her heels
forward reasoning	CAttn	<f>  unk had been a good one
	MultiSAttn	<f>  they were n’t going to kill them
	FAttn	<f>  unk was upset about it
backward reasoning	CAttn	<b>  the unk had been so much more than a year ago
	MultiSAttn	<b>  she was so small
	FAttn	<b>  the little boy was so fond of her
(b) Causality generation case from the COPATest dataset.
Cause	the husband discovered that the husband wife was having an affair
Effect	the husband filed for divorce
forward reasoning	CAttn	<f>  i was n’t sure what to say
	MultiSAttn	<f>  the husband was not in the least concerned
	FAttn	<f>  the husband was disappointed
backward reasoning	CAttn	<b>  the unk had to be careful
	MultiSAttn	<b>  the woman was convicted of the woman
	FAttn	<b>  the woman wanted to get rid of her husband

.

Table 4 presents the study cases from the NOVTest and COPATest datasets, including results for both forward and backward causal reasoning. We can see that the CAttn model, due to its insistence on using global causal attention for all decoding layers, loses hierarchical semantic alignment information. Therefore, CAttn tends to generate less informative outcomes and also has a tendency to produce out-of-vocabulary words. In contrast, the performance of the MultiSAttn and FAttn models, which incorporate multi-layer attention mechanisms, is significantly superior.

Specifically, as an illustration of forward reasoning, as seen in Table 4, given the cause “his wife was having an affair”, FAttn effectively generates a coherent effect “the husband was disappointed”, owing to its integration of comprehensive global causal knowledge that allows it to discern the causal link between “have an affair” and “disappoint”. In contrast, the MultiSAttn model infers a much less plausible effect, that “the husband was not in the least concerned”. However, since the fused causal attention is word-based, which could fail to entirely encapsulate the causal dependencies between events, it potentially results in contextual semantic incongruity during inference.

5. Implications

Our main aim in this study is to address the problem of generation-based commonsense causal reasoning, which has widespread implications for real-life applications such as question answering. While existing methods have shown effectiveness in causality prediction, they encounter difficulties when it comes to handling generation scenarios. To overcome these challenges, we propose a convolutional sequence-to-sequence framework. Firstly, we introduce syntactic causal patterns to automatically extract causalities from a large text corpus. Next, we present a causal attention fusion method to augment the seq2seq model by providing global causal dependency guidance for causality generation. In Section 4.4, we demonstrate the effectiveness of the components integrated within our framework.

A key insight in the generation of causal relations lies in the importance of both global and local causal knowledge. To capture this information, we employ a causal attention mechanism that utilizes external causal knowledge from CausalNet. Furthermore, we introduce a novel attention fusion mechanism that combines the global causal dependency guidance from the causal attention with the local causal dependency obtained from the soft attention mechanism during decoding steps.

6. Conclusions

In this paper, we focus on generation-based commonsense causal reasoning. Specifically, we introduce a novel causal attention mechanism that harnesses external knowledge from CausalNet and a unique fusion mechanism that integrates global causal dependency insights from the causal attention with local causal dependencies obtained through multi-layer soft attention within the CNN seq2seq architecture. The experimental results reveal that within the causal attention fusion framework (based on CNN seq2seq architecture), the FAttn model achieves the highest performance. It attains BLEU-1 scores of 20.06 and 36.94, BLEU-2 scores of 9.98 and 27.78, and human-evaluated accuracy rates of 35% and 52% on two evaluation datasets, respectively, outperforming all other baselines across all metrics on both evaluation datasets. However, the word-based nature of our proposed causal attention mechanism and subsequent fusion mechanism presents a limitation in effectively capturing the intricate causal relationships that arise from events involving multiple agents and complex structures. Consequently, our model may occasionally grapple with contextual and semantic incoherence during the generation of causal relationships. In further work, it is crucial to explore alternative sources of causal knowledge beyond CausalNet, capable of mitigating the constraints associated with word-based mechanisms. One potential direction involves incorporating higher-level representations, such as phrase or sentence-level information, to comprehensively capture complex causal relationships.