Multistage Mixed-Attention Unsupervised Keyword Extraction for Summary Generation

Abstract

Summary generation is an important research direction in natural language processing. Aimed at the problems of redundant information processing difficulties and an inability to generate high-quality summaries from long text in existing summary generation models, BART is the backbone model, an N + 1 coarse–fine-grained multistage summary generation framework is constructed, and a multistage mixed-attention unsupervised keyword extraction summary generation model is proposed (multistage mixed-attention unsupervised keyword extraction for summary generation, MSMAUKE-Summ^N). In the N-coarse-grained summary generation stages, a sentence filtering layer (PureText) is constructed to remove redundant information in long text. A mixed-attention unsupervised approach is used to iteratively extract keywords, assisting summary inference and enriching the global semantic information of coarse-grained summaries. In the 1-fine-grained summary generation stage, a self-attentive keyword selection module (KeywordSelect) is designed to obtain keywords with higher weights and enhance the local semantic representation of fine-grained summaries. Tandem N-coarse-grained and 1-fine-grained summary generation stages are used to obtain long text summaries through a multistage generation approach. The experimental results show that the model improves the ROUGE-1, ROUGE-2, and ROUGE-L metrics by a minimum of 0.75%, 1.48%, and 1.25% over the HMNET, TextRank, HAT-BART, DDAMS, and Summ^N models on summarized datasets such as AMI, ICSI, and QMSum.

1. Introduction

With the continuous development of large language models, text generation becomes a research hotspot in the field of natural language processing. It has gained a wide range of applications for tasks such as article creation, dialog generation, and conference summary. Summary generation is a task of text generation [1], and researchers use different techniques to generate summaries, including rewriting or extracting various information from documents [2]. Currently, lots of research results have been achieved in generating summaries for short texts [3,4]. However, it is still difficult to quickly obtain high-quality summaries from long textual data such as long documents, news interviews, and conference content. Therefore, improving the quality of summaries of long texts is of great research importance [5].

In the task of summary generation for long text, the current summary generation model has the problem of failing to comprehensively extract the important information of long text, and at the same time, there is redundant information in long text. Therefore, in order to better improve the generative ability of the summarization model, sentence filtering is usually used to remove redundant information in long texts during data preprocessing. To address the lack of an ability of summarization models to generate important information, based on deep learning models, supervised methods are designed by researchers to extract keywords, which require a large labeled training dataset. On the other hand, unsupervised methods do not require labeled datasets and have lower dataset requirements [6]. Therefore, unsupervised keyword extraction has been adopted by many scholars, and unsupervised keyword extraction methods based on text embedding, document embedding, and hybrid statistical embedding have emerged, such as MDERank [7], SIFRank [8], and KeyGames [9]. Although the above methods improve the extraction of keywords, the lower-weighted keywords may be used first for summarization inference, while the higher-weighted keywords fail to be applied, resulting in poor quality and readability of the summaries [10].

Therefore, the self-attention mechanism has been used by many scholars to select keywords with higher weights. For example, when generating summaries for long texts or multiple documents, a self-attention mechanism is employed by the Transformer model [11] to capture the dependencies between keywords in the input texts or documents. It achieves a focus on some of the important keywords. However, when the extracted keywords receive a small attentional weight, it affects the complete encoding of the keywords as well as the representation of important information.

The rest of this paper is organized as follows: Section 2 reviews the work related to this study. In Section 3, the framework of the summary generation model MSMAUKE-Summ^N is given and the components of the framework are described in detail. Experimental results and analysis related to the MSMAUKE-Summ^N model are shown in Section 4. Section 5 summarizes the conclusions of this paper and future research directions.

2. Related Work

Currently, Transformer-based summary generation models have made great progress in text summarization [11]. However, due to the large number of words in long texts and the sparse distribution of important information [12], the quality of summarization in long texts is poor as it is difficult for the summary generation model to capture the dependencies between important information in the text. A common solution is to reduce the input long text to short text; the BART model was proposed by Lewis et al. [13], which uses truncation of the input to reduce long text. Another approach is to improve the attention to cope with long text summarization tasks; Local Sensitive Hashing (LSH) attention was utilized by Kitaev et al. [14] to optimize the attention mechanism in Transformer to reduce the effect of quadratic complexity. In addition, a hierarchical self-attention mechanism was used by HMNet [15] and HAT-BART [16] to improve the input constraints of typical self-attention models, thus adapting the models to longer text inputs. Although the above methods can handle text content of a certain length, they are not flexible enough when the length of the text varies greatly.

The multistage modeling framework is capable of adapting to texts of different lengths by varying the number of stages, which is a more flexible way of processing compared to the truncated input and improved self-attention approaches. Therefore, the multistage summary generation model framework has been adopted by many scholars to adequately process information such as long texts or documents. A multistage summary generation framework structure, Summ^N, was proposed by Zhang et al. [17]. It is used to extract summaries of long texts and documents, and the final summary is obtained by N-coarse-grained and one-fine-grained summary generation stages. A staged long-text summary generation approach (PLSGA) was proposed by Fang et al. [18] to obtain the final summary. It includes key information extraction, extraction model training, generative model training, and candidate summary screening stages. A two-stage automatic summary generation model combining dual topic embedding and sentence absolute position embedding was proposed by Ren et al. [19]. It incorporates semantic features by introducing topic embedding in each of the two stages, extracts important information of the text in the extraction stage, and reduces the redundancy of the summary content. In addition, a sentence filtering layer was constructed by Mei et al. [20] for data preprocessing. It removes redundant sentences in long texts, shortens the text length, and provides the model with more streamlined data samples. The above methods reduce the adverse effects of excessively long text by moderating the number of stages, but these methods do not comprehensively consider the keywords and local semantic information of the text, resulting in summaries that omit important information.

The summary generation task needs to consider the importance of keywords for text summarization in addition to the effect of the length of the text. From past research work, it can be observed that with the continuous development of Pretrained Language Models (PLMs), many text generation studies have been centered around the technique of keyword extraction. Therefore, PLMs are used by many researchers as an embedding layer of keyword extraction models [21,22] to improve the extraction of keywords. Keyword extraction models are mainly classified into two categories, supervised and unsupervised. The use of supervised keyword extraction requires a large amount of labeled training data and the models generalize poorly outside the training data domain. Therefore, unsupervised learning was used by Eirini et al. [23] for keyword extraction, where the adjacency matrix corresponding to the word map of a target document is used as a vector representation of the vocabulary of that document, and distributed modeling is performed. To address the inability of embedding-based keyword extraction models to fully utilize PLMs, an unsupervised keyword extraction method, PromptRank, was proposed by Kong et al. [24]. It calculates the probability of keyword generation by the decoder based on prompts. Interpretable neural networks were used by Joshi et al. [25] for unsupervised keyword extraction, defining keywords as iconic words that are favorable for predicting the topic of a document. The above unsupervised approach simplifies the process of keyword extraction, but does not fully consider the importance of the keywords and the semantic relevance of the keywords to the sentences in the document. A mixed-attention model, AttentionRank, was proposed by Ding et al. [26] to extract keywords from documents in an unsupervised manner. It uses self-attention to determine the importance of keywords in the context of a sentence and cross-attention to calculate the semantic relevance between keywords and sentences in a document. The above methods were able to adequately extract the keywords from the text, but failed to enable the keywords to be used correctly in the inference of the summaries, resulting in the absence of important information in the summaries.

The uneven distribution of the extracted keywords in the original text and the lack of a rational method for their use lead to the fact that some keywords with lower weights may be used first in the inference of the summary, affecting the representation and distribution of important information in the summary. Therefore, in order to increase the detailed information of the summary, the targeted selection of keywords with higher weights to aid in inference is a critical step in summary generation. A method of selecting the best keywords was proposed by Tim et al. [27] to iteratively select keywords through an attention scoring function to obtain the best results overall. Meanwhile, in order to select more representative keywords, a probabilistic method was proposed by Akash et al. [28] for selecting representative keywords in a professional domain. A binary mixture model is used to obtain the distribution of keywords and select a representative subset of keywords. A keyword selection method was proposed by Venkatesh et al. [29]. The self-attentive head of the last layer of the pretraining encoder is clustered to select the most popular keywords from the original article. To address the problem of data sparsity in low-resource complex Named Entity Recognition (NER), an attention graph-aware keyword selection method was proposed by Ghosh et al. [30]. It uses attention graph-assisted selective masking to retain named entities in the input sentence and select contextually relevant keywords. The above methods select the most representative keywords in the text, and although they enhance the local semantic representation of the summary, there is some repetition in the use of keywords.

Aimed at the problems of high redundancy and poor comprehensiveness, accuracy, and readability in existing long text summaries, an N + 1 coarse–fine-grained multistage summary generation framework is constructed and a multistage mixed-attention unsupervised keyword extraction summary generation model (MSMAUKE-Summ^N) is proposed. During data preprocessing, the sentence filtering layer, PureText, is set to remove useless information from the data samples to obtain high-quality data samples. A mixture of self-attention and cross-attention is used to consider the importance and semantic relevance of keywords at the sentence level and document level. Multistage iterative extraction of keywords is used to facilitate summarization inference to enrich the global semantic information of coarse-grained summaries, rationalize the use of keywords using the self-attention mechanism, select more weighted keywords to assist the summary, and increase the detailed information of the summary to enhance the local semantic representation of the fine-grained summary.

3. MSMAUKE-Summ^N Model

Aimed at the problems of information redundancy, poor accuracy, comprehensiveness, and readability in current long-text summaries, an N + 1 coarse–fine-grained multistage summary generation framework is constructed, and a multistage mixed-attention unsupervised keyword extraction summary generation model (MSMAUKE-Summ^N) is proposed. The model framework contains two parts: the N-coarse-grained summary generation stage and the 1-fine-grained summary generation stage. In the N-coarse-grained summary generation stage, the sentence filtering layer, PureText, is constructed to remove redundant information from the original text (D) and the target text (T) to provide high-quality data samples for the summary generation model. An unsupervised approach with mixed attention is used to iteratively extract keywords to assist summary inference and enrich coarse-grained summary (Z) global semantic information. In the 1-fine-grained summary generation stage, self-attention is utilized to select keywords with higher weights to enhance the local semantic representation of the fine-grained summary (F). The framework of the MSMAUKE-Summ^N model is shown in Figure 1.

In Figure 1, an N + 1 coarse–fine-grained multistage summary generation framework is constructed, which connects N-coarse-grained and 1-fine-grained summary generation stages in series to obtain summaries of long texts in a multistage summary generation manner. In this case, in the N-coarse-grained summary generation stage, data filtering and segmentation, and coarse-grained summary generation are iterated N times. With the keyword-enhanced coarse-grained summary (Z) and the target text (T), the model is finetuned in the 1-fine-grained summary generation stage to obtain the final fine-grained summary (F). Specifically, in the process of data filtering and segmentation, the text filtering layer, PureText, is used to remove redundant information from the text, and the original text and the target text are segmented and paired using the ROUGE-based greedy algorithm. In the process of coarse-grained summary generation, mixed-attention screening of the initially extracted keywords is performed; the keywords at the sentence level and document level levels are paid full attention. The contextual relevance of the keywords to the sentences is considered. The final score of the keywords is calculated by linear combination to solve the problem of keyword duplication, so that more valuable keywords can assist the summary inference and enhance the global semantic representation of the summary. During fine-grained summary generation, self-attention focuses on selecting keywords with higher weights to aid summary inference and enhance the local semantic representation of the summary.

3.1. N-Coarse-Grained Summary Generation Stage

3.1.1. Data Filtering and Segmentation

In order to remove redundant information in long text, weakly supervised learning is used to train the BERT model in the N-coarse-grained summary generation stage. A versatile and lightweight sentence filtering layer (PureText) is constructed to filter out redundant sentences by utilizing text similarity. The process is to compare the ROUGE score of each sentence with the score of the standard summary sentence and then rank the importance of the sentence, filtering out up to 60% of the sentences in the text. By fine-tuning the BERT model, the ROUGE-1 metric value is utilized to distinguish the importance of different sentences. Since sentences are subunits of text with grammatical structure, it is natural to choose to filter sentences to produce a more concise text. It is hypothesized that the ROUGE-1 of a sentence is an important indicator that influences the evaluation of a summary. This was carried out by calculating the similarity scores between the sentences and the ROUGE-1 metric values of the standard summary sentences, labeling sentences with scores higher than the median as ‘important’ and sentences with scores lower than the median as ‘unimportant’.

After each sentence is labeled by the classifier, the sentences in the text are ranked according to their respective ‘importance’. The reward, $R_i$, is defined according to the importance of the sentence. In the background of the 0–1 Knapsack algorithm, the set of sentences with the highest accumulated reward, ${\displaystyle \sum R_i}$, is found, while the length of the set of sentences does not exceed the input length limit (L) of the summary generation model. Each sentence is then weighted according to its number of tokens. Finally, the best set of sentences (filtered text) is obtained by filtering. In addition, the filtered original text (D) and target text (T) are segmented and matched to obtain the dataset $\left(D_i,T_i\right)$. Fine-tuning summary generation models with high-quality datasets lead to better models. The data filtering and segmentation are shown in Figure 2.

In Figure 2, the sentence filtering layer PureText performs data preprocessing on the original text (D) and the target text (T) to remove redundant sentences. Then, they are segmented to obtain text paragraphs $(D_1,D_2\cdot \cdot \cdot D_n)$ and $(T_1,T_2\cdot \cdot \cdot T_n)$, respectively.

3.1.2. Coarse-Grained Summary Generation

During coarse-grained summary generation, the process of keyword extraction and integration is executed iteratively N times. Keywords are utilized to aid the inference process of summarization; a multistage approach is taken to extract coarse-grained level summary information from long texts. There are five main steps in the process, which are keyword extraction, self-attention accumulation calculation, cross-attention calculation, linear combination score calculation, and keyword incorporation into summary inference.

(1)

Keyword Extraction

The keyword extraction task is accomplished by using the keyword extraction module implemented in EmbedRank, which is capable of the lexical annotation of words. The words labeled NN, NNS, NNP, JJ, VB, VBD, etc., are recognized and the keywords are obtained by phrase extraction using Python’s own NLTK package. For example, for a given sentence ‘The process of creating experiments is minimized by the simple control of running the program code, achieved by editing most of the program’s parameters into a text file’, the keywords extracted from it are ‘program’s parameters’, ‘text file’, ‘program code’, ‘running’, ‘control’, ‘minimized’, and ‘process’. For the keywords initially extracted from the text, it is also necessary to use attention to calculate the importance of the keywords themselves as well as their relevance scores to the context of the sentence and the document, in order to obtain keywords that are of practical value for summary generation.

(2)

Self-Attention Accumulation Calculation

In the process of calculating self-attention accumulation for keywords, the input text segments $(D_1,D_2\cdot \cdot \cdot D_n)$ are lexically annotated and phrase marked to obtain a keyword set, $K$. The pre-trained BERT is utilized to extract $w_i^k$, which denotes the word self-attention weight of the keyword, $k$, in the $i$-th sentence. The accumulated self-attention weight, $a_i^k$, for the keyword, $k$, in the sentence, $i$, is then calculated. Finally, the self-attention of the keywords in each sentence is summed to obtain the document-level self-attention value, $a_k$, for the keyword, k. The self-attention accumulation calculation is shown in Figure 3.

In Figure 3, $a_w$ is obtained from the pre-trained BERT, and $a_w$ denotes the value of the self-attention weights of the words in the sentence; the equation is shown below:

$$a_w={\displaystyle \sum _{w^{\prime }\in s}{a}_{w^{\prime }w}}$$

(1)

where $a_{w^{\prime }w}$ denotes the value of the attentional weight of word $w$ together with all the other words, $w^{\prime }$, in the same sentence, $s$.

Assuming that the sum of the self-attention weight values of keyword k in sentence $i$ is $a_i^k$, the self-attention weight values of the words in $k$ are summed to obtain $a_i^k$; the equation is shown below:

$$a_i^k={\displaystyle \sum _{w\in k}{a}_w}$$

(2)

The document-level self-attention weight value, $a_k$, of keyword $k$ is the sum of the keyword weight values in all sentences; the equation is shown below:

$$a_k={\displaystyle \sum _{i\in d}{a}_i^k}$$

(3)

where $d$ denotes the document and sentence $i$ denotes the $i$-th sentence in the document.

The importance of the keywords is taken into account by calculating the self-attention accumulation of the keywords. However, the evaluation of the keywords is multifaceted and needs to be considered regarding the relevance of the keywords to the context of the sentence and the document.

(3)

Cross-Attention Calculation

To calculate the relevance scores of keywords to sentence context and documents, the network architecture based on the Hierarchical Attention Retrieval (HAR) model and the Bidirectional Attention model is used to design the cross-attention structure. It calculates the relevance between keywords and documents based on the context. A mixture of self-attention and cross-attention is used. The cross-attention calculation is shown in Figure 4.

In Figure 4, keyword $k$ containing $m$ words is denoted as $E^k=\{e_1^k,\dots ,e_m^k\}$, with $e_i\in {ℝ}^H$ denoting the embedding of $w_i$. Similarly, sentence $i$ containing n words is denoted as $E^i=\{e_1^i,\dots ,e_n^i\}$. Calculating document embeddings through cross-attention provides a better measure of the contextual relevance between keywords and sentences in a document.

The equation of the similarity matrix, $S\in {ℝ}^{n\times m}$, between sentence $i$ and keyword $k$ is shown below:

$$S=E^i\cdot E^{kT}$$

(4)

where $E^i\in {ℝ}^{n\times H}$ denotes sentence $i$ and $E^k\in {ℝ}^{m\times H}$ denotes keyword $k$.

The word-based sentence-to-keyword ${\overline{S}}_{i2k}$ and keyword-to-sentence ${\overline{S}}_{k2i}$ similarity calculations are shown in Equations (5) and (6), respectively:

$${\overline{S}}_{i2k}=softma{x}_{row}\left(S\right)$$

(5)

$${\overline{S}}_{k2i}=softma{x}_{column}\left(S\right)$$

(6)

The calculation of cross-attention weights from sentence to keyword, $A_{i2k}$, and word-based keyword to sentence, $A_{k2i}$, are shown in Equations (7) and (8), respectively:

$$A_{i2k}={\overline{S}}_{i2k}\cdot E^k$$

(7)

$$A_{k2i}={\overline{S}}_{i2k}\cdot {\overline{S}}_{k2i}^T\cdot E^i$$

(8)

Based on the cross-attention weights, the new sentence representation, $V^i$, for sentence $i$ is obtained by averaging; the equation is shown below:

$$V^i=AVG\left(E^i,A_{i2k},E^i\odot A_{i2k},E^i\odot A_{k2i}\right)$$

(9)

where $E^i$ denotes the original context of the sentence, $A_{i2k}$, $E^i\odot A_{i2k}$, and $E^i\odot A_{k2i}$ measure the contextual relevance between the sentence and the keywords, and $\odot$ denotes element-by-element multiplication. The new sentence representation is obtained by calculating the average of the sum of the four variables mentioned above, incorporating word-based relationships between keywords and sentences.

A new sentence representation containing $n$ words is $V^i=\left\{v_1^i,\dots ,v_n^i\right\}$, and to highlight the importance of the words, $V^i$ performs self-attention. The average of all columns is calculated to obtain the final representation of the sentence ${\alpha }^i\in {ℝ}^H$. The self-attention and ${\alpha }^i$ of sentence $i$ are shown in Equations (10) and (11), respectively:

$$I=softma{x}_{row}\left(V^i\cdot V^{iT}\right)\cdot V^i$$

(10)

$${\alpha }^i=AVE\left(I\left[:,i\right]\right)$$

(11)

A document, $d$, containing a set of sentences is denoted as $E^d=\left\{{\alpha }^1,\dots ,{\alpha }^i\right\}$, and $E^d$ is subjected to performing self-attention. The average of all columns is calculated as the final document embedding $p^d\in {ℝ}^H$, and the self-attention and $p^d$ of document $d$ are shown in Equations (12) and (13), respectively:

$$P=softma{x}_{row}\left(E^d\cdot E^{dT}\right)\cdot E^d$$

(12)

$$p^d=AVE\left(P\left[:,i\right]\right)$$

(13)

The set of keyword embeddings is denoted as $E^k=\left\{e_1^k,\dots ,e_m^k\right\}$, self-attention is applied to $E^k$, and the average of all the columns is calculated to obtain the final keyword embedding, $p^k\in {ℝ}^H$. The self-attention and $p^k$ for keyword $k$ are shown in Equations (14) and (15), respectively:

$$K=softma{x}_{row}\left(E^k\cdot E^{kT}\right)\cdot E^k$$

(14)

$$p^k=AVE\left(K\left[:,i\right]\right)$$

(15)

The equation of the relevance value, $r_k$, between keyword $k$ and document $d$ is shown below:

$$r_k=\frac{p^k\cdot p^d}{\Vert p^k\Vert \cdot \Vert p^d\Vert }$$

(16)

where $r_k$ is determined by a cosine similarity calculation by $p^k$ and $p^d$. It takes into account both the interrelationship between the keywords and the context of the sentence and the relevance between the keywords and the document, weighing the importance of the keywords from multiple perspectives.

(4)

Linear Combination Score Calculation

The importance of the keywords and the semantic relevance between the keywords and the document sentences are fully taken into account after self-attention accumulation calculation and cross-attention calculation for the preliminary extracted keywords. The final score, $s_k$, of the keywords is calculated by the linear combination method; the equation is shown below:

$$s_k=d\ast a_k+\left(1-d\right)\ast r_k$$

(17)

where $d\in [0,1]$, the normalized self-attention value $a_k$, and the cross-attention based relevance value $r_k$ are calculated accumulatively for each keyword, respectively. The final score of the keywords, $s_k$, is determined by linearly combining $a_k$ and $r_k$. The keywords are further evaluated based on the scores, and a portion of the useless keywords are eliminated. It more effectively integrates the keywords into the summary inference, resulting in an improvement in the quality of the summary.

(5)

Keyword Incorporation into Summary Inference

In the N-coarse-grained summary generation stages, redundant text information is removed through sentence filtering and data segmentation to further streamline the text content. The obtained data samples $\left(D_i,T_i\right)$ are provided to the summary generation model to obtain a coarse-grained paragraph summary, $Z_i^l$, where $l\in \left[1,N\right]$, $l$ denotes the index of the current stage; the equation is shown below:

$$Z_i^l={\mathrm{MSMAUKE}-\mathrm{S}\mathrm{UMM}}_l\left(D_i,T_i\right)$$

(18)

Concatenate $n$ coarse-grained paragraph summaries $(Z_1^l,Z_2^l\cdot \cdot \cdot Z_n^l)$ to obtain $Z^l=Z_1^l\oplus Z_2^l\oplus \cdot \cdot \cdot \oplus Z_n^l$ and use $Z^l$ as a new data sample for the next stage.

MSMAUKE-Summ^N can be applied to summary generation models such as BART and T5 [31], because the BART model performs poorly on summary generation for long texts. Therefore, in order to reflect the framework’s facilitation of BART for long-text summarization, BART is adopted as the backbone model of MSMAUKE-Summ^N.

3.2. 1-Fine-Grained Summary Generation Stage

When $Z^l$ is used as an input source, if the text length of $Z^l$ is less than the maximum input length, $L$, of the summary generation model, it can satisfy the requirements of the summary generation model in the 1-fine-grained stage. In the 1-fine-grained summary generation stage, the summary generation model is trained directly on the dataset $\left(Z^N,T\right)$ from the last coarse-grained stage.

Keyword Selection: a pre-trained encoder model with $x$ self-attentive heads $h_1,h_2,\dots ,h_x$ is used in the 1-fine-grained summary generation stage. Each self-attentive head, h_i, attends to a different part of the input text [32]. Divide each head into $y$ clusters, $C=\left\{c_1,c_2,\dots ,c_y\right\}$, so there are $g=\frac{x}{y}$ heads per cluster. The headers are clustered using a sequential approach and the $m$ most popular keywords are identified from each header. Since a keyword may receive higher attention from more than one head, it may result in an overlap in the set of keywords obtained by each head. To solve the problem of overlapping keywords, up to $g\ast m$ keywords should be obtained from each cluster. Assume that the total number of keywords used for the coarse-grained summary inference process is $n$. By retaining $r$ keywords as topic keywords, the remaining $n-r$ keywords are non-overlapping keywords with less attention. $r$ is a hyperparameter, and it can be derived from the statistical laws that $r$ is typically 10% of $n$. Similar to the inference process of using keywords for coarse-grained summarization, it is also necessary to further select a portion of keywords that are more relevant to the topic for use in the inference process of fine-grained summarization.

Keywords are utilized to assist the inference process of the summary generation model. A fine-grained summary, $F$, is obtained as the final output of MSMAUKE-Summ^N; the equation is shown below:

$$F={\mathrm{MSMAUKE}-\mathrm{S}\mathrm{UMM}}_{N+1}\left(Z^N,T\right)$$

(19)

where $\left(Z^N,T\right)$ denotes the dataset of the last coarse-grained stage, and the dataset $\left(Z^N,T\right)$ is utilized by the MSMAUKE-Summ^N framework to finetune the summary generation model in the (N + 1)-th stage to obtain the final fine-grained summary, $F$.

The pseudo-code for MSMAUKE-Summ^N is shown in Algorithm 1:

Algorithm 1 MSMAUKE-SummN pseudo-code

Input:  $D$, $T$$, Lr$, Coarse-grained beam width, Fine-grained beam width, Input_max_token. Output:  $F$ 1: for n in range(1, N + 1) do 2:    $\mathrm{PureText}\left(D,T\right)$        Summary Generation Model $\leftarrow$ $\mathrm{ROUGE}\left(D_i,T_i\right)$ 3:    $\mathrm{keywords}$ $\Leftarrow$ $\mathrm{NLTK}(D)$ 4:    $a_k$ $\Leftarrow$ ${\mathrm{Accumulation}\left(\mathrm{keywords}\right)}_{\mathrm{Self}-\mathrm{attention}}$ 5:    $r_k$ $\Leftarrow$ $\mathrm{Relevance}{\left(\mathrm{keywords}, \mathrm{sentence}, \mathrm{document}\right)}_{\mathrm{Mixed}-\mathrm{attention}}$ 6:    $s_k$ $\Leftarrow$ $\mathrm{Linear}\_\mathrm{combination}\_\mathrm{score}(a_k,r_k)$ 7:    $Z_i^l$ $\Leftarrow$ $MSMAUKE-SUM{M}_l\left(D_i,T_i\right)$         $(Z_1,Z_2,\cdot \cdot \cdot Z_n)$ $=$ $Z_i^l$ 8:    $Z$ $\Leftarrow$ $\mathrm{Tan}\mathrm{dem}(Z_1,Z_2,\cdot \cdot \cdot Z_n)$        Summary Generation Model $\leftarrow$ $\mathrm{ROUGE}\left(Z_i,T_i\right)$ 9: end for 10: $F$ $\Leftarrow$ $MSMAUKE-SUM{M}_{N+1}\left(Z^N,T\right)$ 11: return $F$

Streamlined data samples are obtained by the sentence filtering process. Mixed-attention keyword extraction is utilized to enrich the global semantic information of coarse-grained summaries. The self-attentive keyword selection module is designed and keywords are selected to assist summary inference to enhance the local semantic representation of fine-grained summaries. An accurate, comprehensive, and readable fine-grained summary, $F$, is obtained.

4. Experimental Results and Analysis

4.1. Experimental Setup and Dataset

In this study, the experimental environment is configured on a cloud server with a single NVIDIA RTX A5000 (NVIDIA, Santa Clara, CA, USA) with 24 GB of video memory. The Python version 3.7 programming language, the Pytorch version 1.8.1 framework, CUDA version 11.1, and fairseq version 0.10.0 are utilized for the experiments.

The experimental hyperparameter settings are shown in

Table 1. Experimental hyperparameter settings.

Serial Number	Parameters	Parameter Value
1	Lr	2 × 10−5
2	Coarse-grained beam width	2
3	Fine-grained beam width	10
4	Input_max_token	1024

.

From Table 1, Lr denotes the learning rate, which is set to 2 × 10⁻⁵. The coarse-grained beam width and fine-grained beam width denote the beam widths of the coarse-grained and fine-grained stages, which are set to 2 and 10, respectively. Input_max_token denotes the maximum number of tokens in the original text, which is set to 1024.

In addition, by looking at the model parameters, we can find that the BART-large model pre-trained on the CNN/DM dataset has the best summary generation capability [33]. Therefore, in this paper, the model parameters of BART-large trained on the CNN dataset are used as initialization parameters.

Both AMI and ICSI are conference scripts generated through automatic speech recognition systems. AMI is a dataset collected from a company’s product design meetings, and ICSI is a dataset collected from an academic group’s meetings. QMSum [34] is a query-based dataset of conference summaries, where each query and example are written by an expert. SummScreen [35] is a dataset consisting of transcripts of TV episodes from TVMegaSite (TMS) and ForeverDream (FD). The summary of each record is a summary of the TMS or FD. GovReport [36] is a large-scale long-document summary dataset containing 19,466 long reports issued by the U.S. Government Accountability Office on national policy issues. These datasets were chosen, in general terms, because they all belong to the long text summarization type of datasets. In terms of separate domains, they are from company meetings, academic conferences, long document reports released by the government, etc., which can improve the generalization ability of the MSMAUKE-Summ^N model and help the proposed summary generation model more. Therefore, these datasets are selected for related experiments in this study.

The statistics of the dataset used for the experiment are shown in

Table 2. Statistics of the dataset.

Dataset	Type	Domain	Size	Original Text Length	Target Text Length	N + 1
AMI	Dialogue	Meetings	137	6007.7	296.6	2
ICSI	Dialogue	Meetings	59	13,317.3	488.5	3
QMSum	Dialogue	Meetings	1808	9069.8	69.6	2
SummScreen	Dialogue	TV shows	26,851	6612.5	337.4	2
GovReport	Document	Reports	19,466	9409.4	553.4	3

.

The original text length and target text length are averaged over the entire dataset, and N denotes the number of coarse-grained summary generation stages.

4.2. Evaluation Metrics

ROUGE is used as an evaluation metric in this study, which is mainly categorized into ROUGE-1 (hereinafter referred to as R-1), ROUGE-2 (hereinafter referred to as R-2), and ROUGE-L (hereinafter referred to as R-L). R-1 is usually used as the standard evaluation metric because higher scores on the R-1 metric usually imply higher scores on the R-2 and R-L metrics as well. Meanwhile, R-1 has a lower time complexity compared to other ROUGE metrics.

The equation of definitions for R-1, R-2, and R-N are shown as follows:

$$R-N=\frac{{\displaystyle {\sum }_{R\in \left\{ReferenceSummeries\right\}}{\displaystyle {\sum }_{gra{m}_n\in R}Coun{t}_{match}\left(gra{m}_n\right)}}}{{\displaystyle {\sum }_{R\in \left\{ReferenceSummeries\right\}}{\displaystyle {\sum }_{gra{m}_n\in R}Count\left(gra{m}_n\right)}}}$$

(20)

where $n$ represents the length of the $gra{m}_n$, $Coun{t}_{match}\left(gra{m}_n\right)$ denotes the maximum number of $gra{m}_n$ that appear in both the candidate and reference summaries, and $R$ denotes the reference summary.

For R-L, L is the longest common subsequence (LCS), and the LCS is used by R-L. R-L is calculated from $R_{lcs}$, $P_{lcs}$, and $F_{lcs}$. $R_{lcs}$ and $P_{lcs}$ denote recall and precision, respectively, while $F_{lcs}$ denotes R-L. R-L is defined by Equations (21)–(23) as follows:

$$R_{lcs}=\frac{LCS\left(X,Y\right)}{m}$$

(21)

$$P_{lcs}=\frac{LCS\left(X,Y\right)}{n}$$

(22)

$$F_{lcs}=\frac{\left(1+{\beta }^2\right)R_{lcs}{P}_{lcs}}{R_{lcs}+{\beta }^2{P}_{lcs}}$$

(23)

where $LCS\left(X,Y\right)$ is the length of the longest common subsequence of $X$ and $Y$. $m$ and $n$ denote the lengths of the reference and auto-summaries, respectively. Since $\beta$ is a very large number, R-L considers $R_{lcs}$ almost exclusively.

4.3. Ablation Experiments

To better validate the usefulness of the modules, the following ablation experiments are designed on the relatively long dataset GovReport. Only the sentence filtering layer PureText is added in Experiment 1. Only the mixed-attention keyword extraction module is incorporated to aid in summary inference in Experiment 2. The sentence filtering layer PureText and the mixed-attention keyword extraction module are added in Experiment 3. The mixed-attention keyword extraction module and the self-attention keyword selection module are incorporated in Experiment 4. The sentence filtering layer PureText, the mixed-attention keyword extraction module, and the self-attention keyword selection module are added in Experiment 5. The ablation experiment module selection is shown in

Table 3. Selection of experimental modules for MSMAUKE-Summ^N ablation experiments.

Experiment Number	Sentence Filtering	Mixed-Attention Keyword Extraction	Self-Attention Keyword Selection
1	√
2		√
3	√	√
4		√	√
5	√	√	√

A “√” in the above table indicates that the corresponding module has been selected.

.

The results of ablation Experiments 1, 2, 3, 4, and 5 on the three metrics R-1, R-2, and R-L are shown in

Table 4. Results of MSMAUKE-Summ^N ablation experiments.

	Evaluation Metrics	R-1 (%)	R-2 (%)	R-L (%)
Experiment Number
1		56.87	23.56	54.12
2		57.23	24.17	54.71
3		57.35	24.33	54.85
4		57.45	24.52	55.02
5		57.52	24.73	55.15

MSMAUKE-Summ^N’s ROUGE metric values on the dataset GovReport are bolded.

.

As can be seen from Table 4, the addition of the sentence filtering layer PureText and the mixed-attention keyword extraction module in Experiments 1 and 2, respectively, resulted in an improvement in the R-1, R-2, and R-L metrics compared to Summ^N. The improvement was more obvious when the latter was added alone, with an improvement of 0.46%, 0.92%, and 0.81%, respectively. Sentence filtering provides a streamlined sample of data without any effective processing of keywords. Therefore, the improvement in summary generation metrics is relatively small, while the mixed-attention mechanism makes self-attention and cross-attention interact with each other, and fully focus on the keywords at the sentence and document levels. It is more beneficial for obtaining the important information of the summary and improving the readability and conciseness of the summary, and the improvement effect is more obvious.

In addition, from Experiment 3, it can be seen that with the simultaneous inclusion of the sentence filtering layer PureText and the mixed-attention keyword extraction module, the quality of summary generation can be improved by them in a mutually reinforcing way. From Experiment 4, it can be seen that the three metrics are improved by 0.68%, 1.27%, and 1.12% with the addition of the mixed-attention keyword extraction module and the self-attention keyword selection module, respectively. This is due to the large number of keywords initially extracted and the fact that the keywords were not filtered by methods such as attention, and there are many duplicates, which is not beneficial to the summarization inference process. Thus, it is not directly useful for summarization inference, while the mixed-attention weighting of keywords in a document is calculated implicitly, taking repetition into account. Semantic relevance between keywords and sentences in the document is also considered to enrich the global semantic information of the summary. On the other hand, keywords with higher weights can be found and utilized by the keyword selection module to assist summary inference. The ability of the summary generation model to represent detailed information is strengthened, and the local semantic representation of the summary is enhanced.

From Experiment 5, it is clear that the enhancement of the ROUGE metric by the three metrics is accumulative. Because redundant information in data samples can be reduced by sentence filtering, the redundancy of summary information can be reduced and the summary generation model can be improved. The summary generation model can be more comprehensively assisted by attention to obtain important keywords. The accuracy, comprehensiveness, and readability of summary information can be improved. Therefore, attention mechanisms can lead to greater enhancement.

4.4. Comparison Experiments

On the AMI, ICSI, and QMSum datasets, the improved modeling framework MSMAUKE-Summ^N is compared with the following comparison models in this paper:

• HMNet [15]: A hierarchical attention structure and cross-domain pretraining are used by HMNet to extract meeting summaries, and a novel abstract summarization network is constructed to adapt to meeting scenarios. Role vectors are designed to cope with the semantic structure and style of different meeting records, and long meeting records are accommodated through a hierarchical structure.
• TextRank [37]: A graph ranking approach is utilized by TextRank to consider the importance of each node’s information. It recursively calculates the importance weight value of each node in the relationship graph from the global semantic information. It counts and ranks important node information for keyword extraction and text summarization.
• HAT-BART [16]: Layered attention is applied by HAT-BART to Transformer. Visual Hierarchical Encoder–Decoder Attention is designed to overcome the uneven distribution of important information in long texts. Hierarchical Attention Transformer’s (HAT) architecture is used to adapt to longer text inputs.
• DDAMS [38]: Relational maps are used by DDAMS to model different discourse relationships; the interactions between discourses in a meeting are explicitly modeled. A relational graph encoder module is constructed, and graph interaction is used to model the interactions between discourses. The semantic relationships and information structure in the summaries are clarified.
• Summ^N [17]: The data samples are segmented by Summ^N. It generates coarse-grained summaries through multiple stages, based on which fine-grained summaries are generated. The framework can handle input text of any length by adjusting the number of stages.

The results of the ROUGE metric values of the MSMAUKE-Summ^N model with HMNet, TextRank, HAT-BART, DDAMS, and Summ^N on the AMI, ICSI, and QMSum datasets are shown in

Table 5. ROUGE metric values on AMI, ICSI and QMSum datasets.

	Dataset	AMI			ICSI			QMSum-All			QMSum-Gold
Model		R-1 (%)	R-2 (%)	R-L (%)	R-1 (%)	R-2 (%)	R-L (%)	R-1 (%)	R-2 (%)	R-L (%)	R-1 (%)	R-2 (%)	R-L (%)
HMNET		52.36	18.63	24.00	45.97	10.14	18.54	32.29	8.67	28.17	36.06	11.36	31.27
TextRank		35.19	6.13	16.70	30.72	4.69	12.97	16.27	2.69	15.41	-	-	-
HAT-BART		52.27	20.15	50.57	43.98	10.83	41.36	-	-	-	-	-	-
DDAMS		53.15	22.32	25.67	40.41	11.02	19.18	-	-	-	-	-	-
SummN		53.44	20.30	51.39	45.57	11.49	43.32	34.03	9.28	29.48	40.20	15.32	35.62
MSMAUKE-SummN		55.19	21.46	53.35	47.54	12.85	45.25	36.29	10.8	31.22	42.2	17.09	38.09

MSMAUKE-Summ^N’s ROUGE metric values on the AMI, ICSI, and QMSum datasets are bolded.

.

For some special datasets, such as the SummScreen dataset, in this paper, the neural model Longformer + ATT and the hybrid model NN + BM25 + Neural proposed by Chen et al. [35] are used as comparison models and experimented on the SummScreen dataset. In addition, for the GovReport dataset, the comparison models are divided into two main categories, one of which is the BART variant, which represents the self-attentive variant with full attention. Another category is BART HEPOS, denoting the encoder variant with head position step encoder–decoder attention. The results of the ROUGE metric values of the MSMAUKE-Summ^N and other models on the SummScreen and GovReport datasets are shown in

Table 6. ROUGE metric values on the SummScreen dataset.

	Dataset	SummScreen-FD			SummScreen-TMS
Model		R-1 (%)	R-2 (%)	R-L (%)	R-1 (%)	R-2 (%)	R-L (%)
Longformer + ATT		25.9	4.2	23.8	42.9	11.9	41.6
NN + BM25 + Neural		25.3	3.9	23.1	38.8	10.2	36.9
SummN		32.48	5.85	27.55	44.64	11.87	42.53
MSMAUKE-SummN		34.34	6.86	28.98	46.38	12.64	44.74

MSMAUKE-Summ^N’s ROUGE metric values on the SummScreen dataset are bolded.

and

Table 7. ROUGE metric values on the GovReport dataset.

	Dataset	GovReport
Model		R-1 (%)	R-2 (%)	R-L (%)
BART Variants
Full (1024)		52.83	20.5	50.14
Stride (4096)		54.29	20.8	51.35
LIN. (3072)		44.84	13.87	41.94
LSH (4096)		54.75	21.36	51.27
Sinkhorn (5120)		55.45	21.45	52.48
BART HEPOS
LSH (7168)		55	21.13	51.67
Sinkhorn (10,240)		56.86	22.62	53.82
SummN		56.77	23.25	53.9
MSMAUKE-SummN		57.52	24.73	55.15

MSMAUKE-Summ^N’s ROUGE metric values on the GovReport dataset are shown in bold.

, respectively.

As can be seen in Figure 5, the model is significantly improved after adding the sentence filtering layer, the mixed-attention keyword extraction module, and the self-attention keyword selection module. The improvement effect is more obvious in Figure 5b compared to (a) because the ICSI dataset is relatively large and more information can be provided for summary generation. Meanwhile, MSMAUKE-Summ^N outperforms models such as Summ^N in all categories of ROUGE because sentence filtering long texts results in high-quality data samples, which helps to improve the quality of summary generation. The importance of keywords at the sentence level and document level can be paid full attention by the mixed-attention keyword extraction module. The extracted keywords are used in the stage of coarse-grained summary generation. The inference process of summary generation is enhanced, and the information extraction ability in the coarse-grained stage is better improved. In addition, keywords with higher weights are used by the self-attentive keyword selection module in the inference process of fine-grained summary generation. It is beneficial to further improve the summary ability of the summary generation model on detailed and focused information.

As can be seen in Figure 6, since SummScreen is a larger-volume dataset in the TV series genre, the MSMAUKE-Summ^N model was able to improve on each of the ROUGE metrics over the other comparison models. However, parameter optimization becomes more difficult due to computational resource constraints, resulting in it being a little harder for the model to improve metrics on larger datasets. It can still be well illustrated that MSMAUKE-Summ^N with BART as the backbone model has a better summary generation ability.

As can be seen in Figure 7, on the document dataset GovReport, the semantic relevance between keywords and the context of sentences and documents is fully considered due to the mixed-attention mechanism. The keywords are considered from multiple perspectives, and the importance of the keywords themselves as well as their semantic relevance to the documents are combined to assist summary inference. The MSMAUKE-Summ^N model has a better summary generation ability compared to other comparison models when dealing with document-like datasets.

In order to further demonstrate the superior performance of the MSMAUKE-Summ^N model relative to other models at different text lengths, text of different lengths is intercepted by us to test the performance of the model for summary generation. The performance difference between the models can be clearly seen through the summary generation time. The performance test of the model with different text lengths is shown in the following figure.

As can be seen from Figure 8, the unit of time on the vertical axis is seconds; text paragraphs of different lengths are intercepted for the document dataset GovReport, with text paragraphs of lengths such as 500, 1000, and 1500 words; and the performance test of the model’s summary generation is accomplished on these text paragraphs of different lengths. As can be seen from the figure, the time taken by each model to generate a summary is very close for shorter texts of 500 words; however, as the text length grows, it can be seen that the difference in performance between the models gradually widens. This is because the longer the length of the text, the more it restricts the model’s ability to generate summaries, unlike the MSMAUKE-Summ^N model because of its ability to flexibly adjust the number of stages to accommodate different lengths of text and to effectively take keywords into account. Therefore, the MSMAUKE-Summ^N model can outperform other models in terms of performance and the quality of summary generation.

At different text lengths, there are large differences in both the time to process the text and the quality of the summaries generated by the different models. When the same length of text is processed, the MSMAUKE-Summ^N model will take less time compared to HAT-BART, DDAMS, and other models, and the difference between the MSMAUKE-Summ^N model and the other models will be more obvious as the length of the text grows. Thus, this is good proof of the advantages of the MSMAUKE-Summ^N model in processing long text summaries. Meanwhile, the MSMAUKE-Summ^N model proposed in this paper is able to adjust the number of generation stages to adapt to different lengths of text, whereas models such as HAT-BART have a more complex and hierarchical structure, which is less adaptive to the task of summarizing long texts. Therefore, the MSMAUKE-Summ^N model is characterized by a simple and flexible structure compared to other models. Similarly, the MSMAUKE-Summ^N model in this paper is tediously trained on various types of long-text summarization datasets, which sufficiently improves the model’s adaptability to different domains and provides a better generalization of the MSMAUKE-Summ^N model compared to other models that are trained on more homogeneous datasets.

5. Conclusions

In the text generation task, the generation of summaries for long texts is an important task; an N + 1 coarse–fine-grained multistage summary generation framework is constructed and a multistage mixed-attention unsupervised keyword extraction summary generation model (MSMAUKE-Summ^N) is proposed. In the N-coarse-grained summary generation stage, the sentence filtering layer, PureText, is constructed to remove redundant information from the text. It improves the summary generation model’s ability to extract important information. Self-attention and cross-attention are mixed. The importance of keywords at the sentence level and document level is fully considered, and keywords are extracted stage-by-stage in an unsupervised manner using mixed attention. The inference process of coarse-grained summary generation is aided and the global semantic information of coarse-grained summaries is enriched. In the 1-fine-grained summary generation stage, the self-attentive keyword selection module is designed to improve the controllability of keyword usage. Keywords with higher weights are incorporated into the summary inference. The local semantic representation of the fine-grained summary is enhanced. By flexibly adapting to texts of different lengths through the N + 1 coarse–fine-grained multistage summary generation framework, the quality of summaries is considered from both global and local semantic perspectives. The accuracy, comprehensiveness, and readability of summary information are improved, and the redundancy of summary information is reduced. The experimental results show that the MSMAUKE-Summ^N model framework is improved over HMNET, TextRank, HAT-BART, DDAMS, and Summ^N models in terms of ROUGE-1, ROUGE-2, and ROUGE-L metrics, with a minimum of 0.75%, 1.48%, and 1.25% improvement in the three metrics across the datasets relative to the Summ^N model. It is demonstrated that MSMAUKE-Summ^N has a better summary generation ability for long texts or documents.

Further research will be conducted in future work on how to generate summaries of multimodal data, which can be data that contains multiple forms such as text and audio, text and image, or text and video, as opposed to a single text type of data. The fusion of MSMAUKE-Summ^N with various improved convolutional neural networks is considered in future work to investigate how to improve applicability to multi-domain data. In addition, how to further improve the accuracy of keyword extraction is also to be solved. In subsequent research work, diversified feature weighting will be considered to improve the self-attention mechanism, and multiple features of keywords are considered comprehensively to improve the accuracy of keyword extraction.