Cross-Domain Document Summarization Model via Two-Stage Curriculum Learning

Abstract

Generative document summarization is a natural language processing technique that generates short summary sentences while preserving the content of long texts. Various fine-tuned pre-trained document summarization models have been proposed using a specific single text-summarization dataset. However, each text-summarization dataset usually specializes in a particular downstream task. Therefore, it is difficult to treat all cases involving multiple domains using a single dataset. Accordingly, when a generative document summarization model is fine-tuned to a specific dataset, it performs well, whereas the performance is degraded by up to 45% for datasets that are not used during learning. In short, summarization models perform well with in-domain cases, as the dataset domain during training and evaluation is the same but perform poorly with out-domain inputs. In this paper, we propose a new curriculum-learning method using mixed datasets while training a generative summarization model to be more robust on out-domain datasets. Our method performed better than XSum with 10%, 20%, and 10% lower performance degradation in CNN/DM, which comprised one of two test datasets used, compared to baseline model performance.

1. Introduction

With the recent development of large-scale artificial intelligence (AI), humans receive more assistance in areas of their daily lives. One of these areas is time-saving text processing via document summarization. Efficient processing of a vast amount of text generated in real-time has become increasingly important due to social media development, and the use of document summarization tasks to shorten long texts using large-scale language models [1,2,3] such as ChatGPT is increasing.

With the increasing demand for document summarization tasks, research on natural language processing (NLP), which utilizes generative language models through various domain document summarization datasets, is actively ongoing. Document summarization is an NLP sub-task that generates natural summary sentences while preserving the key content of the main text. The types of document summarization datasets vary according to the data source and collection method used for the summarization, including news [4,5,6,7], journals [8], and meeting notes [9]. With the development of document generative summarization technology, the latest generative summarization models [2,10,11] have achieved high performance for each of these document summarization datasets.

However, we still need to overcome summarization model performance degradation when applied to an unseen domain. As shown in Section 5, we discovered significant performance degradation of the summarization model that achieved the best performance on individual document summarization datasets. As further discussed in Section 5, it was confirmed that summarization model performance on out-domain data, which did not appear during the training process, significantly decreased. In actual applications, documents on various topics can be inputted. Therefore, there is a need for research on developing a document generative summarization model that mitigates summarization performance degradation for out-domain documents that do not appear during the training process, while not compromising the in-domain performance.

This paper presents the first experimental study on out-domain summarization methodology in a cross-domain setting that combines document summarization data samples from various domains to train a model on mixed training data. When training with cross-domain summarization data, the proposed method uses curriculum learning [12] to improve the performance on both in- and out-domain data for some document summarization datasets. Our contributions can be summarized as follows: (1) We proposed a two-stage abstractive summarization technique that can adapt curriculum learning in a cross-domain environment. (2) We developed a simple method to measure the difficulty of each component of the cross-domain dataset and observed that domain similarity affects the degree of performance degradation in out-domain tasks.

The remainder of this paper is organized as follows. In Section 2, the foundational technologies are discussed to help us understand the training data composition for document summarization and generative language models. Recent research trends related to document generative summarization and the proposed curriculum learning method are discussed in Section 3; the proposed method is further described in Section 4. In Section 5, the two types of curriculum learning methods used in the proposed method are explained. Section 6 presents experiments conducted on the proposed method to evaluate the summarization performance on both in- and out-domain summarization datasets. Finally, Section 7 concludes the paper.

2. Background

In this paper, a training method that mixes datasets with different domain compositions when training a document generative summarization model is proposed. In this section, we present the background knowledge related to document summarization tasks and curriculum learning.

2.1. Text Summarization

Document summarization is an NLP task that generates a shorter, condensed version of a long text while preserving key content [13,14,15]. Document summarization can be broadly divided into extractive and abstractive summarization, depending on how the problem is defined. Extractive summarization involves selecting important sentences from an original document, while abstractive summarization involves rephrasing the core content of the original text into a shorter version [13,14,16,17].

Abstractive summarization is defined as a natural language generation (NLG) problem in which an input document is summarized by generating new natural language sentences [14]. Owing to the nature of document summarization tasks, the input documents are typically very long. Therefore, many training methodologies are proposed to treat a long input length for the summarization model [18,19]. Previously, abstractive summarization did not yield satisfactory results owing to the limitations of NLP technology; thus, related research has not advanced significantly [20].

Abstractive summarization datasets typically consist of the main text and a reference summary created by humans. The goal of generative summarization is to produce a short, coherent summary that captures the essence of the main text [13,14,20], offering better readability than extractive summarization [5,6], and the generation of abstract sentences that are not present in the original text [13].

Most research on abstractive summarization uses lexical overlap and semantic similarity scores between the reference and generated summaries as evaluation metrics. Lexical overlap measures the surface similarity between input pairs, namely, the reference and generated summaries. Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [21] is a lexical metric that calculates the match ratio of word subsequence overlap between them. ROUGE-N measures N-gram overlap, typically using uni-gram or bi-gram metrics such as ROUGE-1 and ROUGE-2 [6,22]. ROUGE-L measures the overlapped Longest Common Subsequence (LCS) to capture the overlap of non-continuous subsequences. Conversely, BERTScore [23] is a semantic similarity score that uses a pre-trained embedding model to measure the pairwise cosine similarity of input pairs. BERTScore uses the BERT [24] embedding score to measure similarity for each token pair.

2.2. Abstractive Summarization Methods

Abstractive Summarization employs many methods like template-based [25] and graph-based [26] methods. Due to the frequent occurrence of reduction and paraphrasing in abstractive summarization, a methodology that uses deep-learning methods [27,28] for text summarization has become popular [29]. Bidirectional Encoder Representations from Transformers (BERT) [24], pre-trained model-based methodologies have shown high performance in various NLP tasks. According to this trend, high-performance pre-trained models [2,10,11] have been fine-tuned and show high performance in various summarization tasks.

After the introduction of BERT [24], pre-trained language models have generally been used to perform document generative summarization tasks. In [10,11], performance improvements in summarization tasks were demonstrated by adopting effective pre-training tasks. BART [10] uses the denoising autoencoder (DAE) method during the pre-training phase, where the objective function involves restoring the original text from the input corrupted with token-level noise. It exploits the token generation ability of various denoising techniques such as random token deletion, substitution, and masking. PEGASUS [11] employs the gap sentence generation (GSG) method as a pre-training objective for generative summarization. It expands the BART token denoising task to the sentence level by masking and restoring all key sentences from the input.

2.3. Curriculum Learning

Curriculum learning [12] mimics the learning process of humans by gradually exposing the model to data samples with increasing difficulty levels, starting from easy samples and progressing to harder ones. There are two main types of curriculum learning based on how difficulty levels are applied during the model training process: data- and model-level methods. Data-level curriculum learning involves providing data samples to the model in an order predefined by their difficulty levels and gradually accumulating them as training data at each step of the learning process. Data-level methods can improve the model convergence when the curriculum is well-constructed. Model-level curriculum learning adjusts the proportion of loss values according to the difficulty level of the data samples when updating the model. Model-level methods need less effort than manually determining the difficulty, but the model might face an overfitting risk.

Self-paced learning (SPL) [30] has been proposed for image-segmentation tasks [31]. Self-paced learning is a model-level curriculum-learning method in which only data samples with loss values below a certain threshold are selected based on the model’s loss values during the learning process. In contrast to data-level curriculum learning, which processively increases data samples at each step of the learning process, self-paced learning gradually increases the difficulty threshold for model updates, incorporating more challenging samples into the model. Self-paced learning may enhance the robustness of the model, but it is challenging to modify the learning strategy as it is designed to solve image tasks but not texts.

3. Related Work

3.1. Abstractive Summarization on Various Domains

Recently, a contrastive learning method has been proposed to align the differences between the pre-training objective functions of sequence-to-sequence (Seq2Seq) language models [10,32] and the summarization metrics introduced in Section 2. Seq2Seq language models maximize the likelihood of word-level occurrences of the reference sentence during the decoding process by Maximum Likelihood Estimation (MLE) training. However, the summarization model performance is measured using sentence-level evaluation metrics, which compare the generated sentence with the reference summary. Ref. [33] proposed a method to mitigate the issue of different application units between language model training and evaluation by contrastive learning. This ensures that the generated summaries are represented closer to the reference summaries in semantic space. In an extension of [33], ref. [34] proposes a method that adjusts the weight loss value according to the occurrence probability of generated summaries in fine-tuned summarization models.

Despite these approaches for further model training using DAE [10] or contrastive learning [33,34], these studies have difficulty applying the training conducted on a specific dataset to other datasets. However, several studies that expand summarization models to work in more generalized settings exist. Ref. [35] tried to detect the out-domain sample among inputs by analyzing the distance features of a pre-trained model. Ref. [36] proposed a learning method for preference models in a low-resource summarization environment. Ref. [37] presented an experimental study of various domain adaptation methods on abstractive summarization. Ref. [37] showed that the gaps in results depended on resource allowance and suggested the need for higher-level domain adaptation methods for abstractive summarization tasks.

3.2. Curriculum Learning in NLP

Several studies incorporated curriculum learning into NLP procedures to enhance performance. Ref. [38] applies curriculum learning to a question-answering task, with the NLP task choosing from candidates the answer that is most relevant to a given passage. It ensembles the choice of curriculum learning results. Ref. [39] performs data-level curriculum learning by choosing one answer among generation results during the training process, which can be relevant to reference sentence lengths. Ref. [40] applied data-level curriculum learning by increasing the length of correct answers from the entire training dataset while training machine-translation models.

Conversely, there are approaches to using curriculum learning for model training efficiency. Ref. [41] guided Transformer-based models with curriculum learning by weighting the candidates to create an efficient learning procedure. Ref. [42] adopted curriculum learning to boost a prompting model’s perturbation, the curriculum strategy following the degree of labeled data.

4. Proposed Method

We propose curriculum learning using cross-domain summarization datasets to alleviate the performance degradation of out-domain data in generative summarization models.

Overall Architecture

In this study, we use a cross-domain summarization dataset as training data for generative summarization models. Figure 1 illustrates the entire training process of the proposed method. The cross-domain summarization dataset is composed of mixed datasets from various domains. By using a cross-domain dataset and two-step curriculum learning, we expect to avoid the performance drop in out-domain scenarios by two-stage curriculum learning. Figure 1 illustrates the entire training process of the proposed method. This approach applies two-stage curriculum learning at both the dataset level and the model level.

For the first stage of curriculum learning for the cross-domain summarization dataset, we use a dataset-level curriculum-learning process to consider the domain difficulty between domains. The dataset-level curriculum is processed to determine each domain’s difficulty for inter-domain difficulty instead of the data-level method of prior works. In Section 5, the method to determine cross-domain dataset difficulty is evaluated using statistical metrics. For each training step of the dataset-level process, data samples at each step are repeatedly accumulated and processed to the next step to train the model.

For the second stage of curriculum learning for the cross-domain summarization dataset, we incorporate a model-level curriculum-learning process to consider intra-domain difficulty on each step’s accumulated data samples. Although data-level curriculum learning can reflect the difficulty level between domains, it may not adequately address the variability in difficulty level within each domain at the individual data-sample levels. We suggest model-level curriculum learning to adjust the amount of learning between data samples during the repetitive training process by focusing on easier examples within the accumulated samples. Our method used SuperLoss [43] as a model-level strategy, to prevent some edge samples from over-influencing and interfering with the model training process. SuperLoss [43] is a curriculum-learning technique that appends an extra loss function on top of the task loss. We applied SuperLoss as a breakwater for stable learning over harsh training conditions within a cross-domain dataset containing texts from various sources. In our process, the SuperLoss support model is used to distinguish difficult samples within mixed data accumulation, selected via dataset-level curriculum learning.

5. Dataset-Level Curriculum Learning

In this study, a cross-domain dataset $D=\{d_1,d_2,\dots ,d_N\}$ was sorted based on the difficulty level of curriculum learning. Here, d represents a single-domain dataset, and D is assumed to be sorted according to difficulty level. In other words, if $i<j$, $d_i$ is considered easier than $d_j$. The process of determining the difficulty level of a single-domain summarization dataset $d_i=\{(do{c}_1^i,re{f}_1^i),(do{c}_2^i,re{f}_2^i),\dots (do{c}_n^i,re{f}_n^i)\}$ is as follows, where n is the number of data in the single-domain summarization datasets d. $doc$ and $ref$ represent the body text and the reference summary of the summarization dataset.

$$\begin{array}{c}\hfill S(do{c}_j^i,re{f}_j^i)=\alpha \times ratioLen\left(d_i\right)-\beta \times R(do{c}_j^i,re{f}_j^i)+\gamma \times stdDev(do{c}_j^i,re{f}_j^i)\end{array}$$

(1)

$$\begin{array}{c}\hfill score\left(d_i\right)=\sum _{j=0}^n{\displaystyle \frac{S(do{c}_j^i,re{f}_j^i)}{N}}\end{array}$$

(2)

Equation (1) evaluates the difficulty level of the j-th data in $d_i$. In this study, we assumed that a smaller value indicates an easier dataset. To determine the difficulty level of dataset $d_i$, the difficulty level was measured for each data point in $d_i$. Equation (2) defines $score\left(d_i\right)$ as the learning difficulty of single-domain dataset $d_i$. The difficulty level of each dataset was averaged to derive the overall difficulty of the dataset.

In Equation (1), the difficulty level measurement for one data point is based on three scores, namely, $(ratioLen,stdDev,R)$. $ratioLen$ is a score based on the ratio of the length of the source document to the reference summary. We assumed that summarizing a long document in a shorter sentence would increase the difficulty level. Therefore, if the length of the reference summary is shorter than that of the source document, we consider it more difficult. $ratioLen(do{c}_j^i,re{f}_j^i)$ is calculated by subtracting the ratio of token numbers $re{f}_j^i/do{c}_j^i$ from unity. This evaluation considers a dataset to be easier if the ratio of lengths between the source document and the reference summary is close to unity, indicating a similarity in length. A smaller value indicates a similarity in length between the source document and the reference summary; therefore, it is an easier dataset. The formula for this process is as follows; ${\sigma }_i$ represents the standard deviation of $ratioLen$ in the single-domain summary dataset $d_i$.

$$\begin{array}{c}\hfill ratioLen(do{c}_j^i,re{f}_j^i)=|1-{\displaystyle \frac{lenght\left(re{f}_j^i\right)}{lenght\left(do{c}_j^i\right)}}|\end{array}$$

(3)

R represents the ROUGE score measured between the source document and reference summary. ROUGE is an evaluation metric used to measure the similarity between sentences. We assume that a lower similarity between the source document and reference summary indicates a higher difficulty level. ROUGE scores are higher when the similarity between sentences is higher and lower when the similarity is lower. We converted the ROUGE score to a negative value so that a higher R indicates an easier dataset in terms of difficulty level.

$$\begin{array}{c}\hfill R(do{c}_j^i,re{f}_j^i)=-ROUGE(do{c}_j^i,re{f}_j^i)\end{array}$$

(4)

$stdDev$ represents the reciprocal of the standard deviation of $ratioLen$ for the single-domain dataset $d_i$, divided by the number of data n. We believe that inconsistent length ratios among the data within a dataset indicate a lack of coherence. Consequently, datasets with a lower consistency are presumed to be more difficult to learn. Therefore, datasets with smaller standard deviation values are considered easier because they exhibit higher consistency among the lengths of the data. Unlike the previous two scores $(ratioLen,ROUGE)$, the standard deviation is a statistical metric at the sentence level. However, for ease of weight adjustment in the weighted sum, the standard deviation is normalized by the number of data points, resulting in measurement difficulty for each data point. The formula for this process is as follows:

$$\begin{array}{c}\hfill stdDev\left(d_i\right)={\displaystyle \frac{{\sigma }_i}{n}}\end{array}$$

(5)

To measure the final difficulty of a single dataset, the weighted sum of each score was calculated, as shown in Equation (2). The weights for each score are denoted by $\alpha ,\beta$, and $\gamma$. In the experiments, the weights were set to $\alpha =0.5,\beta =0.1$, and $\gamma =0.4$.

For dataset-level curriculum learning, an ordered dataset based on the measured difficulty was used, denoted as $D=\{d_1,d_2,\dots ,d_N\}$. If $i<j$, $d_i$ is evaluated as easier than $d_j$. The number of learning steps in curriculum learning is the same as the number of single-domain datasets used, denoted by N. During the N learning steps, in the i-th learning step, only the easy i datasets from the entire dataset are utilized for learning. In other words, in the i-th learning step, $D_{1\sim i}=\{d_1,\dots ,d_i\}$ is used for learning.

6. Experimental Section

In this section, we present the experimental setup, results, and an analysis of the experiments conducted to evaluate the performance of the proposed methodology.

6.1. Model

In the experiment step, PEGASUS [11] was used as the backbone model for abstractive summarization. PEGASUS [11] employs a mix of news and web-crawling data in proportions suitable for the data size during the pre-training process [2,4]. PEGASUS is a pre-trained model fine-tuned on all document summarization datasets used in the experiments and has excellent summarization performance. To put long input summarization document data as a context, we use the ‘pegasus-large’ version in [11] for an experiment baseline. We truncate input document length exceeding 1024 tokens, which is the maximum token length of the baseline model.

6.2. Datasets

In this study, we selected six representative datasets commonly used in abstractive summarization for experimentation.

Table 1. Abstractive Summarization Dataset Description.

Dataset	Domain	Doc/Ref Policy
WikiHow	Web Question Answering	Answer/Topic
ArXiv	Research Paper (ArXiv)	Abstract/Title
Xsum	News Article (BBC News)	Article/Topic
RedditTIFU	Online Community (Reddit)	Article/TLDR
SAMSUM	Dialogue (1:1 Messenger)	History/Summary
CNNDM	News Article (CNN News)	Article/Headline

provides the domain and summary relationship information for the selected datasets.

WikiHow [44] is an extractive summarization dataset that utilizes responses, and the topic is marked as the correct answer on the question-and-answer website WikiHow. ArXiv [8] uses the titles and abstracts of academic papers from the scholarly archive website ArXiv as the source for abstractive summarization. CNN/DM [5] is an extractive summarization dataset that uses news articles from CNN and headlines from each paragraph. XSum [6] is an abstractive summarization dataset using news articles from the BBC with abstract-like summaries, which consist of more abstractive titles than CNN/DM. RedditTIFU [45] is a summarization dataset extracted from conversations in the English-speaking online community Reddit. In [46], the convention of summarizing posts with the keyword ‘TLDR’ at the end of community posts was briefly utilized to generate abstractive summaries. SAMSUM [9] is a 1:1 messenger conversation summarization dataset composed of conversation logs and summary content.

For the experiments in this study, we divided the six datasets [5,6,8,9,44,45] into training and evaluation datasets. Four datasets, namely, WikiHow [44], ArXiv [8], XSum [6], and RedditTIFU [45] were used for training, while a subset of four single-domain datasets was sampled to construct the proposed cross-domain dataset. CNN/DM [5] and SAMSUM [9] were used for evaluation. Using different datasets for training and evaluation, we demonstrate the effectiveness of the proposed methodology in out-domain scenarios. Following the divide-and-conquer approach described in [47], we constructed the training dataset using the first paragraph and correct summary from the entire document of the six summary datasets.

Owing to data imbalance in each dataset and for an efficiency and data diversity balance, we randomly sampled 10,000 samples from the training data of each training dataset. Similarly, for the evaluation dataset, we selected 1000 samples from the test set of each single-domain dataset.

6.3. Experiment Setting

We measure our baseline performance on out-domain data according to which fine-tuning method and training data are used. The hyperparameters used during training are listed in

Table 2. Hyperparameters of Experiment.

Hyperparameter	Value
Epoch	3
Batch size	1
Learning rate	$2\times {10}^{-2}$
SuperLoss-weight $\lambda$	10
Generation length (min/max)	11.64
Length penalty	1.0

. The experiment is performed with the PEGASUS [11] model through Huggingface, using the ‘pegasus-large’ model to allow for the long input context of the experiment dataset. The experiment was run on hardware with two RTX3090 GPUs for 3 h for 3 epochs. We trained the model during noticeable performance changes. Following the other hyperparameter settings employed in [34], we set the learning rate to $2\times {10}^{-2}$, maximum generation length to 128, maximum generation length to 11, and length penalty to 1.0. We regulated the SuperLoss weight parameter $\lambda$ to 10.

6.4. Experimental Result and Analysis

In this experiment, the performance of out-domain scenarios was examined by comparing the results of the proposed methodology with cases that fine-tune the model using a single dataset.

Table 3. Model in-domain performance according to fine-tuning. This table shows model performance transition versus in-domain fine-tuning. FT indicates that the model is fine-tuned to the training data. This table shows examination performance on test data without applying the proposed methodology (model-level and dataset-level curriculum). Theerformance gap between the fine-tuned and non-fine-tuned (non-FT) model and the original version [11] may be affected by the low-resource used during the experiment.

FT	Train		Test	R-1	R-2	R-L
O	ArXiv	->	ArXiv	44.21	16.95	25.67
	WikiHow	->	WikiHow	46.39	22.12	38.41
	RedditTIFU	->	RedditTIFU	27.99	09.81	22.94
	XSum	->	XSum	47.60	24.83	39.64
X	(non-FT)	->	ArXiv	18.25	05.84	15.55
			WikiHow	17.79	05.80	16.48
			RedditTIFU	12.18	01.49	10.45
			XSum	27.14	07.76	21.51

shows backbone model [11] performance on test set data according to the fine-tuning process.

Table 4. Results of Experiments on SAMSUM. This table shows model performance on SAMSUM with out-domain test data. DL and SPL indicate whether dataset-level or model-level curriculum is applied at each row.

DL	Train		Test	SPL	R-1	R-2	R-L
X	ArXiv	->	SAMSUM	X	06.51	00.61	05.80
	WikiHow			X	12.86	02.20	11.95
	RedditTIFU			X	21.02	06.32	19.15
	XSum			X	14.37	02.25	12.39
O	Proposed dataset	->	SAMSUM	X	13.65	02.51	13.57
X	ArXiv	->	SAMSUM	O	25.09	07.32	22.79
	WikiHow			O	25.31	07.44	22.88
	RedditTIFU			O	25.33	07.34	22.90
	XSum			O	24.79	07.46	22.38
O	Proposed dataset	->	SAMSUM	O	25.25	07.32	22.96

and

Table 5. Result of Experiments on CNN/DM. This table shows model performance on CNN/DM with out-domain test data. DL and SPL indicate whether dataset-level or model-level curriculum is applied at each row.

DL	Train		Test	SPL	R-1	R-2	R-L
X	ArXiv	->	CNN/DM	X	10.16	01.98	08.87
	WikiHow			X	13.28	02.26	11.30
	RedditTIFU			X	19.76	07.99	16.91
	XSum			X	15.74	03.27	13.03
O	Proposed dataset	->	CNN/DM	X	12.86	05.03	14.56
X	ArXiv	->	CNN/DM	O	25.24	10.65	21.22
	WikiHow			O	25.28	10.62	21.05
	RedditTIFU			O	25.27	10.66	21.07
	XSum			O	21.44	07.57	18.06
O	Proposed dataset	->	CNN/DM	O	24.03	09.58	20.09

show the out-domain performance of the model according to various training datasets. The sample result of the model trained using the proposed methodology is available in

Table A1. Example of summary in CNN/DM. Example output of summaries generated by baseline and proposed method. The red part of the generated example is the parts that are factually inconsistent. A factually inconsistent summary was generated both by PEGASUS and the model trained with the cross-domain dataset (Ours). Lorraine Valentine is never called a ‘real-life vampire’ literally and it is not certain that she is a mother of four children.

Model	Example
PEGASUS (base)	a woman who has a rare medical condition that makes her allergic to the sun has been named as the ‘real life vampire’.
Ours	Lorraine Valentine suffers from the rare disease erythropoietic protoporphyria (EPP) which causes her skin to burn and become itchy when exposed to sunlight. The 42-year-old went on a holiday to Lanzarote last June but was in hospital for six days as her skin swelled up. The mother-of-four has to keep herself completely covered up in summer as even a small dose of sun leaves her in pain. She was first diagnosed with the incurable condition when she was nine years old.
Gold	Lorraine Valentine, 42, suffers from erythropoietic protoporphyria (EPP) Rare condition means she burns, itches and swells in sunlight or UV light. Was hospitalised for 6 days with burns after a family holiday in Lanzarote. Has to cover herself completely as small amounts of light leave her in pain.

in the Appendix A.

In Table 3, the drop in backbone model [11] performance averaged of 65%. This shows that the original model’s performance is vulnerable to out-domain conditions, which means testing on data unseen while training. This shows that the test data used in this paper is harder than other training datasets, as seen by the 31% or 23% more performance decrease with unseen test data.

Table 4 and Table 5 both show that model out-domain performance significantly increased when model-level curriculum learning is used. However, the RedditTIFU dataset exhibits a smaller performance drop than other training datasets even without model-level curriculum application. This smaller performance drop is likely due to the nature of RedditTIFU [45], which consists of conversational summaries from a community, similar to the dialogue summary dataset SAMSUM [9]. This can indicate that RedditTIFU can positively influence inference on similar out-domain settings like SAMSUM. In summary, the performance degradation without model-level curriculum learning shows that model-level curriculum learning acts as a performance buffer for model inference on unseen domains.

In Table 4, the model trained with the proposed dataset generally exhibited higher performance than XSum, which performed lower than other datasets with model-level curriculum learning. It is notable that the model trained with the proposed dataset achieves a ROUGE-L score of 22.96, which is the highest score among models trained with model-level curriculum learning. However, in Table 5, the model trained with the proposed dataset shows slightly lower performance when using other training data except XSum. This is remarkable because XSum [6] is a dataset from the newspaper domain, similar to CNN/DM [5], as with SAMSUM, as shown in Table 4. It seems that using only XSum may cause overfitting of training data while using mixed-domain data like our proposed dataset prevents the model from being overfitted.

7. Conclusions

In this study, we conducted performance evaluations on in- and out-domain datasets to identify fine-tuning methods for generative summarization models that are applicable across various domains. Applying the two proposed curriculum-learning methods mitigated the drop in performance with out-domain evaluation data during fine-tuning. We discovered that a conversational dataset can act as a buffer for the same conversational test data, although it is not used during the pre-training step. Implementing model-level curriculum learning prevented a significant drop in performance on out-domain performance compared to the models that were not fine-tuned. Utilizing mixed samples with data-level curriculum learning ensured smaller performance variations in the out-domain evaluation dataset [5,9] across different training data samples used in the experiments.

In future work, we plan to refine the criteria for evaluating learning difficulty and expand the variety and quantity of the experimental datasets to verify the generalizability of the proposed methods. In this study, we sampled training and validation datasets for efficiency in the experimental process. Despite the fact that we sampled randomly, sampled data can cause different performance effects to those of using all available data. Therefore, we intend to use the full dataset to directly compare the performance with other studies. In addition to ROUGE, we incorporated semantic similarity-based evaluation metrics and unsupervised evaluation techniques to assess the generalizability of the proposed methods from multiple perspectives.