Automatic Text Simplification for Lithuanian: Transforming Administrative Texts into Plain Language

Abstract

In this study, we present the results of experiments on text simplification for the Lithuanian language, where we aim to simplify administrative-style texts to the Plain Language level. We selected mT5, mBART, and LT-Llama-2 as the foundational models and fine-tuned them for the text simplification task. Additionally, we evaluated ChatGPT for this purpose. Also, we conducted a comprehensive assessment of the simplification results provided by these models both quantitatively and qualitatively. The results demonstrated that mBART was the most effective model for simplifying Lithuanian administrative text, achieving the highest scores across all the evaluation metrics. A qualitative evaluation of the simplified sentences complemented our quantitative findings. Attention analysis provided insights into model decisions, highlighting strengths in lexical and syntactic simplifications but revealing challenges with longer, complex sentences. Our findings contribute to advancing text simplification for lesser-resourced languages, with practical applications for more effective communication between institutions and the general public, which is the goal of Plain Language.

1. Introduction

Text simplification means reducing vocabulary and syntactic complexity while preserving the essential information necessary for understanding. This process allows to increase the accessibility of information for a wide range of groups, including individuals with cognitive disorders, non-native speakers, and children [1]. Additionally, simplifying text is crucial for improving the general public’s comprehension of legal and administrative documents. These types of texts often act as communication bridges between institutions and audiences with varying reading abilities [2].

In this paper, we report the results on text simplification for Lithuanian, targeting the simplification of texts written in administrative (clerical) styles. Public authorities often employ quasilegal language when communicating with their audiences, which can hinder the effective dissemination of information to those groups or individuals who lack specialized knowledge [2]. Text simplification can facilitate reading comprehension by transforming complex natural language into a more straightforward form, which includes vocabulary, sentence structure, and other key elements while maintaining the core content of the original text. As far as we know, this is the first such study for the Lithuanian language.

Our objective is to simplify Lithuanian administrative texts to the level of Plain Language. Plain Language is characterized by clear wording, structure, and design, so the target audience can easily find, understand, and use the information [3], without requiring special education or knowledge [4]. Therefore, we aimed to simplify Lithuanian administrative texts according to the principles of Plain Language. Additionally, we conducted a comprehensive analysis of the simplification results to identify which sentence elements and structures were effectively simplified by our models, as well as to detect potential errors and develop strategies to mitigate them.

For our experiments, we selected mT5, mBART, and LT-Llama-2 as the baseline models and fine-tuned them for the text simplification task. These models were chosen because they support the Lithuanian language, which considers that there is still a lack of large language models that would adequately support lesser-resourced languages. Furthermore, we evaluated the performance of ChatGPT (GPT-4o) in this task. Our approach integrated lexical and syntactic simplification techniques to reduce sentence complexity and replace complex words or phrases with simpler and more commonly used alternatives.

The rest of the paper is structured as follows: in Section 2, we briefly introduce the related work; in Section 3, we describe the data and methods; and in Section 4, we present the results. Finally, in Section 5 we discuss these results and end the paper with conclusions in Section 6.

2. Related Work

In recent years, significant progress has been made in the field of text simplification, moving from rule-based methodologies (e.g., [5,6]) to data-driven methods (e.g., [7,8]). Contemporary research has shown that using these methods texts could be simplified according to diverse goals and perspectives, such as by specifying the desired reading level [9] or directly defining the necessary simplification operations [10]. For example, the BERT model has been used for lexical simplification [11], text simplification using monolingual machine translation [12], and hybrid text simplification combining different methods [13]. Similarly, the T5 model has been applied for controllable text simplification [14,15] and for simplification tasks in low-resource scenarios [16,17]. In particular, the SimpleT5 model, designed to rank and explain complex concepts in a text, outperformed GPT-3 in key metrics related to feedback and time constraints [18]. Additionally, BART has been used for controllable text simplification [14], as well as to simplify entire paragraphs [19] and full documents [20].

Among the various large language models (LLMs), GPT models have attracted particular interest, especially in low-resource settings [21,22,23]. For example, ChatGPT has shown potential in text simplification tasks [24,25]. However, performance evaluations of ChatGPT show that while it can effectively simplify the text, it may miss important information [26] and perform less accurately and in depth than human experts [27]. On the other hand, when provided with additional context, ChatGPT has been observed to simplify complex texts more effectively [28] and provide information that is essentially correct [29].

Various methods have been used to analyze, evaluate, and interpret the results of automatic text simplification. One such approach focuses on controlling and predicting text complexity. A key component of this task is to determine whether a text requires simplification and identify the specific segments that need editing. These tasks can be accomplished using a range of models, including LSTM [30] and BERT [31], alongside model-agnostic techniques such as Local Interpretable Model-Agnostic Explanations (LIME) [32] and SHapley Additive exPlanations (SHAP) [33], which help to gain insight into model decisions [34]. For instance, LIME has been applied with logistic regression (LR) and LSTM classifiers, while SHAP has been used with LR to explain text complexity predictions as demonstrated in [30]. Additionally, they explored the potential of the extraction of adversarial networks for simultaneous prediction and explanation. These methods improve the explainability of text simplification models by providing insights into their behavior and decision-making processes.

Given that text simplification is a complex and multifaceted task, and often treated as a generative task, automatic evaluation metrics alone are not sufficient to capture all its aspects. Consequently, new evaluation metrics for text generation have been proposed. In particular, trained scoring metrics have been developed, such as InstructScore [35], which is based on a fine-tuned Llama model that generates diagnostic reports aligned with human judgment, and TIGERScore [36], which uses Llama-2 to analyze the errors based on the instructions of natural language. Furthermore, LENS [37] has demonstrated stronger correlations with human judgment compared to existing metrics. LENS is a learnable evaluation metric that decomposes and reconstructs sentences to assess both similarity and difficulty within a text simplification system. These examples represent a few of the innovative metrics that have been introduced. Furthermore, new formulas have been proposed to integrate grammaticality, meaning retention, and simplicity into a unified metric for the evaluation of automatically simplified text [38], revealing strong correlations with established metrics such as BERTscore and BLEU.

Moreover, document-level planning has been proposed to ensure controlled simplification and clarity [39]. This approach involved developing a planning model that assigned appropriate simplification operations to each sentence within a document for better control and explanation of the generation process. To further analyze the processes and errors associated with text simplification, an analytical evaluation framework has been introduced, which includes a comprehensive taxonomy of simplification strategies and errors [40]. This study revealed a discrepancy between human and system approaches, where systems often relied on deletions and local changes without incorporating new information essential for meaningful simplification.

Additionally, several studies have explored the use of guided text generation for simplification purposes. For example, SIMSUM [41] directed text generation via the main keywords of a source text. Similarly, research in [10] focused on simplifying text for specific grade levels via the prediction of necessary edit operations. Another approach involved a reinforcement learning system equipped with a readability classifier for iterative simplification until the text reached a desired readability level [42]. Furthermore, the definition and usage of simplification operations have been investigated, highlighting LLMs’ role in the automated recognition of these operations [43].

The Rhetorical Structure Theory (RST) framework proposed by [44] was used to analyze and interpret the results of text simplification, taking into account the contribution of structural aspects to text complexity. Additionally, explainable text simplification was explored through the use of knowledge graphs (KGs), such as KGSimple [45], which simplifies texts at both the graph and text levels, providing explainable progress at each simplification step. This highlights the potential of KGs to increase the interpretability of text simplification models.

Recent research in text simplification has employed a variety of approaches to enhance the explainability and interpretability of model behavior and simplification outcomes. As the field advances, increasing attention is being directed toward addressing the discrepancies between human judgments and the outputs produced by simplification systems. Although simplified texts have been shown to improve reading comprehension [46], existing methods often overlook the inherent complexity of individual inputs, which results in simplifications that require editing and corrections [10]. Additionally, most current text simplification systems do not adequately consider contextual information, which leads to outputs that are difficult to control [47]. Finally, persistent issues related to cultural and commonsense knowledge highlight the necessity for ongoing research in this area [48].

3. Materials and Methods

3.1. Materials

Simplification guidelines. We aimed to simplify the Lithuanian administrative texts so that they would meet the standard of Plain Language, and become more accessible and easier to understand for the general public. To accomplish this, lexical and syntactic simplification rules were developed for our 2 corpora—Parallel Corpus 1 and Parallel Corpus 2. These rules were primarily based on cross-lingual Plain Language principles [49,50]. Additionally, we incorporated text simplification rules from languages with grammatical structures similar to Lithuanian where applicable [51,52]. Furthermore, we developed rules tailored specifically to the Lithuanian language, particularly for managing participles. As a result, the Plain Lithuanian guidelines are organized into three levels of simplification operations, which can be outlined as follows:

• Paragraph-level simplification.

  (a)

  Sentence splitting: sentences longer than 12 words should be broken down into smaller units, preferably by turning embedded relative clauses into independent clauses.

  (b)

  List creation: where appropriate, homogeneous elements need to be transformed into vertical lists, i.e., if there are more than two coordinated elements (e.g., object or subject noun phrases and clauses) with a homogeneous function in a sentence, they need to be converted into vertical lists.

• Lexical-level simplification.

  (a)

  Preference should be given to the more frequent synonyms determined by the Lithuanian Frequency Dictionary [53], even if the normal formal requirements of the register are not followed.

  (b)

  Avoid metaphors and uncommon acronyms.

  (c)

  Define obscure terms in separate sentences.

• Syntactic-level simplification.

  (a)

  Transform passive voice constructions into active voice.

  (b)

  Replace active participle and gerund constructions with relative clauses.

  (c)

  Minimize the use of nominalizations.

  (d)

  Affirmatives are preferred to negatives.

  (e)

  If necessary, introduce demonstrative pronouns and nouns for clarity.

Corpora for Fine-Tuning. To explore the effect of differently prepared data in terms of model performance, we created 2 datasets which we used for fine-tuning mT5, mBART, and LT-Llama-2 (see

Table 1. Basic statistics of corpora.

	Parallel Corpus 1		Parallel Corpus 2
	Original Sentences	Simplified Sentences	Original Sentences	Simplified Sentences
Number of sentences	2142	2521	3123	2999
Number of words	36,404	34,702	64,936	52,382
Average sentence length	14.75	12.43	16.97	13.53
Average word length	7.10	6.79	7.03	6.68

):

• Parallel Corpus 1—a dataset where each original (complex) sentence had 1 simplified equivalent. The complex sentences were taken from websites of governmental and non-governmental public institutions.
• Parallel Corpus 2—a dataset in which Parallel Corpus 1 was augmented with additional complex sentences, some of which had more than 1 simplified counterpart (2–3), based on text simplification corpora such as SimPA [54] and Human Simplification with Sentence Fusion Dataset (HSSF) [55]. For complex sentences, we used the same list of sources as for Parallel Corpus 1.

Both corpora for this study were developed using administrative texts from websites of Lithuanian governmental and non-governmental organizations, covering diverse topics such as social benefits, migration, utilities, copyright, etc. Texts were selected to ensure a broad representation of administrative communication styles and topics. Potential bias related to the over-representation of certain topics was mitigated by diversifying sources and including texts with varying complexity. Additionally, manual review by 4 experts ensured that manually simplified versions of original (complex) sentences in the corpora would follow Plain Lithuanian guidelines.

Data for Testing. We used 554 sentence pairs that were not included in Parallel Corpora 1 and 2, although the list of sources was the same. The dataset was created based on the criteria of topic diversity and sentence complexity. This dataset has been prepared following the same guidelines and procedures as the corpora used for fine-tuning.

3.2. Methods

mT5. For our task, we fine-tuned the mT5 model, a multilingual adaptation of T5 (Text-to-Text Transfer Transformer) [56]. This model reformulates all language-processing tasks as text generation problems, following the foundational principles of the T5 architecture [57]. For its pre-training, the developers of mT5 utilized a multilingual version of the C4 dataset, which comprises textual data in 101 languages, including Lithuanian, sourced from a Common Crawl web scrape. Because the mT5 model has been pre-trained on diverse multilingual data, it is well suited for handling less-resourced languages such as Lithuanian.

mT5 is also a sequence-to-sequence model trained with the span corruption objective [57].

• Input:
  The input X is tokenized into subword units and corrupted by masking spans of tokens.
  The model predicts the masked spans $Z=z_1,z_2,\dots ,z_k$.
• Span corruption function:
  The model is trained to maximize the probability of the output spans:

  $${\mathcal{L}}_{\mathrm{mT}5}=-\sum _{i=1}^{k}logP\left(z_i\right|X,z_{<i})$$

  where P is the probability distribution modeled by mT5, which predicts the likelihood of the next span $z_i$; $z_i$ is the i-th span being predicted; X is the input sequence, which is corrupted by masking spans of tokens; and $z_{<i}$ is all spans predicted prior to $z_i$, ensuring that predictions are made sequentially and contextually.

mBART. We also fine-tuned mBART, a multilingual version of the BART (Bidirectional and Auto-Regressive Transformers) model, which integrates both auto-encoder and auto-regressive components [58]. Although mBART was originally developed for machine translation, its flexible architecture makes it well suited for text simplification tasks as well. It supports Lithuanian as well, even though Lithuanian is only a small part of its pre-training data (only 1835 characters in a 13.7-gigabyte corpus) [59].

The primary training objective of mBART is denoising sequence-to-sequence reconstruction, where the model learns to reconstruct corrupted input sequences [59].

• Encoder–Decoder architecture:
  The encoder processes the input sequence $X=\{x_1,x_2,\dots ,x_n\}$, generating hidden states $H=\{h_1,h_2,\dots ,h_n\}$.
  The decoder generates the output sequence $Y=\{y_1,y_2,\dots ,y_m\}$, conditioned on the encoder’s hidden states and previous decoder outputs.
• Denoising sequence-to-sequence reconstruction:
  The denoising involves corrupting the input sequence X via token masking and sentence permutation. The model minimizes the negative log-likelihood:

  $${\mathcal{L}}_{\mathrm{mBART}}=-\sum _{t=1}^{m}logP\left(y_t\right|y_{<t},H)$$

  where m is the total number of tokens in the output sequence Y, t is the index of the current token being predicted in the output sequence, $y_t$ is the token at position t in the output sequence $Y=\{y_1,y_2,\dots ,y_m\}$, $y_{<t}$ is all tokens in the output sequence that precede the current token $y_t$, H is the encoder’s hidden states, and $P(y_t\mid y_{<t},H)$ is the conditional probability of the current token $y_t$ being correct, given the previous tokens in the sequence ($y_{<t}$) and the encoded input H.

LT-Llama-2. LT-Llama-2 was the last model we fine-tuned for the simplification of Lithuanian administrative texts. It was pre-trained on the Lithuanian question/answer dataset and popular LLM benchmarks translated into Lithuanian—a total dataset of 2 trillion tokens. The model was then fine-tuned on publicly available instructional datasets and supplemented with manually annotated data via Reinforcement Learning with Human Feedback (RLHF). As reported in [60], the performance of LT-Llama-2 was generally comparable with many open-source alternatives.

Llama-2 is an autoregressive transformer-based model [61], and its key mathematical components include the following.

• Autoregressive component:
  The model generates the next token $y_t$ based on the sequence of previous tokens $y_{<t}$:

  $$P\left(Y\right)=\prod _{t=1}^{m}P\left(y_t\right|y_{<t})$$

  where $P\left(Y\right)$ is the probability of generating the entire output sequence $Y=\{y_1,y_2,\dots ,y_m\}$, m is the total number of tokens in the output sequence Y, t is the index of the current token being predicted in the sequence, $y_t$ is the token at position t in the output sequence, $y_{<t}$ is all tokens that have been generated before the current token $y_t$ in the sequence, and $P(y_t\mid y_{<t})$ is the conditional probability of the token $y_t$, given the tokens that precede it in the sequence ($y_{<t}$).
• Transformer architecture:
  The self-attention mechanism computes the attention scores

  $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right)V$$

  where Q, K, and V are the query, key, and value matrices, $d_k$ is the dimensionality of keys, softmax is a normalization function applied to the scaled dot product of Q and $K^{\top }$, and $Q{K}^{\top }$ is the dot product of the query matrix Q and the transpose of the key matrix K, which computes the similarity scores between the query and all keys.
• Optimization:
  Llama-2 uses adaptive optimization techniques and fine-tunes pre-trained weights for specific tasks.

ChatGPT. We evaluated ChatGPT for the text simplification task without fine-tuning. ChatGPT is a variant of the GPT (Generative Pre-trained Transformer) model [62] that leverages a self-attention mechanism that allows the model to understand and process the words in a sentence or document based on their relationship to each other [63,64]. To explore its effectiveness in a low-resource scenario, we tested ChatGPT (GPT-4o) on Lithuanian text simplification using its standard configuration through OpenAI’s browser interface. Although ChatGPT was not specifically designed for the Lithuanian language, it possesses multilingual capabilities to handle general Lithuanian text-processing tasks.

3.3. Evaluation Methods

To achieve a comprehensive assessment, our evaluation consisted of several steps:

• Preliminary evaluation:

  (a)

  Preliminary manual inspection to obtain a general idea of the results of all the models used in this study.

  (b)

  Quantitative assessment of the impact of our 2 corpora in terms of model performance. We used SARI [65], BERTScore [66] and 3 variants of ROUGE (ROUGE-1, ROUGE-2 and ROUGE-L) [67] for this task.

• In-depth evaluation of results of the best performing model:

  (a)

  Quantitative and qualitative approaches.

  We utilized the EASSE [68] and multilingual tseval [69] libraries to facilitate and standardize the automatic evaluation of our best text simplification model. The EASSE library includes reference-based evaluation metrics and methods, such as BLEU [70], SARI [65], and Levenshtein similarity [68]. In contrast, the tseval library provides reference-less simplification assessment metrics and methods, including the proportions of additions and deletions made during the simplification process.

  To complement these quantitative metrics, we conducted a qualitative evaluation of the text simplification outputs based on three widely recognized criteria: simplicity, meaning retention, and grammaticality (whether the model-simplified sentence is grammatical and understandable) [71,72]. Two experts independently evaluated model-simplified sentences. Additionally, these qualitative assessments were integrated with the tseval library to investigate the correlation between automatic evaluation metrics and the qualitative criteria.

  (b)

  Attention analysis:

  To gain more insights into the model’s decision-making process, we also applied BertViz [73] to the best and worst examples of the model’s simplified sentences.

4. Results

4.1. Experimental Setup

We fine-tuned three models, mT5, mBART and LT-Llama-2, for text simplification. All three models were directly fine-tuned separately using Parallel Corpora 1 and 2. The parameter configuration (batch size, learning rate, and number of epochs) was chosen based on the best results observed in previous iterations of fine-tuning. For the LT-Llama-2 model (Lt-Llama-2-7b-instruct-hf (accessible at https://huggingface.co/neurotechnology/Lt-Llama-2-7b-instruct-hf (accessed on 30 October 2024)), we applied 4-bit quantization for resource optimization, preserving performance without excessive memory demands. Additionally, Low-Rank Adaptation (LoRA) [74] with a dropout rate of 0.05 was applied to specific linear layers. Custom prompt formatting was used to guide the model toward producing simplified Lithuanian text responses, in alignment with our simplification task requirements. Finally, we included ChatGPT (GPT-4o) in our study by adding simplification rules from our simplification guidelines to the prompt to test its text simplification capabilities in Lithuanian without fine-tuning.

4.2. Fine-Tuning Process

The fine-tuning focused on the following key aspects:

• Hyperparameters: Based on our previous research, we selected the hyperparameters (batch size and learning rate) that provided the best results.
• Data Augmentation: We evaluated the impact of differently prepared data, using the original data (Parallel Corpus 1) and the updated (augmented) dataset (Parallel Corpus 2), on the results of fine-tuned mT5 and mBART.

The parameter configuration was chosen based on the best results observed in previous iterations of fine-tuning. For mBART, a batch size of 4 and a learning rate of ${10}^{-4}$ were chosen, while for mT5, a batch size of 2 and a learning rate of ${10}^{-4}$ were used. Given that mT5 showed better results with more epochs, the number of epochs was increased from 8 to 16, while for mBART and LT-Llama-2, 8 epochs were used. Also, to ensure efficient model adaptation while balancing memory and computational efficiency, we used a batch size of 4 and a learning rate of $2\times {10}^{-4}$ for LT-Llama-2.

By systematically varying these hyperparameters, we aimed to identify the optimal settings that maximize the performance of models on the text simplification task. This approach allowed us to assess the impact of fine-tuning strategies and data augmentation on the quality of the simplified texts produced by the models and on the desired level of readability, i.e., Plain Language.

4.3. Preliminary Evaluation

4.3.1. Fine-Tuning Results: Parallel Corpus 1 vs. Parallel Corpus 2

We evaluated the performance of fine-tuned mT5 and mBART using a selected set of metrics (SARI, BERTScore and ROUGE) to assess their capacity to simplify text while maintaining the original meaning and intent. This analysis provided insights into the effectiveness of each model in generating high-quality simplified Lithuanian text. A summary of the results is presented in

Table 2. Comparison of mBART and mT5 models for text simplification in Lithuanian language.

Metric	‘New’ mBART *	‘New’ mT5 *	‘Old’ mBART *	‘Old’ mT5 *
SARI	57.2374	54.1182	72.9781	56.0943
BERTScore	0.8633	0.8342	0.9155	0.8498
ROUGE-1	0.6396	0.5931	0.7797	0.6205
ROUGE-2	0.4703	0.4323	0.6753	0.4652
ROUGE-L	0.5993	0.5593	0.7555	0.5875

* New mBART and mT5 were fine-tuned on Parallel Corpus 2, old mBART and mT5—on Parallel Corpus 1.

. For testing our models, we used our test dataset (see Section 3).

Contrary to expectations, the “old” mBART model (SARI score of 72.9781), which was fine-tuned on Parallel Corpus 1, outperformed the “new” mBART model (SARI score of 57.2374) as well as both versions of mT5. This suggests that the “old” mBART was more effective in simplifying text by retaining, adding, and deleting words during the simplification process. Consistent with this trend, the “old” mBART also achieved the highest BERTScore, surpassing the other three models and indicating that its generated simplifications were semantically more aligned with the reference sentences.

In terms of ROUGE scores, which evaluate n-gram overlap between model outputs and reference texts, the “old” mBART consistently scored higher across all ROUGE variants: ROUGE-1 (0.7797), ROUGE-2 (0.6753), and ROUGE-L (0.7555). This shows that the “old” mBART was more effective in capturing the relevant information from the original (complex) sentences.

Overall, the findings show that the “old” mBART was superior to the “new” mBART and both mT5 variants on all evaluation metrics. It generated simplified sentences with higher precision, semantic proximity to the reference texts, and higher readability.

4.3.2. Results of Fine-Tuned LT-Llama-2

We fine-tuned LT-Llama-2 using Parallel Corpus 1, as this configuration had previously given better results with the mT5 and mBART models. However, the outcomes were inadequate. In many cases, instead of simplifying the provided sentences, the fine-tuned model simply expanded them by adding information that was not present in the original complex sentences. Also, while the majority of the generated sentences were grammatically correct, the model exhibited excessive repetition of words and word sequences.

The foundational LT-Llama-2 was primarily fine-tuned for instructional or conversational tasks [60], whereas sentence simplification typically requires restructuring rather than open-ended generation. Therefore, the model appears to be more suited for content expansion or engaging in dialogue than for text simplification. Furthermore, although seven billion parameters of LT-Llama-2 make it a substantial model, the complexity of the text simplification task may demand more nuanced understanding and generation capabilities, which suggests that a larger model might perform the task more effectively. After evaluating the performance and tendencies of the fine-tuned LT-Llama-2, we decided not to use it for further development of our simplification system.

4.3.3. ChatGPT Results

To test out ChatGPT (GPT-4o) in simplifying the Lithuanian administrative texts, we devised a prompt with 18 Plain Lithuanian rules which were accompanied by examples of complex and simplified sentences. The provided rules offered clear indications regarding the replacement of syntactic structures and lexical elements, as well as sentence splitting. However, the experimental results demonstrated that, while ChatGPT responded quite well to a simple and short prompt for simplifying the given text without any further elaboration on the desired outcome, it was prone to disregarding the majority of the rules outlined in a more comprehensive prompt.

While ChatGPT can demonstrate a basic understanding of English grammar, this was not the case for Lithuanian. For example, it was unable to correctly identify the Lithuanian passive voice, nor was it able to convert it into an active voice. This can be explained by the general lack of Lithuanian resources, as studies such as [75] have shown that ChatGPT has limited metalinguistic awareness in the context of minor languages. Hence, while ChatGPT could be employed for non-professional purposes to summarize or simplify a variety of texts, the results of our experiment indicated that its results are too unstable to use in the real-world simplification of Lithuanian administrative texts.

4.4. In-Depth Evaluation

As “old” mBART, fine-tuned on Parallel Corpus 1, provided the best results, we performed a full in-depth evaluation only of this model and its outputs. When a comparison was needed, we compared “old” mBART with “old” mT5 (for the sake of simplicity, in the following subsections, these models will be referred to as mBART and mT5) to obtain more comprehensive insights.

4.4.1. Qualitative Evaluation

To assess the results of fine-tuned mBART and mT5, two linguists, familiar with Plain Lithuanian, independently evaluated model-simplified sentences from our test dataset (see Section 3) according to 3 criteria, indicated in

Table 3. Qualitative evaluation scores (averages).

	Simplicity	Meaning Retention	Grammaticality
mT5	2.67	3.16	3.33
mBART	2.80	3.72	3.88

, i.e., simplicity, meaning retention and grammaticality. The linguists evaluated all the sentences for each criterion from 0 to 5. Though mBART performed best based on quantitative as well as qualitative assessment, we included mT5 in the qualitative evaluation for the sake of comparison.

Table 3 demonstrates that mBART consistently outperformed the other model across all three criteria. A more detailed qualitative analysis revealed that both mBART and mT5 were relatively effective at simplifying shorter sentences. However, their performance varied significantly in handling longer sentences, which are characteristic of the Lithuanian administrative language [76]. Specifically, mT5 tended to perform inadequately with extended passages, often generating nonsensical word repetitions instead of coherent sentences or omitting entire sections of the original text rather than transforming them into new, shorter sentences. Consequently, this significantly reduced its scores across all three evaluation criteria.

Additionally, both models showed higher scores in meaning retention and grammaticality compared to simplicity. This occurred because, on some occasions, the sentences were only slightly simplified, while certain elements that needed simplification were left unchanged. These largely unchanged sentences remained grammatically correct and fully preserved the original meaning, resulting in a score of 5.

During our qualitative analysis, we observed that both models had successfully captured key linguistic patterns characteristic of Plain Lithuanian. These patterns include the insertion of subject and possessive pronouns, the use of demonstratives, and the conversion of passive constructions into active voice. Although transforming passive sentences could potentially create challenges in identifying the subject, particularly when the original passive sentence lacked an oblique agent phrase, this issue was rarely encountered. In most cases, it was sufficient to simply add a first-person plural pronoun in the subject position. Additionally, mBART model excelled at converting participial clauses into relative clauses with finite verbs. However, a significant limitation was that very few sentences achieved perfect scores of 5-5-5. This suggests that our models may need more and more varied training data to achieve the level of robustness required for text simplification.

4.4.2. EASSE Report for mBART Results

Comparison with Baselines

This section provides a comparative analysis of our fine-tuned mBART against two baselines: the Identity baseline and the Truncate baseline [77]. The results were evaluated using various metrics calculated with EASSE library (see

Table 4. System vs. reference comparison table.

	SARI	FKGL	Compression Ratio	Sentence Splits	Levenshtein Similarity	Exact Copies	Additions Proportion	Deletions Proportion	Lexical Complexity Score
System output	49.63	13.84	0.94	1.04	0.84	0.19	0.20	0.23	9.66
Identity baseline	13.43	17.41	1.0	1.0	1.0	1.0	0.0	0.0	9.73
Truncate baseline	23.41	15.98	0.76	0.98	0.85	0.0	0.05	0.26	9.84

). The Identity baseline outputs the input text unchanged. It provides a reference to evaluate how much a system modifies the input. The Truncate baseline generates simplified text by truncating the input text to a specific length or removing less important parts without rewriting.

The mBART model outperformed both the Identity and Truncate baselines across calculated key metrics. Notably, mBART achieved a SARI score of 49.63, which was substantially higher than the Identity baseline’s 13.43 and the Truncate baseline’s 23.41, which indicated the robust simplification capabilities of our model. In terms of readability, the mBART attained FKGL (Flesch–Kincaid Grade Level) [78] score of 13.84 was lower than the Identity baseline’s score (17.41) and the Truncate baseline’s score (15.98), showing the better readability of mBART outputs.

The Compression Ratio [79] for mBART was 0.94, showing that the model effectively preserved content compared to the Truncate baseline’s 0.76. Additionally, mBART recorded a Sentence Splits value of 1.04, which was slightly above the Truncate baseline’s 0.98, which may reflect occasional sentence splitting while maintaining natural text structure.

Regarding text modifications, mBART achieved 0.84 of Levenshtein Similarity [68], indicating substantial but meaningful edits in comparison to the Identity baseline’s 1.0 score while staying close to the Truncate baseline’s 0.85 score. The model ensured that 19% of sentences remained exact copies, where the Identity baseline had 100% of them and the Truncate baseline had 0%, which highlights that simplification was performed effectively.

Furthermore, mBART exhibited 0.20 in the Additions Proportion [80], which was higher than both baselines. This meant that texts were enriched during simplification where it was necessary. Meanwhile, the Deletions Proportion for mBART was 0.23, which was less aggressive than the Truncate baseline’s 0.26, revealing balancing simplicity with informativeness. The Lexical Complexity [68] of mBART simplified sentences decreased to 9.66, compared to the Identity baseline’s 9.73 and the Truncate baseline’s 9.84, which showed the model’s efforts in choosing simpler vocabulary during the simplification.

In summary, the mBART model performed well in simplifying text effectively while retaining essential content and improving readability. It had a balanced approach to content preservation, selective additions and deletions, and vocabulary simplification in comparison to both baselines.

Analysis of Sentence-Length Effect

Table 5. Results by sentence length (characters).

Length	BLEU	SARI	FKGL	Compression Ratio	Sentence Splits	Levenshtein Similarity	Exact Copies	Additions Proportion	Deletions Proportion	Lexical Complexity Score
[8;55]	51.04	58.51	8.77	1.13	1.00	0.87	0.51	0.23	0.13	10.36
[55;102]	41.26	50.77	10.35	0.97	0.98	0.86	0.19	0.21	0.20	10.14
[102;164]	40.04	48.63	13.72	0.92	1.02	0.87	0.15	0.17	0.21	9.59
[164;242]	40.04	51.73	14.13	0.89	1.09	0.81	0.07	0.22	0.29	9.23
[242;4830]	22.10	45.06	16.80	0.81	1.11	0.78	0.04	0.18	0.33	8.88

illustrates the mBART performance across different intervals of sentence lengths, measured in characters. Short sentences ([8;55]) showed more distinguished performance on several metrics, e.g., achieving higher BLEU [70] (51.04) and SARI (58.51) scores compared to longer sentences ([242;4830]), which achieved a BLEU score of 22.10 and SARI score of 45.06. This disparity may be attributed to the inherent complexity of longer sentences [81], which negatively affected the model’s performance. Additionally, simplification metrics such as Levenshtein Similarity and Exact Copies [68] declined with increasing sentence length, which indicated more intensive editing for longer sentences. Specifically, short sentences maintained greater structural integrity with a Levenshtein Similarity of 0.87 and a lower proportion of deletions (0.13) compared to longer sentences (0.78 and 0.33, respectively).

Further analysis revealed that readability, assessed via FKGL score, increased from 8.77 for shorter sentences to 16.80 for longer ones, suggesting that simplifications of longer sentences maintained lower readability due to structural or vocabulary complexity. The Compression Ratio decreased from 1.13 in short sentences to 0.81 in longer sentences, which indicates more intensive length reductions while simplifying longer texts [79]. Sentence Splits showed a slight increase from 1.00 to 1.11 as sentence length increased, reflecting the more frequent splitting of longer sentences during the process of simplification.

The Exact Copies scores significantly decreased from 0.51 for short sentences to 0.04 for long sentences, demonstrating that mBART makes more extensive modifications to longer sentences. On the other hand, the Additions Proportion remained relatively stable across sentence lengths, ranging from 0.17 to 0.23, except for the longest sentences. In contrast, the Deletions Proportion [80] rose significantly from 0.13 for short sentences to 0.33 for longer ones, which highlighted the need for more substantial content removal in longer sentences. Additionally, the Lexical Complexity score decreased from 10.36 for shorter sentences to 8.88 for longer sentences, revealing a more significant reduction in vocabulary complexity during the simplification of longer sentences.

• The mBART model effectively simplified shorter sentences, achieving high BLEU and SARI scores while preserving readability and structural coherence. The metrics indicated a balanced approach to simplification, with minimal length reduction and selective sentence splitting for shorter texts.
• For longer sentences ([242;4830]), mBART’s performance decreased, as showed by lower BLEU and SARI scores, increased FKGL, and greater reliance on deletions and structural modifications. These challenges emphasized the complexity of maintaining both structure and meaning in the simplification of longer sentences.

Analysis of Best and Worst Simplifications

As our analysis revealed, mBART can perform one of the most essential text simplification operations in Lithuanian, i.e., to replace participial constructions, which are typical of the more formal Lithuanian language varieties, but infrequent in spoken and less formal language, with finite verb forms. Lithuanian has a wide variety of passive and active participles [82], so this simplification operation involves correctly identifying the noun phrase arguments of the verb from which the participle is formed, and changing their cases accordingly (Lithuanian has a system of 7 noun cases, with nominative being usually (though not always) used to mark the subject, while the other verb arguments generally are marked with accusative, genitive, or dative cases). This is especially relevant for the transformation of passive voice constructions, which also involve participles, into active clauses. The best simplifications according to SARI (part of EASSE report) exemplify the instances in which the model perform a variety of such operations correctly (Figure 1 and Figure 2).

While case adaptation is not necessary for active participle constructions (Figure 1), it is essential when changing the passive voice into active (Figure 2). The report showed that mBART does not always perform this operation correctly, as in Figure 3 where the highlighted noun suffixes are supposed to be accusative. However, mBART can also successfully add a personal pronoun which boosts the readability of the texts—Lithuanian being a pro-drop language [83], the pronouns in finite clauses are often dropped in administrative-style texts (Figure 2). In general, the best simplifications according to SARI are all examples of fairly successful transformations from administrative style to Plain Language. Thus, the EASSE report provided reliable results in this section.

The examples of the lowest SARI-scoring pairs of sentences did not involve any simplifications, i.e., the text remained unchanged between the original and the simplified versions. The lowest scores were applied both when simplifications were necessary but the model did not apply any, as well as when simplifications were advisable but not required, as a short phrase (usually 2–3 words) was already simple enough. Additionally, some simplifications with significant clause deletion were also assigned low scores. This is not ideal, as clause deletion in itself should not be penalized—Lithuanian administrative texts contain a significant amount of unnecessary information that sometimes should be deleted, instead of the model trying to simplify the titles of specific laws or other documents that are being referred to in the text.

However, quite understandably, EASSE does not have any means to distinguish between the deletions applied to essential information, resulting in incomprehensible sentences, and those that eliminate overspecification. The most significant drawback of the EASSE report on the worst simplifications according to SARI is that they rarely included wrong simplifications that did change the text but failed to produce grammatical and comprehensible sentences. Thus, this part of the report is less reliable.

Meanwhile, simplifications with the most compression provided by EASSE mainly included the following:

• Sentences that were shortened due to clause deletion, which may sometimes result in incomprehensible sentences when essential information is deleted, as already mentioned in the worst SARI-scoring simplifications.
• Sentences that were shortened due to a replacement of a noun phrase with an anaphoric pronoun, which, again, may or may not be a desirable simplification operation, depending on the complexity of the noun phrase and on the importance of the information it provides. Generally, anaphoric pronouns are discouraged in Plain Language guidelines [3].

Regarding the simplifications with the highest amount of paraphrasing, EASSE proposed only a few pairs of sentences with actual paraphrases. Mainly, it did not recognize the same root in both the original and the simplified version, most likely due to the morphological complexity of the Lithuanian language. Morphological complexity is surely also an issue for mBART: it can be seen from ungrammatical verb forms that sometimes appear in simplified texts, such as pakeičiuoja instead of pakeičia ‘changes’, probably per analogy with a common Lithuanian conjugation paradigm that has a -uoja suffix in the 3rd person present (e.g., skaičiuoti ‘to calculate’—skaičiuoja ‘calculates’, and matuoti ‘to measure’—matuoja) but does not apply to the verb pakeisti ‘to change’.

Finally, in some cases, the sentences selected by EASSE as including paraphrasing were those where mBART performed poorly and did not retain the original meaning of the sentence (e.g., 1 žingsnis ’Step 1’ simplified as Antras žingsnis ‘Second step’), or where mBART rewrote dates in words, such as ‘2022.04–2023.03’—nuo 2022 metų balandžio pradžios iki 2023 metų balandžio pabaigos ‘from the beginning of April 2022 until the end of April (sic!) 2023’.

4.4.3. Tseval Results for mBART

To streamline further analysis, we processed qualitatively evaluated model-simplified sentences using a classification approach for simplicity, meaning retention, and grammaticality scores, to explore the correlation of quantitative evaluation metrics and qualitative evaluation criteria. Initially, these scores ranged from 0 to 5, as the evaluators who qualitatively evaluated model-simplified sentences used this evaluation scheme. To classify them, we converted the continuous scores into categorical labels based on defined thresholds:

• Bad: Scores between 0 and 2.5 indicate a low simplification evaluation level.
• Ok: Scores between 2.5 and 3.5 represent a medium or acceptable simplification evaluation level.
• Good: Scores between 3.5 and 5 indicate a high simplification evaluation level.

This transformation enabled us to categorize the evaluation metrics, thereby simplifying their interpretation via a diverse range of methods that do not require reference texts for comparison. To achieve this, we utilized the multilingual tseval library [69], which offered functions for assessing the quality of text simplification, including scores for simplicity, meaning retention, and grammaticality. We conducted this analysis only on the results from our best-performing model, mBART. The label distribution of processed data is shown in Figure 4.

The following tables summarize the key metrics and correlations for each of the three qualitative criteria in assessing text simplification results. For calculating correlation, Pearson correlation coefficients [84] were calculated. Only the metrics that showed statistically significant correlations are presented, i.e., with p $\le 0.05$.

Simplicity

Table 6. Metrics that correlated with the simplicity criterion.

Metric	Pearson	p-Value
BLEUSmoothed	−0.1277	0.0071
AverageCosine	−0.1114	0.0190
LexicalComplexity	−0.1100	0.0206
ProportionDeletedWords	0.1067	0.0247
FleshReadingEase	0.1061	0.0255
AvgPositionWordsFreqTable	−0.1021	0.0317
CompressionRatio	0.1011	0.0334
LemmasInCommon	−0.0988	0.0377
LevenshteinDistance	0.0954	0.0448
LevenshteinSimilarity	−0.0954	0.0448

reports 10 metrics that show statistically significant correlations with the simplicity criterion, although the correlation coefficients are weak (<0.19) [85]. Thus, positive correlations were found for the proportion of deleted words, Flesh Reading Ease (FRE) [86], compression ratio, and Levenshtein distance [87]. This indicated that sentences with higher deletion proportions, better readability scores, shorter lengths, and more substantial editing tended to be simpler.

On the other hand, negative correlations were observed for smoothed BLEU [88], average cosine similarity between pre-trained word embeddings [89] of complex and simplified sentences, lexical complexity, the average position of words in simplified sentences in a frequency table, percentage of common lemmas between complex and simplified sentences, and Levenshtein similarity. These findings suggested that sentences with greater lexical overlap, semantic similarity, or complexity were less simplified.

All in all, the aforementioned metrics aid in evaluating sentence simplicity by measuring the extent of reduction, readability, and semantic alterations between original (complex) and simplified sentences. However, the weak correlations imply that these metrics, while statistically significant, are not strong predictors of simplicity individually.

Meaning Preservation

Table 7. Metrics that correlated with meaning preservation criterion.

Metric	Pearson	p-Value
BLEUSmoothed	0.1726	0.0003
ProportionDeletedWords	−0.1660	0.0005
WordsPerSentence	−0.1648	0.0005
CharactersPerSentence	−0.1648	0.0005
SyllablesPerSentence	−0.1641	0.0005
MaxPositionWordsFreqTable	−0.1461	0.0021
FKGL	−0.1409	0.0030
CharsPerSentenceDifference	−0.1293	0.0064
UnchangedWordsProportion	0.1269	0.0075
LexicalComplexityScore	−0.1130	0.0174
KeptWordsProportion	0.1117	0.0187
HungarianCosine	0.0998	0.0358

presents 12 metrics that reveal statistically significant correlations with the meaning preservation criterion, although with weak correlation coefficients (<0.19). Positive correlations were observed for smoothed BLEU, Levenshtein similarity, the percentage of words left unchanged after simplification, the percentage of shared word forms between complex and simplified sentences, and Hungarian cosine similarity between the pre-trained word embeddings of complex and simplified sentences [89]. These positive correlations demonstrated that greater overlap in content, vocabulary, and semantic similarity contributed to better meaning retention.

Conversely, negative correlations were identified for the percentage of deleted words during simplification, the number of words per sentence, the number of characters per sentence, the number of syllables per sentence, the maximum position of output words in the frequency table, FKGL scores, the difference in the number of characters between original (complex) and simplified sentences, and lexical complexity scores. These negative correlations suggested that excessive content reduction or overly simplified sentences may result in a loss of meaning.

In summary, while these metrics provide valuable insights into factors influencing meaning preservation by assessing reduction extent, readability, and semantic changes between original and simplified texts, the weak correlation scores imply that they are not strong individual predictors of meaning retention. Therefore, a comprehensive evaluation incorporating multiple factors may be necessary.

Grammaticality

Table 8. Metrics that correlated with grammaticality criterion.

Metric	Pearson	p-Value
BLEUSmoothed	0.1651	0.0005
Words	−0.1472	0.0019
Characters	−0.1472	0.0019
SyllablesPerSentence	−0.1463	0.0020
LevenshteinDistance	−0.1386	0.0035
LevenshteinSimilarity	0.1386	0.0035
UnchangedWordsProportion	0.1310	0.0057
WordsPerSentence	−0.1275	0.0072
CharactersPerSentence	−0.1275	0.0072
MaxPositionWordsFreqTable	−0.1258	0.0080
SyllablesPerSentence	−0.1247	0.0086
LemmasInCommon	0.1243	0.0088
ProportionKeptWords	0.1179	0.0131
FKGL	−0.0977	0.0399

presents 14 metrics that demonstrated statistically significant although weak correlations with the grammaticality criterion.

Positive correlations were identified for smoothed BLEU, Levenshtein similarity between complex and simplified sentences [87], the percentage of words left unchanged during simplification, the percentage of lemmas shared between complex and simplified sentences, and the percentage of words retained during simplification. These results suggested that retaining more of the original (complex) sentence structure and vocabulary maintained the grammatical correctness of the simplified sentences.

However, negative correlations were found for the number of words in the simplified text, the number of characters in the simplified text, the number of syllables per sentence, the Levenshtein distance between original and simplified sentences, the number of words per sentence, the number of characters per sentence, the maximum position of output words in the frequency table, the percentage of common lemmas between complex and simplified sentences, the percentage of common word forms between complex and simplified sentences, and FKGL. These negative correlations indicated that increasing the complexity of simplified sentences may lead to grammatical errors.

In summary, while the evaluation metrics provided insights into factors that influenced grammaticality by assessing the preservation of sentence structure and vocabulary, weak correlation scores implied that these metrics are not strong individual predictors of grammatical correctness. Therefore, a more complex approach incorporating multiple metrics may be necessary.

4.4.4. Attention Analysis with BertViz

BertViz facilitates the visualization of attention weights, which enables examining which segments of the input text the model emphasizes during the simplification process [73]. For comparative analysis, BertViz was applied to simplified examples with the highest and lowest SARI scores, representing high-quality and low-quality simplifications, respectively. In transformer models, attention mechanisms allow the model to focus on different parts of the input when generating predictions [64]. Each attention head in the model’s layers calculates a weight matrix that measures how much attention each token gives to the other tokens in the sequence, which determines their relevance. BertViz’s attention plots visualize the weight distributions, showing which specific words, subwords, or other elements the model focuses on for each token in the input sequence. This analysis focuses on the model’s last layer, as it usually has the most relevant information for the fine-tuned task which is text simplification in our case.

Cross-attention, also known as encoder–decoder attention [90], shows how each token in the decoder’s output (the simplified sentence) relates to tokens in the encoder’s input (the complex sentence). This mechanism reveals how the model uses the original sentence to create its simplified version. Strong attention connections indicate which elements in the input sentence influence specific (sub)words and other types of tokens in the output. The effective simplification or paraphrasing is marked by logical alignments between key elements of the input and output. Conversely, misaligned or weak attention may signal difficulties in simplification or loss of important context.

Figure 5 and Figure 6 illustrate cross-attention patterns for sentences with the highest and lowest SARI scores, respectively. In the low-SARI example, attention in most heads relied heavily on delimiters, suggesting the model struggled to identify meaningful parts of the input to focus on [91]. A similar but weaker trend appeared in the high-SARI example. In this case, some attention heads focused on (sub)words that predicted the target (sub)word without directly focusing on the target (sub)word itself, while others distributed attention more evenly across the entire sentence. This suggested that the model was attempting to consider the whole sentence rather than giving priority to specific tokens or segments.

Decoder attention shows how each token in the simplified sentence interacts with other tokens within the same generated sequence. This provides insight into how the model maintains coherence and fluency in its output [92]. For example, it demonstrates how the model ensures grammaticality, such as aligning pronouns with their antecedents or matching verbs with their subjects. Repetitive or diagonal attention patterns often indicate that the model relies on recently generated (sub)words for maintaining fluency.

Figure 7 and Figure 8 display decoder attention for sentences with the highest and lowest SARI scores, respectively. In both cases, attention heads are mostly focused on the same (sub)words or a small number (mostly up to 5) of preceding (sub)words. This pattern was more pronounced in the example with the lowest SARI score, which suggested that the model often copied (sub)words from the input and limited its attention to a few recently generated tokens, instead of considering the broader sequence. While this behavior can benefit text simplification tasks that require minimal changes to the original sentence, it also highlighted potential limitations in the model’s ability to capture long-range dependencies or fully understand the overall context of the input which became apparent in the analysis of model performance in terms of sentence lengths.

Encoder attention visualizes the attention weights within the encoder layers, providing insight into how the model processes the original (complex) sentence before generating a simplified version. This visualization highlights which parts of the input sentence were most important for encoding its meaning [93]. Strong attention weights between specific (sub)words indicate that the model identified these (sub)words as significant or related, which is essential for building a meaningful contextual representation.

Figure 9 and Figure 10 show encoder attention patterns. In the low-SARI example, most attention heads focused on delimiters, which suggests that the model leaned heavily on special tokens to segment the input text. Thus, this reliance on structural cues may have caused the model to overlook key content (sub)words that carry the sentence’s core meaning. In contrast, the high-SARI example showed a different pattern. While some attention heads still focused on delimiters, in many cases they distributed their attention more evenly across all words in the sentence. This suggested that the encoder was working to develop a more balanced understanding of the sentence while taking all components into account to capture the overall meaning or context. Such an approach may improve the model’s ability to generalize to diverse input data, especially when the input does not follow consistent patterns.

5. Discussion

Our study on the simplification of Lithuanian administrative texts to plain language brings several findings related to performance, data augmentation, evaluation metrics, and the capabilities of different language models. These findings are discussed further in this section.

5.1. Data Augmentation and Training Data Quality

• Contrary to expectations, data augmentation did not improve the performance of our text simplification models. The original dataset alone was sufficient to achieve satisfactory results, which highlighted the importance of high-quality and representative data in low-resource language contexts [94]. Instead of enhancing the training data, augmented data may have introduced inconsistencies or reduced the overall quality of the dataset, ultimately decreasing the models’ performance.
• When compared to related studies in text simplification and low-resource NLP tasks, these findings reveal a notable contrast. While data augmentation has been effective for other languages, e.g., English [95], its ineffectiveness in this study points to challenges associated with Lithuanian. This could be due to the broader limitations of simplistic augmentation techniques [96] or specific features of the Lithuanian language.

Unlike studies, targeting large languages, such as English, where many large-scale datasets and pre-trained models are available, we demonstrate the feasibility of simplification in a lower-resource context. Our results underscore the importance of carefully curating datasets to ensure linguistic alignment and high quality. Relying on data augmentation as a universal solution may not always be effective. Instead, the priority should be on developing representative datasets specifically tailored to the unique characteristics of the language to ensure robust model performance.

5.2. Limitations of Automated Metrics

Our study highlighted the limitations of automated metrics like BLEU and ROUGE in evaluating the nuances of text simplification. Human evaluations proved essential for capturing aspects such as grammaticality, semantic accuracy, and the overall quality of simplified outputs, which uncovered trends and patterns that automated metrics missed, such as applying appropriate context in simplifications.

For instance, BLEU prioritized exact matches between simplified and reference sentences and penalized valid paraphrases or structural changes that improved simplification. This limitation stressed the need for more advanced evaluation frameworks. Human evaluators, on the other hand, were able to assess qualitative aspects of simplification that automated metrics overlooked, which accented the importance of integrating human judgment into evaluation processes.

These findings aligned with discussions in the NLP field that advocated for hybrid evaluation approaches that combine automated metrics and human assessments [97,98,99]. Such frameworks combine the scalability of automated metrics with the nuanced insights from human evaluations, which enable a more comprehensive and accurate evaluation of text simplification quality.

5.3. Model Performance

• While mBART performed well in structural simplifications, such as adding pronouns or transforming a passive voice into active voice constructions, it, however, struggled with very long or syntactically complex sentences, which indicated areas where we need further improvements. In comparison, mT5 had difficulties processing longer passages, which likely occurred due to memory limitations or insufficient fine-tuning. This suggested that methods like hierarchical modeling [100] or pre-processing strategies such as sentence splitting could enhance its performance.
• In addition, experiments with LT-Llama-2 revealed limitations, such as the unintended expansion of content, where the model added information not present in the original sentence. These issues likely arose from the model’s design, as it is more suited for conversational or instructional tasks rather than structural rewriting which is more typical for sentence simplification. These findings highlighted the potential need for larger models, alternative architectures, or more targeted fine-tuning techniques to aid in better aligning the model with the requirements of text simplification.
• Finally, ChatGPT, though effective in many general-purpose applications, faced challenges with Lithuanian-specific grammatical rules. This raised a question about the suitability of general-purpose language models for low-resource tasks without extensive customization, such as Reinforcement Learning with Human Feedback (RLHF) [101].

5.4. Sentence-Length Dependency and Structural Changes

• The more in-depth analysis of mBART performance showed a clear dependency on sentence length. The model performed well with short to moderate sentences but struggled with longer, more complex inputs. While techniques like sentence splitting and lexical simplification were effective for shorter sentences, simplifications of longer sentences occasionally resulted in the loss of essential information due to deletions.
• This decline in performance for longer sentences can be attributed to increased syntactic complexity and semantic density. Longer sentences often have complex structures and contain a higher concentration of information [102,103], which makes them more difficult to simplify without losing or altering their meaning. For example, the analysis of input–output pairs revealed that mBART performed well with simple sentence structures but struggled with compound or complex sentences, occasionally producing incomplete simplifications.

To address these challenges, strategies such as iterative simplification [47] or introducing semantic constraints during generation [104] could help reduce information loss when splitting sentences. Additionally, balancing lexical simplification with content preservation is essential. Incorporating semantic similarity metrics into model training [105] could help ensure that simplified text preserves its core meaning while its complexity is decreased.

5.5. Evaluation Metrics and Correlation Analysis

In our in-depth study, we identified weak correlations between automated evaluation metrics and human-judged criteria for simplicity, meaning retention, and grammaticality, which highlighted the challenges of evaluating text simplification, as automated metrics often disagree with human perceptions, i.e., simpler texts often reduced lexical complexity, but for retaining the original meaning a high degree of structural and lexical similarity between the original and simplified versions was required.

These weak correlations likely resulted from a misalignment between automated metrics and human evaluation criteria. Metrics such as BLEU and ROUGE focus on surface-level similarities, like word overlap, and fail to capture deeper aspects [38] such as meaning preservation and consistency, and grammatical correctness. Alternative evaluation methods or task-specific metrics may provide more accurate assessments in capturing contextual and semantic relationships between words. Finally, using diverse perspectives, e.g., including not only domain experts but also target audiences, could improve evaluation frameworks considering the subjective and complex nature of text simplification.

5.6. Attention Mechanisms and Model Behavior

• An analysis of attention mechanisms revealed that mBART effectively linked input and output content but relied too much on delimiters, which impacted semantic richness. Specifically, the encoder attention frequently focused on delimiters, which indicated that the model relied on these structural cues rather than giving sufficient attention to content (sub)words. This reliance may have resulted in not using important information within the text on some occasions.
• On the other hand, decoder attention prioritized fluency by focusing mainly on identical (sub)words or a few preceding (sub)words. This was particularly evident in the examples with lower SARI scores. While this strategy supported grammatical correctness and coherence, it limited the model’s ability to capture long-range dependencies or fully interpret the broader context of the input [106]. This indicated that mBART primarily focused on local context instead of processing and integrating information from the entire sentence, which could limit its ability to produce semantically meaningful and contextually accurate simplifications.

To address these limitations, potential improvements could include implementing hierarchical attention mechanisms [100] or memory augmentation techniques to help the model better capture long-range dependencies, e.g., as in [107,108,109]. Bidirectional decoding strategies could also be explored to achieve a more comprehensive understanding of the input sentence, which would allow the model to generate simplifications that are semantically richer and more accurate. These strategies could improve model performance not only in Lithuanian text simplification but also in other tasks involving low-resource languages.

6. Conclusions

In this study, we investigated text simplification for Lithuanian administrative documents, aiming to improve information accessibility through the use of Plain Language. We fine-tuned the mT5, mBART, and LT-Llama-2 models and evaluated the capabilities of ChatGPT. Among the evaluated models, mBART demonstrated the best performance, effectively handling lexical and syntactic simplifications that corresponded to the requirements of Plain Language.

However, challenges remain in processing longer and more complex sentences, highlighting opportunities for further improvements in sentence-splitting techniques and syntactic complexity management. Also, the results emphasized the importance of high-quality, representative datasets for low-resource languages and underlined the limitations of general-purpose models like ChatGPT for language-specific tasks. Human evaluations proved essential in assessing grammaticality, meaning retention, and simplicity, highlighting the need for task-specific metrics that align with human judgment.

Our future plans will focus on improving model performance through refined fine-tuning techniques, experimenting with larger and more diverse datasets, and optimizing our models for handling longer and more complex texts. While the data we used for fine-tuning the models were carefully curated, the reliance on publicly available administrative texts may introduce sampling bias, particularly in underrepresented domains. The sample size, although sufficient for this study, may limit generalizability to other administrative or informal text types. Therefore, future work will focus on expanding the dataset to include a more comprehensive range of administrative texts.

Additionally, we aim to deepen our understanding of model behavior by analyzing the decision-making processes regarding simplifications, with particular attention to improving factual accuracy and identifying potential biases. These advancements aim to broaden the applicability of text simplification tools in facilitating communication between institutions and the general public.