Exploring GPT-4 Capabilities in Generating Paraphrased Sentences for the Arabic Language

Abstract

Paraphrasing means expressing the semantic meaning of a text using different words. Paraphrasing has a significant impact on numerous Natural Language Processing (NLP) applications, such as Machine Translation (MT) and Question Answering (QA). Machine Learning (ML) methods are frequently employed to generate new paraphrased text, and the generative method is commonly used for text generation. Generative Pre-trained Transformer (GPT) models have demonstrated effectiveness in various text generation tasks, including summarization, proofreading, and rephrasing of English texts. However, GPT-4’s capabilities in Arabic paraphrase generation have not been extensively studied despite Arabic being one of the most widely spoken languages. In this paper, the researchers evaluate the capabilities of GPT-4 in text paraphrasing for Arabic. Furthermore, the paper presents a comprehensive evaluation method for paraphrase quality and developing a detailed framework for evaluation. The framework comprises Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), Lexical Diversity (LD), Jaccard similarity, and word embedding using the Arabic Bi-directional Encoder Representation from Transformers (AraBERT) model with cosine and Euclidean similarity. This paper illustrates that GPT-4 can effectively produce a new paraphrased sentence that is semantically equivalent to the original sentence, and the quality framework efficiently ranks paraphrased pairs according to quality criteria.

1. Introduction

NLP focuses on the interaction between computers and human (natural) languages. It is a field considered to be at the intersection of computer science, Artificial Intelligence (AI), and computational linguistics [1]. AI is defined as “the automation of activities that we associate with human thinking, activities such as decision-making, problem-solving, and learning” [2]. NLP has numerous applications based on the textual form, for instance, a chatbot (chatter robot) [3], autocomplete suggestions [4], MT, QA [5], and paraphrase generation [6]. Paraphrasing transforms a given text into another text while keeping its semantic meaning [7]. Paraphrase generation significantly impacts several NLP applications because it allows the use of different word expressions to create similar text. Therefore, generating an alternative text for the same meaning enhances the performance of other NLP applications such as semantic parsing, machine translation, query formulation in web searching, question answering, and data augmentation [6,8]. ML is a branch of AI that investigates computer learnability based on data [9]. Recent advancements in ML have significantly impacted NLP [10], and there are many ML approaches to generating paraphrasing.

ML approaches include rule-based [11], sequence-to-sequence [12], reinforcement learning [6], transformer model [12], and generative models [13]. GPT is a generative model commonly used for text-generation purposes [14]. However, GPT could be used for purposes exceeding text generation. For instance, Refai et al. (2023) [10] use Arabic GPT-2 (AraGPT-2) for data augmentation, Yang et al. (2022) [15] utilize GPT-3 to improve Knowledge-based Visual Question Answering (VQA), and Sahib et al. (2023) [16] investigate the performance of ChatGPT-3.5 and ChatGPT-4.0 in proofreading and rephrasing a given paragraph in English. To our knowledge, GPT-4’s capability in Arabic paraphrase generation remains underexplored, although Arabic is one of the most widely spoken languages globally. This gap shows a significant research opportunity to evaluate GPT-4 capabilities in producing high-quality, lexically diverse, and semantically equivalent paraphrased sentences for Arabic text, which can be used across various downstream applications. Additionally, our proposed methodologies and approaches can be adapted to languages with similar morphological and syntactic structures.

This research explores and evaluates GPT’s capabilities in paraphrasing Arabic text. To optimize paraphrasing performance, this study identifies effective prompt engineering and inquiries about similarity scores for paraphrased and non-paraphrased pairs. In addition, paraphrase generation is covered for text generally in different domains but without studying the impact of domains [17]; thus, this study investigates the impact of text categories on the quality of generated paraphrased sentences. This research proposes a detailed evaluation framework for assessing paraphrasing quality, comprising lexical diversity, content retention, and semantic similarity. Furthermore, this work includes releasing a new Arabic paraphrasing corpus containing approximately 200 K pairs, which is designed to support future research and advancements in AI and Arabic NLP research. For instance, the corpus could be used to build a new paraphrasing model for text in specific domains, and the corpus might combine with datasets from other languages to generate a multilingual paraphrasing model. On the other hand, the corpus could be used to analyze AI-generated text from linguistic perspectives, such as investigating word selection and sentence structure. This paper is organized as follows. Section 2 reviews related work and fundamental concepts, while Section 3 outlines the research methodology. Section 4 then presents the results and discussion, focusing on the expected corpus statistics and their evaluation.

2. Paraphrasing Theory and Models

This section provides essential definitions and concepts, such as paraphrasing types and evaluation metrics. It also reviews related works by highlighting their research aims and illustrating their methodologies. Furthermore, it gives examples of datasets used in the relevant research papers.

2.1. Fundamental Concepts and Theoretical Foundations of Paraphrasing

In linguistics, there are many techniques for paraphrasing a given text. The most common technique is using synonyms. The synonym approach depends on identifying keywords within a given text and finding appropriate synonyms to substitute for them. The second technique is using a different form of a word. This approach is based on using another form of the word within a given text, such as changing an adjective to a noun, a noun to a verb, etc. The third technique is modifying the used tense from active tense to passive tense, and this approach depends on grammar structures for the target language. The fourth technique is altering the word order; simply, it keeps the words used but modifies the word order. The last technique combines all the previous methods by combining two or three techniques [18].

Table 1. Examples of paraphrasing types from a linguistic perspective.

Paraphrasing Methods	Language	Original Sentence	Paraphrased Sentence
Synonyms substitution	Arabic	قرأ الطفل القصة بمتعة	قرأ الطفل القصة بسرور
	English	The child reads the story with enjoyment	The child reads the story happily
Different forms of a word	Arabic	قرأ الطفل القصة بمتعة	تمتع الطفل بقراءة القصة
	English	The child reads the story with enjoyment	The child enjoyed by reading the story
Changing tense	Arabic	قرأ الطفل القصة بمتعة	قرأت القصة بمتعة
	English	The child reads the story with enjoyment	The story was read with enjoyment
Altering the word order	Arabic	قرأ الطفل القصة بمتعة	بمتعة قرأ الطفل القصة
	English	The child reads the story with joy.	With joy, the child reads the story.

illustrates paraphrasing methods by mentioning examples. Paraphrasing can be expressed on many levels: word level, phrase level, sentence level, paragraph level, or document level. LD is a primary principle in paraphrase evaluation [19], which is a measure to assess how diverse words or vocabulary are within a text [20]. Evaluating the quality of paraphrasing requires language experts to perform human evaluations.

Human evaluation is a critical component in numerous NLP tasks [7]. However, it remains a challenging stage across various tasks, which include machine summarization [21], machine translation [22], and text paraphrasing [19]. Human evaluation is both time-consuming and costly because it often requires human labor (a language expert) [22]. Additionally, human experts might evaluate a given text differently, leading to score variability. Furthermore, human resources are not reusable, meaning each evaluation cycle requires additional time, effort, and cost. Therefore, Papineni et al. (2002) [22] considered finding an automatic approach to evaluating textual data by a machine rather than a human. Next, other researchers conducted new methods for assessing textual data using a machine; these evaluators are called automatic metrics. Automatic metrics in NLP are essential tools for evaluating the quality of generated text, particularly in tasks such as machine translation and summarization. These metrics include widely recognized ones like BLEU, which is used for machine translation, and ROUGE, which is developed for machine summarization [21,22].

BLEU measures how close the given machine-generated translation (candidate) is to human translation (reference), which could be one, two, or three references. BLEU depends on calculating precision based on n-grams by finding the total number of overlapped n-grams between the machine-generated text and reference texts [23]. BLEU computes the geometric mean of the precision scores for many n-grams from unigram (1-gram) to four-gram (4-gram). Additionally, BLEU considers sentence length by applying a brevity penalty to overcome the issue of low short translation scores [24]. The metric has been widely adopted in various NLP tasks beyond translation, including text summarization, image captioning, and text paraphrasing [12,25]. ROUGE could be calculated in multiple n-grams, such as unigrams and bigrams, and includes many variants, such as ROUGE-L and ROUGE-N. ROUGE-L calculates the longest common subsequence between two given sequences (text and summary are in token format), which is used to capture the structure of sentences. Furthermore, ROUGE-N calculates the overlap of n-grams, while ROUGE-1 evaluates unigrams and ROUGE-2 assesses bigrams [26]. The score of ROUGE involves calculating recall, precision, and F1 scores based on n-gram matches [27]. ROUGE-Precision counts the numbers of n-grams in the candidate summary that appear in the reference text, while ROUGE-Recall calculates the number of n-grams in the reference text that are in the candidate. ROUGE-F1 is the harmonic mean of ROUGE-Precision and ROUGE-Recall [26].

Table 2. Formulas of BLEU and ROUGE.

Metric	Formula
BLEU-Precision	$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{Count of matching n-grams }{Total n-grams in candidate}}$
ROUGE-Recall	$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{Count of matching n-grams }{Total n-grams in reference}}$
ROUGE-Precision	$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{Count of matching n-grams }{Total n-grams in candidate}}$
ROUGE-F1 Score	$\mathrm{F}1 \mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}={\displaystyle \frac{2\ast ROUGE-Precision\ast ROUGE-Recall }{ROUGE-Precision+ROUGE-Recall}}$

displays the formulas of both BLEU and ROUGE. However, BLEU and ROUGE cannot capture the semantic meaning of two given texts; therefore, other metrics measure how the meanings of two given texts are closely related [28]. BLEU and ROUGE range from zero to one [0–1], and higher values indicates better quality.

Semantic similarity is a crucial consideration in NLP, and it is used for various NLP applications, including text paraphrasing [19], data augmentation [10], text summarization [29], and machine translation [30]. Many metrics are used for semantic similarity, such as cosine similarity, Euclidean distance, and the Jaccard score.

Table 3. Formula of similarity metrics.

Metric	Formula	Description
Cosine Similarity	${\displaystyle \frac{\sum _{i=1}^n{A}_i\ast B_i}{\sqrt{\sum _{i=1}^n{A_i}^2}\ast \sqrt{\sum _{i=1}^n{B_i}^2}}}$	The cosine score is calculated by finding the angle between two vectors (A and B) and focusing on their orientation rather than magnitude.
Euclidean Distance	$\sqrt{{\displaystyle \sum _{i=1}^n}{(A_{i }-B_{i }) }^2}$	Euclidean distance is computing the straight-line distance between two points (vectors) on a plane.
Jaccard Score	$J\left(A,B\right)={\displaystyle \frac{|A\cap B|}{|A\cup B|}}$	The Jaccard score is calculated by finding the intersection between two given sets of tokens.

provides the formula for each similarity metric. Cosine similarity depends on finding the angle between two given vectors regarding their direction. Its value ranges from −1 to 1, where 1 means identical meaning, 0 means dissimilar, and −1 means the opposite [31,32]. Euclidean distance calculates the distance between two vectors, where zero means identical vectors and a smaller distance means more semantic similarity [8]. At the same time, Jaccard similarity computes the intersections between two given sets of tokens, which are primarily words [33]. Cosine similarity and Euclidean distance are commonly used with word embedding.

Word embedding is an NLP method representing vocabulary words and phrases into numerical vectors [30,34]. For text embedding, the Bi-directional Encoder Representation from the Transformers (BERT) model [13] and the AraBERT model [35] are commonly used in a variety of NLP applications. The BERT model can understand the context of the words in a given sentence [36]. Moreover, it demonstrates a powerful language representation due to its ability to capture semantic and contextual meaning. BERT has led to significant advancements in multiple NLP tasks, such as sentiment analysis [37], sarcasm identification [38], information retrieval [39], question answering [36], machine translation [40], and classifying paraphrased text [41]. BERT utilizes the concept of deep bidirectional training of transformer architectures. BERT aims to capture the context of words in a sentence by moving in both directions (left and right) simultaneously [13]. BERT is generated using English text [42,43]. Other researchers depend on the BERT architecture to develop different versions of BERT using various datasets in different languages.

Antoun et al. (2020) [35] pre-trained the BERT model for three Natural Language Understanding (NLU) tasks, which are Sentiment Analysis (SA), Name Entity Recognition (NER), and QA. Moreover, they cover both Modern Standard Arabic (MSA) (formal language) and Dialectal Arabic (DA) (informal language). The AraBERT model is publicly available and could be used as a baseline for other NLP tasks because it is trained using 24 GB data with 64 K vocabulary tokens [35]. In addition, AraBERT is developed to capture the structural aspects of Arabic by utilizing the text tokenization stage. AraBERT shows its outperformance in various Arabic NLP tasks [35], for example, toxic tweet classification [44], emotion analysis [45], fake news detection [46], textual similarity [47], sentiment analysis [35], named entity recognition [35], and question-answering [35]. Therefore, AraBERT can understand and produce Arabic text more effectively [48], and its performance is the most advanced for many NLP tasks [34]. AraBERT is used in word embedding to represent each word into a vector of size 768 [30,34]. Similarity metrics and AraBERT embedding could be used to evaluate the performance of text paraphrasing models. Moreover, several approaches exist to generate new paraphrased text using AI methods.

These AI approaches include rule-based, sequence-to-sequence, reinforcement learning, and deep generative models. In the rule-based method, the AI model depends on evolving a system that could eliminate the number of possible alternative texts using defined constraints. The model focuses on the text that matches the defined constraints, and the model must identify which word is suitable for synonym substitution [11,49]. In a sequence-to-sequence (seq2seq) method, the AI model is developed based on the encoder–decoder paradigm [6]. This model contains three main parts: encoder, decoder, and context layer. A given input goes into the encoder to result in a vector, which is then used as input for the context layer. The context layer is a context vector of function h, which forwards the essence of the input to the decoder. Finally, the decoder creates a task-specific output as a sequence [50]. More complex additional steps are taken in the reinforcement learning and deep generative model.

The reinforcement learning model depends on two main mechanisms: the statistical mechanism and the dynamic programming mechanism. Both mechanisms enhance decision-making worldwide by assessing the benefit of taking action. The model works based on reward feeding after it chooses an action and then informs the agent of the reward and the updated state [51]. Its major aim is to find the ideal reward function. More specifically, paraphrasing is built based on the sequence-to-sequence model and adds the optimal reward function, which learns from a given pair of paraphrased texts [6]. The deep generative model is built based on the sequence-to-sequence model, and the model evolves into additional layers. The intermediate layers differ from one research to another. For illustration, Iyyer et al. (2018) [52] inserted a parser generator within the encoder–decoder. Mahmoud and Zrigui [53] preferred a hybrid neural network comprising the Convolutional Neural Network (CNN) with an attention model for the Arabic text. GPT is one of the trendiest generative models in AI.

GPT was introduced in 2018 by OpenAI, which shows a significant shift in language model development. GPT is built based on transformer architecture using the attention technique to understand and generate human-like text. GPT comes in many versions, such as GPT-1 [54,55], GPT-2 [55], GPT-3 [56], and GPT-4 [57]. Its main idea is to utilize a vast and diverse textual corpus to train the model using an unsupervised learning approach (unlabeled data), followed by fine-tuning downstream tasks using supervised learning (labeled data) [54]. A clear and specific prompt must be provided to guide the GPT behavior and ensure its generated outcomes meet the wanted requirements. The prompt means the instructions are provided to the GPT, and the prompt significantly enriches the model’s capabilities. Besides, each task requires a different prompt based on its requirements and context [58]. The next part mentions various works using multiple AI approaches for text generation. In addition, it spotlights different uses of GPT in numerous applications and cites some examples of GPT prompts. Furthermore, the next part covers some common datasets used in building AI models for text generation aims.

2.2. AI-Based Methods for Paraphrasing

A range of studies have employed diverse AI methodologies for text generation. In the context of a rule-based approach, McKeown (1983) [11] developed a rule-based paraphrase system for English text, Salloum and Habash (2011) [49] worked on improving the quality of Arabic–English statistical machine translation of dialectal Arabic text. Salloum and Habash [49] depend on morphological knowledge to define a lightweight rule-based approach to producing MSA paraphrasing of dialectal Arabic words. Regarding the seq2seq approach, Nagoudi et al. (2022) [17] introduced pre-trained transformer-based seq2seq models. They also investigate a variety of tasks, such as news title generation, question answering, machine translation from Arabic to a foreign language, and paraphrase generation. Their model is called AraT5, which is highly significant in Arabic NLP due to its effectiveness in tasks such as MT. Ormazabal et al. (2022) [59] focused on paraphrasing generation tasks using round-trip machine translation. More specifically, in their system, the encoder eliminates the unrelated information from a given translation sentence, and then the decoder uses the result to reconstruct the paraphrased sentence. Gudkov et al. (2020) [60] produced the first Russian paraphrase corpus and developed a paraphrasing model. Fu et al. (2019) [61] evolved a paraphrase generation system for the latent bag-of-words in English text.

In the context of the reinforcement approach, Sancheti et al. (2022) [62] developed a reinforcement learning-based paraphrase generation task using English text. Li et al. (2017) [6] enhanced paraphrase generation by evolving a novel deep reinforcement learning using an English textual dataset. In the context of the deep generative approach, Mahmoud and Zrigui (2021) [53] worked on detecting paraphrasing in Arabic, and their model is based on a CNN with attention model. Li et al. (2021) [27] employed a deep learning model for paraphrase detection from an English text. Iyyer et al. (2018) [52] evolved a model for English that introduces a novel approach to generating a paraphrase from a given sentence by defining a particular syntactic form.

Table 4. A comparison summary of the AI approaches used in paraphrasing.

Paper	AI Approach	Language	Aim
McKeown [11]	Rule-based	English	He develops a rule-based paraphrase system by using pre-defined constraints.
Salloum and Habash [50]	Rule-based	Dialectal Arabic text	They aim to improve the quality of Arabic–English statistical machine translation of the dialectal Arabic text.
Nagoudi et al. [17]	Seq2seq	Arabic	They generate a novel Arabic benchmark for Arabic paraphrase generation.
Ormazabal et al. [60]	Seq2seq	English and France	They focus on paraphrasing generation tasks by using round-trip machine translation.
Gudkov et al. [60]	Seq2seq	Russian	They produce the first Russian corpus for paraphrase generation
Fu et al. [61]	Seq2seq	English	They evolve a paraphrase generation system for the latent bag-of-words in English text.
Sancheti et al. [62]	Reinforcement learning	English	They aim to initiate a new task of entailment relation-aware paraphrase generation.
Li et al. [6]	Reinforcement learning	English	They enhance paraphrasing generation by evolving a novel deep-reinforcement
Mahmoud and Zrigui [53]	Deep generative	Arabic	They work on detecting paraphrases
Li et al. [27]	Deep generative	English	They work on paraphrase detection
Iyyer et al. [52]	Deep generative	English	They develop a model for paraphrase generation

outlines a comparison of the AI approaches used in paraphrasing. All GPT versions can produce paraphrasing but with various capabilities, and GPT can be employed in different applications, which will be explained in the following section.

GPT has been applied across many fields for different purposes. For example, Ding et al. (2023) [63] found that GPT-3 could be used as a general-purpose data annotator in NLP tasks. Surameey and Shakor (2023) [64] used GPT-3.5 as a debugging tool, and they concluded their experiment by asking to add GPT as part of a comprehensive debugging tool. Goyal et al. (2022) [65] used GPT-3 as a summarization tool, and Gutierrez et al. (2022) [66] investigated GPT-3’s ability in Information Extraction (IE) and Relation Extraction (RE) for bioinformatics datasets. Yang et al. (2022) [15] examined GPT-3’s ability to answer visual questions by converting each question into an equivalent caption, facilitating the model’s comprehension. Each piece of research defines a different prompt based on research objectives. For illustration, Ding et al. (2022) [63] defined three different prompts: (1) they provide GPT with a sentence and ask GPT to give a sentiment label (positive or negative), (2) they ask GPT to create a new sentence based on a required sentiment label, and (3) they give GPT a title and the required sentiment label then ask GPT to produce a new sentence in that title with that sentiment label. Furthermore, Goyal et al. (2022) [65] posted an entire article to the GPT and asked it to summarize the article into one, two, or three sentences. However, the GPT could produce a new dataset or expand an existing one. ML could generally be used to create new data items, which serve in further modeling stages.

For instance, Rafai et al. (2023) [10] used AraGPT-2 to expand the dataset and release a new version of the corpus; the produced corpus is then employed in sentiment text classification. Their experiments show the effectiveness of using AraGPT-2, which improves modeling performance by 7% to 13% for different datasets. Moreover, for the paraphrasing task by Nagoudi et al. (2021) [17] in their common AraT5 model, the MT model produces a new paraphrased sentence for a given text in English. This releases a new corpus of paraphrased pairs where the original text is the Arabic text that is human-generated, and its equivalent paraphrased text is MT-generated. The produced corpus is used to build a model for text paraphrasing, but they found that using MT to make a new paraphrased pair can capture the meaning slightly. Other researchers follow several methods and use multiple sources to collect paraphrasing pairs. The following part highlights some of the commonly used corpora.

Paraphrasing research suffers from the lack of parallel corpora [67]. Due to the complex morphology of the Arabic language, finding high-quality and well-structured data for paraphrasing is challenging [68]. Therefore, researchers utilize an open dataset in developing paraphrasing models. For instance, Arabic Language Generation (ARGEN) is an Arabic paraphrasing dataset containing 123.6 K pairs provided by adopting semantic similarity datasets and MT datasets [17]. Mahmoud and Zrigui (2021) [53] used Open-Source Arabic Corpora (OSAC) and the King Saud University Corpus of Classical Arabic (KSUCCA) to produce paraphrased pairs by substituting synonyms, which releases 1 K sentence pairs. BAR and Dershowitz (2021) [69] created a corpus of similar documents by extracting text from published news from several agencies. The extraction process is based on picking one published document and then finding one that achieves the highest similarity score, which launches 100 correctly paired documents. There are various datasets for other languages, such as English and Spanish.

Quora Question Pair (QQP) is an English dataset created from Quora questions, where a pair is annotated as a paraphrased question pair or not. The researchers depend on identifying the duplicated questions to be labeled as a paraphrased pair [70], and this delivers 150 K paraphrase pairs [7]. Twitter Uniform Resource Locator (TwitterURL) is a sentential paraphrase dataset in English extracted from Twitter. It groups tweets as paraphrased sentence pairs when tweets refer to the same URL to produce a corpus with 51 K sentence pairs [68]. Para-Phrase DataBase (PPDB) is a paraphrase dataset that includes phrasal and lexical paraphrases but does not contain sentence paraphrases [71]. It comprises 220 M paraphrase pairs, which are 8 M lexical paraphrases and 73 M phrasal paraphrases [7]. This dataset covers the English language but has an extended version for the Spanish language. The Spanish version contains 196 M paraphrase pairs, which are 33 M lexical paraphrases [71].

Table 5. Summarization of dataset details.

Dataset	Language	Size
ARGEN [17]	- Arabic	123.6 K paraphrase pairs
Merged OSAC and KSUCCA [54]	- Arabic	1 K sentence pairs
Arabic corpus BD [70]	- Arabic	100 documents correctly pairing
QQP [71]	- English	150 K paraphrase pairs
TwitterURL [68]	- English	51 K sentence pairs
PPDB [72]	- English - Spanish	220 M phrasal and lexical paraphrases

summarizes dataset details; it is noticeable that there are limited corpora with tiny sizes. In summary, AI approaches are used to produce paraphrasing pairs, but paraphrasing suffers from a lack of paraphrasing corpora. The following section argues for the methodology used to investigate GPT capabilities in producing Arabic paraphrasing sentences and evaluate paraphrase quality using many metrics to release a high-quality paraphrasing corpus.

3. Methodology

This study investigates the capabilities of GPT in paraphrasing, with a focus on Arabic language text. A structured methodology was designed to achieve the research aim, comprising the following essential phases: (1) data selection, (2) data processing, (3) modeling, and (4) evaluation. Each phase plays a critical role in ensuring the reliability and validity of the generated paraphrasing. The following subsections illustrate each phase intensively, and Figure 1 shows an overview of the research methodology.

3.1. Data Selection

This phase requires an exploration of the open Arabic dataset and then the identification of a suitable dataset with the research aim. This phase includes (1) an exploration of open Arabic datasets, (2) an exploration of dataset format and statistics, (3) an investigation of text quality, and (4) a selection of the data. Intensive exploration of the dataset found several challenges in open datasets. For instance, (1) the text file is a mixture of HTML tags and Arabic sentences, (2) the dataset is formed of words rather than complete sentences or articles, (3) the dataset contains informal language such as dialectical phrases found in tweets, and (4) the corpus comprises very short sentences, some of which do not exceed three words. This research utilizes the Single-Labelled Arabic News Articles Dataset (SANAD). SANAD is an MSA textual data for classification purposes created by Einea et al. (2019) [72], and it is freely available. It contains 200 K articles from three newspapers (website-based): AlKhaleej, AlArabiya, and Akhbarona. Additionally, articles of the entire corpus are classified into seven categories: Culture, Finance, Medical, Politics, Religion, Sports, and Technology. This research selects articles from AlKhaleej only because its contents are manually explored and comprise an equal number of articles, which is 6500. Furthermore, this research scope includes culture, technology, and sports articles due to the limitation of research time. The dataset is cleaned, and there are no HTML tags, URLs, etc.; each article contains plain text without a title. Therefore, the dataset is ready for the next phase, which is the data processing phase.

3.2. Data Processing

The data processing phase covers (1) reading the article, (2) tokenization into sentences, and (3) saving each sentence separately. White space removal is needed before sentence tokenization. Tokenization means splitting a text into tokens, such as words, phrases, sentences, numbers, or punctuation [2,73]. Punctuation marks are used to identify the edges of sentences, which could be terminated by a full stop (.), question mark (؟), or exclamation mark (!), depending on the context of the sentence [74]. SANAD [72,75] data are collected in the document format. Thus, the data are tokenized into sentences by determining the edge of each sentence. For this task, we first depended on using NLTK [75] library sentence tokenization for the culture and technology categories. However, NLTK could not cover all possible cases in the dataset, such as the number of goals between circle brackets for the sports category. Later, tokenization uses regular expressions to catch up with Arabic sentence endings and consider the category’s nature. Additionally, the text requires cleaning steps, which include removing extra white spaces between the article’s sentences and spaces before and after punctuation. The phase produces a pool of sentences; each sentence is saved separately in a text file format. Every sentence is used individually in the modeling phase.

3.3. Modeling

Modeling is the process of using GPT-4 to produce a new paraphrased sentence. The modeling phase involves (1) reading a sentence, (2) asking GPT-4 to paraphrase it, and (3) saving the sentence and its paraphrasing as a pair. Prompt engineering plays a vital role in GPT results because the different prompts lead to differences in the performance of the Large Language Model (LLM) [76]. Additionally, the prompt must identify the needed outcome clearly and precisely. For instance, Goyal et al. (2022) [65] use three prompts of summarization for a given article by asking GPT to summarize a given article into a summary of one, two, and three sentences. This research conducts many primary experiments to identify a suitable prompt for this research. The experiments aim to answer the following questions: (1) Which language, Arabic or English, is most appropriate for writing a prompt? (2) What Arabic prompt is suitable for producing a new paraphrased sentence? (3) Can GPT paraphrase an entire article in a single API call? (4) If a sentence is given to the GPT and requests to paraphrase, does the GPT produce the same paraphrased sentence for 10 API calls? (5) Can GPT understand the different types of lexical paraphrasing methods, such as using different word forms or synonyms, or changing sentence tense?

The experiments found the following. (1) GPT could interpret paraphrasing commands and responses even if we use English or Arabic prompts to rephrase Arabic sentences. (2) GPT can understand many Arabic prompts to paraphrase a sentence, such as “أعد الصياغة” (rephrase it) and “اكتبها بطريقة أخرى” (rewrite it uses another way). However, GPT misunderstands “أعد الكتابة” (rewrite it), and it repeats the sentence without rephrasing it. (3) GPT can paraphrase an entire article once its length is within its range of capabilities, which means GPT cannot paraphrase a long article using an API call (meaning without splitting it into parts). (4) Calling GPT to paraphrase a sentence rapidly leads to creating a new paraphrased sentence for each call using different synonyms and sentence structures. However, GPT lost its ability to paraphrase and copy the original sentence as a paraphrased sentence after many calls. Thus, calling GPT several times to paraphrase a sentence does not lead to generating the best-paraphrased sentence. Sometimes, GPT produces more than one paraphrased sentence for one GPT API call, which releases a single original sentence and multiple paraphrased sentences, and this means there is no unification in results. Accordingly, an attached prompt is essential to force GPT to produce only one paraphrased sentence for each call.

After the primary experiment to explore the GPT prompt, this research selects a single prompt, which is, in Arabic “أعد صياغة التالي في جملة واحدة”, and its translation in English “rephrase the following in one sentence”. After exploring the Google Search engine and Google Schooler engine for paraphrasing in Arabic, it is noticeable that “أعد الصياغة” (rephrase it) and “أعد الكتابة” (rewrite it) are the most commonly used prompts, but “أعد الصياغة” (rephrase it) is the most used in academic research; thus, “أعد الصياغة” (rephrase it) is chosen as GPT prompt. Moreover, specifying the number of sentences needed, “جملة واحدة” (one sentence) is appended to the prompt.

Table 6. An example of prompt engineering.

Language	Prompt	Example
Arabic	أعد صياغة التالي في جملة واحدة	أعد صياغة التالي في جملة واحدة: “علم البيانات هو مجموعة من التقنيات التي تستخدم لاستخراج قيمة من البيانات.”
English	Rephrase the following in one sentence.	Rephrase the following in one sentence: “Data science is a collection of techniques used to extract value from data” [9].

demonstrates an example of a prompt in Arabic and the equivalent translation in English. The modeling stage is done by calling GPT API for each sentence separately, which means a single API call for each sentence is run in isolation from other related sentences (sentences from the same article). This research uses the gpt-4o-2024-05-13 version to have fixed results for all API calls. In addition, this research uses a Google Colab environment [77] with RAM [1.21–12.67] GB and T4 GPU for Python programming language (version 3.11.11). This research requires splitting the corpus into smaller parts and processing 200–500 articles daily using a Python script due to GPT API limitations. This phase releases an Arabic paraphrasing corpus, where the original sentence is human-generated text, and its paraphrased sentence is model-generated text. Exhaustive evaluation steps are taken to ensure the quality of the paraphrased pair.

3.4. Evaluation

This research depends on regular NLP metrics, LD, semantic similarity, and manual evaluation by Arabic language experts to evaluate the quality of paraphrased sentences and examine GPT-4 performance. More specifically, the evaluation phase comprises (1) calculating BELU and ROUGE, (2) finding similarity, (3) comparing sentence scores with thresholds, and (4) setting level. Furthermore, investigation by this research leads to defining the quality ranking framework for paraphrasing. For primary judgments, Arabic experts employ manual evaluation in this research to assess the quality of paraphrasing. A random corpus subset is chosen, and the research selects 120 pairs from every category, meaning 420 paraphrased pairs. The evaluators are chosen based on their academic qualifications, with a minimum requirement of a master’s degree or higher in Arabic linguistics.

Furthermore, each evaluator must have had work experience in Arabic for at least two years. The research implements many steps to ensure reliability and minimize potential bias or tampering. These steps are (1) multiple evaluations, (2) independent evaluation, (3) clear evaluation criteria, and (4) anonymous review. Three Arabic language experts were chosen for evaluation, and each expert evaluated the sample separately without communicating with other experts or seeing other evocators’ results. The evaluation criteria are defined clearly and in isolation from each other. In addition, an orientation session is done for each evaluator individually to ensure their understanding of linguistic evaluation criteria. An anonymous review is essential to reduce personal bias in evaluating AI-generated sentences. Therefore, the source of the original and paraphrased sentences is not provided, and the paraphrased sentence is not explicitly defined as a paraphrased sentence of the original sentence. Instead, the evaluation survey defines the sentences as sentence A and sentence B, then asks the expert to identify their semantic meaning and determine if they are equivalent. As a result, the research maintains objectivity and eliminates subjective opinions in human evaluation tasks.

The human evaluation combines multiple paraphrasing criteria from a linguistic perspective. Evaluation criteria for paraphrasing pairs include semantic similarity, fluency, syntax, structure, synonym substitution, and changing word forms. In addition, the evaluators are asked to rank the quality of the paraphrasing based on the overall criteria. For all criteria, the evaluator has to choose from four options: (1) excellent or applicable, (2) good, (3) poor, and (4) very poor or not applicable. Human evaluation for 420 pairs indicates the overall performance of GPT, but using other NLP metrics is an essential stage in evaluating the entire corpus. Appendix A shows some samples of paraphrased sentences in the human evaluation survey.

The evaluation of the whole corpus covers several steps: (1) finding the average score for all regular metrics and LD, (2) measuring similarity, and (3) defining the evaluation framework. The research utilizes the most prevalent NLP, BLEU, and ROUGE for word unigram. Additionally, LD is measured in two ways: calculating the average LD score of the original text and paraphrased text and the difference between their LD scores. The following explains semantic similarity and the paraphrase quality ranking framework.

3.5. Semantic Similarity

Regarding similarity, the semantic meaning of pairs is studied using word embedding with cosine similarity and Euclidean distance. Additionally, this research calculates the similarity score using a common metric, the Jaccard similarity. AraBERT is chosen for word embedding due to its efficiency in understanding context. More specifically, the “AraBERTv0.2-base” version is chosen for this research because it employs a special pre-segmentation approach that can interpret the complexity of Arabic morphology. This research adopts semantic similarity with word embedding to explore the following: (1) measuring semantics for each of the original and paraphrased sentences, and (2) studying similarity scores for paraphrased and non-paraphrased pairs. This research computes the similarity scores among sentences to discover the impact of paraphrase pairing on similarity scores. It considers three cases: (1) each sentence in the category with its equivalent paraphrased pair, (2) two sentences from the same category but not a paraphrased pair, and (3) two sentences from different categories that are not paraphrased pair. The three cases provide a full examination view for paraphrased pairs and non-pairs regarding semantic similarity. Therefore, 40 sentences are selected randomly with their equivalent paraphrased pairs, and then finding different combinations leads to 360 sentence pairs for all three cases.

3.6. Quality Ranking Framework for Paraphrasing

This research develops a quality ranking framework to evaluate paraphrasing pairs by finding the rounded average score for each metric. More specifically, these scores serve as thresholds to rank the quality of paraphrasing corresponding to the achievement of thresholds. The threshold value for each metric is presented in the Section 4. The quality ranking contains four levels: level-0, level-1, level-2, and level-3. Level-3 is for pairs with eight, seven, or six thresholds. Level-2 is assigned to pairs that have achieved five, four, or three of the thresholds. Level-1 is for pairs that have achieved two thresholds or one threshold. Lastly, level-0 is given when no thresholds are achieved. The quality ranking framework is shown in Algorithm 1. This framework is used to study the effect of the categories on text paraphrasing quality.

Algorithm 1: A quality ranking algorithm for each paraphrasing pair.
1:	//Calculate the score of paraphrasing quality
2:	//Input: two sentences that are original text and paraphrased text
3:	//Output: the final score of paraphrasing quality level
4:	Start,
5:	Ori_tokens ← word tokenization of original sentence
6:	Par_tokens ← word tokenization of paraphrased sentence
7:	Ori_vector ← embedding using AraBERT of original sentence
8:	Par_vector ← embedding using AraBERT of paraphrased sentence
9:	BLEU_1 ← calculate BLEU for Ori_tokens and Par_tokens
10:	ROUGE_1_recall ← calculate ROUGE-Recall for Ori_tokens and Par_tokens
11:	ROUGE_1_ precision ← calculate ROUGE-precision for Ori_tokens and Par_tokens
12:	ROUGE_1_ F1 ← calculate ROUGE-F1 for Ori_tokens and Par_tokens
13:	LD_avg ← calculate the average score of LD for Ori_tokens and Par_tokens
14:	Jaccard_sim ← calculate Jaccard similarity for Ori_tokens and Par_tokens
15:	Cosine_sim ← calculate cosine similarity between Ori_vector and Par_vector
16:	Euclidean_sim ← calculate Euclidean similarity between Ori_vector and Par_vector
17:	BELU_threshold  ← values range [0.4, 1) \\ 1 is excluded
18:	ROUGE_recall_threshold  ← values range [0.6, 1) \\ 1 is excluded
19:	ROUGE_precision_threshold  ← values range [0.5, 1) \\ 1 is excluded
20:	ROUGE_F1_threshold  ← values range [0.5, 1) \\ 1 is excluded
21:	Jaccard_threshold  ← values range [0.5, 1) \\ 1 is excluded
22:	Cosine_threshold  ← values range [0.8, 1) \\ 1 is excluded
23:	Euclidean_threshold  ← values range (0, 3.5] \\ 0 is excluded
24:	Checking threshold for all metrics:
25:	If eight, seven, or six of them are achieved ← set quality level for the pair is Level-3
26:	If five, four, or three of them are achieved ← set quality level for the pair is Level-2
27:	If one or two of them are achieved ← set quality level for the pair is Level-1
28:	Else ←  set quality level for the pair is Level-0
29:	Return quality level
30:	Stop.

4. Results and Discussion

This section shows the results of the modeling and evaluation stages. The modeling stage produces a paraphrased pair: the sentence from the article is the original sentence (Ori_sentence) (human-generated text), and the other is the newly generated sentence that is generated by GPT (Par_sentence). This stage produces 48,486 pairs for the culture category while releasing 74,323 for the technology category. Additionally, the sports category comes with 67,360 paraphrased pairs. Therefore, the total number of pairs is 190 K sentences, and

Table 7. Corpus statistics for the three categories.

Category	Original Sentence	Paraphrase Sentence
Culture	48,486	48,486
Technology	74,323	74,323
Sport	67,360	67,360
Total	190,169	190,169

displays corpus statistics. The next part illustrates evaluation methods.

4.1. Human Evaluation

All criteria for human evaluation results are presented in

Table 8. Criterion of semantic similarity.

	Evaluator #1				Evaluator #2				Evaluator #3
	0 *	1	2	3	0	1	2	3	0	1	2	3
Culture	0.71	1.43	2.14	95.71	1.43	7.86	5.00	85.71	2.14	1.43	10.71	85.71
Sport	0.00	0.00	1.43	98.57	0.00	6.43	5.00	88.57	2.86	2.86	7.14	87.14
Technology	4.29	5.00	2.86	87.86	8.57	5.00	10.00	76.43	6.43	5.71	3.57	84.29

* Quality levels, where 0 represents the lowest quality, and 3 represents the highest.

,

Table 9. Criterion of sentence fluency.

	Evaluator #1				Evaluator #2				Evaluator #3
	0 *	1	2	3	0	1	2	3	0	1	2	3
Culture	0.71	0.00	12.14	87.14	0.00	0.00	1.43	98.57	2.14	4.29	23.57	70.00
Sport	0.71	2.86	12.14	84.29	0.00	0.71	2.14	97.14	1.43	2.86	30.71	65.00
Technology	1.43	6.43	9.29	82.86	2.86	1.43	7.86	87.86	6.43	6.43	22.14	65.00

* Quality levels, where 0 represents the lowest quality, and 3 represents the highest.

,

Table 10. Criterion of following Arabic grammar and rules.

	Evaluator #1				Evaluator #2				Evaluator #3
	0 *	1	2	3	0	1	2	3	0	1	2	3
Culture	0.71	0.00	7.86	91.43	0.00	0.00	2.14	97.86	2.86	2.86	7.86	86.43
Sport	0.00	1.43	5.71	92.86	0.71	0.71	1.43	97.14	2.14	1.43	15.00	81.43
Technology	2.14	2.14	10.00	85.71	2.86	1.43	8.57	87.14	11.43	3.57	10.00	75.00

* Quality levels, where 0 represents the lowest quality, and 3 represents the highest.

,

Table 11. Criterion of sentence restructure.

	Evaluator #1				Evaluator #2				Evaluator #3
	0 *	1	2	3	0	1	2	3	0	1	2	3
Culture	67.14	16.43	12.86	3.57	0.00	0.71	1.43	97.86	57.86	10.00	21.43	10.71
Sport	69.29	14.29	12.14	4.29	1.43	0.71	2.14	95.71	50.00	8.57	20.71	20.71
Technology	71.43	16.43	9.29	2.86	0.71	2.86	5.71	90.71	43.57	12.86	18.57	25.00

* Quality levels, where 0 represents the lowest quality, and 3 represents the highest.

,

Table 12. Criterion of changing word form.

	Evaluator #1				Evaluator #2				Evaluator #3
	0 *	1	2	3	0	1	2	3	0	1	2	3
Culture	53.57	21.43	19.29	5.71	0.71	6.43	6.43	86.43	75.71	2.86	15.71	5.71
Sport	61.43	22.86	12.86	2.86	0.71	4.29	5.00	90.00	86.43	2.14	7.86	3.57
Technology	67.14	18.57	12.86	1.43	2.14	7.86	15.00	75.00	87.14	0.71	7.14	5.00

* Quality levels, where 0 represents the lowest quality, and 3 represents the highest.

,

Table 13. Criterion of synonym substitution.

	Evaluator #1				Evaluator #2				Evaluator #3
	0 *	1	2	3	0	1	2	3	0	1	2	3
Culture	27.86	25.71	28.57	17.86	0.71	5.71	7.14	86.43	46.43	0.00	13.57	40.00
Sport	34.29	23.57	22.86	19.29	1.43	4.29	5.71	88.57	46.43	4.29	12.14	37.14
Technology	49.29	17.86	22.86	10.00	1.43	7.86	15.71	75.00	64.29	1.43	7.86	26.43

* Quality levels, where 0 represents the lowest quality, and 3 represents the highest.

and

Table 14. Overall quality rating.

	Evaluator #1				Evaluator #2				Evaluator #3
	0 *	1	2	3	0	1	2	3	0	1	2	3
Culture	0.71	1.43	12.86	85.00	1.43	2.86	6.43	89.29	4.29	4.29	26.43	65.00
Sport	0.71	1.43	17.14	80.71	2.14	2.86	2.86	92.14	2.86	4.29	33.57	59.29
Technology	0.71	7.14	11.43	80.71	1.43	3.57	12.86	82.14	7.86	7.14	22.14	62.86

* Quality levels, where 0 represents the lowest quality, and 3 represents the highest.

. Each table gives a descriptive analysis of evaluators’ responses for every evaluation criterion individually. In addition, the columns labeled 0, 1, 2, and 3 correspond to distinct quality levels, where 0 represents the lowest quality and 3 represents the highest. Specifically, each cell within these columns shows the average number of pairs assessed and classified under the respective quality level during the evaluation process. For illustration, Table 8 shows that Evaluator #1 assessed 95.71% of the sentence pairs as high quality, whereas Evaluator #2 and Evaluator #3 assessed 85.71% of the pairs as high quality within the culture category.

Regarding semantic similarity, all three evaluators highly rated paraphrased sentences for the different categories. We conclude that GBT 4 produces paraphrased sentences that are semantically equivalent, as presented in Table 9. Regarding fluency, most sentences are rated as excellent or good for all categories based on Table 10; thus, GBT 4 can generate sentences that look fluent and natural. The evaluators evaluate sentences as grammatically correct, as shown in Table 11. Evaluators find that GPT-4 follows Arabic grammar and rules because the sentence components are correctly formed. In Table 12, the first and third evaluators find the paraphrased sentences are not much restructured compared to the original text’s structure. More specifically, GPT does not change sentence structure much and preserves its original structure. In contrast, the second evaluator believes there are many changes in sentence structure and rates most of them as excellent. Based on the first and third evaluators’ opinions in Table 13, most sentences do not use different forms of words and paraphrasing. Thus, GPT does not change word forms to create paraphrased sentences. Still, GPT sometimes uses synonym substitution, as illustrated in Table 14, which reveals synonyms appearing in half of the pairs and rated as excellent or good. The overall quality of paraphrased pairs is ranked as excellent for all categories, as provided in

Table 15. Results of regular NLP metrics.

Category	BLEU-1	ROUGE-1-Precision	ROUGE-1-Recall	ROUGE-1-F1	AVG LD	Diff LD
Culture	0.49	0.53	0.63	0.57	0.94	0.05
Sport	0.55 *	0.58	0.62	0.59	0.96	0.03
Technology	0.55	0.58	0.68	0.62	0.95	0.05
Average	0.53	0.56	0.64	0.59	0.95	0.04

* Bold means the highest value in the column.

.

4.2. Regular NLP Metrics

The newly generated paraphrased text is evaluated for the entire corpus using BLEU and ROUGE.

Table 16. Summary of previous work results.

Paper	BLEU_1	ROUGE-1-Recall
Li et al. (2017), Quora-1 dataset [6]	[36–43]	[58–64]
Li et al. (2017), Quora-2 dataset [6]	[26–34]	[47–57]
Li et al. (2017), Twitter dataset [6]	[30–45]	[30–42]
Engomwan and Chali (2019) [12]	[35–40]
Fu et al. (2019), Quora dataset [61]	[55–72]	[58–72]
Fu et al. (2019), MSCOCO dataset [61]	[72–80]	[42–49]
Nagoudi et al. (2022), paraphrasing dataset [17]	[17–19]
This research	[50–60]	[50–70]

shows the BLEU, ROUGE, and LD scores. The average BLEU score is 0.5, ROUGE-1 precision is 0.7, ROUGE-1 recall is 0.6, and ROUGE-1 F1 is 0.6. The overall performance of the metric shows a high degree of overlap between the Ori_sentence and Par_sentence, with scores of 0.5 or higher. That performance means the quality of paraphrased sentences is satisfactory. In other words, for the three categories, BLEU and ROUGE scores achieved a moderate level of word unigram overlap between the Ori_sentence and Par_sentence, which is considered a sensible degree of similarity due to the use of synonyms. On the other hand, the average LD score is almost 0.8 for all three categories, which means that both Ori_sentence and Par_sentence are rich in vocabulary and phrases. The difference between their LDs is extremely low, which means one sentence has a slightly more diverse vocabulary than the other. After intensively investigating Ori_sentence and Par_sentence content, sentence length is the main cause of the difference in LD because Par_sentence length is often shorter than Ori_sentence. The metric results are explored by comparing research result ranges with those reported in previous studies for text paraphrasing in Arabic and other languages. Results of BLEU range from 0.5 to 0.6, which is similar to the Fu et al. (2019) paper [61], and higher than Li et al. (2017) [6] and Engomwan and Chali (2019) [12]. However, Nagoudi et al. (2021) [17] show results lower than other research due to the dependence on MT to produce the pair. ROUGE-1-recall scores for this research are close to the results of Li et al. (2017) [6] and Fu et al. (2019) [61], both of whom use the Quora dataset. Table 3 shows metric results, and Table 17 presents the investigation results.

4.3. Corpus Semantic Similarity

Examination of semantic similarities between each pair is summarized in Table 17. The average cosine similarity score is 0.89, which means that Ori_sentence and Par_sentence are similar in meaning and context. In other words, the vector of Ori_sentence and the vector of Par_sentence point in almost the same direction. The Euclidean distance is 3.16, which shows an overlap between the pair, but it is not identical due to the use of a variety of vocabulary and phrases. The Jaccard score is 0.35, which presents a small ratio of overlap, which is considered a good quality of paraphrasing because GPT-4 uses different terms to express the same meaning.

Table 17. Similarity scores for the three categories.

Category	Cosine Similarity	Euclidean Distance	Jaccard Similarity
Culture	0.89	3.21	0.33
Sport	0.89	3.1	0.35
Technology	0.89	3.16	0.36
Average	0.89	3.16	0.35

Bold means the best value for each similarity metric.

Table 17. Similarity scores for the three categories.

Category	Cosine Similarity	Euclidean Distance	Jaccard Similarity
Culture	0.89	3.21	0.33
Sport	0.89	3.1	0.35
Technology	0.89	3.16	0.36
Average	0.89	3.16	0.35

Bold means the best value for each similarity metric.

4.4. Semantic Similarity for Paraphrase Pairing and Non-Pairing

Three cases are considered to explore the similarities between paraphrased pairs and non-pairs and investigate their impact on semantic measurement scores, as demonstrated in

Table 18. Semantic similarity for the three cases.

Case	Category	Cosine Similarity	Euclidean Distance	Jaccard Similarity
Its pair	Culture	0.92	2.61	0.39
	Sport	0.92	2.68	0.38
	Technology	0.94	2.33	0.45
	Average	0.93	2.54	0.40
Same category and not pair	Culture	0.76	4.72	0.04
	Sport	0.73	5.01	0.04
	Technology	0.72	5.19	0.05
	Average	0.74	4.97	0.04
Different categories and not pair	Culture	0.68	5.51	0.03
	Sport	0.66	5.66	0.02
	Technology	0.61	6.10	0.04
	Average	0.65	5.76	0.03

Bold means the best value for each similarity metric.

. The cosine similarity can capture the paraphrased pair by giving high and moderate scores for sentences from the same categories but low scores for sentences from different categories. The Euclidean distance is 2.5 for the paraphrased pair, and the distance increases for sentences from the same category, while it reaches 5.6 for sentences from different categories. Jaccard similarity shows very low scores for unpaired sentences either from the same category or other categories, which are 0.04 and 0.03, respectively. This experiment concludes that the similarity metrics scores can capture each paraphrased pair’s semantic similarity.

4.5. Results of Quality Ranking Framework for Paraphrasing

This research produces a new corpus for Arabic paraphrasing, which contains 190 K pairs. After employing the quality ranking algorithm, the entire corpus is ranked into four levels based on their quality. The corpus statistics are shown in

Table 19. Corpus statistics based on quality-level ranking.

Category	Level-0	Level-1	Level-2	Level-3	Total
Culture	392	7011	15,564	25,519	48,486
Technology	324	5508	17,407	51,084	74,323
Sport	66	7831	15,656	43,807	67,360
Total	782	20,350	48,627	120,410	190,169

below. Most corpus pairs are labeled Level-3, representing 52% of culture, 68% of technology, and 65% of sport. The content of each category and level is explored to identify its effects on the paraphrasing quality. The low level for culture is due to the inclusion of some poetry texts, in which each line of poetry is tokenized individually, and GPT produces a very short sentence that rewrites a line in a prose sentence. Additionally, the NLTK sentence tokenizer could not handle some cases in which an acronym contains many single letters, each followed by a dot, for instance, “م.ق” (M.K). Thus, the culture category includes some sentences that contain only one letter, which explains its low scores. The technology text is a mixture of English and Arabic words with some computing terminology. GPT attempts to rephrase a given sentence with limited structures or phrases in computing topics because the computing words appear in both Ori_sentence and Par_sentence. Subsequently, the similarity scores are increased for Cosine and Jaccard similarity, and Euclidean distance decreases; the technology category dominated the highest scores. However, including English and abbreviations in technology leads to inaccurate tokenization and modeling issues. The content of the text is manually filtered to remove unexpected tokens, and

Table 20. Corpus statistics after removing unexpected tokens in the technology category.

Category	Level-0	Level-1	Level-2	Level-3	Total
Culture	392	7011	15,564	25,519	48,486
Technology	266	5508	17,407	51,084	74,265
Sport	66	7831	15,656	43,807	67,360
Total	724	20,350	48,627	120,410	190,111

shows its statistics. The sports category achieves fewer pairs in Level-0 due to an accurate tokenization method specifically defined for sports. GPT can analyze a given sentence in sports, generating a new sentence summarizing the match result. However, GPT misunderstands some sports texts that include the players’ numbers by treating player numbers as the number of goals in a match. Some examples of illustrations are presented in

Table 21. Illustration examples for text contents.

Category	Ori_sentence	Par_sentence	Explanation
Culture	مُغلقةٌ حاويات النواياوراء ضبابٍ كتيمٍ تقيمُبقعر الظنونْ.	تقبعُ حاويات النوايا مُغلقةً في قعر الظنون، وراء ضبابٍ كتيم.	This case contains poetry text; GPT rewrites it as a sentence.
Sport	وأشادت بعض الصحف المكسيكية بمنتخب بلادها، حيث تصدرت عناوين الانتصار والإشادة، صفحات بعض الجرائد هناك، مثل صحيفتي «ريفورما» و«يونيفيرسال»، فيما اكتفت صحيفة «إكسيلسيور» بتخصيص عمود صغير للتذكير بالنتيجة (3-1)	أشادت بعض الصحف المكسيكية بمنتخب بلادها بتصدُّر عناوين الانتصار والإشادة صفحتي «ريفورما» و«يونيفيرسال»، بينما اكتفت «إكسيلسيور» بتخصيص عمود صغير لتذكير القراء بالنتيجة (3-1)	The sport text shows match results in two circle packets. This shows the specific defined tokenization method for sport category works effectively in splitting sentence ending by a circle packet.
Sport	وبات فيتوريا غيمارايش يبتعد بفارق 3 نقاط امام سبورتينغ لشبونة وصيف البطل اثر خسارة الاخير امام يونياو ليريا بهدف لليدسون (86) مقابل اربعة اهداف لبول سيزار (15 و19) ونغال (83) وكادو (90)	اتسع الفارق بين فيتوريا غيمارايش وسبورتينغ لشبونة وصيف البطل إلى 3 نقاط بعد خسارة الأخير أمام يونياو ليريا بنتيجة 1-4، حيث سجل لليدسون (86) وليرييا بول سيزار (15 و19)، ونغال (83)، وكادو (90).	GPT can understand sport text that contains number of goals or player numbers. Additionally, in the Ori_sentence, the number of goals is written as text, and GPT analyzes the text and rewrites it as a number between a hyphen such as 4-1
Technology	يُشار إلى أن حُزمة تريند مايكرو Data Loss Prevention for Endpoint تفوقت في اختبارات مماثلة أجرتها الدورية العالمية المتخصصة Network World؛ كما منحتها دورية SC Magazine لقبَ أفضل الحلول في مجال الحول دون فقدان البيانات.	تفوقت حُزمة تريند مايكرو Data Loss Prevention for Endpoint في اختبارات الدورية العالمية Network World، وحصلت على لقب أفضل حلول منع فقدان البيانات من دورية SC Magazine	The pair is ranked as Level-3 due to the high overlap in English computing words such as ‘Data Loss Prevention for Endpoint’.

.

5. Conclusions and Future Work

Paraphrasing has a significant impact on other NLP applications, and GPT could be used to produce new paraphrased text. This paper explores GPT-4 performance in producing new paraphrased text at sentence level. Prompt engineering plays a prominent role in optimizing GPT performance. Thus, this research discovers multiple prompts for Arabic paraphrasing and identifies a suitable prompt that is both simple and understandable by GPT. The generated paraphrased sentences have been evaluated through numerous stages and for various purposes. This research investigates similarity scores for paraphrased and non-paraphrased pairs and finds that similarity metrics with AraBERT embedding can capture the semantics for paraphrased and non-paraphrased pairs. The regular NLP metrics find paraphrased pairs that contain a balanced scale of word unigram overlaps, with an average score of 0.5. In addition, the impact of text categories on paraphrasing has been studied, and the research finds that some of the technology and culture texts challenge GPT abilities due to the use of computing terminologies in English and inclusion of poetry text. This finding encourages a future investigation of other categories, such as finance and political articles. In addition, the developed quality framework effectively ranks paraphrasing into levels to produce a high-quality paraphrased corpus containing 190 K pairs. The new Arabic paraphrasing corpus could evolve a new paraphrasing model for Arabic or combine it with other languages’ corpora to generate a multilingual paraphrasing model. Furthermore, the corpus can be employed to study AI-generated text from linguistic perspectives, including word selection and sentence structure. This framework inspires the potential development of a new paraphrased quality algorithm that considers linguistic aspects such as synonym usage and sentence restructure between pairs.