A Comprehensive Analysis of Various Tokenizers for Arabic Large Language Models

Abstract

Pretrained language models have achieved great success in various natural language understanding (NLU) tasks due to their capacity to capture deep contextualized information in text using pretraining on large-scale corpora. Tokenization plays a significant role in the process of lexical analysis. Tokens become the input for other natural language processing (NLP) tasks, like semantic parsing and language modeling. However, there is a lack of research on the evaluation of the impact of tokenization on the Arabic language model. Therefore, this study aims to address this gap in the literature by evaluating the performance of various tokenizers on Arabic large language models (LLMs). In this paper, we analyze the differences between WordPiece, SentencePiece, and BBPE tokenizers by pretraining three BERT models using each tokenizer while measuring the performance of each model on seven different NLP tasks using 29 different datasets. Overall, the model pretrained with text tokenized using the SentencePiece tokenizer significantly outperforms the other two models that utilize WordPiece and BBPE tokenizers. The results of this paper will assist researchers in developing better models, making better decisions in selecting the best tokenizers, improving feature engineering, and making models more efficient, thus ultimately leading to advancements in various NLP applications.

1. Introduction

In natural language processing (NLP), tokenization is the fundamental preprocessing step for breaking down text into smaller, yet more manageable units called tokens that facilitate machines to appropriately process and generate human language. These tokens can be as small as characters or as long as words. These tokens serve as the building blocks for the sequences that are passed to language models during pretraining or fine-tuning stages. Therefore, the choice of tokenizer significantly impacts the performance of language models [1,2,3,4]. Recently, transformer-based architectures have emphasized the importance of tokenization to limit the vocabulary size, particularly for managing large datasets. Tokenizations are employed in many transformer-based architectures such as BERT [5], DistillBERT [6], GPT-2 [7], and RoBERTa [8].

There are many tokenization algorithms and techniques, including word tokenization, character tokenization, and subword tokenization. Word tokenization algorithms involve splitting the text into individual words, whereas character tokenization represents text as a sequence of individual characters. Subword tokenization is one of the most common tokenization algorithms due to its widespread use with large language models (LLMs). It provides a solution for issues arising from word and character-based tokenization algorithms. Subword tokenization breaks text into units smaller than whole words but that are larger than individual characters. Additionally, it was designed to work for most languages while keeping the vocabulary size manageable. This is because it captures commonalities between words, even in different languages, thus reducing the need for a massive list of unique tokens. Furthermore, the algorithm resolves the issues associated with word-based tokenization such as large vocabulary sizes, out-of-vocabulary tokens, and the different meanings of very similar words. It also overcomes the problems related to character-based tokenization such as long sequences and meaningless individual tokens. However, the subword tokenization approach introduces different types of issues, such as nonsensical subwords. Some of the popular subword-based tokenization algorithms are WordPiece [9], Byte-Pair Encoding (BPE) [10], Unigram [10], Byte-level BPE [11], and SentencePiece [12].

The selection of an appropriate tokenization strategy depends on the specific language and the NLP application. However, there is a critical gap in the literature regarding the influence of various tokenization algorithms on different NLP tasks. According to Bostrom and Durrett [13], there is no direct evaluation of the impact of tokenization over language models like BERT in terms of training efficiency and model performance on different downstream NLP tasks. This lack of investigation presents a significant opportunity to improve the performance of language models on NLP tasks by investigating how various tokenization algorithms would impact language models on different tasks. Moreover, little work has been conducted to evaluate language-specific tokenizers for the Arabic language [1]. Thus, we aim to evaluate the influence of various tokenizers on BERT-based Arabic models, and for this task, we have chosen the most prevalent and widely used tokenizers on the HuggingFace platform: WordPiece, SentencePiece, and BBPE. To the best of our knowledge, this is the first study to evaluate the preference for various tokenizers for Arabic LLMs, particularly BERT-based Arabic models.

Contribution: In this paper, we compare the performance of three subword tokenizers for the Arabic language: WordPiece, SentencePiece, and BBPE. We achieve this by pretraining three separate BERT models, with each using a different tokenizer. We then evaluate the performance of each model on seven different NLP tasks using 29 datasets. The findings of this study will be beneficial to researchers in several ways. They will assist them in developing better models, making informed decisions when selecting the best tokenizers, improving feature engineering techniques, and ultimately leading to more efficient models that advance various NLP applications.

Outline:The remainder of the article is organized as follows. The related works are covered in Section 2. In Section 3, we describe the implementation of the three selected tokenizations. Section 4 describes the experiment and the employed datasets. Section 5 discusses the experiment’s results. Section 6 presents the concluding remarks and future work.

2. Related Works

The literature contains some studies comparing the performance of tokenization algorithms across diverse languages on NLP tasks, including Tunisian Arabic [3], the Turkish language [14], and dialect-agnostic Arabic [1], as well as multilingual applications [15,16]. For example, Mekki et al. [3] compared three tokenization algorithms for the Tunisian Arabic dialect on three machine learning models: conditional random fields, support vector machines, and deep neural networks. The authors used several public datasets such as the Tunisian media corpus [17] and the Tunisian Arabic dialect [18] to compare the model’s performance based on their ability to accurately tokenize text, thus considering the challenges posed by Tunisian Arabic’s morphological complexity. The three models achieved good performance, but the conditional random fields model achieved the best results, with an F measure of 88.9%. In the Turkish language, Erkaya et al. [14] compared the performance of three tokenization algorithms. The three used tokenization algorithms were Byte-pair Encoding [19], WordPiece [20], and the Unigram Language Model [10] on the Open Super-large Crawled Aggregated coRpus (OSCAR) [21]. The results showed that WordPiece tends to produce more single-word tokens, along with morphology-compatible tokens. Also, the authors found that increasing the size of the training corpus for a tokenizer leads to producing more and shorter tokens. In the Arabic language, Alyafeai et al. [1] introduced three new tokenization algorithms for Arabic and then compared them to the other three baselines. The three tokenizers used in the comparison were SentencePiece, Character, and Word tokenizer. The evaluation was conducted on three supervised classification tasks: sentiment analysis, news classification, and poetry classification. Four datasets were employed: AJGT [22], Metrec [23], LABR [24], and DSAC [25]. The result showed that the word tokenizer achieved the best results for small vocabulary sizes only for news classification. However, for small and larger vocabulary sizes, the character tokenizer achieved the best results for poetry classification.

A growing body of research has investigated the impact of different tokenization methods on NLP tasks in multilingual contexts. Domingo et al. [15] conducted several experiments using five different tokenizer algorithms across ten language pairs to evaluate their impact on the quality of the final translation. The five tokenizers employed were SentencePiece, Mecab, StanfordWord Segmenter [26], OpenNMT [27], and Moses [28]. The results showed that tokenization significantly affects the quality of the translation, and the selection of the best tokenizer differs based on the selected language pairs. Also, Rust et al. [16] investigated the performance of multilingual models on nine different languages across five different tasks (NER, SA, QA, Dependency Parsing, and Part-of-Speech Tagging), thus comparing them to their equivalent monolingual model. The paper also investigated the impact of the adopted tokenization approach on the performance of multilingual models on monolingual tasks. The results showed that using a tokenizer that is designed for a particular language significantly improves the model’s performance on downstream tasks for almost every task and language. Moreover, Wei et al. [29] also performed a comparison between byte-level and character-level tokenizers in vocabulary building for multilingual pretrained language models. Several case studies were conducted on the multilingual XNLI tasks in seven languages such as English, Arabic, French, and Russian using the NEZHA architecture [30]. The authors found that the NEZHA model trained with byte-level subwords consistently outperformed the multilingual BERT model [5].

Subword tokenizers have also been explored for various languages and applications. In large language models (LLMs), Chirkova and Troshin [31] investigated the effect of unigram [10] and and BPE [32] subtokenization algorithms on the performance of LLMs trained on source code using the PLBART model [33] as a testing ground. The study revealed that reducing the subtokenization length up to 17% could improve the model training efficiency. Additionally, using the right subtokenizer (in this case: unigram) improved the quality of the trained LLM by 0.5–2%. Similarly, Bostrom and Durrett [13] conducted a study to evaluate the performance of pretrained language models using BPE and unigram tokenizers. They pretrained four RoBERTa-base models: two on an English text corpus of 3 billion tokens and two on a Japanese text corpus of 0.6 billion tokens. Each language group had one model pretrained with BPE tokenization and another with unigram tokenization. They concluded that the English model that used the unigram tokenizer slightly outperformed its counterpart model pretrained with a BPE tokenizer on various NLP tasks. However, the unigram Japanese language model showed a more than 10% improvement over its BPE counterpart in the Japanese QA task. Furthermore, Beinborn and Pinter [34] analyzed the relationship between the output of the tokenizer and the time and accuracy of human performance on a lexical decision task. The results showed that the Unigram produced outputs that were less aligned with human cognition than BPE and WordPiece.

Compared to current related works, there has been limited investigation into the impact of subword tokenizers on the Arabic language. Additionally, there has been limited exploration of how different tokenizers affect LLMs like BERT, particularly for the Arabic language. Thus, this paper aims to bridge this gap by comparing various tokenization algorithms for the Arabic language. We evaluated three subword tokenizers: WordPiece, Byte-Level BPE, and SentencePiece. We pretrained three BERT models using each tokenizer and then evaluated their performance on range of NLP tasks. The outcomes of this paper will assist researchers in developing better models, thus making better decisions in selecting the appropriate tokenizer for their needs, improving feature engineering, and making models more efficient, which will ultimately lead to advancements in various NLP applications.

3. Tokenization

3.1. WordPiece

The WordPiece (WP) tokenizer is a subword tokenization technique originally designed for solving Japanese–Korean segmentation problems in the Google speech recognition system [20]. Wu et al. [9] introduced this tokenization algorithm to maximize the likelihood of the training data when given an evolving word definition. WordPiece selects a symbol pair that will lead to the largest increase in likelihood upon merging. This is similar to the method used by Schuster and Nakajima [20] to handle uncommon/rare words to generate a segmentation for any word. One of the benefits of using the WordPiece algorithm is that it reduces the vocabulary size, thus resulting in more efficient encoding of text data. Additionally, the algorithm is considered a language-agnostic tokenizer, which means that it can be applied to different languages, thus making it adaptable for multilingual applications. First, the algorithm breaks down the input words into individual units called wordpieces. It then counts the most frequently repeated two-character, three-character, and longer sequences in the training corpus. The resulting vocabulary list will contain full words, individual characters, and subwords. To distinguish between a whole word and a subword token, a special symbol, usually ‘##’, is added as a prefix to each subword token. This indicates that the token is part of a larger word rather than a complete word on its own. For example, the sentence “I was uncomfortable” might be split into “I”, “was”, “un”, “##comfortable”. In contrast to BPE, WordPiece selects the one that maximizes the likelihood of the training dataset given the current vocabulary, not the most common symbol pair. The wordpiece tokenizer has been used in many prominent language models such as BERT [5], DistillBERT [6], and in almost all Arabic langauge models such as Arabic-BERT [35], ARBERT [36], AraPoemBERT [37], and CAMELBERT [38].

3.2. Byte-Level BPE (BBPE)

Byte-Pair Encoding (BPE) is a popular subword tokenization algorithm introduced by Sennrich et al. [32]. The pretokenization stage of BPE includes separating the text into words and then creating a set of unique words, along with the frequency of each word in the training text corpus. In the tokenization stage, the algorithm starts by creating a base vocabulary list that contains all the characters and symbols that are in the unique word set. From there, the algorithm starts merging the starting tokens in the base vocabulary list, which are individual characters to form new tokens to be added to the vocabulary list. The algorithm starts by merging the most frequent tokens, and it keeps repeating the merging process until the number of tokens in the vocabulary list reaches the predetermined vocabulary size. The BPE tokenizer was used in the original OpenAI GPT model [39]. Both the BPE and WordPiece algorithms follow a bottom-up approach in building the vocabulary list.

Byte-level BPE (BBPE): BBPE is a recent variation of the original BPE tokenization algorithm introduced by Wang et al. [11]. The main difference between the two is that BBPE builds the base vocabulary list using a 256-byte set instead of actual characters. This approach allows the vocabulary to include all possible based characters while maintaining the overall vocabulary size within a reasonable limit, and it ensures that there is no need for an out-of-vocabulary token. In contrast to Sentencepiece and Wordpiece, BBPE is byte-level, which builds a vocabulary based on bytes encoding, not based on individual characters, in the training data. It also offers a universal vocabulary of bytes, which allows for better handling of many languages and for solving the issue of out-of-vocabulary. The BBPE tokenizer was used in many prominent language models such as GPT2 [39], RoBERTa [8], JABER and SABER [40], and ARAMUS [41].

3.3. SentencePiece

SentencePiece (SP) is an unsupervised subword tokenizer and detokenizer designed for neural-based text processing [12]. Because it is an unsupervised tokenizer, it does not need prelabeled data, and thus, it learns how to segment words directly from raw sentences. SentencePiece is considered to be language-Independent, since it treats the sentences just as sequences of Unicode characters (i.e., it does not extract words based on the whitespace character). It encompasses four main elements: the normalizer, trainer, encoder, and decoder. The task of the normalizer is to convert semantically equivalent Unicode (Unicode- NFKC) characters into their standard forms. The task of the trainer is to train the corpus by learning the subword segmentation strategy. SentencePiece supports two tokenization algorithms—byte pair encoding (BPE) [32] and the unigram language model [10]—to construct the appropriate vocabulary. The task of the encoder is to prepare the text for processing by converting it to a sequence of IDs, while the task of the decoder is to convert the output back to human-readable text. The model starts by creating vocabulary of a predetermined size from the basic character set. Then, a unique ID is assigned to each subword in its vocabulary. In contrast to WordPiece and BPE, SentencePiece adds spaces and nontext symbols in its vocabulary, thus allowing it to tokenize raw text directly. The SentencePiece tokenizer was used in many prominent language models such as ALBERT [42], SaudiBERT [43], XLNet [44], and T5 [45].

4. Experiments

In this section, we present the steps for text preprocessing, the pretraining configurations of the presented models, and the fine-tuning procedures. Additionally, we briefly elaborate on the evaluation metrics used to assess different models and provide a short description of the evaluation datasets used in this study.

All experiments were conducted on a local machine with AMD Ryzen-9 7950x processor, 64 GB DDR5 memory, and two GeForce RTX 4090 GPUs with 24 GB memory each. The software environment was set up on the Ubuntu 22.04 operating system, and we used the HuggingFace Transformers library [46] for pretraining and fine-tuning the models, in addition to downloading many of the datasets used in the experiments from the HuggingFace hub.

4.1. Model Pretraining

The goal of this project is to evaluate the performance of various tokenizers for Arabic LLms. We analyzed the differences between WordPiece (WP), SentencePiece (SP), and Byte-level BPE (BBPE) tokenizers by pretraining three BERT models using each tokenizer (BERT-WP https://huggingface.co/faisalq/bert-base-arabic-wordpiece (accessed on 21 May 2024), BERT-SP https://huggingface.co/faisalq/bert-base-arabic-senpiece (accessed on 21 May 2024), and BERT-BBPE https://huggingface.co/faisalq/bert-base-arabic-bbpe (accessed on 21 May 2024)) while measuring the performance of each model on seven different NLP tasks. The pretraining text corpus underwent a few preprocessing operations using PyArabic [47], a Python library for preprocessing and normalizing Arabic text. The preprocessing steps included only removing diacritics and Arabic letters’ extension (Tatweel), without removing any punctuation, symbol characters, or English words. The final corpus size came out to 35.7 GB (29.6 GB from CC-100-AR [48], and 6.1 GB from Wikipedia dump and Al-Shamela library [49] that was already compiled and made public on Kaggle [50]).

All three models followed the same architecture as the original BERT model, with 12 encoder layers, 12 attention heads per layer, and a hidden layer size of 768 units, and the vocabulary size was set to 50,000 wordpieces. Furthermore, we used the following configurations for pretraining the models: masking 15% of the input tokens, a maximum sequence length of 128, a batch size of 128, and the ‘AdamW’ optimizer, with a learning rate of 5 × ${10}^{-5}$. Finally, all three models were pretrained for 2 epochs (around 2 million steps). Figure 1 provides an overview of the text preprocessing and tokenizer preparation steps, along with the model pretraining procedures.

4.2. Evaluation Metrics

We evaluated the performance of the new models by measuring their performance across all tasks using the F1 score (1) metric. Additionally, we included the Accuracy (1) metric for the majority of the evaluation tasks, whereas the Exact Match (EM) (5) metric was used only for the QA task. EM is a binary evaluation metric that assesses whether the predicted answer matches the referenced answer or not. The following formulas were used to determine the performance of each classification model:

$$\mathrm{F1-score}=\frac{2\ast Precision\ast Recall}{Precision+Recall}=\frac{2\ast TP}{2\ast TP+FP+FN}$$

(1)

$$\mathrm{Precision}=\frac{TP}{TP+FP}$$

(2)

$$\mathrm{Recall}=\frac{TP}{TP+FN}$$

(3)

$$\mathrm{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$

(4)

$$\mathrm{EM}=\frac{NumberofExactMatches}{T}$$

(5)

where $TP$ is True Positive, $TN$ is True Negative, $FP$ is False Positive, and $FN$ is False Negative. Also, EM represents the average of all individual exact match scores in the set, and T represents the total number of predictions in the set.

4.3. Datasets

In this section, we briefly describe all datasets used for each NLP task. The evaluation process was conducted by measuring the performance of all three models across seven NLP tasks, with a total of 29 datasets. Some datasets were annotated for two or more tasks. Thus, we added an extra identifier in front of the name of those datasets.

4.3.1. Natural Language Inference (NLI) and Stance Detection (SD)

NLI and SD are both subtasks of NLP that deal with understanding the relationships between two sentences associated with a label. However, the NLI task involves determining the relationship between two given sentences. The first sentence is typically called the “premise”, and the second sentence is known as the “hypothesis”. The model’s objective is to classify whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) based on the information in the premise. On the other hand, SD involves identifying the author’s stance in a text concerning a specific target. The typical stances identified are ‘favor’, ‘against’, and ‘neutral’. For this task group, we employed five datasets. Four consist of Modern Standard Arabic (MSA) text and one dialectal Arabic (DA) dataset, called Mawqif. Descriptions of employed five datasets are provided below:

• Cross-lingual Natural Language Inference (XNLI): XNLI is a crowdsourced dataset designed to evaluate crosslingual representations [51]. It consists of 393 k sentence pairs, with 2490 in the validation set and 5010 in the test set. Each pair was annotated with one of three labels: entailment, contradiction, or neutral. The dataset has been machine-translated into 14 languages, including Arabic.
• The Multi-Genre Natural Language Inference (MultiNLI): Similarly, MultiNLI is also a crowdsourced dataset designed for crosslingual representations tasks [52]. It consists of 84 k sentence pairs with 21 k in the validation set. The dataset has been machine-translated into 15 languages, including Arabic.
• Arabic Stance Detection dataset (AraStance):AraStance is a collection of Arabic news websites annotated with three fact-checking stances [53]. The dataset consists of 4063 articles covering claims from different domains such as health, politics, and sports.
• Arabic News Stance Corpus (ANS-Stance): ANS-Stance is a collection that includes Arabic news titles, paraphrased titles, and corrupted titles designed for news claim verification and stance prediction [54]. It consists of 3786 titles, with each associated with pairs of claims and evidence.
• A Multi-label Arabic Dataset for Target-specific Stance Detection (Mawqif): Mawqif is a multilabel Arabic dataset that was crawled from Twitter and manually annotated for four different NLP tasks: stance detection, sarcasm detection, sentiment analysis, and text classification [55]. It consists of 4121 tweets in multidialectal Arabic, thus covering three topics: digital transformation, the COVID-19 vaccine, and women’s empowerment.

4.3.2. Sarcasm Detection (SaD)

SaD task typically involves identifying whether a given sentence is sarcastic or not. The task is challenging, because it requires an understanding of subtle hints and context to identify sarcasm. For the SaD task, we employed four datasets, including Mawqif that was briefly described in the NLI and SD task group. All datasets in this task group primarily consist of DA text, along with some MSA text.

• Saudi Dialect Irony Dataset (SaudiIrony): SaudiIrony is a collection of tweets specifically designed to help with detecting irony in the Saudi dialect [56]. The dataset contains about 19,800 tweets.
• iSarcasmEval: iSarcasmEval is a collection of Arabic sarcastic texts specifically designed to help detect intended sarcasm in text [57]. It contains 4887 texts from different dialects such as the Nile Basin, the Gulf area, and North Africa.
• ArSarcasmV2: ArSarcasm is an Arabic SaD dataset [58]. It is an extension of the original ArSarcasm dataset [59]. The expanded corpus contains 15,548 tweets, with each labeled for sarcasm, and which are manually annotated with one of three sentiments: positive, negative, or neutral.

4.3.3. Sentiment Analysis (SA)

In the SA task, the model’s objective is to identify the sentiment of input text data, which can be positive, negative, or sometimes neutral. For this task group, we used six datasets. Two datasets, HARD and ASTD, consist of mixed text in MSA and DA. One dataset, BRAD, consists only of MSA. The other two datasets, Mawqif (briefly described in the NLI and SD task groups) and ArSarcasmV2 (briefly described in the SaD task group), consist of DA. Descriptions of the remaining four datasets are provided below:

• Hotel Arabic-Reviews Dataset (HARD): HARD is a large dataset composed of customer reviews of various hotels that were written in both MSA and various DA forms [60]. The corpus contains 409,562 reviews collected from Booking.com. The dataset contains 2 versions: balanced and unbalanced. The the balanced version contains 105,698 reviews annotated with either positive or negative. In this paper, both versions were included in the experiments.
• Book Reviews in Arabic Dataset (BRAD): BRAD is a large dataset of Arabic book reviews [61] collected from goodreads.com. It contains 510,600 reviews associated with a rating on a scale of 1 to 5 stars for sentiment analysis.
• Arabic Sentiment Tweets Dataset (ASTD): ASTD is a collection of Arabic posts collected from Egyptian Twitter accounts [62]. It contains approximately 10,000 tweets.

4.3.4. Topic Classification (TC)

Topic classification (TC) is a key task in natural language processing. In TC, texts are automatically categorized based on their content. The purpose of this task is to measure the impact of different tokenizers on model performance in analyzing a large body of text. For the TC task, we employed four datasets, all of which consist of news articles written in MSA. Descriptions of the four datasets used are provided below:

• Single-labeled Arabic News Articles Dataset (SANAD): SANAD is a large collection of single-labeled Arabic news articles [63]. It contains 194,797 articles covering different topics such as sports, culture, finance, politics, and medicine. The articles were collected from three different news websites: AlKhaleej, AlArabiya, and Akhbarona. In this paper, we used the AlKhaleej subset labeled into seven categories, and it contains 45 K articles.
• Open Source Arabic Corpora (OSAC): OSAC is a large collection of Arabic datasets that were collected from various sources, including news articles (BBC and CNN Arabic news), cooking recipe books, and religious books [64]. The corpora are divided into three sets: BBC Arabic, CNN Arabic, and OSAC. In total, the corpus contains 22,429 articles covering different topics on economics, astronomy, law, etc. In this paper, we used the CNN Arabic corpus, which contains 5070 articles labeled into 6 different categories: business, entertainment, Middle East news, science and technology, sports, and world news.
• A Moroccan News Articles Dataset (MNAD v2): MNAD is an Arabic dataset that was collected from four Moroccan news articles websites [65]. It contains 418 K articles in total covering 19 different categories such as sports, finance, politics, and business.
• An Arabic News Text (ANTCorp v1.1): ANTCorp is a large collection of Arabic news articles that were collected from Tunisian RSS Feeds [66]. The corpora contain about 6000 articles covering different categories: economy, culture, local News, sports, etc.

4.3.5. Text Classification (TxC)

The text classification (TxC) task is similar to TC, where input texts are categorized into different classes. However, the main difference from TC is that the input text is much shorter, which typically consists of a sentence or two. For the TxC task, we employed seven datasets, including Mawqif, iSarcasmEval, and ArSarcasmV2, which were described previously. All datasets in this group were written in DA, except the ASND dataset. Descriptions of the remaining four datasets are provided below:

• Arabic Social Media News Dataset (ASND): ASND is an Arabic dataset collected from the official Aljazeera news accounts on Twitter, Facebook, and YouTube [67]. It contains about 10 k posts covering different categories such as business, economy, education, politics, spirituality, etc.
• Arabic Influencer Twitter Dataset (AITD): AITD is an Arabic dataset collected from 60 Arab influencers on Twitter [67]. For each influencer, the last 3200 tweets were collected and classified into one of ten categories.
• Dangerous dataset: The Dangerous dataset is an Arabic dataset collected from various Twitter accounts. It includes samples of dangerous speech, such as offensive or hateful language, alongside nondangerous speech [68]. In total, the corpus contains approximately 110.3 million posts collected from 399,000 users.
• Adult content detection: Adult content detection is a collocation of Twitter posts of adult content [69]. In total, the corpora contain about 50 k posts.

4.3.6. Question Answering (QA)

In the QA task, the model is expected to extract the correct answer to a given question from input text passages. For this task, we employed four datasets that contain only MSA text. Descriptions of these datasets are provided below:

• Arabic Reading Comprehension Dataset (ARCD): ARCD is an Arabic QA dataset that was built to solve the issue of open domain factual Arabic QA [70]. It contains 1395 unique question–answer pairs posed by crowdworkers on Wikipedia articles and a machine translation of another Arabic QA dataset. The articles were originally collected from Wikipedia articles written in Arabic.
• Cross-lingual Question Answering Dataset (XQuAD): XQuAD is a benchmark dataset designed to evaluate the preference of crosslingual question answering [71]. It consists of 240 paragraphs and 1190 question–answer pairs developed from SQuAD v1.1 [72]. It covers 11 diverse languages. The dataset has been translated into ten languages: Arabic, Russian, Greek, Turkish, Vietnamese, etc. In this study, we only included the Arabic subset of XQuAD benchmark.
• A Benchmark for Information-Seeking Question Answering (TyDiQA): TyDiQA [73] is a multilingual QA benchmark for 11 different languages, including Arabic. Originally, the dataset consisted of 204 K question–answer pairs. However, we extracted the Arabic subset only, which comprises 15,726 question–answer pairs.
• Evaluating Cross-lingual Extractive Question Answering (MLQA): Similar to XQuAD, MLQA is a benchmark dataset to evaluate the preference of crosslingual question answering [74]. It consists of between 5000 and 6000 question–answering instances in each language, thus presented in SQuAD format in seven languages: Arabic, German, Vietnamese, English, etc. The Arabic subset contains 5852 question–answer pairs.

4.3.7. Named Entity Recognition (NER)

In the NER NLP task, we measure the model’s ability to identify and categorize named entities within unstructured text such as people’s names, locations, dates, and times. For the NER task, we employed six datasets that contain only MSA text. Descriptions of the employed four datasets are provided below:

• Wikipedia-based Fine-grained Arabic Named Entity (NewsFANE_Gold): NewsFANE_Gold is a fine-grained Arabic NER corpora that was manually annotated from the Arabic Newswire [75]. It contains 170 K tokens categorized into 88 classes.
• Wikipedia-based Fine-grained Arabic Named Entity Corpus (WikiFANE_Gold): WikiFANE_Gold is a fine-grained Arabic NER corpora that was manually annotated from the Arabic Wikipedia [75]. It contains about 500 K tokens that are annotated manually into 102 classes.
• Wikipedia-derived corpus (WDC): WDC is a dataset specifically designed for training models to identify named entities in Arabic text [76]. In the WDC corpus, only sentences with at least three annotated named entity tokens were included. It consists of 165,119 sentences consisting of around 6 M tokens categorized into 9 classes.
• WikiAnn: WikiAnn is a dataset collected from Wikipedia articles in 282 Languages. It was designed for the task of crosslingual naming and linking [77]. The Arabic subset contains 40 K sentences, while the tokens are categorized into 7 classes.
• American and Qatari Modeling of Arabic (AQMAR): AQMAR dataset was collected from Wikipedia for Arabic NER task [78]. It consists of 3000 sentences in four domains: history, science, sports, and technology. The tokens are categorized into 26 classes.
• Arabic NER corpora (ANERcorp): ANERcorp is a dataset designed to be used in Arabic NER tasks [79]. The data were collected from news websites and other web sources [80]. It consists of 316 articles and 150,286 tokens. Each token was annotated based on four categories: person, location, organization, and miscellaneous or other.

5. Results

This section presents the models’ evaluation results across the target NLP tasks.

Table 1. Comparison of BERT models using different tokenizers: WordPiece (BERT-WP), SentencePiece (BERT-SP), and Byte-level BPE (BERT-BBPE) for the NLI and SD tasks.

		BERT-WP		BERT-SP		BERT-BBPE
Dataset	Type	Acc.	F1	Acc.	F1	Acc.	F1	Train Size	Validation Size	#Labels
XNLI	MSA	74.1	74.22	74.53	74.54	73.57	73.64	393 K	7.5 K	3
MultilingualNLI	MSA	66.45	65.81	66.86	66.13	65.46	64.74	84 K	21 K	3
AnsStance	MSA	71.52	65.49	78.67	74.99	77.88	73.46	3.03 K	0.75 K	3
AraStance	MSA	84.53	79.16	82.24	77.38	84.35	79.75	3.49 K	0.56 K	4
Mawqif (stance)	DA	77.31	66.93	77.31	68.45	75.03	65.88	2.8 K	0.7 K	3
Avg		74.78	70.32	75.92	72.29	75.25	71.49	-	-	-

presents the results of the models across all datasets employed for NLI and SD tasks. The table includes training and validation sizes for each dataset, along with the number of labels. For this experiment, we trained the BERT model on five datasets. The results indicate that BERT-WP achieved the highest accuracy scores only on the AraStance and Mawqif datasets, with scores of 84.53% and 77.31%, respectively, while BERT-BBPE achieved the highest F1 score on the AraStance dataset alone. However, BERT-SP demonstrated superiority across the majority of datasets, thus achieving the highest accuracy and F1 scores for most tasks, with overall average scores of 75.92% for accuracy and 72.29% for F1. The results suggest that BERT-SP is a more effective tokenizer for NLI and SD tasks compared to BERT-WP and BERT-BBPE. Moreover, no significant differences were observed between the MSA and DA types.

For the SaD task,

Table 2. Comparison of BERT models using different tokenizers: WordPiece (BERT-WP), SentencePiece (BERT-SP), and Byte-level BPE (BERT-BBPE) for the SaD tasks.

		BERT-WP		BERT-SP		BERT-BBPE
Dataset	Type	Acc.	F1	Acc.	F1	Acc.	F1	Train Size	Validation Size	#Labels
Mawqif (sarcasm)	DA	96.00	68.53	96.43	72.48	94.43	67.59	2.8 K	0.7 K	2
SaudiIrony	DA	71.63	70.45	71.70	70.45	70.84	69.97	15.7 K	3.9 K	2
iSarcasmEval (sarcasm)	DA	72.64	61.83	74.14	63.01	66.21	57.11	3.1 K	1.4 K	2
ArSarcasmV2 (sarcasm)	DA	77.26	70.84	76.63	70.32	75.96	70.78	12.54 K	3 K	2
Avg		79.38	67.91	79.72	69.06	76.86	66.36	-	-	-

presents the results of BERT models evaluated across all four datasets included. It also provides the details for the training and validation sizes for each dataset. Overall, BERT-SP has outperformed the other two models on the majority of tasks, achieving the highest average accuracy and F1 scores at 79.72% and 69.06%, respectively. BERT-WP achieved the second highest scores for both accuracy and F1, while BERT-BBPE had the lowest overall average scores.

For the SA task,

Table 3. Comparison of BERT models using different tokenizers: WordPiece (BERT-WP), SentencePiece (BERT-SP), and Byte-level BPE (BERT-BBPE) for the SA task.

		BERT-WP		BERT-SP		BERT-BBPE
Dataset	Type	Acc.	F1	Acc.	F1	Acc.	F1	Train Size	Validation Size	#Labels
Mawqif (sentiment)	DA	71.46	69.82	71.61	68.96	71.32	68.49	2.8 K	0.7 K	3
ArSarcasmV2 (sentiment)	DA	68.23	63.92	67.83	63.80	66.70	62.72	12.54 K	3 K	3
HARD (balanced)	MSA + DA	89.04	86.84	89.10	86.81	89.04	86.84	84.55 K	21.14 K	2
HARD (unbalanced)	MSA + DA	90.44	85.55	90.42	85.65	90.30	85.52	327.64 K	81.91 K	3
ASTD	MSA + DA	74.27	55.68	75.42	57.40	75.22	56.43	8 K	2 K	4
BRAD	MSA	52.90	51.79	52.73	51.71	52.55	51.62	408.47 K	102.12 K	5
Avg		74.39	68.93	74.51	69.05	74.18	68.60	-	-	-

shows that the performance of all models was quite similar. We fine-tuned the BERT models on six datasets, each with different training and validation sizes. BERT-SP achieved the highest overall average accuracy and F1 scores, at 74.51% and 69.05%, respectively. Meanwhile, BERT-WP achieved the highest F1 score on four individual datasets and the highest accuracy on three. Moreover, no significant differences were observed between the MSA and DA types.

For the TC task,

Table 4. Comparison of BERT models using different tokenizers: WordPiece (BERT-WP), SentencePiece (BERT-SP), and Byte-level BPE (BERT-BBPE) for the TC task.

		BERT-WP		BERT-SP		BERT-BBPE
Dataset	Type	Acc.	F1	Acc.	F1	Acc.	F1	Train Size	Validation Size	#Labels
SANAD (AlKhaleej)	MSA	98.60	98.60	98.57	98.57	98.60	98.60	36 K	9.1 K	7
OSAC (CNN)	MSA	94.87	94.38	94.97	94.27	94.97	94.32	4.05 K	1.01 K	6
MNADv2	MSA	80.98	73.43	81.06	73.13	81.62	74.61	855 K	213 K	20
ANTCorpv1.1	MSA	84.30	80.12	82.88	80.51	84.30	78.95	8.12 K	2.03 K	9
Avg		89.68	86.63	89.37	86.62	89.87	86.62	-	-	-

shows that BERT-BBPE achieved a slightly better accuracy score than the other two models. This advantage is due to its better compression rate, which requires fewer tokens when tokenizing specific texts compared to the WordPiece and SentencePiece tokenizers. We will further demonstrate this at the end of this section. This feature is particularly significant, as the input text for the topic classification task often exceeds the maximum allowed sequence length of 128 tokens. However, all three models achieved nearly the same F1, with a score of 86.63%.

For the TcX task,

Table 5. Comparison of BERT models using different tokenizers: WordPiece (BERT-WP), SentencePiece (BERT-SP), and Byte-level BPE (BERT-BBPE) for the TxC task.

		BERT-WP		BERT-SP		BERT-BBPE
Dataset	Type	Acc.	F1	Acc.	F1	Acc.	F1	Train Size	Validation Size	#Labels
Mawqif (target classification)	DA	99.85	99.85	100	100	99.85	99.85	2.8 K	0.7 K	3
ArSarcasmV2 (dialect detection)	DA	72.20	38.53	66.70	38.82	72.10	36.33	12.54 K	3 K	5
iSarcasmEval (dialect detection)	DA	55.85	42.28	57.21	44.51	56.14	39.94	3.1 K	1.4 K	5
ASND	MSA	82.33	78.67	81.19	78.27	81.37	78.95	6.67 K	1.67 K	12
ATID	DA	97.10	96.85	97.26	96.86	97.78	97.46	92.55 K	23.13 K	10
Dangerous	DA	82.33	77.53	82.43	78.42	81.33	77.00	4 K	1 K	2
Adult	DA	95.89	90.29	95.96	90.53	95.81	90.06	35 K	15 K	2
Avg		83.65	74.85	82.96	75.34	83.48	74.22	-	-	-

shows that BERT-SP outperformed the other two models on the majority of tasks by achieving the highest accuracy score on four out of seven tasks and the highest F1 score on five tasks. Additionally, the model achieved the highest overall average F1 score at 75.34%, thus significantly outperforming the other two models. However, the low average accuracy score is due to the poor performance on the ArSarcasmV2 task, which heavily affected the overall average accuracy.

For the QA task,

Table 6. Comparison of BERT models using different tokenizers: WordPiece (BERT-WP), SentencePiece (BERT-SP), and Byte-level BPE (BERT-BBPE) for the QA task.

		BERT-WP		BERT-SP		BERT-BBPE
Dataset	Type	EM	F1	EM	F1	EM	F1	Train Size	Validation Size
ARCD	MSA	18.66	47.06	20.79	47.66	17.94	42.84	693	702
XQuAD	MSA	17.64	27.15	19.32	28.23	20.16	27.18	952	238
TyDiQA	MSA	54.50	77.62	50.59	75.57	51.57	74.94	14.8 K	921
MLQA	MSA	23.40	39.73	23.98	40.69	26.30	42.23	5.34 K	517
Avg		28.55	47.89	28.67	48.03	28.99	46.79

shows that all three models achieved comparable results across all datasets, which is due to the high complexity of the QA NLP tasks. On average, BERT-SP achieved a slightly better F1 score of 48.03%, while BERT-BBPE recorded the highest EM score at 28.99%.

For the NER task,

Table 7. Comparison of BERT models using different tokenizers: WordPiece (BERT-WP), SentencePiece (BERT-SP), and Byte-level BPE (BERT-BBPE) for the NER task. The results show the F1 score for each model.

		BERT-WP	BERT-SP	BERT-BBPE
Dataset	Type	F1	F1	F1	Train Size	Validation Size	#Labels
NewsFANEGold	MSA	85.65	85.71	71.53	3.8 K	952	88
WikiFANEGold	MSA	79.34	80.38	67.15	12.6 K	3.15 K	102
WDC	MSA	78.66	79.51	78.83	151 K	37.77 K	9
WikiANN	MSA	94.31	94.13	94.13	30 K	10 K	7
AQMAR	MSA	76.56	80.92	79.64	1.96 K	425	26
ANERCorp	MSA	81.11	83.59	81.17	4.15 K	961	9
Avg		82.60	84.04	78.74	-	-	-

presents the F1 scores for each model on the datasets used. The results indicate that BERT-SP significantly outperformed the other models on five out of six tasks, thus achieving the highest overall average score of 84.04%. This suggests that the SentencePiece tokenizer is the most effective approach for NER tasks, regardless of the size of the dataset or the number of labels.

In our comparative analysis of BERT models using different tokenizers—WordPiece (BERT-WP), Byte-level BPE (BERT-BBPE), and SentencePiece (BERT-SP)—we measured several performance metrics, including the pretraining time, tokenizer training time, and minimum training loss.

Table 8. Comparative performance of BERT models utilizing different tokenization methods: WordPiece, Byte-level BPE, and SentencePiece. Best results in bold.

	BERT-WP	BERT-BBPE	BERT-SP
Model Pretraining Time (GPU hours)	140	133	147
Tokenizer Training Time (minutes)	7:59	6:02	7:21
Corpus Tokenization Time	1:22:38	1:08:41	1:35:36
Avg. No. Tokens per Sequence	35.216	35.128	37.902
Min. Training Loss	2.3755	2.4632	2.1534

shows that BERT-BBPE was the most efficient in terms of the training and tokenization time, thus achieving the shortest model pretraining time at 133 GPU hours, tokenizer training time at 6:02 min, and corpus tokenization time at 1:08:41. These results indicate that BERT-BBPE is highly effective in terms of speed and processing efficiency. Additionally, the BBPE tokenizer achieved the best compression rate, thus requiring fewer tokens to encode input texts. However, BERT-BBPE recorded the highest training loss during model pretraining, thus leading it to underperform compared to the other two models in the majority of evaluation tasks, except for the topic classification task, where its efficient compression rate was beneficial.

On the other hand, despite its longer tokenization and training time, BERT-SP achieved the lowest training loss at 2.1534 over the same number of training epochs as the other models, thus demonstrating its effectiveness in learning from the training data. Figure 2 illustrates the training loss over the number of steps for each model. In contrast, BERT-WP, while not excelling in any specific category, showed consistent performance across all metrics. This analysis highlights the trade-offs between the tokenizer choice and aspects of the model efficiency, accuracy, and training speed. Both BERT-BBPE and BERT-SP exhibited strengths in different areas of the training process.

6. Conclusions

Tokenization serves as a crucial foundation for many NLP tasks involving the breakdown of text streams into basic meaningful units called tokens. These tokens are essential inputs for subsequent NLP tasks such as semantic parsing and language modeling. Despite its importance, there is a notable lack of research on how different tokenization approaches affect the performance of Arabic language models. This gap obstructs our understanding of the impact of various tokenization methods on NLP effectiveness and performance. This study addresses this gap by evaluating the effectiveness of three different tokenizers—SentencePiece, WordPiece, and BBPE—by pretraining three Arabic BERT language models (BERT-SP, BERT-WP, and BERT-BBPE). Each model was pretrained using the same corpus and configurations, but they employed different tokenizers. We assessed the impact of each tokenizer by analyzing the performance of each model across seven NLP task groups using 29 Arabic datasets. Our results show that BERT-SP significantly outperformed the other models in most tasks, thus achieving the highest overall average F1 scores: 84.04% in NER, 48.03% in QA, 75.34% in text classification, 69.05% in SA, 69.06% in SaD, and 72.29% in NLI. On the other hand, the BBPE tokenizer demonstrated superior efficiency, thus recording the shortest pretraining time (133 GPU hours), tokenizer training time (6:02 min), and corpus tokenization time (1:08:41). It also achieved the best compression rate, thus requiring fewer tokens per input sequence and enhancing its speed and processing efficiency. This capability allows it to efficiently handle large input texts. Meanwhile, the WordPiece tokenizer showed consistent performance across all evaluated metrics without excelling in any specific area. These findings contribute to the development of more efficient models and better tokenizer selection, thus enhancing feature engineering and overall advancements in NLP applications. Future work should include a more in-depth analysis to understand how various Arabic dialects affect and influence tokenizer performance, as well as evaluating the performance of various tokenizers with generative language models.